A mouse face video dataset making system and method for deep learning

Abstract

The application provides a mouse face video dataset making system and method for deep learning, an infrared camera collects mouse face video data, and is connected with a PC end video processing module to transmit and store the video data in the PC end video processing module; an experimental condition control module sends a stimulation condition to the mouse and sends a level signal to an Arduino development board; the Arduino development board sends pin feedback information to the PC end video processing module, and the PC end video processing module controls the infrared camera based on an OpenCV library to label color labels in real time in a video frame. The dataset making method and system provided by the application are automatic processes, only need to provide a video data storage path, save the problems of time and labor cost caused by manual labeling, and greatly improve the dataset making efficiency and quality.

Description

Technical Field

[0001] This invention relates to the field of deep learning image classification data technology, specifically to a system and method for creating a mouse facial video dataset for deep learning. Background Technology

[0002] In recent years, with the continuous emergence of deep learning methods and deep neural network models, algorithm performance has been constantly improved, and the application fields of computer vision technology have become increasingly broad. In the field of neuroscience, when exploring the function of a certain brain region, there is no good way to quantify animal behavior when designing animal experiments. It can only rely on subjective observation by human eyes, which is highly subjective. At the same time, it is difficult to observe subtle changes in animal behavior when experimental conditions change. Computer vision technology fills this gap. Computer vision technology quantitatively explains the intrinsic connection between machine vision behavior understanding and brain neural networks, and can establish stable mapping models, forming a new research paradigm for discovering behavioral neural circuits based on large-scale computer vision behavior detection and brain neural signal correlation.

[0003] However, applying computer vision technology to analyze the facial behavior of mice requires the collection of a large number of high-precision datasets. Since different experimental conditions may be changed in mouse experiments, the collected videos need to be labeled immediately when the conditions are triggered in order to distinguish the facial behavior of mice collected under different conditions. Manual labeling not only consumes a lot of time and labor costs, but also cannot guarantee timely labeling when the conditions are triggered, resulting in a lack of temporal coupling between video data and electrophysiological data; this leads to a decline in data quality and affects the analysis results of biological experiments.

[0004] Therefore, there is an urgent need for a dataset acquisition system and method that can record mouse video data in real time during the triggering of experimental conditions. Summary of the Invention

[0005] To overcome the problems of the prior art, the present invention provides a system for creating mouse facial video datasets for deep learning.

[0006] This invention provides the following technical solutions:

[0007] A system for creating a mouse facial video dataset for deep learning includes an infrared camera, an experimental condition control module, an Arduino development board, and a PC-based video processing module. The infrared camera acquires mouse facial video data and connects to the PC-based video processing module to transmit and store the video data. The experimental condition control module sends stimulus conditions to the mouse and sends level signals to the Arduino development board. The Arduino development board sends pin feedback information to the PC-based video processing module, which controls the infrared camera based on the OpenCV library and labels the video frames with color tags.

[0008] Furthermore, the experimental condition control module controls the release of the stimulus condition, which is the release of a stimulating gas or a voltage release. When the stimulus condition is released, a low-level signal is sent to the A0-A5 pins of the Arduino development board.

[0009] Furthermore, the start and end of video recording can be controlled by the S and E keys on the PC-side video processing module, or by the Arduino start pin 8 and end pin 13. When the experimental condition control module starts, it sends a low-level signal to the Arduino development board start pin 8, and when the experimental condition control module terminates, it sends a low-level signal to the Arduino development board end pin 13.

[0010] Furthermore, the Arduino development board is connected to the experimental condition control module via DuPont wires. The Arduino development board detects the low-level signals of each pin and establishes a one-to-one mapping relationship between each pin and the feedback information of different pins. The pin feedback information is a number or an English character. The condition trigger pin A0 corresponds to the character "0", the condition trigger pin A1 corresponds to the character "1", the condition trigger pin A2 corresponds to the character "2", the condition trigger pin A3 corresponds to the character "3", the condition trigger pin A4 corresponds to the character "4", the condition trigger pin A5 corresponds to the character "5", the start pin 8 and the end pin 13 correspond to the characters "s" and "e" respectively. The character signals are sent to the PC video processing module through serial communication. The low-level transmission signal time is coupled with the experimental condition trigger time.

[0011] Furthermore, the PC-side video processing module establishes a one-to-one mapping relationship between different numeric characters and different color labels. The PC-side video processing module controls video recording, calls the camera to start the recording function, receives video data, detects the serial port status, and marks the corresponding video frame with a color label when the numeric character is successfully matched. The PC-side video processing module stores the recorded consecutive frame video data, and the pin feedback information is coupled with the successful matching time and the label marking time.

[0012] Furthermore, the PC-side video processing module starts the timing function when it detects the character "s" or the S key on the keyboard is triggered; it records the corresponding time when different numeric characters are detected; and it ends the timing function when it detects the character "e" or the E key on the keyboard is triggered, generating and storing a table of experimental condition triggers and time points.

[0013] Furthermore, the PC-side video processing module extracts video frames with the same color label and saves them as video files.

[0014] The method for creating video datasets using the aforementioned mouse facial video dataset creation system includes the following steps:

[0015] Step 1: The PC-side video processing module controls the infrared camera to turn on and capture video of the mouse's face. The video data is stored in the PC-side video processing module.

[0016] Step 2: Start the experimental condition control module. The experimental condition control module sends a low-level signal to the start pin 8 of the Arduino development board. After the Arduino development board detects the low-level signal of the start pin 8, it sends the character "s" to the PC video processing module through serial communication. The PC video processing module calls the camera to start the recording function and receives the video data.

[0017] Step 3: The experimental condition control module controls the release of the stimulus condition and sends a low-level signal to the A0-A5 pins of the Arduino development board. The Arduino development board detects the low-level pulse and sends a digital character signal to the video processing module on the PC through serial communication.

[0018] Step 4: The PC-side video processing module detects the serial port status. After a successful character match, it marks the corresponding color label in the current video frame. The PC-side video processing module then stores the recorded consecutive video frames.

[0019] Step 5: When the experiment ends, if the PC video processing module detects the character "e", the PC video processing module will record the video file and save it to the specified path.

[0020] Furthermore, in step two, the PC-side video processing module enables the timing function to record the start time; in step four, it records the trigger time points for different experimental conditions; and in step five, it records the end time, generating and storing a table of experimental condition triggers and time points.

[0021] Furthermore, step six involves the PC-side video processing module reading the video, detecting pixel values ​​of different colors at the pre-reserved positions on the collected data, classifying the data according to the detected pixel values, extracting video frames with the same color labels, and saving the video files accordingly.

[0022] By adopting the above technical solution, the present invention has the following beneficial effects:

[0023] (1) The dataset production method and system proposed in this invention is an automated process. It only requires providing the video data storage path, which saves the time and labor costs caused by manual annotation and greatly improves the efficiency of dataset production.

[0024] (2) The annotation action and the conditional trigger action of this invention are highly coupled in time, which improves the quality of the dataset in the sense of biological experiment, reduces experimental errors caused by human error, ensures that subsequent analysis will not be affected by time delay, and plans a standard dataset production process, which is of great significance for the application of deep learning methods in the field of neuroscience. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of a mouse facial video dataset system;

[0026] Figure 2 This is a time table diagram of the experimental results;

[0027] Figure 3 This is a schematic diagram of the video frame extraction experiment results. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the structural diagrams and specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0029] Example 1

[0030] This invention provides a system for creating mouse facial video datasets for deep learning, such as... Figure 1 As shown, the system includes an infrared camera, an experimental condition control module, an Arduino development board, and a PC-based video processing module. The infrared camera acquires video data of the mouse's face and connects to the PC-based video processing module to transmit and store the video data. The experimental condition control module sends stimuli to the mouse and sends level signals to the Arduino development board. The Arduino development board sends pin feedback information to the PC-based video processing module, which controls the infrared camera based on the OpenCV library and adds color labels to the video images.

[0031] The experimental condition control module controls the release of stimulus conditions, including the release of different proportions of stimulating gases, the release of different voltages, or other stimulus experimental conditions that need to be set.

[0032] The Arduino development board is connected to the experimental condition control module via DuPont wires. The Arduino development board detects the low-level signals of each pin and establishes a one-to-one mapping relationship between each pin and the feedback information of different pins. The pin feedback information can be numbers, English characters, or other characters. In this embodiment, numbers are used for the condition trigger pins, and English characters are used for the start pins. For example, condition trigger pin A0 corresponds to the character "0", condition trigger pin A1 corresponds to the character "1", condition trigger pin A2 corresponds to the character "2", condition trigger pin A3 corresponds to the character "3", condition trigger pin A4 corresponds to the character "4", condition trigger pin A5 corresponds to the character "5", start pin 8 and end pin 13 correspond to the characters "s" and "e" respectively. The number of pins can be adjusted to correspond one-to-one with the trigger conditions. The character signals are sent to the PC video processing module via serial communication, and the low-level transmission signal time is coupled with the experimental condition trigger time.

[0033] By defining the mapping relationship between different stimulus conditions and characters through software, the rigor of the experiment design and data collection is strictly ensured. When designing the experiment, only the color and stimulus condition need to be matched, and the triggering of the condition and the different color markings can be ensured to have zero error during the experiment. The character signal and the low-level signal are triggered simultaneously, ensuring consistency in time.

[0034] The experimental condition control module controls the release of stimulus conditions, including the release of stimuli such as gases and voltages. When a stimulus condition is released, a low-level signal is sent to pins A0-A5 of the Arduino development board. Video recording is started and stopped by the S and E keys on the PC's video processing module, or by the Arduino's start pin 8 and end pin 13. The experimental condition control module sends a low-level signal to the Arduino development board's start pin 8 when starting and a low-level signal to the Arduino development board's end pin 13 when stopping. The recording of video frames can be manually started or stopped, or controlled by the experimental condition control module, increasing the flexibility of experimental progress control. The experimental condition control module can also completely control the entire process without requiring manual intervention.

[0035] The PC-side video processing module establishes a one-to-one mapping between different numeric characters and different color labels, as shown in Table 1. The PC-side video processing module controls video recording. The OpenCV library calls the camera to start recording. The module receives video data, checks the serial port status, and when a numeric character matches successfully, it uses the OpenCV function `cv.addWeighted()` to label the corresponding video frame with a color. The module stores the recorded consecutive frames of video data. The pin feedback information for successful matching is coupled with the label labeling time. The correspondence between stimulus conditions and color labels is shown in Table 1. The PC-side video processing module receives character signals and labels labels simultaneously, ensuring temporal consistency and guaranteeing the accuracy and completeness of the video data under experimental stimulus conditions.

[0036]

[0037] Table 1

[0038] The PC-based video processing module starts timing when it detects the character "s" or the S key on the keyboard; it records the corresponding time when different numeric characters are detected, indicating the triggering of the experimental condition; and it stops timing when it detects the character "e" or the E key on the keyboard, generating and storing a table of experimental condition triggers and time points. This table is generated for subsequent experimental data analysis, providing a temporal perspective on the relationship between animal behavior and neuroscience, and also serving as a data log.

[0039] The OpenCV library is used to call the video reading function to extract video frames with the same color label and save them as video files.

[0040] Example 2

[0041] This invention provides a method for creating video datasets using a mouse facial video dataset creation system, comprising the following steps:

[0042] Step 1: The PC-side video processing module controls the infrared camera to turn on and capture video of the mouse's face. The video data is stored in the PC-side video processing module.

[0043] Step 2: Start the experimental condition control module. The experimental condition control module sends a low-level signal to the start pin 8 of the Arduino development board. After the Arduino development board detects the low-level signal of the start pin 8, it sends the character "s" to the PC video processing module through serial communication. The PC video processing module calls the camera to start the recording function and receives the video data.

[0044] Step 3: The experimental condition control module controls the release of the stimulus condition and sends a low-level signal to the A0-A5 pins of the Arduino development board. The Arduino development board detects the low-level pulse and sends a digital character signal to the video processing module on the PC through serial communication.

[0045] Step 4: The PC-side video processing module detects the serial port status. After a successful character match, it marks the corresponding color label in the current video frame. The PC-side video processing module then stores the recorded consecutive video frames.

[0046] Step 5: When the experiment ends, if the PC video processing module detects the character "e", the PC video processing module will record the video file and save it to the specified path.

[0047] Preferably, in step two, the PC-side video processing module enables the timing function and records the start time; in step four, it records the trigger time points for different experimental conditions; and in step five, it records the end time, generating and storing a table of experimental condition triggers and time points.

[0048] Preferably, the method also includes step six, in which the video processing module on the PC reads the video, performs pixel value detection of different colors at the reserved positions of the collected data, classifies the data according to the detected pixel values, extracts video frames with the same color labels, and saves the video files according to the classification.

[0049] The stimulation conditions in this embodiment are shown in Table 1 as an example. The purpose was to obtain the facial movements of mice under six experimental stimuli. Different proportions of gas and different voltages were released on the mice. Each stimulus was performed 100 times, with a stimulation time of 2 seconds and an interval of 5 seconds. The total video frame length was approximately 80 minutes, and a total of 600 video data segments of six categories were generated.

[0050] The generated table of experimental condition triggers and time points is shown below. Figure 2 As shown. The result of randomly selecting a frame of image from 600 segments of data across six categories is as follows. Figure 3 As shown, the color label is placed in the upper right corner.

[0051] The method of this invention only requires ensuring a one-to-one correspondence between color labels and stimulus conditions. When starting the device, only the video save path needs to be entered. Upon completion of the experiment, a time record table and video segments categorized by color are automatically generated. This saves the time and labor costs associated with manual annotation, significantly improving the efficiency of dataset creation. Due to the coupling of the various signals, the accuracy and completeness of the extracted video segments under the stimulus conditions are guaranteed.

[0052] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A system for creating a mouse facial video dataset for deep learning, comprising an infrared camera, an experimental condition control module, an Arduino development board, and a PC-based video processing module, characterized in that, The infrared camera acquires video data of the mouse's face and connects to a PC-side video processing module to transmit and store the video data. The experimental condition control module sends stimulus conditions to the mouse and sends level signals to the Arduino development board. The Arduino development board sends pin feedback information to the PC-side video processing module, which controls the infrared camera based on the OpenCV library and marks color labels in the video frames. The experimental condition control module controls the release of stimulus conditions, which are either the release of an irritant gas or a voltage release. When the stimulus condition is released, it sends a low-level signal to pins A0-A5 of the Arduino development board. The Arduino development board is connected to the experimental condition control module via DuPont wires. The Arduino development board detects the low-level signals of each pin and establishes a one-to-one mapping relationship between each pin and the feedback information of different pins. The pin feedback information is a number or an alphanumeric character: condition trigger pin A0 corresponds to the character "0", condition trigger pin A1 corresponds to the character "1", condition trigger pin A2 corresponds to the character "2", condition trigger pin A3 corresponds to the character "3", condition trigger pin A4 corresponds to the character "4", and condition trigger pin A5 corresponds to the character "5". Start pin 8 and end pin 13 correspond to the characters "s" and "e" respectively, sending the character signals to the PC-side video processing module via serial communication. The low-level transmission signal time is coupled with the experimental condition trigger time. The PC-side video processing module establishes a one-to-one mapping relationship between different numeric characters and different color labels. The PC-side video processing module controls video recording, calls the camera to start the recording function, receives video data, detects the serial port status, and marks the corresponding video frame with a color label when a numeric character is successfully matched. The PC-side video processing module stores the recorded consecutive frames of video data, and the pin feedback information matching success time is coupled with the label marking time. The PC-side video processing module extracts video frames with the same color label and saves them as video files according to their categories.

2. The mouse facial video dataset creation system according to claim 1, characterized in that, The start and end of video recording can be controlled by the S and E keys on the PC-side video processing module, or by the start pin 8 and end pin 13 of the Arduino. When the experimental conditions control module starts, it sends a low-level signal to the start pin 8 of the Arduino development board, and when the experimental conditions control module terminates, it sends a low-level signal to the end pin 13 of the Arduino development board.

3. The mouse facial video dataset creation system according to claim 1, characterized in that, The PC-side video processing module starts timing when it detects the character "s" or the S key on the keyboard is triggered; it records the corresponding time when different numeric characters are detected, based on the trigger time of the corresponding experimental conditions. The timing function ends when the character "e" is detected or the E key on the keyboard is triggered, and a table of experimental condition triggers and time points is generated and stored.

4. The method for creating a video dataset using the mouse facial video dataset creation system according to any one of claims 1-3, characterized in that, Includes the following steps: Step 1: The PC-side video processing module controls the infrared camera to turn on and capture video of the mouse's face. The video data is stored in the PC-side video processing module. Step 2: Start the experimental condition control module. The experimental condition control module sends a low-level signal to the start pin 8 of the Arduino development board. After the Arduino development board detects the low-level signal of the start pin 8, it sends the character "s" to the PC video processing module through serial communication. The PC video processing module calls the camera to start the recording function and receives the video data. Step 3: The experimental condition control module controls the release of the stimulus condition and sends a low-level signal to the A0-A5 pins of the Arduino development board. The Arduino development board detects the low-level pulse and sends a digital character signal to the PC video processing module through serial communication. Step 4: The PC-side video processing module detects the serial port status. After a successful character match, it marks the corresponding color label in the current video frame. The PC-side video processing module stores the recorded consecutive video frames. Step 5: When the experiment ends, if the PC video processing module detects the character "e", the PC video processing module will record the video file and save it to the specified path.

5. The video dataset creation method according to claim 4, characterized in that, In step two, the PC-side video processing module enables the timing function and records the start time; in step four, it records the trigger time points for different experimental conditions; in step five, it records the end time and generates and stores a table of experimental condition triggers and time points.

6. The video dataset creation method according to claim 4, characterized in that, The process also includes step six, where the PC-side video processing module reads the video, detects the pixel values ​​of different colors at the reserved positions on the tags, classifies the data according to the detected pixel values, extracts video frames with the same color tags, and saves the video files accordingly.

Images