Animal behavior pattern mining system and method, computer equipment and storage medium

Abstract

The invention relates to an animal behavior pattern mining system and method, computer equipment and a storage medium. The system comprises PC equipment and an experiment box body, wherein an infrared touch screen, a transparent box body, a camera, a thermal infrared probe and a brain endoscope are respectively arranged in the experiment box body; the transparent box body is used for placing an experimental animal, the infrared touch screen, the camera and the thermal infrared probe are respectively used for collecting track data, video images and body temperature data of the experimental animal, and the brain endoscope is used for collecting neuron signals of the experimental animal; and the PC equipment is used for acquiring the behavior pattern category of the experimental animal in combination with the trajectory data, the video image, the body temperature data and the neuron signal. According to the embodiment of the invention, the synchronous recording of multi-modal data is realized, the influence of illumination, noise, camera jitter and other factors on the video image is avoided, and the recognition accuracy of the animal behavior mode is improved.

Description

Technical Field

[0001] The present application belongs to the field of biological technology, and in particular, relates to an animal behavior pattern mining system, method, computer equipment and storage medium. Background Art

[0002] Animal behavior research is extremely important in the field of biology. By observing and recording animal behavior, we can study the internal states of animals such as animal psychology, neuroscience, and pharmacology. When observing and recording animal behavior, certain methods and means need to be adopted. In the prior art, the mainstream method of animal behavior recording is camera recording. For example, patent document CN202111628940.1 introduces a small animal behavior recording device with precise visual stimulation, which uses a camera module above to perform a two-dimensional top-down image recording of the animal behavior below. Another patent document CN201611174778.X introduces a system for synchronously recording animal neuronal signals and behaviors, which mainly uses a CCD (charge coupled device) camera to record video images of experimental animals when they produce corresponding behaviors, and judge whether corresponding signals are generated in the brain area of ​​the experimental subject based on the video images.

[0003] In summary, the animal behavior recording method in the prior art only stays at two-dimensional video recording. The video images shot are too single and will be affected by factors such as lighting, noise and camera shake, which will have a certain impact on the quality of the final video image. In addition, there is a lack of multimodal signal combination and the assistance of advanced algorithms. In addition, the different actions recorded by the two-dimensional video are not independent and single to each other, and there are certain patterns and rules between them. Simple two-dimensional video recording plus human subjective judgment can easily overlook various details, and lack the ability of objective synthesis at a macro level, which affects the accuracy of judging animal behavior patterns. Summary of the invention

[0004] The present application provides an animal behavior pattern mining system, method, computer device and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0005] In order to solve the above problems, this application provides the following technical solutions:

[0006] An animal behavior pattern mining system comprises a PC device and an experimental box, wherein the PC device is located outside the experimental box, and the experimental box is also provided with an infrared touch screen, a transparent box, a camera, a thermal infrared probe and a brain endoscope; the infrared touch screen is arranged at the bottom of the experimental box, the transparent box is arranged above the infrared touch screen, the camera and the thermal infrared probe are respectively fixed on the transparent box, and the infrared touch screen, the camera, the thermal infrared probe and the brain endoscope are respectively connected to the PC device; the transparent box is used to place experimental animals, the infrared touch screen, the camera and the thermal infrared probe are respectively used to collect trajectory data, video images and body temperature data of the experimental animals, and the brain endoscope is used to collect neuron signals of the experimental animals; the PC device is used to obtain the behavior pattern category of the experimental animals in combination with the trajectory data, video images, body temperature data and neuron signals.

[0007] The technical solution adopted in the embodiment of the present application also includes: the experimental box is a square structure with an aluminum alloy as the frame, the six surfaces of the experimental box are covered with opaque acrylic panels, the inner sides of the six surfaces are all affixed with sound-absorbing panels, and one side of the experimental box is a detachable structure for placing experimental animals during experiments.

[0008] The technical solution adopted in the embodiment of the present application also includes: the experimental box also includes a development board, a sound wave generator and a display, the development board is arranged on the upper surface of the inner side of the experimental box, the sound wave generator and the display are respectively closely attached to the upper inner side of the experimental box, and are connected to the development board and the PC device in sequence through wires; the PC device is used to control the development board to generate stimulation signals, and make the sound wave generator and the display connected thereto generate auditory and visual stimulations respectively.

[0009] The technical solution adopted in the embodiment of the present application also includes: the experimental box also includes a data acquisition card, the infrared touch screen, camera, thermal infrared probe and brain endoscope are respectively connected to the data acquisition card, and the data acquisition card is connected to the PC device; the data acquisition card is used to synchronize and package the trajectory data, video images, body temperature data and neuronal signals, and then transmit the packaged data to the PC device.

[0010] The technical solution adopted in the embodiment of the present application also includes: the PC device includes a deep convolutional network and a cluster mining module, and the PC device is used to combine trajectory data, video images, body temperature data and neuron signals to obtain the behavior pattern category of the experimental animal, specifically:

[0011] Automatically identifying the action category of the experimental animal in each frame of the video image through the deep convolutional network, and outputting the action category sequence of the video image after correcting the action category through trajectory data, body temperature data and neuron signals;

[0012] The cluster mining module combines trajectory data, body temperature data and neuron signals to cluster the action category sequence, and corresponds the clustered action category sequence to the time series of the video image to obtain the behavior pattern category sequence of the experimental animal.

[0013] The technical solution adopted by the embodiment of the present application also includes: the action category of the experimental animal in each frame of the video image is automatically identified by the deep convolutional network:

[0014] Extracting all frame images from the video image, and randomly selecting a set number of frame images as a data set that needs to be manually labeled;

[0015] Frame images of various categories are sorted out from the data set for manual labeling, and the manually labeled training set images are input into a deep convolutional network for training; wherein the deep convolutional network selects ResNet50 as the backbone network;

[0016] All frames of the video image are input into a trained deep convolutional network, and the action category of the experimental animal in each frame is automatically identified by the deep convolutional network;

[0017] The fully connected layer of the deep convolutional network is trained by using the trajectory data, body temperature data and neuron signals, and the action category sequence of the video image is output after the action category of the experimental animal in each frame is corrected.

[0018] The technical solution adopted by the embodiment of the present application also includes: the cluster mining module includes a synchronization program, a clustering program and a visualization program, and the cluster mining module is used to cluster the action category sequence by combining trajectory data, body temperature data and neuron signals. Specifically:

[0019] Using the synchronization program to retrieve the packaged data, synchronizing all the packaged data onto a time axis according to the start and end time of the packaged data, the time marking signal, and the wavelet transform detection, and outputting a multidimensional array including the time;

[0020] The action category sequence and the multidimensional array are input into a clustering program for automatic clustering. The clustering program uses the correlation difference between arrays of different dimensions between different action categories as a distinction, finds the action categories of the same category that are closely related from the multidimensional data to generate behavior pattern categories, and corresponds the behavior pattern categories to the time series of the video image to obtain the behavior pattern category sequence;

[0021] The behavior pattern category sequence is converted into a subtitle file through the visualization program, and the subtitle file is displayed in the video image.

[0022] Another technical solution adopted by the embodiment of the present application is: a method for mining animal behavior patterns, comprising:

[0023] The trajectory data, video images, body temperature data and neuron signals of experimental animals are collected through infrared touch screens, cameras, thermal infrared probes and brain endoscopes;

[0024] Uploading the trajectory data, video images, body temperature data, and neuron signals to a PC device;

[0025] The behavior pattern category of the experimental animal is obtained by combining the trajectory data, video images, body temperature data and neuron signals through the PC device.

[0026] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0027] The memory stores program instructions for implementing the animal behavior pattern mining method;

[0028] The processor is used to execute the program instructions stored in the memory to control the animal behavior pattern mining method.

[0029] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the animal behavior pattern mining method.

[0030] Compared with the prior art, the beneficial effects produced by the embodiments of the present application are as follows: the animal behavior pattern mining system, method, computer equipment and storage medium of the embodiments of the present application utilize a variety of electronic components to realize the synchronous recording of multimodal data such as video images, infrared temperature, real-time position and neuronal signals, which can capture the behavioral state of experimental animals in an all-round and three-dimensional manner, and avoid the influence of factors such as light, noise and camera jitter on the video image, and then use a deep convolutional network to automatically identify the action category of the experimental animal in each frame of the video image, and combine the action category, trajectory data, body temperature data and neuronal signals to predict the behavior pattern category of the experimental animal, thereby making full use of the patterns and laws between different actions in the video image, which is conducive to improving the recognition accuracy of animal behavior patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a structural diagram of an animal behavior pattern mining system according to an embodiment of the present application;

[0032] Figure 2 It is a deep convolutional network workflow diagram of an embodiment of the present application;

[0033] Figure 3 is a workflow diagram of a cluster mining module according to an embodiment of the present application;

[0034] Figure 4 is a flow chart of the animal behavior pattern mining method of an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the computer device structure of an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0039] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0040] See also Figure 1, is a schematic diagram of the structure of the animal behavior pattern mining system of the embodiment of the present application. The animal behavior pattern mining system of the embodiment of the present application includes a PC device 10 and an experimental box 20. The PC device 10 is located outside the experimental box 20. The experimental box 20 is a square structure with an aluminum alloy frame. The six surfaces of the experimental box are covered with opaque acrylic plates, and the inner sides of the six surfaces are all affixed with sound-absorbing panels (not shown). One side of the experimental box 20 is a detachable structure for placing experimental animals during experiments. The experimental box 20 also includes a development board 21, a sound wave generator 22, a display 23, a data acquisition card 24, an infrared touch screen 25, a transparent box 26, a camera 27, a thermal infrared probe 28 and a brain endoscope 29; wherein the development board 21 is arranged on the upper surface of the inner side of the experimental box 20, the sound wave generator 22 and the display 23 are respectively attached to the upper inner side of the experimental box 20, and are connected to the development board 21 and the PC device 10 in turn through wires; the data acquisition card 24 is attached to the left surface of the inner side of the experimental box 20, and is connected to the PC device 10 through wires; the infrared touch screen 25 It is arranged at the bottom of the experimental box 20 and is connected to the data acquisition card 24 through wires; the transparent box 26 is a device composed of three transparent planes and one transparent curved surface, and is arranged above the infrared touch screen 25. The camera 27 and the thermal infrared probe 28 are respectively fixed on the two transparent surfaces of the transparent box 26 by fixing components (not shown in the figure), and the focal points of the camera 27 and the thermal infrared probe 28 are respectively aligned with the center position of the transparent box 26, and are connected to the data acquisition card 24 through wires; the brain endoscope 29 is placed in the experimental box 20 and is connected to the data acquisition card 24 through wires.

[0041] Specifically, the development board 21 is an Arduino development board, and the PC device 10 is used to control the development board 21 to generate specific signals, so that the sound wave generator 22 and the display 23 generate auditory and visual stimulations respectively, and the programs of the sound wave generator 22 and the display 23 can be designed by themselves according to the experimental requirements through languages ​​such as Python or C++. The above control operations can be set by the computer programming of the PC device 10. The frame rate of the infrared touch screen 25 is set to 120fps, which is used to record the trajectory data of the experimental animals. The camera 27 is a USB wide-angle camera with a resolution of 1960*1080 and a frame rate of 60fps, which is used to collect the intuitive behavior of the experimental animals. The frame rate of the thermal infrared probe 28 is 60fps, and the background temperature noise is removed during operation, which is used to record the body temperature data of the experimental animals in real time. The fixing components used to fix the camera 27 and the thermal infrared probe 28 include but are not limited to screws. The brain endoscope 29 is used to observe the neuronal activity of the experimental animals through optical signal detection, and collect neuronal signals corresponding to the neuronal activity of the experimental animals. It can be understood that the product types and parameters of the above-mentioned development board 21, sound wave generator 22, display 23, data acquisition card 24, infrared touch screen 25, camera 27, thermal infrared probe 28 and brain endoscope 29 can be selected according to the actual application scenario.

[0042] In the embodiment of the present application, the PC device 10 includes but is not limited to electronic devices such as desktop computers, all-in-one computers, laptop computers or tablet computers, and a single-chip microcomputer can also be used as an alternative to PC devices. The six sides of the experimental box 20 are covered with opaque acrylic panels, and sound-absorbing panels are attached to the inside, so as to effectively isolate external noise and other interfering phonemes while ensuring the airtightness of the experimental box, and avoid interference with the experiment caused by other environmental factors. It can be understood that the experimental box 20 can also be equipped with or replaced with other special materials for other special environments such as high / low temperatures.

[0043] Further, the implementation process of the animal behavior pattern mining system of the embodiment of the present application is as follows: before the experiment begins, the PC device 10 is started, and the sound wave generator 22, the display 23, the infrared touch screen 25, the camera 27, the thermal infrared probe 28 and the brain endoscope 29 are driven by this. At the beginning of the experiment, the panel on one side of the experimental box 20 is opened, the brain endoscope 29 placed in the experimental box 20 is inserted into the skull opening of the experimental animal, and the experimental animal is placed in the transparent box 26 in the experimental box 20, wherein the experimental animal is injected with an experimental virus to mark specific proteins in its neurons. Then, each electronic device in the experimental box 20 is debugged, and it is tested whether the display 23 displays an image, whether the sound wave generator 22 makes a sound, whether the camera 27 and the thermal infrared probe 28 capture the picture in the transparent box 26, whether the infrared touch screen 25 collects the position of the experimental animal, and whether the brain endoscope 29 collects the neuron signal of the experimental animal. After the test is completed, the panel on one side of the experimental box 20 is closed and the experiment is started. After the experiment starts, the PC device 10 controls the development board 21 to generate a specific stimulation signal, so that the sound wave generator 22 and the display 23 connected thereto generate auditory and visual stimulation respectively. At the same time, the trajectory data of the experimental animal is tracked by the infrared touch screen 25 using infrared reflection, the video image of the experimental animal is collected in real time by the camera 27, the body temperature data of the experimental animal is collected by the thermal infrared probe 28, and the neuronal signal of the experimental animal is collected by the brain endoscope 29. Then, the acquired multimodal data such as trajectory data, video images, body temperature data and neuron signals are uploaded to the data acquisition card 24 in real time for synchronization and packaging processing, and then the packaged data is transmitted to the PC device 10 through the data acquisition card 24. The PC device 10 automatically identifies the action category of the experimental animal in each frame of the video image through the deep convolution network trained by the artificially labeled pictures, and corrects the action category through the trajectory data, body temperature data and neuron signals, and outputs the action category sequence of the video image, and then clusters the action category sequence by combining the trajectory data, body temperature data and neuron signals through the clustering mining module, and corresponds the clustered action category sequence to the time series of the video image to obtain the behavior pattern category sequence of the experimental animal. It can be understood that the experimental animals include but are not limited to small experimental animals such as rats, mice, guinea pigs, or large experimental animals such as cats, dogs, monkeys, etc., and the size of the experimental box 20 and the transparent box 26 can be adjusted according to the size of the experimental animals.

[0044] Specifically, the main body of the deep convolutional network is composed of a convolutional neural network. After the manually labeled pictures are input into it for training, it can automatically identify the action category corresponding to each frame of the video image taken by a small camera. In order to deploy the deep convolutional network and embed it into the system, the embodiment of the present application optimizes and improves the conventional deep neural network operation facilities. The relevant program is written in python and is run by a workstation computer equipped with GeForce RTX 3050. The video image is first preprocessed and split into pictures with different numbers of frames. Then a certain number of pictures are randomly selected as the data set to be marked, and the data set is input into the ResNet50 network for training. After the training is completed, the captured video image is input into the ResNet network, and the predicted value is output as the output result of the experimental animal action category. It can be understood that the ResNet50 network can also be replaced by other types of deep convolutional networks such as ResNet101.

[0045] For further information, see Figure 2 , is a flowchart of the deep convolutional network workflow of an embodiment of the present application. The action category recognition process of the deep convolutional network specifically includes:

[0046] The first step is to extract all the frames from the video image and randomly select a certain number of frames as the data set that needs to be manually labeled;

[0047] In the second step, the program's GUI (Graphical User Interface) is used to sort out the frame images of each category from the data set for manual labeling, and the manually labeled training set images are input into the deep convolutional network for training; in particular, the frame image sample size needs to be increased through the preprocessing method of data augmentation to ensure that the number of labeled images of each category can meet the network training requirements; the deep convolutional network selects ResNet50 as the backbone network, sets the learning rate, the number of samples per training, and the update function, and then starts the training mode; during the training process, the loss function and the accuracy change curve will be displayed on the screen and recorded in the document in real time. After the training is completed, the previous parameters are adjusted according to its performance and iterated again;

[0048] The third step is to adjust the trained deep convolutional network to the test mode, and input all the frames of the video image into the deep convolutional network, so as to automatically identify the action category of the experimental animal in each frame through the deep convolutional network;

[0049] The fourth step is to train the fully connected layer of the deep convolutional network by using the trajectory data, body temperature data, and neuron signals in the packaged data, and after correcting the action categories of the experimental animals, output the action category sequence of the video image.

[0050] Further, the cluster mining module includes a synchronization program, a clustering program and a visualization program. The synchronization program is used to normalize and synchronize the packaged data in the data acquisition card 22, and output a multidimensional array including time and each recorded data sequence. The clustering program is rewritten by a clustering method based on Toeplitz inverse covariance (Toeplitzinverse covariance-based clustering, referred to as TICC), the core of which is deep learning and Markov process, which is used to distinguish the correlation difference between different dimensional arrays of different action categories, automatically find the same type of closely related action categories from the multidimensional array, generate behavior pattern categories, and finally output sequences marked with different behavior pattern categories. The visualization program is used to display the behavior pattern category sequence in the form of subtitles on the time series in the original video image, so that researchers can determine whether the behavior pattern recognition is accurate, and adjust the parameter settings of the clustering program accordingly. It can be understood that the TICC clustering algorithm can also be replaced by other multidimensional data clustering algorithms such as K-means.

[0051] Specifically, Figure 3 The figure is a workflow diagram of the cluster mining module of the embodiment of the present application. The cluster mining module matches the output result of the deep convolutional network to the matching process in the corresponding time series of the video image, which specifically includes:

[0052] The first step is to use the synchronization program to retrieve the packaged data uploaded by the data acquisition card, synchronize all the packaged data to a time axis according to the start and end time of the packaged data, the time marking signal and the wavelet transform detection, and output a multidimensional array including multimodal data such as time;

[0053] The second step is to input the action categories output by the deep convolutional network together with the multidimensional array into the clustering program, and after setting the parameters of the clustering program such as the number of clusters, convergence criteria, and interval duration, run the clustering program to automatically cluster and generate behavior pattern categories, and correspond the behavior pattern categories to the time series of the video image to obtain the behavior pattern category sequence;

[0054] In the third step, the behavior pattern category sequence is converted into a subtitle file through a visualization program and displayed in the video image. The researchers use the video image and subtitle file to determine whether the behavior pattern recognition is correct, and adjust the relevant parameters of the clustering program and iterate again until the behavior pattern category is consistent with the researchers' empirical judgment. The parameters of the clustering program are fixed for subsequent data clustering.

[0055] Based on the above, the animal behavior pattern mining system of the embodiment of the present application uses a variety of electronic components to realize the synchronous recording of multimodal data such as video images, infrared temperature, real-time position and neuron signals, which can capture the behavioral state of experimental animals in an all-round and three-dimensional manner, and avoid the influence of factors such as light, noise and camera jitter on the video image, and then use the deep convolution network to automatically identify the action category of the experimental animal in each frame of the video image, and combine the action category, trajectory data, body temperature data and neuron signals to predict the behavior pattern category of the experimental animal, thereby making full use of the patterns and laws between different actions in the video image, which is conducive to improving the recognition accuracy of animal behavior patterns. The present application is convenient to configure, has no restrictions on the use time, and can be widely used in experimental animals of different types and sizes.

[0056] See also Figure 4 , is a flow chart of the animal behavior pattern mining method of the embodiment of the present application. The animal behavior pattern mining method of the embodiment of the present application comprises the following steps:

[0057] S1: Open the panel on one side of the experimental box, insert the brain endoscope into the skull opening of the experimental animal, and place the experimental animal into the transparent box inside the experimental box;

[0058] S2: Debug each electronic device in the experimental box and test whether each electronic device can run normally under the program drive of the PC device. If yes, execute S3; otherwise, continue to execute S2;

[0059] In this step, the electronic equipment in the experimental box includes a sound wave generator, a display, an infrared touch screen, a camera, a thermal infrared probe and a brain endoscope. The sound wave generator and the display are used to generate auditory and visual stimulation respectively, the camera is used to collect the intuitive behavior of the experimental animals, the infrared touch screen is used to record the behavioral trajectory of the experimental animals, the thermal infrared probe is used to record the body temperature data of the experimental animals in real time, and the brain endoscope is used to detect the neuronal activity of the experimental animals through optical signals and collect the corresponding neuronal signals. Testing whether each electronic device can operate normally under the program drive of the PC device is specifically: testing whether the display displays an image, whether the sound wave generator makes a sound, whether the camera and the thermal infrared probe capture the picture in the transparent box, whether the infrared touch screen collects the position of the experimental animals, and whether the brain endoscope collects the neuronal signals of the experimental animals.

[0060] S3: Collect multimodal data of experimental animals through various electronic devices in the experimental box, upload the multimodal data to the data acquisition card in real time for synchronization and packaging, and then transmit the packaged data to the PC device;

[0061] In this step, the multimodal data includes trajectory data, video images, body temperature data, and neuronal signals. The data acquisition card is attached to the inside of the experimental box and connected to the PC device through wires. It is used to collect multimodal data collected by various electronic devices in the experimental box and transmit the multimodal data to the PC device for behavioral pattern recognition of experimental animals.

[0062] S4: receiving the packaged data through a PC device, splitting the video image in the packaged data into frame images, inputting the split frame images into a deep convolutional network trained by manually labeled images for action recognition, and outputting the action category sequence of the experimental animal in the video image;

[0063] In this step, the main body of the deep convolutional network is composed of a convolutional neural network. After the artificially marked pictures are input into it for training, the action category corresponding to each frame of the video image taken by the camera can be automatically identified. The video image is first pre-processed and split into images of different frames, and then a part of the frame pictures are randomly selected as the data set that needs to be marked, and the data set is input into the deep convolutional network for training. After the training is completed, the captured video image is input into the deep convolutional network, and the predicted value is output as the output result of the experimental animal action category. In the embodiment of the present application, it can be understood that the deep convolutional network selects the ResNet50 network as the backbone network, and the ResNet50 network can also be replaced by other types of deep convolutional networks such as ResNet101.

[0064] Specifically, the behavior pattern category recognition process of the deep convolutional network includes:

[0065] The first step is to extract all the frames from the video image and randomly select a certain number of frames as the data set that needs to be manually labeled;

[0066] In the second step, the program's GUI is used to sort out pictures of each category from the data set for manual labeling, and the manually labeled training set pictures are input into the deep convolutional network for training. In this process, the image sample size needs to be increased through the preprocessing method of data augmentation to ensure that the number of labeled pictures of each category is more than one thousand. After setting parameters such as the network learning rate, the number of samples for each training, and the update function, the training mode is started. During the training process, the loss function and the accuracy change curve will be displayed on the screen and recorded in the document in real time. After the training is completed, the previous parameters will be adjusted according to its performance and iterated again.

[0067] The third step is to adjust the trained deep convolutional network to the test mode, and input all the frames of the video image into the deep convolutional network, so as to automatically identify the action category of the experimental animal in each frame through the deep convolutional network;

[0068] The fourth step is to train the fully connected layer of the deep convolutional network by using the trajectory data, body temperature data, and neuron signals in the packaged data, and after correcting the action categories of the experimental animals, output the action category sequence of the video image.

[0069] S5: clustering the action category sequence by combining the trajectory data, body temperature data and neuron signals through a cluster mining module to generate behavior pattern categories, and corresponding the behavior pattern categories to the time series of the video images to obtain the behavior pattern category sequence of the experimental animal;

[0070] In this step, the cluster mining module is composed of a synchronization program, a clustering program, and a visualization program. The synchronization program is used to normalize and synchronize the packaged data in the data acquisition card, and output a multidimensional array including time and each recorded data sequence. The clustering program is rewritten from a clustering method based on Toeplitz inverse covariance. Its core is deep learning and Markov process. It is used to distinguish between different action categories by the correlation difference of different dimensional arrays. It automatically finds the same type of closely related action categories from the multidimensional array to form a behavior pattern category, and finally outputs a time series marked with different behavior pattern categories. The visualization program is used to display the behavior pattern category sequence in the form of subtitles on the time series in the video image, which is convenient for researchers to determine whether the behavior pattern recognition is accurate, and adjust the parameter settings of the clustering program accordingly. It can be understood that the TICC clustering algorithm can also be replaced by other multidimensional data clustering algorithms such as K-means.

[0071] Specifically, the cluster mining module combines trajectory data, body temperature data and neuron signals to cluster action category sequences, including:

[0072] The first step is to use the synchronization program to retrieve the packaged data uploaded by the data acquisition card, synchronize all the packaged data to a time axis according to the start and end time of the packaged data, the time marking signal and the wavelet transform detection, and output a multidimensional array including multimodal data such as time;

[0073] The second step is to input the action category sequence output by the deep convolutional network together with the multidimensional array into the clustering program, and after setting the parameters of the clustering program such as the number of clusters, convergence criteria, and interval duration, run the clustering program to automatically cluster and generate behavior pattern categories, and correspond the behavior pattern categories to the time series of the video image to obtain the behavior pattern category sequence;

[0074] In the third step, the behavior pattern category sequence is converted into a subtitle file through a visualization program and displayed in the original video image. The researchers use the video image and subtitle file to determine whether the behavior pattern recognition is correct, and adjust the relevant parameters of the clustering program and iterate again until the behavior pattern category is consistent with the researchers' experience. Then, the parameters of the clustering program are fixed for subsequent data clustering.

[0075] Based on the above, the animal behavior pattern mining method of the embodiment of the present application uses a variety of electronic components to realize the synchronous recording of multimodal data such as video images, infrared temperature, real-time position and neuron signals, which can capture the behavioral state of experimental animals in an all-round and three-dimensional manner, and avoid the influence of factors such as light, noise and camera jitter on the video image. Then, the deep convolutional network is used to automatically identify the action category of the experimental animal in each frame of the video image, and the action category, trajectory data, body temperature data and neuron signal are combined to predict the behavior pattern category of the experimental animal, thereby making full use of the patterns and laws between different actions in the video image, which is conducive to improving the recognition accuracy of animal behavior patterns. The present application is convenient to configure, has no restrictions on the use time, and can be widely used in experimental animals of different types and sizes.

[0076] See also Figure 5 , is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes:

[0077] A memory 51 storing executable program instructions;

[0078] A processor 52 connected to the memory 51;

[0079] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: respectively collect the trajectory data, video images, body temperature data and neuron signals of the experimental animals through the infrared touch screen, camera, thermal infrared probe and brain endoscope; upload the trajectory data, video images, body temperature data and neuron signals to the PC device; and obtain the behavior pattern category of the experimental animals through the PC device in combination with the trajectory data, video images, body temperature data and neuron signals.

[0080] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0081] See also Figure 6 , is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program instruction 61 that can implement the following steps: collect the trajectory data, video images, body temperature data and neuron signals of the experimental animals respectively through an infrared touch screen, a camera, a thermal infrared probe and a brain endoscope; upload the trajectory data, video images, body temperature data and neuron signals to a PC device; obtain the behavior pattern category of the experimental animals through the PC device in combination with the trajectory data, video images, body temperature data and neuron signals. Among them, the program instruction 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions to enable a device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media that can store program instructions, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet. Among them, the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0082] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0083] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An animal behavior pattern mining system, characterized in that: It comprises a PC device and an experimental box, wherein the PC device is located outside the experimental box, and an infrared touch screen, a transparent box, a camera, a thermal infrared probe and a brain endoscope are respectively arranged inside the experimental box; the infrared touch screen is arranged at the bottom of the experimental box, and the transparent box is arranged above the infrared touch screen, the camera and the thermal infrared probe are respectively fixed on the transparent box, and the infrared touch screen, the camera, the thermal infrared probe and the brain endoscope are respectively connected to the PC device; the transparent box is used to place experimental animals, the infrared touch screen, the camera and the thermal infrared probe are respectively used to collect trajectory data, video images and body temperature data of the experimental animals, and the brain endoscope is used to collect neuron signals of the experimental animals; the PC device is used to obtain the behavior pattern category of the experimental animals in combination with trajectory data, video images, body temperature data and neuron signals.

2. The animal behavior pattern mining system according to claim 1, characterized in that: The experimental box is a square structure with an aluminum alloy frame. The six surfaces of the experimental box are covered with opaque acrylic panels, and the inner sides of the six surfaces are all affixed with sound-absorbing panels. One side of the experimental box is a detachable structure for placing experimental animals during experiments.

3. The animal behavior pattern mining system according to claim 2, characterized in that: The experimental box also includes a development board, a sound wave generator and a display. The development board is arranged on the upper surface of the inner side of the experimental box. The sound wave generator and the display are respectively attached to the upper inner side of the experimental box and are connected to the development board and the PC device in sequence through wires. The PC device is used to control the development board to generate stimulation signals, and make the sound wave generator and the display connected thereto generate auditory and visual stimulations respectively.

4. The animal behavior pattern mining system according to claim 3, characterized in that: The experimental box also includes a data acquisition card, the infrared touch screen, camera, thermal infrared probe and brain endoscope are respectively connected to the data acquisition card, and the data acquisition card is connected to a PC device; the data acquisition card is used to synchronize and package the trajectory data, video images, body temperature data and neuron signals, and then transmit the packaged data to the PC device.

5. The animal behavior pattern mining system according to any one of claims 1 to 4, characterized in that: The PC device includes a deep convolutional network and a cluster mining module. The PC device is used to combine trajectory data, video images, body temperature data and neuron signals to obtain the behavior pattern category of the experimental animal, specifically: Automatically identifying the action category of the experimental animal in each frame of the video image through the deep convolutional network, and outputting the action category sequence of the video image after correcting the action category through trajectory data, body temperature data and neuron signals; The cluster mining module combines trajectory data, body temperature data and neuron signals to cluster the action category sequence, and corresponds the clustered action category sequence to the time series of the video image to obtain the behavior pattern category sequence of the experimental animal.

6. The animal behavior pattern mining system according to claim 5, characterized in that: The action category of the experimental animal in each frame of the video image automatically identified by the deep convolutional network is specifically: Extracting all frame images from the video image, and randomly selecting a set number of frame images as a data set that needs to be manually labeled; Frame images of various categories are sorted out from the data set for manual labeling, and the manually labeled training set images are input into a deep convolutional network for training; wherein the deep convolutional network selects ResNet50 as the backbone network; All frames of the video image are input into a trained deep convolutional network, and the action category of the experimental animal in each frame is automatically identified by the deep convolutional network; The fully connected layer of the deep convolutional network is trained by using the trajectory data, body temperature data and neuron signals, and the action category sequence of the video image is output after the action category of the experimental animal in each frame is corrected.

7. The animal behavior pattern mining system according to claim 6, characterized in that: The cluster mining module includes a synchronization program, a clustering program and a visualization program. The cluster mining module is used to cluster the action category sequence by combining trajectory data, body temperature data and neuron signals. Specifically: Using the synchronization program to retrieve the packaged data, synchronizing all the packaged data onto a time axis according to the start and end time of the packaged data, the time marking signal, and the wavelet transform detection, and outputting a multidimensional array including the time; The action category sequence and the multidimensional array are input into a clustering program for automatic clustering. The clustering program uses the correlation difference between arrays of different dimensions between different action categories as a distinction, finds the action categories of the same category that are closely related from the multidimensional data to generate behavior pattern categories, and corresponds the behavior pattern categories to the time series of the video image to obtain the behavior pattern category sequence; The behavior pattern category sequence is converted into a subtitle file through the visualization program, and the subtitle file is displayed in the video image.

8. A method for mining animal behavior patterns, characterized in that: include: The trajectory data, video images, body temperature data and neuron signals of experimental animals are collected through infrared touch screens, cameras, thermal infrared probes and brain endoscopes; Uploading the trajectory data, video images, body temperature data, and neuron signals to a PC device; The behavior pattern category of the experimental animal is obtained by combining the trajectory data, video images, body temperature data and neuron signals through the PC device.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the animal behavior pattern mining method according to claim 8; The processor is used to execute the program instructions stored in the memory to control the animal behavior pattern mining method.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the animal behavior pattern mining method described in claim 8.

Images