A method and system for mouse behavior recognition and classification based on deep learning

Abstract

The present invention belongs to the technical field of neuroscience and pharmacology mouse behavior analysis, and discloses a mouse behavior recognition and classification method and system based on deep learning; the method comprises: obtaining a mouse behavior video through a video acquisition system; inputting the behavior video into a pre-trained key point detection model to obtain key point results of continuous frames; pre-processing the key point results of continuous frames to obtain continuous frame feature data; inputting the continuous frame feature data into a pre-trained behavior classification network model to obtain a continuous frame classification sequence; performing data processing on the continuous frame classification sequence to obtain processed continuous frame classification information; the reliability of objective data based on mouse research in the fields of neuroscience and pharmacology is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of neuroscience and pharmacology mouse behavior analysis, and more specifically, to a method and system for mouse behavior recognition and classification based on deep learning. Background Art

[0002] With the in-depth research of neuroscience and pharmacology, mouse behavior analysis has become an important means to understand the function of the nervous system and the effects of drugs. Traditional mouse behavior analysis mainly relies on manual observation and manual annotation, which is time-consuming and labor-intensive, and is easily affected by subjective factors, limiting the efficient processing of large-scale data. Therefore, there is an urgent need for an efficient, accurate and automated mouse behavior analysis method to accelerate the research process and improve the reliability of data.

[0003] In recent years, deep learning technology has achieved remarkable results in the field of computer vision, providing a powerful tool for image recognition and classification. Applying deep learning technology to the field of mouse behavior analysis has become a promising research direction. Deep learning models are able to learn complex spatiotemporal features, thereby more accurately identifying and classifying various behaviors of mice under experimental conditions. This method not only improves the efficiency of analysis, but also reduces subjective errors, providing researchers with more reliable experimental results.

[0004] The existing announcement number CN113569675B discloses a method for analyzing the behavior of mice in an open field experiment based on a ConvLSTM network. A key point detection model is used to output a feature map containing semantic information and position information of mouse key points, and then a feature map sequence composed of adjacent frames is input into a recognition and classification model established based on the ConvLSTM network to achieve behavior classification. Finally, a majority filter is used to correct the misclassified results, thereby obtaining the behavioral parameters in the open field experiment of mice. The invention realizes the automatic recognition of animal behavior and the automatic calculation of animal behavior indicators, which can not only reduce the workload of researchers, provide them with a quantitative behavior analysis method, and improve the objectivity of the experiment, but also, the refined behavior analysis method can help researchers capture some imperceptible behavior patterns and improve the reliability of the analysis results. At the same time, compared with the existing mouse behavior recognition method, this method has higher accuracy, is less affected by environmental and lighting changes, and has higher robustness.

[0005] The existing technology still has the following problems:

[0006] Applying deep learning to mouse behavior analysis still requires overcoming a series of challenges, including the difficulty of automatic data annotation, the dynamic posture characteristics of mouse behavior and their subtle differences. Most of the existing technical means for analyzing mouse behavior are not efficient, accurate and automated, and lack detailed posture recognition of various parts of the mouse's limbs, which limits the accurate capture of mouse behavior details, and thus affects the reliability of objective data based on mouse research in the fields of neuroscience and pharmacology.

[0007] In view of this, the present invention proposes a mouse behavior recognition and classification method and system based on deep learning to solve the above problems. Summary of the invention

[0008] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a method for identifying and classifying mouse behavior based on deep learning, comprising:

[0009] Obtain the behavioral video of mice through the video acquisition system;

[0010] Input the behavior video into the pre-trained key point detection model to obtain the key point results of consecutive frames;

[0011] Preprocess the key point results of continuous frames to obtain continuous frame feature data;

[0012] Inputting the continuous frame feature data into the pre-trained behavior classification network model to obtain a continuous frame classification sequence;

[0013] Data processing is performed on the continuous frame classification sequence to obtain processed continuous frame classification information.

[0014] Further, the key point result is a sequence of key point information; the key point information is the plane coordinate information corresponding to each key part of the mouse; the coordinate system of the plane coordinate information is a two-dimensional image coordinate system; the origin of the two-dimensional image coordinate system is the upper left corner of the image, the horizontal axis is increasing horizontally to the right, and the vertical axis is increasing vertically downward, and the unit is pixel; the key points include the head, left ear, right ear, neck, back, buttocks, tail, left front paw, right front paw, left hind paw and right hind paw; the representation method of the key point result P includes:

[0015] P=[[x mouth ,y mouth ],[x l-ear ,y l-ear ],[x r-ear ,y r-ear ],[x neck ,y neck ][x back ,y back ],[xpygal ,y pygal ],[x tail ,y tail ],

[0016] [x l-front-paw ,y l-front-paw ],[x r-front-paw ,y r-front-paw ],[x l-rear-paw ,y l-rear-paw ],[x r-rear-paw ,y r-rear-paw ]];

[0017] In the formula, x is the horizontal coordinate in the key point information; y is the vertical coordinate in the key point information; [x mouth ,y mouth ] is the plane coordinate information of the head; [x l-ear ,y l-ear ] is the plane coordinate information of the left ear; [x r-ear ,y r-ear ] is the plane coordinate information of the right ear; [x neck ,y neck ] is the plane coordinate information of the neck; [x back ,y back ] is the plane coordinate information of the back; [x pygal ,y pygal ] is the plane coordinate information of the hip; [x tail ,y tail ] is the plane coordinate information of the tail; [x l-front-paw ,y l-front-paw ] is the plane coordinate information of the left front paw; [x r-front-paw ,y r-front-paw ] is the plane coordinate information of the right front paw; [x l-rear-paw ,y l-rear-paw ] is the plane coordinate information of the left hind paw; [x r-rear-paw ,y r-rear-paw ] is the plane coordinate information of the right hind paw.

[0018] Furthermore, the training method of the key point detection model includes:

[0019] Obtain v groups of historical key point data sets, where v is a positive integer greater than 1, and the key point data sets include continuous frame image data split from historical behavior videos and corresponding key point result annotation data; use the key point data set as a sample set, divide the sample set into a training set and a test set, build a key point detection model, and perform multi-class regression prediction on each pixel point in each frame image, wherein the regression prediction includes the two-dimensional plane coordinates (x, y) of the key point; use the continuous frame image data in the training set as input data, and the key point result annotation data in the training set as output data, train the key point detection model, test the initial model using the test set, and output a model that meets the preset accuracy as the key point detection model, wherein the key point detection model is an MR-Pose deep learning model.

[0020] Furthermore, the training method of the behavior classification network model includes:

[0021] Acquire t groups of behavior data sets, where t is a positive integer greater than 1, and the behavior data sets include continuous frame feature data and annotated continuous frame classification sequences; the continuous frame classification sequence is a sequence of continuous frame mouse behaviors; the mouse behaviors include 10 different behaviors such as sniffing, grooming, climbing, supported standing, unsupported standing, resting, walking, eating, chewing and drinking water; use the behavior data set as a sample set, divide the sample set into a training set and a test set, construct a classifier, use the continuous frame feature data in the training set as input data, use the annotated continuous frame classification sequence in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, output a classifier that meets a preset accuracy as a behavior classification network model, and the classifier is a multilayer perceptron; the multilayer perceptron has an input layer, three hidden layers and an output layer; the number of input layers is 165; the number of hidden layers is 256, 512 and 1024 respectively; the number of output layers is 10.

[0022] Furthermore, the method for constructing the MR-Pose deep learning model includes:

[0023] The structure of MR-Pose includes a backbone network, a neck network and a decoupling head; the backbone network is a feature extraction network; the feature extraction network includes a GC module, a CSP module, a GSPPF module and a SA attention module; the GC module includes a phantom convolution layer, batch normalization and a ReLU activation function;

[0024] The design of the phantom convolution layer is as follows:

[0025] Step A1: Obtain an initial feature map through conventional convolution operation;

[0026] Step A2: apply a first new feature weight to the initial feature map of each channel, and combine the weight with the original feature map to obtain an enhanced initial feature map; the first new feature weight is obtained through a 3*3 or 5*5 convolution kernel.

[0027] Furthermore, the processing flow of the CSP module includes:

[0028] Step B1, input features are passed through the first GC module to perform feature extraction operations to generate a first feature map; the input features are multi-channel feature maps initially extracted by the backbone network;

[0029] Step B2: Generate a second new feature weight through a phantom convolution operation;

[0030] Step B3: Based on the first feature map and in combination with the second new feature weight, a second feature map is generated through a second GC module;

[0031] Step B4: Fusing the first feature map and the second feature map in the channel dimension to generate an overall feature map;

[0032] Step B5: pass the fused overall feature map to the third GC module to further extract feature information in the overall feature map.

[0033] Furthermore, the processing flow of the GSPPF module includes:

[0034] Step C1, performing initial feature learning through the fourth GC module to obtain local information and global information of each part of the mouse body on the image plane;

[0035] Step C2: downsample the features through four maximum pooling modules to obtain new local information and new global information corresponding to each of the four maximum pooling modules;

[0036] Step C3, fusing the new local information of the four maximum pooling modules with the new global information at the channel level to generate fused local information and fused global information;

[0037] Step C4: Optimize the fusion of local information and global information through the fifth GC module to obtain high-level features.

[0038] Furthermore, the processing flow of the SA attention module includes:

[0039] Step D1: Let the input of the feature layer be X∈R C×H×W ;

[0040] Where X is the feature information output by the CSP module or the high-level features output by the GSPPF module; R is a real number set; C is the number of channels; W and H are the width and height of the corresponding feature layer;

[0041] Step D2: Divide the total feature layer into N sub-feature layers, expressed as X = [X1, X2, ... X N ]; Random sublayer X k ∈R C / N×H×W ; k is a positive integer less than or equal to N; N is a positive integer greater than or equal to 2;

[0042] Step D3: Divide the random sublayer into two input sublayers X k1 ,X k2 ∈R C / 2N×H×W ;

[0043] Step D4: Perform SA attention analysis on X k1 Complete the channel attention operation and obtain the first output feature X o1 : The method for obtaining the first output feature includes: X o1 =σ(w·avg(X k1 )+b)·X k1 ;

[0044] Where avg(·) is the global pooling operation; w is the convolution weight; b is the bias; σ is the ReLU function;

[0045] X k2 Complete the channel attention operation and obtain the second output feature X o2 : The method for obtaining the second output feature includes: X o2 =σ(w′·GN(X k2 )+b′)·X k2 ;

[0046] Where GN(·) is the normalization operation; w′ is the convolution weight, b′ is the bias; σ is the ReLU function;

[0047] Step D5: Fuse the first output feature and the second output feature in the channel dimension to generate a new output sublayer X o ∈R C×H×W .

[0048] Furthermore, the number of frames of the behavior video is s; s is a positive integer;

[0049] The pretreatment method comprises:

[0050] Step E1: Take the key point information corresponding to the neck [x neck ,y neck ] is the coordinate origin, the key point information p is vectorized, and the first feature vector p of a single frame is obtained. vector , p vector =p-[x neck ,y neck ];

[0051] Step E2: single frame feature vector p vector Flatten it and sum it with the confidence of each key point to obtain the second feature vector p of a single frame vector-33 ;

[0052] Step E3: Add a frame label to the second feature vector of a single frame to obtain the feature data p of the single frame. vector-33-num ; num is the current frame number of the behavior video; num is a positive integer less than or equal to s;

[0053] Step E4: If the total number of frames of the action video s is less than 5, the expression of the continuous frame feature data feature is feature = [p vector-33-0 ,p vector-33-1 ,...n*p vector-33-s ]; where n = 6-s; if the total number of frames of the action video s ≥ 5, the expression of the continuous frame feature data feature is feature = [p vector-33-m ,...p vector-33-s ], where m=s-5.

[0054] A mouse behavior recognition and classification system based on deep learning, implementing the mouse behavior recognition and classification method based on deep learning, comprising:

[0055] The data acquisition module acquires the behavior video of mice through the video acquisition system;

[0056] The feature analysis module inputs the behavior video into the pre-trained key point detection model to obtain the key point results of consecutive frames;

[0057] The feature processing module pre-processes the key point results of continuous frames to obtain the feature data of continuous frames;

[0058] The behavior classification module inputs the continuous frame feature data into the pre-trained behavior classification network model to obtain the continuous frame classification sequence;

[0059] The data cleaning module processes the continuous frame classification sequence and obtains the processed continuous frame classification information.

[0060] The technical effects and advantages of the mouse behavior recognition and classification method and system based on deep learning of the present invention are as follows:

[0061] The present invention first collects the behavior video of the mouse, and obtains the key point results of continuous frames by inputting the video into the pre-trained MR-Pose deep learning model, thereby realizing the continuous and accurate detection of the key points of the mouse; then the key point results are pre-processed to obtain continuous frame feature data, which are input into the pre-trained behavior classification network model to obtain the continuous frame classification sequence, thereby realizing the automatic labeling of the mouse behavior in the mouse behavior video; finally, the continuous frame classification sequence is processed to obtain the processed continuous frame classification information, thereby avoiding the interference of short-term behavior on the analysis results; the overall technical solution effectively improves the accurate capture of the mouse behavior details and the subsequent automatic classification processing, thereby greatly improving the reliability of objective data based on mouse research in the fields of neuroscience and pharmacology.

[0062] The present invention uses and optimizes the MR-Pose model and adopts lightweight modules (GC module, CSP module, GSPPF module and SA attention module) to reduce the computational complexity while maintaining efficient feature extraction capabilities; the CSP module and the GSPPF module can extract global and local features at different scales and effectively fuse them, greatly improving the ability to capture the details of mouse behavior; the SA attention module improves the sensitivity to the temporal and spatial characteristics of mouse behavior by focusing on the significant areas in the features, making dynamic behavior recognition more accurate; at the same time, the optimized MR-Pose model is highly versatile and is not only suitable for mouse behavior recognition, but can also be migrated to behavior analysis tasks of other animals or dynamic targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of a mouse behavior recognition and classification system based on deep learning according to Example 1 of the present invention;

[0064] Figure 2 This is a flow chart of a method for mouse behavior recognition and classification based on deep learning according to Embodiment 2 of the present invention;

[0065] Figure 3 This is a schematic diagram of the structure of a video acquisition system according to Embodiment 1 of the present invention;

[0066] Figure 4 This is a schematic diagram of the MR-Pose deep learning model structure of Example 1 of the present invention;

[0067] Figure 5 Flow chart of the processing method of the SA attention module of embodiment 1 of the present invention;

[0068] Figure 6 Schematic diagram of the structure of a multi-layer perceptron according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Example 1

[0071] See also Figure 1 As shown, the mouse behavior recognition and classification system based on deep learning described in this embodiment includes: a data acquisition module, a feature analysis module, a feature processing module, a behavior classification module and a data cleaning module, and each module is connected by wired and / or wireless connection.

[0072] The data acquisition module acquires the behavior video of the mouse through a video acquisition system; the frame number of the behavior video is s; s is a positive integer.

[0073] See also Figure 3 As shown, the video acquisition system includes:

[0074] A camera, a profile, an experimental box, a box and a fill light; the profile includes a clamping position 1 and a clamping position 2, which are convenient for adjusting and fixing the angle and position of the camera; the camera is fixed at the clamping position 2 to ensure clear and stable video recording during the experiment; the experimental box is located in the box; the size of the experimental box is adjustable to provide a variety of experimental environments to capture the behavioral performance of mice in different spaces; the box is of a fixed size and is used to place the experimental box to ensure the neatness and order of the experimental environment; the fill light is located above the box to provide appropriate lighting conditions to ensure that the camera obtains a sufficiently bright image, which helps to accurately capture the various behaviors of mice.

[0075] The feature analysis module inputs the behavior video into the pre-trained key point detection model to obtain the key point results of continuous frames; the key point results are a sequence of key point information; the key point information is the plane coordinate information corresponding to each key part of the mouse; the coordinate system of the plane coordinate information is a two-dimensional image coordinate system; the origin of the two-dimensional image coordinate system is the upper left corner of the image, the horizontal axis is horizontally increasing to the right, and the vertical axis is vertically increasing downward, and the unit is pixel; the key points include the head, left ear, right ear, neck, back, buttocks, tail, left front paw, right front paw, left hind paw and right hind paw; the representation method of the key point result P includes:

[0076] P=[[x mouth ,y mouth ],[x l-ear ,y l-ear ],[xr-ear ,y r-ear ],[x neck ,y neck ][x back ,y back ],[x pyga,l y pyga，l ],[x tai,l y tai，l ],

[0077] [x l-front-paw ,y l-front-paw ],[x r-front-paw ,y r-front-paw ],[x l-rear-paw ,y -lrear-paw ],[x r-rear-paw ,y r-rear-paw ]];

[0078] In the formula, x is the horizontal coordinate in the key point information; y is the vertical coordinate in the key point information; [x mouth ,y mouth ] is the plane coordinate information of the head; [x l-ear ,y l-ear ] is the plane coordinate information of the left ear; [x r-ear ,y r-ear ] is the plane coordinate information of the right ear; [x neck ,y neck ] is the plane coordinate information of the neck; [x back ,y back ] is the plane coordinate information of the back; [x pygal ,y pygal ] is the plane coordinate information of the hip; [x tail ,y tail ] is the plane coordinate information of the tail; [x l-front-paw ,y l-front-paw ] is the plane coordinate information of the left front paw; [x r-front-paw ,y r-front-paw ] is the plane coordinate information of the right front paw; [x l-rear-paw ,y l-rear-paw ] is the plane coordinate information of the left hind paw; [x r-rear-paw ,y r-rear-paw ] is the plane coordinate information of the right hind paw.

[0079] The training method of the key point detection model includes:

[0080] Obtain v groups of historical key point data sets, where v is a positive integer greater than 1, and the key point data sets include continuous frame image data split from historical behavior videos and corresponding key point result annotation data; use the key point data set as a sample set, divide the sample set into a training set and a test set, build a key point detection model, and perform multi-class regression prediction on each pixel point in each frame image, wherein the regression prediction includes the two-dimensional plane coordinates (x, y) of the key point; use the continuous frame image data in the training set as input data, and the key point result annotation data in the training set as output data, train the key point detection model, test the initial model using the test set, and output a model that meets the preset accuracy as the key point detection model, wherein the key point detection model is an MR-Pose deep learning model.

[0081] The method for constructing the MR-Pose deep learning model includes:

[0082] See also Figure 4 As shown in the figure, MR-Pose is a lightweight structure with strong adaptability. The structure of MR-Pose includes a backbone network, a neck network and a decoupling head; the backbone network is a feature extraction network; the feature extraction network includes a GC module, a CSP module, a GSPPF module and a SA attention module.

[0083] The GC module comprises a phantom convolution layer, batch normalization and a ReLU activation function. After completing the phantom convolution, the GC module further enhances the stability and nonlinear expression capability of the network through batch normalization and the ReLU activation function.

[0084] The design of the phantom convolution layer is as follows:

[0085] Step A1: Obtain the initial feature map through conventional convolution operations to ensure that the network can learn the key basic features of the mouse behavior video in the early stage.

[0086] Step A2: apply a first new feature weight to the initial feature map of each channel, and combine the weight with the original feature map to obtain an enhanced initial feature map; the first new feature weight can be obtained through a 3*3 or 5*5 convolution kernel.

[0087] It should be noted that the design of the phantom convolution layer in this scheme is different from the traditional phantom convolution layer; the traditional phantom convolution layer generates a new feature map and concatenates the original feature map with the newly generated feature map in the channel dimension to achieve identity mapping; however, this scheme adopts the method of assigning new feature weights to the original feature map; this different design may help to adjust the feature weights more flexibly to adapt to different tasks or optimization objectives; the structure of the entire GC module is designed to effectively perform feature extraction and nonlinear transformation through the synergy of phantom convolution layers, batch normalization and ReLU activation functions.

[0088] The processing flow of the CSP module includes:

[0089] Step B1, input features are passed through the first GC module to perform feature extraction operations to generate a first feature map, which is used to capture the local features and overall contour information of subtle differences in various parts of the mouse in the video; the input features are multi-channel feature maps initially extracted by the backbone network; the multi-channel feature map is a deep feature representation obtained by processing the continuous frame image data split from the mouse behavior video through convolution, pooling, etc. of the backbone network.

[0090] Step B2: Generate a second new feature weight through phantom convolution operation, so that the system maintains a low amount of calculation while enhancing the nonlinear feature expression.

[0091] It should be noted that the phantom convolution operation in the CSP module also follows the principle of the phantom convolution layer in the GC module, but the timing of calling and fusion is different.

[0092] Step B3: Based on the first feature map and in combination with the second new feature weight, a second feature map is generated through a second GC module.

[0093] Step B4: Fuse the first feature map and the second feature map in the channel dimension to generate an overall feature map.

[0094] Step B5: pass the fused overall feature map to the third GC module to further extract feature information in the overall feature map.

[0095] The CSP module helps maintain efficient computing performance during processing while enhancing nonlinear features and channel-level information fusion.

[0096] The processing flow of the GSPPF module includes:

[0097] Step C1: Perform initial feature learning through the fourth GC module to obtain local and global information of each part of the mouse's body on the image plane.

[0098] Step C2: Downsample the features through four maximum pooling modules, focusing on the key parts of the mouse, such as the head, four paws, and tail, from different scales, to ensure that significant behavioral features can be extracted in multi-level space, and obtain new local information and new global information corresponding to each of the four maximum pooling modules.

[0099] Step C3: Fuse the new local information of the four maximum pooling modules with the new global information at the channel level to generate fused local information and fused global information.

[0100] It should be noted that integrating new local information at different scales with new global information improves the sensitivity of the network in capturing complex behavior patterns.

[0101] Step C4: Optimize the fused local information and the fused global information through the fifth GC module to extract high-level features that can represent the temporal and spatial information of mouse behavior.

[0102] The GSPPF module helps to represent the input data more efficiently through dimensionality reduction and fusion operations while maintaining global and local information.

[0103] The neck network adopts a hierarchical fusion idea to hierarchically fuse the underlying features and upper features in the feature extraction backbone network, and then input the hierarchically fused features into the decoupling head; the underlying features mainly capture low-order visual clues such as edges, corners, textures, etc. in the image to help identify the preliminary shapes and local details of various parts of the mouse's body; the upper-level features focus more on the extraction of overall structure and global semantics, such as the main outline of the mouse's body and movement posture.

[0104] The decoupling head includes two parts. The first part is the area for identifying the mouse; the second part is the key point information for identifying the mouse's head, left ear, right ear, neck, back, buttocks, tail, left front paw, right front paw, left hind paw, and right hind paw.

[0105] See also Figure 5 As shown, the processing flow of the SA attention module includes:

[0106] Step D1: Let the input of the feature layer be X∈R C×H×W .

[0107] Where X is the feature information output by the CSP module or the high-level features output by the GSPPF module; R is a real number set, indicating that the range of feature values ​​is real numbers; C is the number of channels, indicating the dimension of multi-channel features extracted from the video; W and H are the width and height of the corresponding feature layer.

[0108] Step D2: Divide the total feature layer into N sub-feature layers, expressed as X = [X1, X2, ... XN ]; Random sublayer X k ∈R C / N×H×W ; k is a positive integer less than or equal to N; N is a positive integer greater than or equal to 2.

[0109] Step D3: Divide the random sublayer into two input sublayers X k1 ,X k2 ∈R C / 2N×H×W .

[0110] Step D4: Perform SA attention analysis on X k1 Complete the channel attention operation and obtain the first output feature X o1 :

[0111] X o1 =σ(w·avg(X k1 )+b)·X k1 ;

[0112] Where avg(·) is the global pooling operation, which is used to extract the average value of the channel and capture the global information in the behavior video; w is the convolution weight, b is the bias, which is used to learn the weight distribution of features; σ is the ReLU function, which improves the nonlinear expression ability of the key point detection model.

[0113] X k2 Complete the channel attention operation and obtain the second output feature X o2 :

[0114] X o2 =σ(w′·GN(X k2 )+b′)·X k2 ;

[0115] Where GN(·) is a normalization operation used to stabilize feature distribution and improve robustness to mouse behavior details; w′ is the convolution weight, b′ is the bias; and σ is the ReLU function.

[0116] Step D5: Fuse the first output feature and the second output feature in the channel dimension to generate a new output sublayer X o ∈R C×H×W .

[0117] The feature processing module pre-processes the key point results of continuous frames to obtain the feature data of continuous frames.

[0118] The pretreatment method comprises:

[0119] Step E1: Take the key point information corresponding to the neck [x neck ,y neck ] is the coordinate origin, the key point information p is vectorized, and the first feature vector p of a single frame is obtained. vector, p vector =p-[x neck ,y neck ].

[0120] Step E2: single frame feature vector p vector Flatten it and sum it with the confidence of each key point to obtain the second feature vector p of a single frame vector-33 .

[0121] Step E3: Add a frame label to the second feature vector of a single frame to obtain the feature data p of the single frame. vector-33-num ; num is the current frame number of the behavior video; num is a positive integer less than or equal to s.

[0122] Step E4: If the total number of frames of the action video s is less than 5, the expression of the continuous frame feature data feature is feature = [p vector-33-0 ,p vector-33-1 ,...n*p vector-33-s ]; where n = 6-s; if the total number of frames of the action video s ≥ 5, the expression of the continuous frame feature data feature is feature = [p vector-33-m ,...p vector-33-s ], where m=s-5.

[0123] The behavior classification module inputs the continuous frame feature data into the pre-trained behavior classification network model to obtain a continuous frame classification sequence result; the continuous frame classification sequence is a sequence of continuous frames of mouse behaviors; the mouse behaviors include 10 different behaviors such as sniffing, grooming, climbing, standing with support, standing without support, resting, walking, eating, chewing and drinking water; the behavior numbers corresponding to the mouse behaviors are integers 0-9 in sequence.

[0124] For example, result = [0, 1, 1, 3] means that in the behavior video, the mouse is sniffing in the first frame, grooming in the second frame, grooming in the third frame, and standing with support in the fourth frame.

[0125] The training method of the behavior classification network model includes:

[0126] Obtain t sets of behavior data sets, where t is a positive integer greater than 1, and the behavior data sets include continuous frame feature data and annotated continuous frame classification sequences; use the behavior data sets as sample sets, divide the sample sets into training sets and test sets, build a classifier, use the continuous frame feature data in the training set as input data, use the annotated continuous frame classification sequences in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, and output a classifier that meets the preset accuracy as a behavior classification network model, where the classifier is a multi-layer perceptron; refer to Figure 6As shown, the multilayer perceptron has an input layer, three hidden layers and an output layer; the input of the input layer is feature, and the number of input layers is 165; the hidden layers are 256, 512 and 1024 respectively; the output layer is 10, which represents the probability distribution of 10 different behavior categories. After normalization by the softmax function, the behavior category is predicted frame by frame.

[0127] The data cleaning module processes the continuous frame classification sequence and obtains the processed continuous frame classification information.

[0128] The method for acquiring the continuous frame classification information comprises:

[0129] Step F1, grouping the continuous frame classification sequence according to the continuous same classification values ​​to obtain the first classification information; each group represents a continuous same behavior of the mouse.

[0130] Step F2, setting the shortest behavior occurrence frame number min_frame to filter abnormal short behaviors;

[0131] Step F3, initialize a new array result_new=[], clean the continuous frame classification sequence, obtain the continuous frame classification information, and store the continuous frame classification information in the new array.

[0132] The cleaning method comprises:

[0133] Step G1, start traversal from result[1]. If the length of result[1] is greater than min_frame, add the corresponding traversal value to result_new; if the length of result[1] is less than or equal to min_frame, perform noise replacement; the traversal value corresponds to the behavior number of the mouse in the continuous frame classification sequence.

[0134] The noise replacement method comprises:

[0135] If the length of result_new is greater than 0, replace the currently traversed value with the number of the mouse's most recent long-lasting behavior in result_new.

[0136] Otherwise, add the current traversal value to result_new.

[0137] Step G2: After the traversal is completed, all values ​​of the array added to result_new and whose length is less than min_frame are replaced with the value of the first array whose length is greater than min_frame.

[0138] Exemplarily, the continuous frame classification sequence is result = [0,1,1,1,3,6,6,2,5,9,9,7]; then the first classification information is result = [[0], [1,1,1], [3], [6,6], [2], [5], [9,9], [7]]; if the shortest number of behavior occurrence frames is set to 3, the continuous frame classification information after cleaning is result = [0,1,1,1,1,1,1,1,1,1,1,1,1].

[0139] Example 2

[0140] See also Figure 2 As shown, this embodiment provides a mouse behavior recognition and classification method based on deep learning, including:

[0141] Obtain the behavioral video of mice through the video acquisition system;

[0142] Input the behavior video into the pre-trained key point detection model to obtain the key point results of consecutive frames;

[0143] Preprocess the key point results of continuous frames to obtain continuous frame feature data;

[0144] Inputting the continuous frame feature data into the pre-trained behavior classification network model to obtain a continuous frame classification sequence;

[0145] Data processing is performed on the continuous frame classification sequence to obtain processed continuous frame classification information.

[0146] Example 3

[0147] The electronic device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of an embodiment of a method for identifying and classifying mouse behavior based on deep learning are implemented, such as Figure 2 Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 1 Functionality of the modules shown.

[0148] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0149] Those skilled in the art will appreciate that these are merely examples of electronic devices and do not constitute a limitation of the electronic device. The electronic device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0150] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0151] The memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. Furthermore, the memory may include both an internal storage unit and an external storage device of the terminal device. The memory is used to store the computer program and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0156] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for mouse behavior recognition and classification based on deep learning, characterized in that: include: Obtain the behavioral video of mice through the video acquisition system; Input the behavior video into the pre-trained key point detection model to obtain the key point results of consecutive frames; Preprocess the key point results of continuous frames to obtain continuous frame feature data; Inputting the continuous frame feature data into the pre-trained behavior classification network model to obtain a continuous frame classification sequence; Data processing is performed on the continuous frame classification sequence to obtain processed continuous frame classification information.

2. A method for mouse behavior recognition and classification based on deep learning according to claim 1, characterized in that: The key point result is a sequence of key point information; the key point information is the plane coordinate information corresponding to each key part of the mouse; the coordinate system of the plane coordinate information is a two-dimensional image coordinate system; the origin of the two-dimensional image coordinate system is the upper left corner of the image, the horizontal axis is increasing horizontally to the right, and the vertical axis is increasing vertically downward, and the unit is pixel; the key points include the head, left ear, right ear, neck, back, buttocks, tail, left front paw, right front paw, left hind paw and right hind paw; the representation method of the key point result P includes: P=[[x mouth ,and mouth ],[x l-ear ,and l-ear ],[x r-ear ,and r-ear ],[x neck ,and neck ][x back ,and back ],[x pygal ,and pygal ],[x tail ,and tail ], [x l-front-paw ,and l-front-paw ],[x r-front-paw ,and r-front-paw ],[x l-rear-paw ,and l-rear-paw ],[x r-rear-paw ,and r-rear-paw ]]; In the formula, x is the horizontal coordinate in the key point information; y is the vertical coordinate in the key point information; [x mouth ,y mouth ] is the plane coordinate information of the head; [x l-ear ,y l-ear ] is the plane coordinate information of the left ear; [x r-ear ,y r-ear ] is the plane coordinate information of the right ear; [x neck ,y nec k] is the plane coordinate information of the neck; [x back ,y back ] is the plane coordinate information of the back; [x pygal ,y pygal ] is the plane coordinate information of the hip; [x tail ,y tail ] is the plane coordinate information of the tail; [x l-front-paw ,y l-front-paw ] is the plane coordinate information of the left front paw; [x r-front-paw ,y r-front-paw ] is the plane coordinate information of the right front paw; [x l-rear-paw ,y l-rear-paw ] is the plane coordinate information of the left hind paw; [x r-rear-paw ,y r-rear-paw ] is the plane coordinate information of the right hind paw.

3. A method for mouse behavior recognition and classification based on deep learning according to claim 1, characterized in that: The training method of the key point detection model includes: Obtain v groups of historical key point data sets, where v is a positive integer greater than 1, and the key point data sets include continuous frame image data split from historical behavior videos and corresponding key point result annotation data; use the key point data set as a sample set, divide the sample set into a training set and a test set, build a key point detection model, and perform multi-class regression prediction on each pixel point in each frame image, wherein the regression prediction includes the two-dimensional plane coordinates (x, y) of the key point; use the continuous frame image data in the training set as input data, and the key point result annotation data in the training set as output data, train the key point detection model, test the initial model using the test set, and output a model that meets the preset accuracy as the key point detection model, wherein the key point detection model is an MR-Pose deep learning model.

4. A method for mouse behavior recognition and classification based on deep learning according to claim 1, characterized in that: The training method of the behavior classification network model includes: Acquire t groups of behavior data sets, where t is a positive integer greater than 1, and the behavior data sets include continuous frame feature data and annotated continuous frame classification sequences; the continuous frame classification sequence is a sequence of continuous frame mouse behaviors; the mouse behaviors include 10 different behaviors such as sniffing, grooming, climbing, supported standing, unsupported standing, resting, walking, eating, chewing and drinking water; use the behavior data set as a sample set, divide the sample set into a training set and a test set, construct a classifier, use the continuous frame feature data in the training set as input data, use the annotated continuous frame classification sequence in the training set as output data, train the classifier to obtain an initial classifier, test the initial classifier using the test set, output a classifier that meets a preset accuracy as a behavior classification network model, and the classifier is a multilayer perceptron; the multilayer perceptron has an input layer, three hidden layers and an output layer; the number of input layers is 165; the number of hidden layers is 256, 512 and 1024 respectively; the number of output layers is 10.

5. A method for mouse behavior recognition and classification based on deep learning according to claim 3, characterized in that: The method for constructing the MR-Pose deep learning model includes: The structure of MR-Pose includes a backbone network, a neck network and a decoupling head; the backbone network is a feature extraction network; the feature extraction network includes a GC module, a CSP module, a GSPPF module and a SA attention module; the GC module includes a phantom convolution layer, batch normalization and a ReLU activation function; The design of the phantom convolution layer is as follows: Step A1: Obtain an initial feature map through conventional convolution operation; Step A2: apply a first new feature weight to the initial feature map of each channel, and combine the weight with the original feature map to obtain an enhanced initial feature map; the first new feature weight is obtained through a 3*3 or 5*5 convolution kernel.

6. A method for mouse behavior recognition and classification based on deep learning according to claim 5, characterized in that: The processing flow of the CSP module includes: Step B1, input features are passed through the first GC module to perform feature extraction operations to generate a first feature map; the input features are multi-channel feature maps initially extracted by the backbone network; Step B2: Generate a second new feature weight through a phantom convolution operation; Step B3: Based on the first feature map and in combination with the second new feature weight, a second feature map is generated through a second GC module; Step B4: Fusing the first feature map and the second feature map in the channel dimension to generate an overall feature map; Step B5: pass the fused overall feature map to the third GC module to further extract feature information in the overall feature map.

7. A method for mouse behavior recognition and classification based on deep learning according to claim 5, characterized in that: The processing flow of the GSPPF module includes: Step C1, performing initial feature learning through the fourth GC module to obtain local information and global information of each part of the mouse body on the image plane; Step C2: downsample the features through four maximum pooling modules to obtain new local information and new global information corresponding to each of the four maximum pooling modules; Step C3, fusing the new local information of the four maximum pooling modules with the new global information at the channel level to generate fused local information and fused global information; Step C4: Optimize the fusion of local information and global information through the fifth GC module to obtain high-level features.

8. The method for mouse behavior recognition and classification based on deep learning according to claim 5, characterized in that: The processing flow of the SA attention module includes: Step D1: Let the input of the feature layer be X∈R C×H×W ; Where X is the feature information output by the CSP module or the high-level features output by the GSPPF module; R is a real number set; C is the number of channels; W and H are the width and height of the corresponding feature layer; Step D2: Divide the total feature layer into N sub-feature layers, expressed as X = [X1, X2, ... X N ]; Random sublayer X k ∈R C / N×H×W ; k is a positive integer less than or equal to N; N is a positive integer greater than or equal to 2; Step D3: Divide the random sublayer into two input sublayers X k1 ,X k2 ∈R C / 2N×H×W ; Step D4: Perform SA attention analysis on X k1 Complete the channel attention operation and obtain the first output feature X o1 : The method for obtaining the first output feature includes: X o1 =σ(w·avg(X k1 )+b)·X k1 ; Where avg(·) is the global pooling operation; w is the convolution weight; b is the bias; σ is the ReLU function; X k2 Complete the channel attention operation and obtain the second output feature X o2 : The method for obtaining the second output feature includes: X o2 =σ(w′·GN(X k2 )+b′)·X k2 ; Where GN(·) is the normalization operation; w′ is the convolution weight, b′ is the bias; σ is the ReLU function; Step D5: Fuse the first output feature and the second output feature in the channel dimension to generate a new output sublayer X o ∈R C×H×W .

9. A method for mouse behavior recognition and classification based on deep learning according to claim 1, characterized in that: The number of frames of the behavior video is s; s is a positive integer; The pretreatment method comprises: Step E1: Take the key point information corresponding to the neck [x neck ,y neck ] is the coordinate origin, the key point information p is vectorized, and the first feature vector p of a single frame is obtained. vector , p vector =p-[x neck ,y neck ]; Step E2: single frame feature vector p vector Flatten it and sum it with the confidence of each key point to obtain the second feature vector p of a single frame vector-33 ; Step E3: Add a frame label to the second feature vector of a single frame to obtain the feature data p of the single frame. vector-33-num ; num is the current frame number of the behavior video; num is a positive integer less than or equal to s; Step E4: If the total number of frames of the action video s is less than 5, the expression of the continuous frame feature data feature is feature = [p vector-33-0 ,p vector-33-1 ,...n*p vector-33-s ]; where n = 6-s; if the total number of frames of the action video s ≥ 5, the expression of the continuous frame feature data feature is feature = [p vector-33-m ,...p vector-33-s ], where m=s-5.

10. A mouse behavior recognition and classification system based on deep learning, implementing a mouse behavior recognition and classification method based on deep learning as described in any one of claims 1 to 9, characterized in that: include: The data acquisition module acquires the behavior video of mice through the video acquisition system; The feature analysis module inputs the behavior video into the pre-trained key point detection model to obtain the key point results of consecutive frames; The feature processing module pre-processes the key point results of continuous frames to obtain the feature data of continuous frames; The behavior classification module inputs the continuous frame feature data into the pre-trained behavior classification network model to obtain the continuous frame classification sequence; The data cleaning module processes the continuous frame classification sequence and obtains the processed continuous frame classification information.

Images