Method and device for recognizing social behavior of rats, computer device and storage medium

Abstract

Embodiments of the present application provide a rat social behavior recognition method and device, computer equipment and a storage medium, relating to the technical field of animal behavior recognition, wherein the method comprises: acquiring a video comprising two rats, and determining key points of each rat in each frame of the video; in each frame of the image, according to the key points of each rat, determining the coordinate information and the skeletal posture of the key points of each rat; in each frame of the image, according to the coordinate information and the skeletal posture of the key points of each rat, extracting the behavior features of the two rats; inputting the behavior features of the two rats in each frame of the image into a behavior recognition model, and the behavior recognition model outputs the social behavior type of the two rats in each frame of the image, and the behavior recognition model is obtained by training a machine learning component with historical behavior features and corresponding social behavior types as samples. The scheme can improve the efficiency and accuracy of rat social behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of animal behavior recognition, and in particular to a rat social behavior recognition method and device, computer equipment and a storage medium. BACKGROUND

[0002] Autism spectrum disorder is a heterogeneous neurodevelopmental condition characterized by early-onset social impairment, stereotyped behavior and interest. Autism patients have social impairment and stereotyped behavior as common characteristics, and may also have different degrees of intellectual impairment, language impairment, attention deficit hyperactivity disorder, epilepsy, anxiety, sleep disorders, personality disorders, digestive system problems and other complications.

[0003] Considering the technical limitations of neuroscience and the constraints of experimental ethics, clinical experiments on autism are mostly non-invasive operations. It is difficult to allow in-depth central nervous system mechanism research on living human individuals. However, animal models can test the effects of specific pathogenic factors on neuropathology and behavior by controlling variables. Due to the high similarity of gene sequences to humans, small individual differences and sufficient quantity, rats have become one of the most commonly used model animals in neuroscience research. The behavior characteristics of autism rat models have also been proven to have a certain correspondence with the clinical phenotypes of autism.

[0004] There are two experimental paradigms for evaluating rat social behavior, namely three-box social preference test and social interaction test. The three-box test has a relatively mature computer-aided analysis system, but the test may have false negatives and false positives. In contrast, the social interaction test has a higher degree of restoration of rat social behavior in a natural state and fewer errors in the paradigm itself. Currently, the classification of rat social behavior in the social interaction test is evaluated by humans. However, the evaluation of rat behavior in the social interaction test requires strong expertise, and the social behavior between rats is more complex than the behavior of a single rat. It requires well-trained experimenters to spend a lot of time and effort to complete double-blind experiments before the classification and evaluation of social behavior can be completed. The evaluation is inefficient, and is affected by factors such as professional differences and human errors, which cannot ensure the accuracy of the evaluation. SUMMARY

[0005] Therefore, the embodiments of the present application provide a rat social behavior recognition method to solve the technical problems of low efficiency and low accuracy in evaluating rat behavior in the prior art. The method comprises:

[0006] obtaining a video including two rats, and determining key points of each rat in each frame of the video;

[0007] In each frame of image, the coordinate information and the skeletal pose of the key points of each rat are determined according to the key points of each rat.

[0008] In each frame of image, the behavior features of the two rats are extracted according to the coordinate information and the skeletal pose of the key points of each rat.

[0009] The behavior recognition model outputs the social behavior type of the two rats in each frame of image, wherein the behavior recognition model is obtained by training a machine learning component with historical behavior features and corresponding social behavior types as samples.

[0010] The embodiment of the present application also provides a rat social behavior recognition device to solve the technical problems of low efficiency and low accuracy in evaluating rat behavior in the prior art.

[0011] The key point determination module is configured to acquire a video including two target rats and determine key points of each rat in each frame of image of the video.

[0012] The skeletal information determination module is configured to determine, in each frame of image, the coordinate information and the skeletal pose of the key points of each rat according to the key points of each rat.

[0013] The feature extraction module is configured to extract, in each frame of image, the behavior features of the two rats according to the coordinate information and the skeletal pose of the key points of each rat.

[0014] The behavior type recognition module is configured to input the behavior features of the two rats in each frame of image into a behavior recognition model, and the behavior recognition model outputs the social behavior type of the two rats in each frame of image, wherein the behavior recognition model is obtained by training a machine learning component with historical behavior features and corresponding social behavior types as samples.

[0015] The embodiment of the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements any of the above rat social behavior recognition methods when executing the computer program, to solve the technical problems of low efficiency and low accuracy in evaluating rat behavior in the prior art.

[0016] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program for executing any of the above rat social behavior recognition methods, to solve the technical problems of low efficiency and low accuracy in evaluating rat behavior in the prior art.

[0017] Compared with the prior art, the at least one technical scheme adopted by the embodiment of the present specification can achieve the beneficial effects at least including: it is proposed to determine the key points of each rat in each frame image of the video, and determine the coordinate information and the skeletal pose of the key points of each rat according to the key points of each rat, and then extract the behavior features of the two rats according to the coordinate information and the skeletal pose of the key points of each rat, and finally input the behavior features of the two rats in each frame image into a behavior recognition model, and the behavior recognition model can output the social behavior type of the two rats in each frame image. The image processing is performed on each frame image of the video to determine the key points, the coordinate information and the skeletal pose of the key points, and then the coordinate information and the skeletal pose of the key points are input into the behavior recognition model of deep learning, so that the social behavior type of the two rats in each frame image can be recognized, and an automatic and intelligent rat social behavior recognition process is realized. Compared with the scheme of manually evaluating the rat social behavior in the prior art, the efficiency of rat social behavior recognition can be improved; at the same time, the influence of human factors such as personal professional difference and human error is avoided, which is beneficial to improve the accuracy of rat social behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a flowchart of a rat social behavior recognition method provided by the embodiment of the present application;

[0020] Figure 2 is a key point labeling schematic diagram provided by the embodiment of the present application;

[0021] Figure 3 is a labeling schematic diagram of identity information provided by the embodiment of the present application;

[0022] Figure 4 is a behavior schematic diagram of latency provided by the embodiment of the present application;

[0023] Figure 5 is a schematic diagram of separation behavior provided by the embodiment of the present application;

[0024] Figure 6 is a schematic diagram of normal one-way social behavior provided by the embodiment of the present application;

[0025] Figure 7 is a schematic diagram of abnormal one-way social behavior provided by the embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a bidirectional social behavior provided by an embodiment of the present application;

[0027] Figure 9 is a structural block diagram of a computer device provided by an embodiment of the present application;

[0028] Figure 10 is a structural block diagram of a rat social behavior recognition device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application will be described in detail below with reference to the drawings.

[0030] The embodiments of the present application will be described in detail below with reference to the drawings.It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. The present application can be implemented or applied in other different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0031] In an embodiment of the present application, a rat social behavior recognition method is provided, as shown in the figure, the method comprises: Figure 1

[0032] Step S101: acquiring a video comprising two target rats, and determining key points of each rat in each frame image of the video;

[0033] Step S102: in each frame image, determining coordinate information and a skeletal pose of the key points of each rat according to the key points of each rat;

[0034] Step S103: in each frame image, extracting behavior features of the two rats according to the coordinate information and the skeletal pose of the key points of each rat;

[0035] Step S104: inputting the behavior features of the two rats in each frame image into a behavior recognition model, the behavior recognition model outputs a social behavior type of the two rats in each frame image, wherein the behavior recognition model is obtained by training a machine learning component with historical behavior features and corresponding social behavior types as samples.

[0036] In an embodiment of the present application, a rat social behavior recognition device is provided, as shown in the figure, the device comprises: Figure 1It can be learned from the flowchart that in the embodiment of the present application, image processing is performed on each frame of image of the video to determine the key points, the coordinate information of the key points and the skeleton posture, and then the coordinate information of the key points and the skeleton posture are input into the behavior recognition model of deep learning, so that the social behavior type of the two rats in each frame of image can be recognized, and an automatic and intelligent rat social behavior recognition process is realized. Compared with the scheme of manually evaluating the rat social behavior in the prior art, the efficiency of rat social behavior recognition can be improved. At the same time, the influence of human factors such as personal professional difference and human error is avoided, and the accuracy of rat social behavior recognition is improved.

[0037] In specific implementation, the video including two rats can be a video of interaction between two rats in an experimental environment, or a video of interaction between two rats in a conventional and normal scene. The two rats can be experimental rats or conventional and ordinary rats, and the form of the rats is not specifically limited.

[0038] In specific implementation, the video for rat social behavior recognition can be a video collected directly, or a video segment extracted at equal intervals from the collected video. For example, for each video, 600 frames are extracted at equal intervals, and the key points of all rats in each frame are labeled to identify the social behavior type of the rats.

[0039] In specific implementation, in order to improve the accuracy of rat social behavior recognition, in the embodiment, as shown in the figure, Figure 2 The key points of each rat can include head, front torso, rear torso, tail base, middle tail and tail end, etc.

[0040] In specific implementation, in order to accurately determine the coordinate information and the skeleton posture of the key points of each rat and reduce the calculation amount, in the embodiment, a method for determining the coordinate information and the skeleton posture of the key points of each rat is proposed, for example,

[0041] Mapping the preset image region of each frame of image into a heat image, the preset image region including the key points of the two rats;

[0042] Inputting the heat image into a detection model, the detection model outputting a heat map and a vector map, wherein the heat map includes the coordinate information of the key points in the heat image, and the vector map includes the connection vector information between the key points in the heat image, and the detection model is obtained by training a multi-scale feature fusion network with historical heat images, historical heat maps and historical vector maps as samples;

[0043] Generating a skeleton posture according to the vector map by using a minimum spanning tree algorithm;

[0044] Map the coordinate information of the key points in the heat map and the skeleton pose to the frame image corresponding to the heat map.

[0045] Specifically, the detection model can be trained based on a ResNet-50 network, and the process of determining the coordinate information of the key points and the skeleton pose using the detection model can include the following steps:

[0046] Map a preset image region of 1 / 4 size of each frame image to a heat image, the preset image region including key points of each rat; and then input the heat image into the ResNet-50 network, select features in a bottleneck of a third unit in four blocks in the ResNet-50 network, and concatenate the features after convolution, and finally output a predicted heat map (the heat map including coordinate and probability of the key points) and a vector map (the vector map including connection vector information between the key points) using two inverse convolution layers;

[0047] A focal loss loss function is used to further reduce the imbalance problem of positive and negative samples (pixels of the key points are positive samples, and pixels of non-key points are negative samples) in predicting the category probability of the rat key points, as shown in the following formula:

[0048]

[0049] wherein FL (p, y) is a loss function, y belongs to {+1, -1}, and is a category label of positive and negative samples, p belongs to a predicted probability of the model for the category y = 1, is a modulation factor, used to weaken the contribution of easily distinguishable negative samples to the loss function, so that the loss function focuses on these difficult-to-distinguish samples, is a constant, for example, in training The value can be 2, is a category balancing hyperparameter, and is a constant, which can be 0.25 in training.

[0050] The vector map between the rat key points is predicted using a Huber loss loss, as shown in the following formula:

[0051]

[0052] wherein y is a real label of a sample, f(x) is a predicted probability of the model, and δ is a hyperparameter of the HuberLoss, which adopts a square error when a prediction deviation is less than δ, and adopts a linear error when the prediction deviation is greater than δ;

[0053] ​The cost weight of the connection between the key points is calculated based on the predicted vector diagram, and the optimal bone posture of each rat is obtained through a minimum spanning tree algorithm. Finally, the coordinate information of the key points and the bone posture are mapped into each frame of image.

[0054] In implementation, in order to improve the accuracy of rat social behavior recognition, in the embodiment, the behavior features of two rats include the body shape features of each rat, the motion state features of each rat, the distance features between the two rats, and the interaction tendency features between the two rats.

[0055] In implementation, in order to improve the accuracy of rat social behavior recognition, in the embodiment, a method for extracting the behavior features of two rats in each frame of image is proposed according to the coordinate information of the key points and the bone posture of each rat, for example, the body shape features of each rat in the current frame of image are determined according to the coordinate information of the key points and the bone posture of each rat in the preset frame of image, wherein the body shape features include the distance between the key points, the angle formed by the bone vector, and the occlusion situation of the key points. Specifically, the distance between the body parts of the individual rat in the body shape features of each rat can depict the state of the rat's own body, such as curling, stretching, or self-grooming state. At the same time, the occlusion situation of the key points of the rat's body parts in the body shape features is helpful to determine whether the mouse is stepped on or suppressed.

[0056] According to the coordinate information of the key points and the bone posture of each rat in the preset frame of image, the motion state features of each rat in the current frame of image are determined, wherein the motion state features include the motion speed, the motion direction, and the displacement. Specifically, the movement distance information (for example, the movement distance between adjacent frames) of the rat itself in the motion state features can depict whether the rat is moving at this time, the speed and intensity of the movement, which information plays an important role in judging whether the rat is in the chasing state or the sniffing behavior of the rat is intense.

[0057] According to the coordinate information of the key points and the bone posture of each rat in the preset frame of image, the distance features between the two rats in the current frame of image are determined, wherein the distance features include the distance between the two rats and the distance between the body parts of the two rats. Specifically, the distance between the two rats can provide a basis for judging whether the rats are in a separated state or an interactive state. The distance between the two rats not only includes the distance between the two rats (which can depict the position state of the two rats), but also includes the specific distance between the body parts of the two rats (which has an important influence on depicting the chasing and sniffing state of the rats).

[0058] According to the coordinate information and the skeleton posture of the key points of each rat in the preset frame image, an interaction tendency feature between the two rats in the current frame image is determined, wherein the interaction tendency feature includes an angle (for example, angle direction, size, and the like) between a center of mass of one of the two rats and a body part (for example, which can be represented by a head key point, a center of mass, or a tail base key point) of the other rat, and the preset frame image includes the current frame image, a first preset number of frame images before the current frame image, and a second preset number of frame images after the current frame image. Specifically, the angle in the interaction tendency feature can be an angle formed by an orientation or a body curve of the two rats. For the angle between the two rats, it can be determined whether the two rats are in an interactive state or a non-interactive state. If interactive, the relative angle between the two rats is small, otherwise, the relative angle between the two rats is large.

[0059] In specific implementation, after the behavior features of the two rats are extracted, in order to further distinguish the social behavior of each rat, in this embodiment, identity information of each rat in each frame image is also determined. The extracted behavior features of the two rats are associated with the identity information of each rat, so as to determine which rat initiates the social behavior based on the identity information.

[0060] In specific implementation, in this embodiment, the identity information of each rat can be identified by the following steps, for example,

[0061] In each frame image, a detection box position of each rat is captured, wherein the detection box includes all key points of the corresponding rat.

[0062] The detection box position of each rat is associated and matched with the position of each rat to generate at least one short track of each rat.

[0063] For two short tracks generated by each identity confusion situation, a first weight of the identity confusion situation is determined according to feature information of the identity confusion situation, and a second weight of the identity confusion situation is determined according to a frequency of the identity confusion situation in all identity confusion situations. The product of the first weight and the second weight is taken as an edge weight of the two short tracks generated by the identity confusion situation.

[0064] A network flow minimization algorithm is used to splice at least one short track of each rat into a long track according to the edge weight of each short track, and identity information of each rat is labeled in each frame image of a video corresponding to the long track.

[0065] Specifically, the identity information of the two rats at any time in the video can be obtained based on a long-time multi-target tracking algorithm, for example,

[0066] All the key points of the rat are surrounded using an error ellipse drawing algorithm to represent the position of the detection box of the rat;

[0067] The position information of the rat in motion is captured using a Kalman filter;

[0068] The detection box in pose estimation is associated and matched with the position predicted by the Kalman filter using a Hungarian algorithm to generate a short track of rat tracking;

[0069] For identity confusion between rats during tracking, different matching strategies between short tracks are used according to different identity confusion situations. For example, short track matching is performed according to the characteristic information of different identity confusion situations;

[0070] For the identity confusion situation of rat stepping and occlusion between rats, the distance between frames of short tracks can be used as the matching weight, i.e., the first weight mentioned above;

[0071] For the identity confusion situation of rat crossing between rats, the change value of the head movement direction angle of the rat can be used as the matching weight, i.e., the first weight mentioned above;

[0072] For the identity confusion situation of rat moving in the same direction between rats, the change value of the tail movement direction angle of the rat can be used as the matching weight, i.e., the first weight mentioned above;

[0073] At the same time, different matching strategies are given additional frequency weights according to the frequency of various identity confusion situations, i.e., the second weight mentioned above. The product of the first weight and the second weight corresponding to each identity confusion situation is used as the edge weight of the two short tracks generated by each identity confusion situation;

[0074] Finally, the network flow minimization algorithm is used to complete the splicing of the short track of each rat according to the edge weight of each short track to obtain a long track and obtain accurate rat identity information. The identity information of each rat is labeled in each frame of the video corresponding to the long track, as shown in Figure 3 The identity information of each rat can be represented by a number, for example, the identity information of one rat can be represented by 1, and 1 can be labeled on each key point to clearly indicate the identity information of the one rat. The identity information of another rat can be represented by 2, and 2 can be labeled on each key point to clearly indicate the identity information of the other rat. The identity information of different rats can also be represented by characters or any different codes.

[0075] In specific implementation, after obtaining the behavior characteristics of the two rats in each frame of image, the behavior characteristics of the two rats in each frame of image can be input into the above behavior recognition model to obtain the social behavior type of the two rats in each frame of image.

[0076] In practice, the types of social behavior include latency, separation behavior, one-way social behavior, and two-way social behavior. One-way social behavior can be further divided into normal one-way social behavior and abnormal one-way social behavior (i.e., aggressive one-way social behavior). Two-way social behavior refers to two rats actively initiating any one or more of the above-mentioned one-way social behaviors towards each other.

[0077] In practice, to improve the accuracy of rat social behavior recognition, during the training of the behavior recognition model, the corresponding social behavior type can be determined from the historical behavioral features of each frame of the image, and the social behavior type corresponding to each frame of the image can be labeled. The behavioral features and corresponding social behavior types of each frame of the image are then used as samples to train the machine learning component to obtain the aforementioned behavior recognition model. For example, the social behavior type corresponding to the historical behavioral features of each frame of the image can be determined using the following method:

[0078] 1) Incubation period: such as Figure 4 As shown ( Figure 4 (a) and (b) show two different rat states during the latency period; the latency period is the time from the start of video recording to the moment when one of the rats first initiates social behavior; 2) Separation behavior: such as Figure 5 As shown ( Figure 5 Figures (a) and (b) show two different rat states during separation behavior (each white area in the figure represents a body area of ​​a rat); separation behavior is the behavior of two rats in a separated state where no approach event occurs; 3) Normal one-way social behavior: such as Figure 6 As shown ( Figure 6 (a) and (b) show two different rat states in normal one-way social behavior. Normal one-way social behavior in rats can include behaviors such as approaching and smelling. Approaching is defined as the behavior when one rat walks directly towards another rat, and the two are within two body-lengths of each other; smelling is defined as the behavior when one rat stretches forward and smells another rat (e.g., smelling the head, face, nose, neck, side tail, anus, etc.), and the distance between the tip of one rat's nose and the other rat's nose is less than 5mm; 4) Abnormal one-way social behavior: such as Figure 7 As shown ( Figure 7The abnormal one-way social behavior in the rat social behavior can include any one or any combination of behaviors such as scratching, pouncing, pushing down, chasing, attacking, and pinning behaviors. Among them, the scratching is defined as the behavior that one rat quickly extends the forepaw to the other rat for oblique stabbing motion when the two rats are within a body position distance range; the pouncing is defined as the behavior that one rat quickly moves forward from a body position distance above to the other rat; the pushing down is defined as the behavior that one rat approaches and holds the other rat from behind, and quickly pushes down; the chasing is defined as the behavior that one rat quickly chases the other rat; the attacking is defined as the behavior that one rat continuously presses down the other rat and limits the movement of the other rat, and the back of the other rat is upward; the pinning is defined as the behavior that one rat continuously presses down the other rat and limits the movement of the other rat, and the abdomen of the other rat is upward; the biting is defined as the behavior that one rat bites the back, side, tail, face, head, or other body parts of the other rat; 5) bidirectional social behavior: as shown in (a) and (b) of FIG. 6, Figure 8 Figure 8 The bidirectional social behavior in the rat social behavior is defined as the behavior that both rats actively initiate any one or more behaviors in the above one-way social behavior to each other. The corresponding behavior category is labeled for each frame in the training video.

[0079] In specific implementation, the machine learning component for training the behavior recognition model can be a random forest classification model.

[0080] In specific implementation, after obtaining the social behavior type of each frame image, the total number of frame images corresponding to each social behavior type can be counted to determine the time length corresponding to each social behavior type, and then the percentage of the time length corresponding to each social behavior type in the video time length can be calculated.

[0081] In specific implementation, the process of implementing the above rat social behavior recognition method is introduced in detail below, which can include the following steps:

[0082] Step 1, install the equipment, collect the video clips of rat social behavior for training and testing, and take the video of rat social behavior in the experimental scene as an example.

[0083] Step 1.1, build a free social field in the behavior room, disinfect the test box (39 cm x 25 cm x 20 cm) with 95% ethanol, install a camera on the ceiling to take a picture of the test box, adjust the position of the test box and the camera, and obtain a clear and complete view.

[0084] ​Step 1.2, four groups of behavior videos are recorded simultaneously in four test boxes, and the boxes are separated by opaque partitions to prevent the experimental mice in adjacent boxes from seeing each other; the behavior room is kept quiet, dark, and undisturbed throughout the experiment.

[0085] Step 1.3, the subject mice and model mice are taken out and allowed to adapt to the new environment in the behavior room for thirty minutes.

[0086] Step 1.4, place one subject mouse from the model group in each of the four test boxes, and allow it to adapt and freely explore in the completely new unlimited empty test box for ten minutes.

[0087] Step 1.5, place another wild-type model mouse of the same strain, matching in age, gender, and weight, into each test box under dark and quiet conditions, and allow it to freely socialize for ten minutes after closing the behavior room door, and keep the video recording for analysis.

[0088] Step 2, label the key point positions of the rat in the video frames of the collected training video dataset.

[0089] Step 2.1, for each video, take 600 frames at equal intervals, and label the key points of all rats in each frame image; the key points of each rat can include head, front torso, back torso, tail base, middle tail, and tail end, etc.

[0090] Step 3, obtain the skeletal pose of the rat through a bottom-up rat key point position recognition technique and a key point connection cost calculation method.

[0091] Step 3.1, the input of the rat key point detection model is a 1 / 4 size heat map image of the video frame image; the detection model is based on the ResNet-50 network, and the features in the bottleneck of the third unit in each of the four blocks of ResNet-50 are selected and spliced after convolution, and finally two inverse convolution layers are used to output the predicted heat map (which includes the coordinates and probabilities of the key points) and vector map (which includes the connection vectors between the key points).

[0092]

[0093]

[0094] The Huber loss function is used to predict the vector map between the rat key points, as shown in the following formula:

[0095] ​​

[0096] where y represents the true class, p represents the class predicted by the model; f(x) represents the predicted value of the model.

[0097] Step 3.3, based on the predicted vector diagram, the cost weight of the connection between the key points is calculated, and the best skeleton posture of the rat is obtained by the minimum spanning tree algorithm, and the coordinate information of the key points and the skeleton posture are mapped to the corresponding frame image.

[0098] Step 4, using a long-time multi-target tracking algorithm to obtain the identity information of the two rats at any time in the video.

[0099] Step 4.1, using the error ellipse drawing algorithm to enclose all the key points of the rat to represent the position of the detection box of the rat.

[0100] Step 4.2, using Kalman filter to capture the position information of the moving rat.

[0101] Step 4.3, using the Hungarian algorithm to associate and match the detection box in the posture estimation with the position predicted by the Kalman filter, generating a short track for each rat tracking.

[0102] Step 4.4, for the identity confusion between rats in the tracking process, different matching strategies between short tracks are used according to different confusion situations.

[0103] Step 4.5, for the identity confusion of the rat stepping and occlusion, the distance between the frames of the short tracks is used as the matching weight, that is, the first weight mentioned above.

[0104] Step 4.6, for the identity confusion of the rat interlaced and walking, the change value of the head movement direction angle of the rat is used as the matching weight, that is, the first weight mentioned above.

[0105] Step 4.7, for the identity confusion of the rat interlaced and walking, the change value of the tail movement direction angle of the rat is used as the matching weight, that is, the first weight mentioned above.

[0106] Step 4.8, according to the frequency of various identity confusion situations, an additional frequency weight is given to different matching strategies, that is, the second weight mentioned above, for each identity confusion situation, the product of the first weight and the second weight is used as the edge weight of the short track generated by this identity confusion situation.

[0107] Step 4.9, using network flow minimization algorithm, according to the edge weight of the short track, the splicing of the short track of the rat is completed, a long track is obtained, and the identity information of each rat in the accurate long track is obtained.

[0108] Step 5: Extract the behavioral features of the two rats based on the coordinates of key points, skeletal posture, and identity information. Input the behavioral features of the two rats and the labeled social behavior types into the random forest classification model for training to obtain the above-mentioned behavior recognition model.

[0109] Step 5.1: Set the coordinates of the key points of the rat. The normalization process is performed using the following formula:

[0110]

[0111] The video size is The coordinates of a key point are , The width of the video is respectively The reason for enlarging the normalized coordinates is that if the coordinates are decimals, the computational cost and error will increase. Enlarging improves the accuracy and efficiency of the calculation.

[0112] Step 5.2: Based on the distance between key points of the rat, the angle formed by the skeletal vectors, and the occlusion of key points, calculate and extract the body morphology features of each rat in the current frame and in the time period before and after the current frame.

[0113] Step 5.2: Based on the rat's body parts and the rat's overall movement speed, direction, and displacement, calculate and extract the movement state features of each rat in the current frame and for a period of time before and after the current frame.

[0114] Step 5.3: Based on the centroid distance between the two rats and the distances between various body parts of the two rats, calculate and extract the distance features between the two rats in the current frame and in the time before and after the current frame.

[0115] Step 5.4: Based on the angle between the head of one rat and the body parts (e.g., head, centroid, or tail base keypoint) of the other rat, with the centroid of one rat as the vertex, extract the interaction tendency features of the two rats in the current frame and for a period of time before and after the current frame.

[0116] Step 5.5, the social behavior of rats can be divided into: 1) latency: the latency of rat social behavior is defined as the time from when the model rat is placed in the experimental box to when the test rat initiates social behavior for the first time; 2) separation behavior: the separation behavior of rat social behavior is defined as the behavior of two rats in a state of separation without approaching events; 3) normal one-way social behavior: normal one-way social behavior of rat social behavior includes approaching and sniffing. Among them, approaching is defined as the behavior of the observed rat walking straight to another rat, and the two are within the distance range of two body positions; Sniffing is defined as the behavior of the observed rat stretching forward and sniffing the head, face, nose, neck, side body tail, and anus of another rat, with a distance between the tip of the nose and the other rat of less than 5mm; 4) abnormal one-way social behavior: abnormal one-way social behavior of rat social behavior includes scratching, pouncing, pushing, chasing, attacking and pinching behavior. Among them, scratching is defined as the behavior of the observed rat quickly extending the front paw to another rat for oblique stabbing, and the two are within the distance range of one body position; Pouncing is defined as the behavior of the observed rat moving quickly forward from a distance of more than one body position to another rat; Pushing is defined as the behavior of the observed rat approaching and grabbing another rat from behind, and quickly pushing it down; Chasing is defined as the behavior of the observed rat chasing another rat quickly; Attack is defined as the behavior of the observed rat continuously pressing down on another rat and restricting its movement, with the latter's back up; Pinching is defined as the behavior of the observed rat continuously pressing down on another rat and restricting its movement, with the latter's abdomen up; Biting is defined as the behavior of the observed rat biting the back, side, tail, face, and head of another rat; 5) two-way social behavior: two-way social behavior of rat social behavior is defined as the behavior of both rats actively initiating any of the above one-way social behaviors towards each other. Each frame of image in the training video is labeled with the corresponding social behavior type.

[0117] Step 5.6, the rat behavior features extracted from each frame of image of the training video and the corresponding social behavior type are input into the random forest classification model for training, to obtain the above behavior recognition model.

[0118] Step 6, according to the coordinates of the rat key points, the rat skeletal posture and the rat identity information obtained from each frame of image of the test video, the behavior features of the two rats are calculated and input into the trained random forest classification model to obtain the social behavior type of the two rats in each frame of image of the test video.

[0119] Step 7, the classified social behavior is statistically analyzed to calculate the percentage of the duration of each social behavior type in the total duration of the video.

[0120] Step 7.1, the percentage of the duration of appropriate social behavior in the one-way social behavior initiated by the test rat in the video is calculated.

[0121] Step 7.2, statistics of the percentage of the time length of aggressive social behavior in the one-way social behavior initiated by the subject mouse in the video in the total time length of the video.

[0122] Step 7.3, statistics of the percentage of the time length of appropriate social behavior in the one-way social behavior initiated by the model mouse in the video in the total time length of the video.

[0123] Step 7.4, statistics of the percentage of the time length of aggressive social behavior in the one-way social behavior initiated by the model mouse in the video in the total time length of the video.

[0124] Step 7.5, statistics of the percentage of the time length of appropriate two-way interaction behavior in the video in the total time length of the video.

[0125] Step 7.6, statistics of the percentage of the time length of aggressive two-way interaction behavior in the video in the total time length of the video.

[0126] Step 7.7, statistics of the percentage of the time length of self-grooming behavior of the subject mouse in the video in the total time length of the video.

[0127] Step 7.8, statistics of the percentage of the time length of self-grooming behavior of the model mouse in the video in the total time length of the video.

[0128] In the implementation, the above rat social behavior recognition method provides an automatic and intelligent analysis method for inter-rat social behavior, can realize efficient and accurate recognition of inter-rat social behavior types, is beneficial to providing accurate and reliable behavior analysis data for researchers, and can solve the problem of high labor cost of manual social behavior analysis.

[0129] In this embodiment, a computer device is provided, which, as shown in the accompanying drawings, comprises a memory 901, a processor 902, and a computer program stored in the memory and executable on the processor, and the processor implements any of the above rat social behavior recognition when executing the computer program. Figure 9

[0130] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0131] In this embodiment, a computer readable storage medium is provided, which stores a computer program for executing any of the above rat social behavior recognition.

[0132] ​In particular, computer readable storage media tangibly embody the software and / or firmware programming code that is executable by a computer and causes a computer to perform a method. The computer readable storage media can be implemented in numerous types of media, in or out of the computer, which are suitable for storing this type of programmable code. The computer readable storage media can be implemented using all or any portion of memory such as volatile or non-volatile computer memory or removable computer memory such as a random access memory (RAM), a read-only memory (ROM), a flash memory included in a personal digital assistant (PDA), a computer-readable medium such as a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), or any other memory storage technology. The computer readable storage media can be tangibly embodied in one or more computer- readable storage media components or media devices available to a computer. The computer readable storage media described here can store data which is accessible by a computer.

[0133] Based on the same inventive concept, the embodiments of the present application also provide a rat social behavior recognition device, as described in the following embodiments. Since the rat social behavior recognition device solves the problem by the same principle as the rat social behavior recognition method, the implementation of the rat social behavior recognition device can refer to the implementation of the rat social behavior recognition method, and the repeated parts will not be described here. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0134] Figure 10 is a structural block diagram of the rat social behavior recognition device of the embodiments of the present application, as shown in Figure 10 The device comprises:

[0135] A key point determination module 1001 is configured to acquire a video comprising two target rats and determine key points of each rat in each frame image of the video.

[0136] A skeleton information determination module 1002 is configured to determine, in each frame image, coordinate information and a skeleton pose of the key points of each rat according to the key points of each rat.

[0137] A feature extraction module 1003 is configured to extract, in each frame image, behavior features of the two rats according to the coordinate information and the skeleton pose of the key points of each rat.

[0138] The behavior type recognition module 1004 is configured to input the behavior features of the two rats in each frame of image into a behavior recognition model, and the behavior recognition model outputs the social behavior type of the two rats in each frame of image, wherein the behavior recognition model is obtained by training a machine learning component with historical behavior features and corresponding social behavior types as samples.

[0139] In one embodiment, the skeleton information determination module is configured to map a preset image region of each frame of image as a heat map image, the preset image region including the key points of the two rats; input the heat map image into a detection model, the detection model outputting a heat map and a vector map, wherein the heat map includes coordinate information of the key points in the heat map image, and the vector map includes connection vector information between the key points in the heat map image, the detection model being obtained by training a multi-scale feature fusion network with historical heat map images, historical heat maps and historical vector maps as samples; generate a skeleton pose according to the vector map by using a minimum spanning tree algorithm; and map the coordinate information of the key points in the heat map and the skeleton pose to the frame of image corresponding to the heat map image.

[0140] In one embodiment, the feature extraction module is configured to determine, according to the coordinate information of the key points and the skeleton pose of each rat in a preset frame of image, a body shape feature of each rat in a current frame of image, wherein the body shape feature includes a distance between key points, an angle formed by a skeleton vector, and an occlusion situation of a key point.

[0141] In one embodiment, the feature extraction module is configured to determine, according to the coordinate information of the key points and the skeleton pose of each rat in a preset frame of image, a motion state feature of each rat in a current frame of image, wherein the motion state feature includes a motion speed, a motion direction, and a displacement.

[0142] In one embodiment, the feature extraction module is configured to determine, according to the coordinate information of the key points and the skeleton pose of each rat in a preset frame of image, a distance feature between the two rats in a current frame of image, wherein the distance feature includes a center of mass distance between the two rats and a distance between body parts of the two rats.

[0143] In one embodiment, the feature extraction module is configured to determine, according to the coordinate information of the key points and the skeleton pose of each rat in a preset frame of image, an interaction tendency feature between the two rats in a current frame of image, wherein the interaction tendency feature includes an angle between a head of one of the two rats and a body part of the other rat, with a center of mass of the one rat as a vertex, and the preset frame of image includes the current frame of image, a first preset number of frame of images before the current frame of image, and a second preset number of frame of images after the current frame of image.

[0144] In one embodiment, the apparatus further includes:

[0145] An identity information determination module is configured to determine identity information of each rat in each frame of image.

[0146] An information association module is configured to associate the extracted behavior features of the two rats with the identity information of each rat.

[0147] In an embodiment, the identity information determination module is configured to capture a detection box position of each rat in each frame of image, wherein the detection box comprises all key points of the corresponding rat; associate and match the detection box position of each rat with the position of each rat to generate at least one short track of each rat; for two short tracks generated in each identity confusion situation, determine a first weight of the identity confusion situation according to the feature information of the identity confusion situation, and determine a second weight of the identity confusion situation according to the frequency of the identity confusion situation in all identity confusion situations, and take the product of the first weight and the second weight as the edge weight of the two short tracks generated in the identity confusion situation; and use a network flow minimization algorithm to splice at least one short track of each rat into one long track according to the edge weight of each short track, and label the identity information of each rat in each frame of image of the long track corresponding video.

[0148] In an embodiment, the device further comprises:

[0149] A statistical module is configured to count the total number of frames of image corresponding to each social behavior type to determine the time length corresponding to each social behavior type.

[0150] The embodiment of the present application achieves the following technical effects: the key points of each rat are determined in each frame of image of the video, and the coordinate information and the skeletal posture of the key points of each rat are determined according to the key points of each rat, and then the behavior features of the two rats are extracted according to the coordinate information and the skeletal posture of the key points of each rat, and finally the behavior recognition model is input with the behavior features of the two rats in each frame of image, and the behavior recognition model can output the social behavior type of the two rats in each frame of image. The key points, the coordinate information and the skeletal posture of the key points are determined by image processing of each frame of image of the video, and then the coordinate information and the skeletal posture of the key points are input into the behavior recognition model of deep learning, so that the social behavior type of the two rats in each frame of image can be recognized, and an automatic and intelligent rat social behavior recognition process is realized. Compared with the scheme of manually evaluating rat social behavior in the prior art, the efficiency of rat social behavior recognition can be improved; at the same time, the influence of human factors such as personal professional difference and human error is avoided, which is conducive to improving the accuracy of rat social behavior recognition.

[0151] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be realized by a general computing device, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different orders, or each module can be manufactured as an individual integrated circuit module, or multiple modules or steps can be manufactured as a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0152] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recognizing social behavior in rats, characterized in that, include: Acquire videos of two rats and identify key points for each rat in each frame of the video; In each frame of the image, the coordinate information and skeletal posture of the key points of each rat are determined based on the key points of each rat. In each frame of the image, behavioral features of two rats are extracted based on the coordinate information of the key points of each rat and the skeletal posture. The behavioral features of the two rats in each frame image are input into the behavior recognition model, and the behavior recognition model outputs the social behavior type of the two rats in each frame image. The behavior recognition model is obtained by training machine learning components with historical behavioral features and corresponding social behavior types as samples. In each frame, based on the keypoints of each rat, the coordinate information of the keypoints and the skeletal pose of each rat are determined, including: A preset image region for each frame is mapped to a thermal image, and the preset image region includes key points of two rats; The thermal image is input into the detection model, and the detection model outputs a heatmap and a vector map. The heatmap includes the coordinate information of key points in the heatmap, and the vector map includes the connection vector information between key points in the heatmap. The detection model is obtained by training a multi-scale feature fusion network using historical heatmaps, historical thermal images, and historical vector maps as samples. The minimum spanning tree algorithm is used to generate the skeleton pose based on the vector graph; The coordinate information of key points in the heatmap and the skeletal pose are mapped to the frame image corresponding to the heatmap.

2. The rat social behavior recognition method as described in claim 1, characterized in that, In each frame, behavioral features of the two rats are extracted based on the coordinates of key points and skeletal posture of each rat, including: Based on the coordinate information of key points and skeletal posture of each rat in the preset frame image, the body morphology features of each rat in the current frame image are determined, wherein the body morphology features include the distance between key points, the angle formed by the skeletal vectors, and the occlusion status of key points. Based on the coordinate information of key points and skeletal posture of each rat in the preset frame image, the motion state characteristics of each rat in the current frame image are determined, wherein the motion state characteristics include motion speed, motion direction and displacement; Based on the coordinate information of key points and skeletal posture of each rat in the preset frame image, the spacing features between two rats in the current frame image are determined, wherein the spacing features include the centroid distance between the two rats and the distance between each body part of the two rats; Based on the coordinate information of key points and skeletal posture of each rat in the preset frame image, the interaction tendency features between the two rats in the current frame image are determined. The interaction tendency features include the angle between the head of one rat and the body part of the other rat, with the centroid of one rat as the vertex. The preset frame image includes the current frame image, a first preset number of frame images before the current frame image, and a second preset number of frame images after the current frame image.

3. The rat social behavior recognition method according to any one of claims 1 to 2, characterized in that, Also includes: In each frame of the image, the identification information of each rat is determined; The behavioral characteristics of the two rats were extracted and associated with the identity information of each rat.

4. The rat social behavior recognition method as described in claim 3, characterized in that, In each frame of the image, the identification information of each rat is determined, including: In each frame of the image, the location of the detection box for each rat is captured, wherein the detection box includes all key points of the corresponding rat; The detection box position of each rat is associated and matched with the position of each rat to generate at least one short trajectory for each rat; For each type of identity confusion, two short trajectories are generated. The first weight of the identity confusion is determined based on the feature information of the identity confusion. The second weight of the identity confusion is determined based on the frequency of the identity confusion among all identity confusion. The product of the first weight and the second weight is used as the edge weight of the two short trajectories generated by the identity confusion. The network flow minimization algorithm is used to stitch together at least one short trajectory of each rat into a long trajectory based on the edge weights of each short trajectory segment, and the identity information of each rat is labeled in each frame of the video corresponding to the long trajectory.

5. The rat social behavior recognition method according to any one of claims 1 to 2, characterized in that, The types of social behavior include latency, separation behavior, one-way social behavior, and two-way social behavior.

6. The rat social behavior recognition method according to any one of claims 1 to 2, characterized in that, Also includes: Count the total number of frames corresponding to each type of social behavior, and determine the duration of each type of social behavior.

7. A device for recognizing social behavior in rats, characterized in that, include: The key point determination module is used to acquire a video including two target rats and determine the key points of each rat in each frame of the video; The skeletal information determination module is used to determine the coordinate information and skeletal posture of the key points of each rat in each frame of the image, based on the key points of each rat. The feature extraction module is used to extract the behavioral features of two rats in each frame of the image based on the coordinate information of the key points of each rat and the skeletal posture. The behavior type recognition module is used to input the behavioral features of the two rats in each frame image into the behavior recognition model, and the behavior recognition model outputs the social behavior type of the two rats in each frame image. The behavior recognition model is obtained by training the machine learning component with historical behavioral features and corresponding social behavior types as samples. A skeletal information determination module is used to map a preset image region of each frame image into a heatmap image, wherein the preset image region includes key points of two rats; the heatmap image is input into a detection model, and the detection model outputs a heatmap and a vector map, wherein the heatmap image includes the coordinate information of the key points in the heatmap image, and the vector map includes the connection vector information between the key points in the heatmap image; the detection model is obtained by training a multi-scale feature fusion network using historical heatmap images, historical heatmap images, and historical vector maps as samples; a minimum spanning tree algorithm is used to generate skeletal pose based on the vector map; and the coordinate information of the key points in the heatmap image and the skeletal pose are mapped to the frame image corresponding to the heatmap image.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the rat social behavior recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the rat social behavior recognition method according to any one of claims 1 to 6.

Images