Machine learning-driven behavior feature classification method, storage medium and processor

Abstract

The present application is applied to the technical field of video recognition and classification, and provides a machine learning-driven behavior feature classification method, a storage medium and a processor. According to the method, on the basis of an extraction algorithm which can be used for estimating animal behavior fine features, by acquiring video data of animals at multiple angles and in multiple directions, and analyzing skeleton dynamic features of different fine actions in videos, specific accurate features and all possible action features of different animals in specific states can be accurately identified, and can be further fitted to obtain complex behaviors of animals. In addition, on the basis of three-dimensional behavior analysis and in light of multi-dimensional determination of feature dynamic analysis of different animals in different states, animal behavior modes that are difficult to distinguish due to extremely similar action features can be distinguished.

Description

Machine learning-driven behavioral feature classification method, storage medium, and processor Technical Field

[0001] The present application belongs to the field of video recognition and classification technology, and in particular relates to a machine learning-driven behavior feature classification method, storage medium, and processor. Background Art

[0002] The behavior of different animals in different states is not only complex but often exhibits significant individual variation. However, in actual classification, the judgment of different animals in different states lacks suitable biomarkers, and is often determined only through experience or scale scoring. Existing methods for assessing animal behavioral traits mostly rely solely on a simple analysis of their primary behavioral states, resulting in a very crude assessment of behavioral postures. On the one hand, most rely on experienced behavioral neuroscientists trained to distinguish using the naked eye, which not only has low resolution efficiency but is also fraught with subjectivity. Furthermore, due to technical limitations in video information extraction, there is no mature method for objectively and precisely quantifying the behavioral phenotypes of different animals in different states. Existing technologies are insufficient.

[0003] Summary of the Invention

[0004] The purpose of this application is to provide a machine learning driven behavioral feature classification method, storage medium and processor, aiming to solve the technical problems that the existing animal behavioral feature analysis is one-sided and the analysis results are relatively subjective.

[0005] In one aspect, the present application provides a machine learning driven behavioral feature classification method, the method comprising the following steps:

[0006] s1. Set up at least two cameras in a specific area to record the animal's behavior from different angles;

[0007] s2. Perform frame-by-frame video recognition, extract body parts that represent animal behavior in the video, and perform three-dimensional behavior analysis;

[0008] s3. Use deep learning technology to track multiple body parts of the animal in the video and reconstruct the three-dimensional motion skeleton of different model animals under different behavioral states;

[0009] s4. Align the three-dimensional motion skeleton of each frame of the animal for coordinate correction based on the data generated by the fine-grained behavioral analysis; obtain comparable three-dimensional skeleton information at the same spatial scale; calculate the various motion posture features of the animal by analyzing the dynamic motion parameters of the three-dimensional motion skeleton to achieve motion posture feature extraction;

[0010] s5. Based on the three-dimensional reconstruction of the action posture characteristics, a three-dimensional fine feature data set of animal behavior is obtained, and then a three-dimensional fine behavioral feature map of different animals under different states and a complete behavioral map is constructed;

[0011] s6. Based on the three-dimensional fine behavioral feature map, further analyze the various behavioral characteristics of different animals in different states, and establish a high-dimensional correspondence between their behavioral characteristics and different states.

[0012] On the other hand, the present application also provides a storage medium, which stores a program file that can implement the above-mentioned machine learning-driven behavior feature classification method.

[0013] On the other hand, the present application also provides a processor for running a program, wherein the program executes the above-mentioned machine learning-driven behavioral feature classification method when running.

[0014] This application proposes a machine learning-driven behavioral feature classification method based on a set of fine feature extraction algorithms that can be used to estimate different animals in different states. By collecting animal video data from multiple directions and angles and analyzing the skeletal dynamics of different fine movements in the video, it can accurately identify the precise characteristics unique to different animals in specific states and all possible movement characteristics, and further fit the characteristics and complete behavior of the animals. At the same time, based on three-dimensional behavioral analysis combined with multi-dimensional judgment of the dynamic analysis of the characteristics of different animals in different states, it can distinguish animal behavior patterns that are difficult to distinguish due to very similar movement characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a basic flow chart of a behavioral feature classification method driven by machine learning provided in Example 1 of the present application;

[0016] FIG2 is a schematic diagram of the main process of identifying complex animal behaviors in this application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0018] The following describes the specific implementation of this application in detail with reference to specific embodiments:

[0019] Example 1:

[0020] Figure 1 shows the implementation process of the machine learning-driven behavioral feature classification method provided in Example 1 of this application, and Figure 2 shows a schematic diagram of the main process of identifying complex animal behaviors in this application. For ease of explanation, only the parts related to this embodiment of the application are shown, and the details are as follows:

[0021] As shown in Figures 1-2, this application provides a machine learning-driven behavioral feature classification method, which includes the following steps:

[0022] s1. Set up at least two cameras in a specific area to record the animal's behavior from different angles;

[0023] s2. Perform frame-by-frame video recognition, extract body parts that represent animal behavior in the video, and perform three-dimensional behavior analysis;

[0024] s3. Use deep learning techniques to track multiple body parts of animals in videos and reconstruct the 3D motion skeletons of different animal models under different behavioral states;

[0025] s4. Based on the data generated by the detailed behavioral analysis, the 3D motion skeleton of each frame of the animal is aligned to achieve coordinate correction. Comparable 3D skeleton information at the same spatial scale is obtained. By analyzing the dynamic motion parameters of the 3D motion skeleton, the characteristics of each animal's motion posture are calculated to achieve motion posture feature extraction.

[0026] s5. Perform 3D reconstruction based on motion and posture features to obtain a 3D fine feature dataset of animal behavior, and then construct a 3D fine behavioral feature map of different animals in different states and a map of their complete behavior;

[0027] s6. Based on the three-dimensional detailed behavioral feature map, further analyze the various behavioral characteristics of different animals in different states, and establish a high-dimensional correspondence between their behavioral characteristics and different states.

[0028] In specific implementation, in view of the current lack of effective means to extract three-dimensional, accurate and objective posture information of animals from videos, this application uses three-dimensional fine behavior analysis technology to objectively evaluate the physiological state patterns of animals and identify abnormal and specific movement and posture characteristics of animals without introducing additional stimulation.

[0029] Preferably, step s3 includes:

[0030] Use deep learning technology to integrate the corresponding tracking information of multiple body parts in different camera videos to perform detailed behavioral analysis; reconstruct its 3D skeleton based on the analysis results.

[0031] Preferably, the posture features include posture features such as the length, height, curvature of the body skeleton, amplitude of body shaking, degree of body curling, and movement speed of the animal in the video.

[0032] Preferably, in step s2, the results of the recognition of each frame of the image are connected in series to constitute continuous tracking of the body parts of the animal, thereby obtaining the spontaneous behavioral data of the animal.

[0033] Preferably, in step s3, the reconstruction of the three-dimensional skeleton includes the following steps:

[0034] w1. Extract the 3D coordinates of 16 key points of the animal's body in each frame of video;

[0035] w2. Align the animal's 3D skeleton in each frame and unify the horizontal axis orientation of the body;

[0036] w3. Scale the animal skeleton in each frame to the same scale;

[0037] w4. Calculate the posture features of the action based on the action labels divided by the results of detailed behavioral analysis.

[0038] Preferably, the detailed behavioral analysis includes the following steps:

[0039] j1. In three-dimensional space, analyze the duration of each animal's action within a predetermined time range and determine the time percentage of each action:

[0040] j2. In three-dimensional space, count the number of times each action occurs within a predetermined time range and obtain the frequency ratio of each action;

[0041] j3 in three-dimensional space, statistics of the duration of each action within a predetermined time range, to obtain the average duration of each action;

[0042] j4. Calculate the distribution duration of each action at different positions in the behavioral box within a predetermined time range to obtain the spatiotemporal distribution characteristics of each action.

[0043] Preferably, the animals include animals of different genotypes and / or heterozygotes and homozygotes based on the same gene mutation.

[0044] Preferably, corresponding action posture feature model databases are established for model animals of different genotypes and / or heterozygotes and homozygotes of the same gene mutation.

[0045] This application proposes a machine learning-driven behavioral feature classification method based on a set of fine feature extraction algorithms that can be used to estimate the characteristics of different animals in different states. By collecting video data of animals from multiple directions and angles and analyzing the skeletal dynamics of different fine movements in the video, it can accurately identify the precise characteristics unique to different animals in specific states and all possible movement characteristics, and further fit the characteristics and complete behavior of the animals. At the same time, based on three-dimensional behavioral analysis combined with multi-dimensional judgment of the dynamic analysis of the characteristics of different animals in different states, it can distinguish animal behavior patterns that are difficult to distinguish due to very similar movement characteristics.

[0046] The refined behavioral analysis methods used in the above analysis process are not limited to Behavior Atlas. Similar analysis results can be obtained by using similar refined behavioral analysis software such as Moseq and LEAP, as well as multi-site animal body tracking methods such as DeepLabCut and EthoVision to reconstruct the animal's 3D skeleton and then applying the posture feature extraction algorithm proposed in this application.

[0047] Example 2:

[0048] On the other hand, the present application also provides a storage medium, which stores a program file that can implement the above-mentioned machine learning-driven behavior feature classification method.

[0049] Those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0050] Example 3:

[0051] On the other hand, the present application also provides a processor for running a program, wherein the program executes the above-mentioned machine learning-driven behavioral feature classification method when running.

[0052] In the embodiments of the present application, the machine learning-driven behavioral feature classification method can be implemented by corresponding hardware or software units. Each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit. This is not intended to limit the present application. The specific implementation of each unit can be referred to the description of Example 1 and will not be repeated here.

[0053] This application uses multiple cameras to capture continuous video and reconstruct the animal's 3D skeleton based on the video to observe the animal's 3D movements and postures. The application also observes the characteristics and various behaviors of different animals in their natural state, without external stimuli that cause behavioral variations, and thus better reflects the animal's behavior in its true state. By digitizing the various posture characteristics of the animal through calculation, it can more accurately describe the animal's posture characteristics and provide quantitative indicators for detailed analysis of behavior and movements.

[0054] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A machine learning-driven behavioral feature classification method, characterized in that, The method includes the following steps: s1. Set at least two cameras in a specific area, and record the behaviors of animals in videos from different angles. s2. Perform video recognition frame by frame, extract the body sites representing animal behaviors in the video, and conduct three-dimensional behavior analysis. s3. Use deep learning technology to track multiple body sites of animals in the video, and reconstruct the three-dimensional action skeletons of different animals in different behavior states. s4. For the data generated by fine behavior analysis, align the three-dimensional action skeletons of each frame of the animal to achieve coordinate correction; obtain three-dimensional skeleton information that can be compared at the same spatial scale; calculate the action posture characteristics of the animal by analyzing the dynamic motion parameters of the three-dimensional action skeleton, and achieve action posture feature extraction. s5. Based on the action posture characteristics, perform three-dimensional reconstruction to obtain a three-dimensional fine feature dataset of animal behaviors, and then construct three-dimensional fine behavior feature maps and complete behavior maps of different animals in different states. s6. Further analyze various behavior characteristics of different animals in different states based on the three-dimensional fine behavior feature maps, and establish a high-dimensional correspondence relationship between their behavior characteristics and different states.

2. The method according to claim 1, characterized in that, The step s3 includes: Use deep learning technology to integrate the corresponding tracking information of multiple body sites in videos from different cameras, and conduct the fine behavior analysis; reconstruct the three-dimensional skeleton of animals in complex physiological state patterns based on the analysis results.

3. The method according to claim 2, wherein The posture characteristics include posture characteristics such as the body length, body height, degree of body skeleton bending, body shaking amplitude, body curling degree, and movement speed of the animal in the video.

4. The method according to claim 3, wherein In the step s2, the results of the recognition of each frame of the image are concatenated to form a continuous tracking of the body sites of the animal, and ethological data of the animal are obtained.

5. The method according to claim 2, wherein In the step s3, the reconstruction of the three-dimensional skeleton includes the following steps: w1. Extract the three-dimensional spatial coordinates of 16 body key points of the animal in each frame of the video. w2. Align the three-dimensional skeletons of the animal in each frame and unify the orientation of the body horizontal axis. w3. Scale the animal skeletons in each frame to the same scale size. w4. Calculate the posture characteristics of the action according to the action labels divided by the results of the fine ethological analysis.

6. The method according to claim 5, wherein The fine ethological analysis includes the following steps: j1. In three-dimensional space, analyze the duration of each action of the animal within a predetermined time range to obtain the time proportion of the action. j2. In three-dimensional space, count the number of occurrences of each action within a predetermined time range to obtain the frequency proportion of each action. j3. In three-dimensional space, count the duration of each action within a predetermined time range to obtain the average duration of each action. j4. Calculate the distribution duration of each action at different positions in the ethological box within a predetermined time range to obtain the spatio-temporal distribution characteristics of each action.

7. The method according to claim 1, characterized in that, The animals include different genotypes and / or heterozygotes and homozygotes based on the same gene mutation.

8. The method according to claim 7, wherein Establish corresponding action posture feature model databases for different genotype model animals and / or heterozygotes and homozygotes of the same gene mutation respectively.

9. A storage medium, characterized in that, The storage medium stores a program file capable of implementing the machine learning-driven behavioral feature classification method described in any one of claims 1 to 8.

10. A processor, characterized in that, The processor is used to run the program, wherein when the program runs, it executes the machine learning-driven behavioral feature classification method described in any one of claims 1 to 8.

Images