Skeleton behavior recognition method and device, and electronic device

Abstract

The application provides a skeleton behavior recognition method and device and electronic equipment, wherein the skeleton behavior recognition method comprises: determining skeleton data of an animal to be recognized; performing global feature extraction on the skeleton data based on a preset skeleton behavior recognition model, fusing the skeleton data based on the relevance of first skeleton behavior features obtained through the global feature extraction and second skeleton behavior features, and performing skeleton behavior recognition based on the fused features. Since the first skeleton behavior features and the second skeleton behavior features are both obtained through global feature extraction, the real behavior of the skeleton of the animal to be recognized in the time and space dimensions can be retained, and the accuracy of the skeleton behavior recognition can be effectively improved through the fusion of the skeleton data based on the relevance of the first skeleton behavior features and the second skeleton behavior features, so as to cover various limb actions involved in the skeleton behavior.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a skeleton behavior recognition method, device, and electronic device. Background Technology

[0002] Skeleton behavior recognition, as an important research branch in the field of computer vision, includes not only human skeleton behavior recognition but also animal skeleton behavior recognition. Since recognizing animal behavior (such as recognizing the skeleton behavior of monkeys) is of great significance and application prospects for the evaluation of new drugs, how to accurately recognize animal skeleton behavior has become a popular research direction.

[0003] In related technologies, animal skeleton behavior recognition methods are similar in principle to human skeleton behavior recognition methods. Animal skeleton behavior recognition methods typically consist of two stages. First, a pose estimation model is used to identify the skeleton information containing animal key points in the video. Then, the skeleton information is analyzed to obtain the corresponding behavioral state. For example, skeleton behavior recognition methods can use the Spatio-Temporal Graph Convolutional Networks (ST-GCN) model.

[0004] However, since the ST-GCN model can only extract features from a few frames of video images in the skeleton information, the extracted information only involves local features of the animal, resulting in low accuracy in animal skeleton behavior recognition. Summary of the Invention

[0005] This invention provides a skeletal behavior recognition method, device, and electronic device to address the shortcomings of existing technologies where the features extracted based on the ST-GCN model only involve local features of the animal, resulting in low accuracy in animal skeletal behavior recognition. This invention aims to achieve efficient and accurate recognition of animal skeletal behavior.

[0006] This invention provides a skeleton behavior recognition method, comprising:

[0007] Determine the skeletal data of the animal to be identified;

[0008] Based on a preset skeleton behavior recognition model, global features are extracted from the skeleton data. Based on the correlation of the first skeleton behavior features and the second skeleton behavior features obtained from the global feature extraction, the skeleton data is fused, and skeleton behavior recognition is performed based on the fused features.

[0009] According to a skeleton behavior recognition method provided by the present invention, the step of extracting global features from the skeleton data based on a preset skeleton behavior recognition model includes:

[0010] Based on the preset ST-GCN sub-model in the preset skeleton behavior recognition model, channel transformation is performed on the skeleton data;

[0011] Global feature extraction is performed on the skeleton data after channel transformation according to the preset channel dimensions.

[0012] According to a skeleton behavior recognition method provided by the present invention, when the number of first skeleton behavior features is greater than the number of second skeleton behavior features, the method of fusing the skeleton data based on the correlation of the first skeleton behavior features obtained from the global feature extraction and the second skeleton behavior features includes:

[0013] Based on the correlation between the various first skeleton behavior features obtained from the global feature extraction, a correlation matrix is ​​determined;

[0014] The skeleton data is fused based on the correlation matrix and the second skeleton behavior features.

[0015] According to a skeleton behavior recognition method provided by the present invention, determining a correlation matrix based on the correlation of the first skeleton behavior features obtained from the global feature extraction includes:

[0016] Each first skeleton behavior feature extracted based on the global features is subjected to dimensionality reduction processing to obtain the corresponding first target skeleton behavior feature;

[0017] Calculate the correlation matrix of the behavioral features of the first target skeleton.

[0018] According to a skeleton behavior recognition method provided by the present invention, the step of fusing the skeleton data based on the correlation matrix and the second skeleton behavior feature includes:

[0019] The second skeleton behavior features are subjected to dimensionality reduction processing to obtain the second target skeleton behavior features;

[0020] Based on the aforementioned correlation matrix, the behavioral features of the second target skeleton are reweighted;

[0021] The reweighted features are then subjected to dimensionality-upgrading processing.

[0022] Based on the features obtained after the dimensionality upgrade, the skeleton data is fused.

[0023] According to a skeletal behavior recognition method provided by the present invention, determining the skeletal data of the animal to be identified includes:

[0024] Acquire raw video data representing the daily behavior of the animal to be identified;

[0025] The original video data is preprocessed and labeled with behaviors to obtain the behavior-labeled video data;

[0026] Based on a preset animal pose estimation model, skeleton information is extracted from the video data after the behavior is labeled to obtain the skeleton data of the animal to be identified; wherein, the preset animal pose estimation model is obtained by training a human pose estimation model.

[0027] According to a skeleton behavior recognition method provided by the present invention, the training process of the preset skeleton behavior recognition model includes:

[0028] Based on a pre-set animal pose estimation model, frame-by-frame key point information is labeled on the sample animal videos to obtain sample skeleton behavior video data.

[0029] Based on the sample skeleton behavior video data, the initial skeleton behavior recognition model is trained to obtain the preset skeleton behavior recognition model; wherein, the initial skeleton behavior recognition model is constructed based on the initial ST-GCN sub-model and the initial global spatiotemporal coding module.

[0030] According to the skeleton behavior recognition method provided by the present invention, the training process of the preset animal pose estimation model includes:

[0031] Preprocess the raw video data representing the daily behavior of the animals to be identified;

[0032] The preprocessed video data is frame-by-frame labeled with preset key points, and the human pose estimation model is trained using the video data with preset key point labels to obtain a preset animal pose estimation model.

[0033] The present invention also provides a skeleton behavior recognition device, comprising:

[0034] The determining unit is used to determine the skeletal data of the animal to be identified;

[0035] The recognition unit is used to extract global features from the skeleton data based on a preset skeleton behavior recognition model, fuse the skeleton data based on the correlation of the first skeleton behavior feature and the second skeleton behavior feature obtained from the global feature extraction, and perform skeleton behavior recognition based on the fused features.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the skeleton behavior recognition method as described above.

[0037] The present invention provides a skeletal behavior recognition method, apparatus, and electronic device. The skeletal behavior recognition method first determines the skeletal data of the animal to be identified. Then, based on a preset skeletal behavior recognition model, global feature extraction is performed on the skeletal data. The skeletal data is fused based on the correlation of a first skeletal behavior feature and a second skeletal behavior feature obtained from the global feature extraction. Skeletal behavior recognition is then performed based on the fused features. Since both the first and second skeletal behavior features are obtained through global feature extraction, they can both retain the true temporal and spatial behaviors of the animal's skeleton covered by the skeletal data. Furthermore, by fusing the skeletal data through the correlation of the first and second skeletal behavior features, the method can cover the movements of each limb involved in skeletal behavior, thereby effectively improving the accuracy of skeletal behavior recognition. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the skeleton behavior recognition method provided by the present invention;

[0040] Figure 2 This is a schematic diagram of the overall process of the skeleton behavior recognition method provided by the present invention;

[0041] Figure 3 This is a schematic diagram of the skeleton behavior recognition device provided by the present invention;

[0042] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] Skeleton behavior recognition, as an important research branch in the field of computer vision, includes not only human skeleton behavior recognition but also animal skeleton behavior recognition. Since recognizing animal behavior (such as recognizing the skeleton behavior of monkeys) is of great significance and application prospects for the evaluation of new drugs, how to accurately recognize animal skeleton behavior has become a popular research direction.

[0045] The following explanation uses the skeletal behavior recognition of monkeys as an example.

[0046] Since human skeleton behavior recognition is similar to monkey skeleton behavior recognition, and human skeleton behavior recognition usually uses a pose estimation model to extract the skeleton information of people in the video, and then uses a skeleton behavior recognition model (such as the ST-GCN model) to perform intra-frame spatial convolution and inter-frame temporal convolution to extract spatiotemporal features, and finally classifies the extracted features.

[0047] Based on the above process, this invention further explores the feasibility and advantages of monkey skeleton behavior recognition on the basis of the ST-GCN model, and proposes a new global spatiotemporal coding method suitable for monkey skeleton behavior recognition, so as to achieve the goal of efficient and accurate recognition of animal skeleton behavior.

[0048] The following is combined Figures 1-3 This invention describes a skeleton behavior recognition method, apparatus, and electronic device, wherein the execution subject of the skeleton behavior recognition method can be a terminal device, which can be a personal computer (PC), portable device, laptop computer, smartphone, tablet computer, portable wearable device, or other electronic device. This invention does not limit the specific form of the terminal device.

[0049] It should be noted that the execution subject of the following method embodiments can be part or all of the terminal device. The following method embodiments are described using a terminal device as the execution subject.

[0050] Figure 1 This is a flowchart illustrating the skeleton behavior recognition method provided by the present invention, as shown below. Figure 1 As shown, the skeleton behavior recognition method includes the following steps:

[0051] Step 110: Determine the skeletal data of the animal to be identified.

[0052] The animal to be identified can be an animal in a specific scenario, such as a caged monkey, and the cage can be an iron cage; the skeletal data of the animal to be identified can represent data on other behaviors of the caged animal, such as creeping, squatting, walking, jumping up, jumping down, climbing up, climbing down, hanging, standing, and clinging.

[0053] Specifically, when the animal to be identified is a monkey and it is placed in a cage, the monkey can be filmed by placing a camera on the cage. The filming time can be several days, a week, several consecutive days, or several consecutive months; no specific limitation is made here. Then, based on the processing of the filmed video, the skeletal data of the animal to be identified is determined. This skeletal data is a three-dimensional array of C*T'*V, where C is the number of channels in the convolutional layer of the preset skeletal behavior recognition model; T' is the number of video frames in the filmed video; and V is the number of preset key points in each video frame. When the number of preset key points is 17, the 17 preset key points can be the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0054] Step 120: Based on the preset skeleton behavior recognition model, perform global feature extraction on the skeleton data, fuse the skeleton data based on the correlation of the first skeleton behavior feature and the second skeleton behavior feature obtained from the global feature extraction, and perform skeleton behavior recognition based on the fused features.

[0055] Specifically, a preset skeleton behavior recognition model is used to identify the skeleton data of the animal to be identified. This involves inputting the skeleton data into the preset skeleton behavior recognition model, which first performs global feature extraction on the skeleton data, and then fuses the skeleton data using the correlation of the first skeleton behavior feature extracted globally and the second skeleton behavior feature. This allows for skeleton behavior recognition based on the fused features. The global feature extraction process involves extracting global features from each video frame image of the animal in the cage based on preset keypoints, thus obtaining the first and second skeleton behavior features. Both the first and second skeleton behavior features have the same dimension and are three-dimensional, for example (C*T'*V), where C is the number of channels in the convolutional layer of the preset skeleton behavior recognition model, T' is the number of video frames in the captured video, and V is the number of preset keypoints in each video frame.

[0056] The skeletal behavior recognition method provided by this invention first determines the skeletal data of the animal to be identified. Then, based on a preset skeletal behavior recognition model, global feature extraction is performed on the skeletal data. The skeletal data is fused based on the correlation of a first skeletal behavior feature and a second skeletal behavior feature obtained from the global feature extraction. Skeletal behavior recognition is then performed based on the fused features. Since both the first and second skeletal behavior features are obtained through global feature extraction, they can both retain the true temporal and spatial behaviors of the animal's skeleton covered by the skeletal data. Furthermore, by fusing the skeletal data through the correlation of the first and second skeletal behavior features, the method can cover the movements of each limb involved in skeletal behavior, thereby effectively improving the accuracy of skeletal behavior recognition.

[0057] Optionally, the implementation process of step 110 may include:

[0058] First, raw video data representing the daily behavior of the animal to be identified is acquired; then, the raw video data is preprocessed and labeled with behavior to obtain the labeled video data; finally, based on a preset animal posture estimation model, key point information is extracted frame by frame from the labeled video data to obtain the skeleton data of the animal to be identified; wherein, the preset animal posture estimation model is obtained by training a human posture estimation model.

[0059] Specifically, when the animal to be identified is a monkey, multiple monkeys of different species, sexes, and ages can be placed in a metal cage. Cameras mounted on the cages are used to film the daily behavior of the monkeys of different sexes, species, and ages. During filming, the monkeys in the cages can be rotated periodically while maintaining a constant total number. The age range of the monkeys can include young, middle-aged, and old; the sex can include males and females; and the species can include rhesus macaques and cynomolgus monkeys. The total filming time can be two months. When the filming time is two months, the monkeys in the cages can be rotated approximately every week. For example, if the cycle is four days, a young rhesus macaque can be replaced with a middle-aged or young cynomolgus monkey. After filming in this way for the required total time, raw video data representing the daily behavior of the animal to be identified is obtained and sent to the terminal device. Among them, the daily behaviors of the animals to be identified may include creeping, squatting, walking, jumping up, jumping down, climbing up, climbing down, hanging, standing, clinging, etc.

[0060] Furthermore, the terminal device first preprocesses the received raw video data representing the daily behavior of the animal to be identified. This involves instructing the user to manually delete low-quality video segments with poor lighting (such as those shot on cloudy days or at night). When the raw video data is in .TS format, considering that .TS format video data has a small file size but suffers from significant information loss, it is necessary to transcode the preprocessed raw video data to .MP4 format to ensure the accuracy of skeleton behavior recognition. Then, by instructing the user to manually annotate, the transcoded raw video data is annotated with the daily behaviors of the animal to be identified, thus obtaining the behavior-annotated video data.

[0061] Finally, based on the preset animal posture estimation model, skeleton information is extracted from the behavior-annotated video data. The skeleton information includes the coordinates and corresponding confidence scores of each preset key point on the corresponding video frame image in the behavior-annotated video data, thereby obtaining the skeleton data of the animal to be identified. The preset key points can be nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0062] The skeleton behavior recognition method provided by this invention first preprocesses and annotates the original video data representing the daily behavior of the animal to be identified, and then extracts skeleton information from the behavior-annotated video data based on a preset animal posture estimation model. This ensures the richness and completeness of the obtained skeleton data of the animal to be identified, thereby providing a reliable guarantee for the accurate identification of skeleton behavior in the future.

[0063] Optionally, the global feature extraction of the skeleton data based on the preset skeleton behavior recognition model in step 120 may include the following process:

[0064] First, based on the preset ST-GCN sub-model in the preset skeleton behavior recognition model, the skeleton data is transformed by channel transformation; then, global feature extraction is performed on the skeleton data after channel transformation according to the preset channel dimensions.

[0065] Specifically, the preset skeleton behavior recognition model is the model obtained after training the initial skeleton behavior recognition model. The initial skeleton behavior recognition model includes the initial ST-GCN sub-model and the initial GlobalSpatial Temporal Encoder Module (GSTEM). When the initial ST-GCN sub-model is trained, it becomes the preset ST-GCN sub-model, and when the initial GSTEM is trained, it becomes the preset GSTEM.

[0066] Based on this, when skeleton data is input into a preset skeleton behavior recognition model, the preset GSTEM can be instructed to use a preset ST-GCN sub-model to perform channel transformation on the skeleton data. That is, when skeleton data with dimension (C*T'*V) is input into the preset skeleton behavior recognition model, it can be transformed through the convolutional layers in different preset ST-GCN sub-models. For example, the number of channels in the skeleton data can be tripled to obtain the channel-transformed skeleton data. Then, global feature extraction is performed on the channel-transformed skeleton data according to the preset channel dimension. That is, the channel-transformed skeleton data is divided into 3 equal parts. The dimensions of the 3 skeleton behavior features extracted in this way are the same as the dimensions of the skeleton data.

[0067] The skeleton behavior recognition method provided by this invention uses a preset skeleton behavior recognition model to first perform channel transformation on the skeleton data, and then extract global features from the transformed skeleton data. This method, combined with a preset ST-GCN sub-model and a global spatiotemporal coding module to extract global features, ensures that the extracted features retain global characteristics, thereby ensuring the accuracy of subsequent skeleton behavior recognition.

[0068] Optionally, when the number of the first skeleton behavior features is greater than the number of the second skeleton behavior features, the process of fusing the skeleton data based on the correlation of the first skeleton behavior features obtained from the global feature extraction and the second skeleton behavior features in step 120 may include:

[0069] First, a correlation matrix is ​​determined based on the correlation of each first skeleton behavior feature obtained from the global feature extraction; then, the skeleton data is fused based on the correlation matrix and the second skeleton behavior feature.

[0070] Specifically, skeleton data of dimension (C*T'*V) enters a preset skeleton behavior recognition model for channel variation and global feature extraction. This process yields more than one first skeleton behavior feature and one second skeleton behavior feature that represent global characteristics. Each first skeleton behavior feature and each second skeleton behavior feature represents a global feature and has a dimension of (C*T'*V). Then, by analyzing the correlation between each first skeleton behavior feature, a correlation matrix representing the correlation between each first skeleton behavior feature is determined. Furthermore, based on this correlation matrix and the second skeleton behavior feature, the skeleton data is fused, for example, by using the correlation matrix to weight the second skeleton behavior feature.

[0071] It should be noted that when skeleton data of dimension (C*T'*V) enters the preset skeleton behavior recognition model, the preset GSTEM uses the corresponding preset ST-GCN sub-model to perform channel changes and global feature extraction. When performing global feature extraction, more than one second skeleton behavior feature and one first skeleton behavior feature representing global characteristics can also be extracted. At this time, the skeleton data can be fused using the correlation matrix representing the degree of correlation between the various second skeleton behavior features and the first skeleton behavior feature.

[0072] The skeleton behavior recognition method provided by this invention improves the reliability and accuracy of skeleton behavior recognition by fusing the skeleton data through a correlation matrix that characterizes the correlation between the various first skeleton behavior features extracted from the skeleton data.

[0073] Optionally, determining the correlation matrix based on the correlation of the first skeleton behavior features obtained from the global feature extraction includes:

[0074] First, each first skeleton behavior feature extracted based on the global features is subjected to dimensionality reduction processing to obtain the corresponding first target skeleton behavior feature; then, the correlation matrix of the first target skeleton behavior feature is calculated.

[0075] Specifically, skeleton data of dimension (C*T'*V) enters the preset skeleton behavior recognition model, where it undergoes channel transformation and global feature extraction. This extracts two first skeleton behavior features and one second skeleton behavior feature, each representing global characteristics. The dimensions of both the first and second skeleton behavior features are (C*T'*V). Further, the two first skeleton behavior features are dimensionality-reduced, transforming their dimensions to (T'V, C), resulting in two first target skeleton behavior features, each with dimensions (T'V, C). Then, the correlation matrix M of the two first target skeleton behavior features is calculated. T’V*T’V Its calculation formula includes:

[0076]

[0077] Among them, the two first target skeleton behavioral features are x1 and x2, respectively, where the superscript T represents the transpose operation, x i Let be the i-th data in each row of the (1*T'V) dimensional vector of the matrix obtained by multiplying x1 and x2 (T'V*T'V), and n be the total number of data in each row of the (T'V*T'V) dimensional vector of the matrix (T'V*T'V).

[0078] It should be noted that the dimensionality reduction processing of the two first skeleton behavior features and the calculation of the correlation matrix of the two first target skeleton behavior features can also be completed by the preset GSTEM in the preset skeleton behavior recognition model used by the terminal device.

[0079] The skeleton behavior recognition method provided by this invention improves the stability and reliability of global feature extraction by calculating the correlation matrix of the first target skeleton behavior features obtained after dimensionality reduction processing of each first skeleton behavior feature extracted from the skeleton data, thereby laying the foundation for the accuracy of subsequent fusion processing.

[0080] Optionally, fusing the skeleton data based on the correlation matrix and the second skeleton behavior features includes:

[0081] First, the second skeleton behavior features are dimensionality reduced to obtain the second target skeleton behavior features; then, based on the correlation matrix, the second target skeleton behavior features are reweighted; next, the reweighted features are dimensionality increased; finally, the skeleton data are fused based on the dimensionality-increased features.

[0082] Specifically, for the two first skeleton behavior features and one second skeleton behavior feature representing global characteristics extracted, while reducing the dimensions of both first skeleton behavior features, the second skeleton behavior feature can also be reduced in dimension. That is, by transforming the dimension of the second skeleton behavior feature, which is (C*T'*V), into (T'V, C), the second target skeleton behavior feature is obtained. Then, the second target skeleton behavior feature is reweighted using a correlation matrix. The reweighting formula is as follows:

[0083] x′3=M T’V*T’V *x3; x′3 is the reweighted feature, and x3 is the skeleton behavior feature of the second target.

[0084] Since the dimension of the reweighted feature x′3 is still (T'V, C), to ensure the normal execution of subsequent fusion, the dimension of the reweighted feature x′3 can be increased, that is, its dimension can be transformed to (C*T'*V), thus obtaining the dimension-increased feature x″3. Finally, based on the dimension-increased feature, the skeleton data is fused, and the fusion formula is as follows:

[0085] x′=x+wx″3; x′ is the fusion result, x is the skeleton data with dimension (C*T'*V); w is the preset weight coefficient, the value of w is between 0 and 1, and the preferred value of w is 0.2.

[0086] It should be noted that the dimensionality reduction of the second skeleton behavior features, the reweighting of the second target skeleton behavior features, the dimensionality increase of the reweighted features, and the fusion of the skeleton data based on the dimensionality increase features can all be accomplished by the preset GSTEM in the preset skeleton behavior recognition model used by the terminal device.

[0087] The skeleton behavior recognition method provided by this invention targets the first skeleton behavior features and the second skeleton behavior features extracted from the skeleton data. By using the correlation matrix of each first skeleton behavior feature, the second skeleton behavior features after dimensionality reduction are first reweighted. Then, the reweighted features are transformed to the same dimension as the skeleton data and fused with the skeleton data. This ensures the accuracy and reliability of the fusion, thereby improving the accuracy of skeleton behavior recognition.

[0088] Optionally, the training process of the preset skeleton behavior recognition model includes:

[0089] First, based on a preset animal pose estimation model, preset key point information is annotated frame by frame in the sample animal video to obtain sample skeleton behavior video data; then, based on the sample skeleton behavior video data, an initial skeleton behavior recognition model is trained to obtain a preset skeleton behavior recognition model; wherein, the initial skeleton behavior recognition model is constructed based on the initial ST-GCN sub-model and the initial global spatiotemporal coding module.

[0090] Specifically, the initial skeleton behavior recognition model in this invention is a model formed by connecting the initial ST-GCN sub-model and the initial GSTEM. The initial ST-GCN sub-model and the initial GSTEM can be in a one-to-one connection relationship, a many-to-one connection relationship, or a one-to-many connection relationship. Here, there is no specific limitation on the number and connection relationship of the initial ST-GCN sub-model and the initial GSTEM.

[0091] For training the initial skeleton behavior recognition model, sample animal videos are first acquired. These sample animal videos can be the pre-processed original video data mentioned above, i.e., the pre-processed video data representing the daily behavior of the animals to be identified; or they can be video data obtained by filming the daily behavior of monkeys of different sexes, breeds, and ages in a cage for two months, with the total number of monkeys in the cage kept constant during the filming period. The pre-processed video data is then used as the sample animal videos. The filming process here can be performed synchronously with the filming process in the aforementioned embodiment, and both the filming process and the pre-processing process can refer to the aforementioned embodiment, which will not be repeated here.

[0092] Next, frame-by-frame keypoint extraction is performed on the sample animal videos. This involves manually annotating 17 preset keypoints in each frame of the sample animal video to obtain sample skeleton behavior video data. The sample skeleton behavior video data consists of the skeleton behavior sequence information of multiple sample video frames. The skeleton behavior sequence information of each sample video frame can be represented as a tensor with dimension Y*N*U, where U=3 represents the feature dimension of each preset keypoint, Y is the total number of sample video frames contained in the sample skeleton behavior video data, and N is the total number of preset keypoints to be extracted from each frame of the sample animal video.

[0093] The initial skeleton behavior recognition model is further trained using sample skeleton behavior video data. The training epochs of the initial skeleton behavior recognition model are set to M, for example, M can be 100. After each epoch of training, the accuracy of the corresponding trained model is calculated and saved. When the initial skeleton behavior recognition model is trained to M times using sample skeleton behavior video data, the model with the highest accuracy among the M trained models is selected as the preset skeleton behavior recognition model. That is, the initial skeleton behavior recognition model can become the preset skeleton behavior recognition model after it is trained. At this time, each initial ST-GCN sub-model is also trained and becomes a preset ST-GCN sub-model, and each initial GSTEM is also trained and becomes a preset GSTEM. The preset GSTEM is used to extract global skeleton behavior features of monkeys through global spatiotemporal coding method. The preset GSTEM can be used as a plug-and-play module, which has the advantage of being lightweight and does not affect the use of the preset ST-GCN sub-model. The preset GSTEM and the preset ST-GCN sub-model are combined to form the preset skeleton behavior recognition model.

[0094] The skeleton behavior recognition method provided in this invention trains an initial skeleton behavior recognition model constructed from an initial ST-GCN sub-model and an initial GSTEM using sample skeleton data obtained by annotating key points frame by frame in sample animal videos. This results in a preset skeleton behavior recognition model, which, combined with deep learning methods, achieves the goal of accurately predicting the posture of the animal to be identified in a specific scene, thereby improving the accuracy of animal skeleton behavior recognition.

[0095] Optionally, the training process of the preset animal pose estimation model includes:

[0096] First, the raw video data representing the daily behavior of the animal to be identified is preprocessed; then, the preprocessed video data is annotated frame by frame with preset key points, and the human pose estimation model is trained using the video data with preset key points to obtain the preset animal pose estimation model.

[0097] Specifically, for training the human pose estimation model, firstly, pre-defined key points are extracted frame by frame from the pre-processed raw video data. That is, by instructing the user to manually annotate, 17 pre-defined key points are annotated for each frame of the pre-processed raw video data. The human pose estimation model is then trained using the video data with the pre-defined key points annotated. The training epoch of the human pose estimation model is set to M', for example, M' can also be 100. After each epoch of training, the accuracy of the corresponding trained model is calculated and saved. When the human pose estimation model has been trained to M' using the video data with the pre-defined key points annotated, the model with the highest accuracy among the M' trained models is selected as the pre-defined animal pose estimation model.

[0098] It should be noted that the human pose estimation model mentioned in the embodiments of the present invention can be an existing human pose estimation model, and no specific limitation is made here.

[0099] The skeleton behavior recognition method provided by this invention trains a human pose estimation model using video data annotated with preset key points to obtain a preset animal pose estimation model for animal pose estimation. This model is then combined with the human pose estimation model and deep learning methods to generate a preset animal pose estimation model for predicting the pose of an animal to be identified in a specific scenario, thereby improving the reliability and flexibility of animal pose recognition.

[0100] Based on the foregoing embodiments, Figure 2 This is a flowchart of the skeleton behavior recognition method provided by the present invention, as follows: Figure 2 As shown, the skeleton behavior recognition method includes:

[0101] Step 210: Obtain raw video data representing the daily behavior of the animal to be identified, and preprocess and label the raw video data to obtain video data with labeled behavior.

[0102] Step 220: Based on the preset animal posture estimation model, extract skeleton information from the video data after behavior annotation to obtain the skeleton data of the animal to be identified.

[0103] Step 230: When inputting skeleton data into the preset skeleton behavior recognition model, the preset GSTEM uses the convolutional layer of the preset ST-GCN sub-model to triple the number of channels in the skeleton data, thus obtaining the skeleton data after channel transformation.

[0104] Step 240: Divide the skeleton data after channel transformation into three equal parts according to the preset channel dimension, and use them as two first skeleton behavior features and one second skeleton behavior feature, each with a dimension of (C*T'*V). Then reduce the dimension of each feature to (T'V, C) to obtain two first target skeleton behavior features and one second target skeleton behavior feature.

[0105] Step 250: Calculate the correlation matrix of the behavioral features of the two first target skeletons.

[0106] Step 260: Use the correlation matrix to reweight the behavioral features of the second target skeleton, and then increase the dimension of the reweighted features to (C*T'*V) to obtain the features after dimension increase.

[0107] Step 270: Use the upgraded features to fuse the skeleton data to obtain the fused features, which is the fusion result of the preset ST-GCN sub-model and the preset GSTEM.

[0108] Step 280: Perform skeleton behavior recognition based on the fused features.

[0109] It should be noted that when comparing the performance of the method of this invention with that of the existing ST-GCN model for animal skeleton behavior recognition, the accuracy of the method of this invention for animal skeleton recognition is 0.735, while the accuracy of the existing ST-GCN model is 0.698. The accuracy of the method of this invention is 3.7% higher than that of the existing ST-GCN method. Therefore, this invention verifies the feasibility of using skeleton behavior recognition methods for monkey skeleton behavior recognition, and the proposed animal skeleton behavior recognition method has a higher skeleton behavior recognition accuracy compared to the existing ST-GCN method.

[0110] The skeletal behavior recognition method provided in this invention first determines the skeletal data of the animal to be identified. Then, based on a preset skeletal behavior recognition model, global feature extraction is performed on the skeletal data. The skeletal data is fused based on the correlation of a first skeletal behavior feature and a second skeletal behavior feature obtained from the global feature extraction. Skeletal behavior recognition is then performed based on the fused features. Since both the first and second skeletal behavior features are obtained through global feature extraction, they can both retain the true temporal and spatial behaviors of the animal's skeleton covered by the skeletal data. Furthermore, by fusing the skeletal data through the correlation of the first and second skeletal behavior features, the method can cover the movements of each limb involved in skeletal behavior, thereby effectively improving the accuracy of skeletal behavior recognition.

[0111] The skeleton behavior recognition device provided by the present invention is described below. The skeleton behavior recognition device described below can be referred to in correspondence with the skeleton behavior recognition method described above.

[0112] Reference Figure 3 The skeleton behavior recognition device provided in this application embodiment, in Figure 3 The skeleton behavior recognition device 300 includes:

[0113] The determining unit 310 is used to determine the skeletal data of the animal to be identified.

[0114] The recognition unit 320 is used to extract global features from the skeleton data based on a preset skeleton behavior recognition model, fuse the skeleton data based on the correlation of the first skeleton behavior feature and the second skeleton behavior feature obtained from the global feature extraction, and perform skeleton behavior recognition based on the fused features.

[0115] Optionally, the determining unit 310 can be used to acquire raw video data representing the daily behavior of the animal to be identified; preprocess and label the raw video data to obtain the behavior-labeled video data; and extract skeleton information from the behavior-labeled video data based on a preset animal posture estimation model to obtain the skeleton data of the animal to be identified; wherein the preset animal posture estimation model is obtained by training a human posture estimation model.

[0116] Optionally, the recognition unit 320 can be used to perform channel transformation on the skeleton data based on the preset ST-GCN sub-model in the preset skeleton behavior recognition model; and to perform global feature extraction on the channel-transformed skeleton data according to the preset channel dimensions.

[0117] Optionally, the recognition unit 320 can also be used to determine a correlation matrix based on the correlation between the various first skeleton behavior features extracted from the global features; and to fuse the skeleton data based on the correlation matrix and the second skeleton behavior features.

[0118] Optionally, the recognition unit 320 can also be used to perform dimensionality reduction processing on each first skeleton behavior feature extracted based on the global features to obtain the corresponding first target skeleton behavior feature; and calculate the correlation matrix of the first target skeleton behavior feature.

[0119] Optionally, the recognition unit 320 can also be used to perform dimensionality reduction processing on the second skeleton behavior features to obtain the second target skeleton behavior features; reweight the second target skeleton behavior features based on the correlation matrix; perform dimensionality increase processing on the reweighted features; and fuse the skeleton data based on the dimensionality increase processing features.

[0120] Optionally, the device may further include a training unit for annotating sample animal videos frame by frame with preset key point information based on a preset animal pose estimation model to obtain sample skeleton behavior video data; and for training an initial skeleton behavior recognition model based on the sample skeleton behavior video data to obtain a preset skeleton behavior recognition model; wherein the initial skeleton behavior recognition model is constructed based on an initial ST-GCN sub-model and an initial global spatiotemporal coding module.

[0121] Optionally, the training unit can also be used to preprocess the raw video data representing the daily behavior of the animal to be identified; to annotate the preprocessed video data frame by frame with preset key points, and to train the human pose estimation model using the video data with preset key points to obtain the preset animal pose estimation model.

[0122] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a skeleton behavior recognition method, which includes:

[0123] Determine the skeletal data of the animal to be identified;

[0124] Based on a preset skeleton behavior recognition model, global features are extracted from the skeleton data. Based on the correlation of the first skeleton behavior features and the second skeleton behavior features obtained from the global feature extraction, the skeleton data is fused, and skeleton behavior recognition is performed based on the fused features.

[0125] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the skeleton behavior recognition method provided by the above methods, the method comprising:

[0127] Determine the skeletal data of the animal to be identified;

[0128] Based on a preset skeleton behavior recognition model, global features are extracted from the skeleton data. Based on the correlation of the first skeleton behavior features and the second skeleton behavior features obtained from the global feature extraction, the skeleton data is fused, and skeleton behavior recognition is performed based on the fused features.

[0129] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the skeleton behavior recognition method provided by the above methods, the method comprising:

[0130] Determine the skeletal data of the animal to be identified;

[0131] Based on a preset skeleton behavior recognition model, global features are extracted from the skeleton data. Based on the correlation of the first skeleton behavior features and the second skeleton behavior features obtained from the global feature extraction, the skeleton data is fused, and skeleton behavior recognition is performed based on the fused features.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A skeleton behavior recognition method, characterized in that, include: Determine the skeletal data of the animal to be identified; Based on a preset skeleton behavior recognition model, global feature extraction is performed on the skeleton data. Based on the correlation of the first skeleton behavior feature and the second skeleton behavior feature obtained from the global feature extraction, the skeleton data is fused, and skeleton behavior recognition is performed based on the fused features. The step of extracting global features from the skeleton data based on a preset skeleton behavior recognition model includes: Based on the preset ST-GCN sub-model in the preset skeleton behavior recognition model, channel transformation is performed on the skeleton data; Global feature extraction is performed on the skeleton data after channel transformation according to the preset channel dimensions; When the number of the first skeleton behavior features is greater than the number of the second skeleton behavior features, the process of fusing the skeleton data based on the correlation of the first skeleton behavior features obtained from the global feature extraction and the second skeleton behavior features includes: Based on the correlation between the various first skeleton behavior features obtained from the global feature extraction, a correlation matrix is ​​determined; The skeleton data is fused based on the correlation matrix and the second skeleton behavior features; The determination of the correlation matrix based on the correlation of the first skeleton behavior features obtained from the global feature extraction includes: Each first skeleton behavior feature extracted based on the global features is subjected to dimensionality reduction processing to obtain the corresponding first target skeleton behavior feature; Calculate the correlation matrix of the behavioral features of the first target skeleton; The process of fusing the skeleton data based on the correlation matrix and the second skeleton behavior features includes: The second skeleton behavior features are subjected to dimensionality reduction processing to obtain the second target skeleton behavior features; Based on the aforementioned correlation matrix, the behavioral features of the second target skeleton are reweighted; The reweighted features are then subjected to dimensionality-upgrading processing. Based on the features obtained after the dimensionality upgrade, the skeleton data is fused.

2. The skeleton behavior recognition method according to claim 1, characterized in that, The determination of the skeletal data of the animal to be identified includes: Acquire raw video data representing the daily behavior of the animal to be identified; The original video data is preprocessed and labeled with behaviors to obtain the behavior-labeled video data; Based on a preset animal pose estimation model, skeleton information is extracted from the video data after the behavior is labeled to obtain the skeleton data of the animal to be identified; wherein, the preset animal pose estimation model is obtained by training a human pose estimation model.

3. The skeleton behavior recognition method according to claim 1 or 2, characterized in that, The training process of the preset skeleton behavior recognition model includes: Based on a pre-set animal pose estimation model, frame-by-frame key point information is labeled on the sample animal videos to obtain sample skeleton behavior video data. Based on the sample skeleton behavior video data, the initial skeleton behavior recognition model is trained to obtain the preset skeleton behavior recognition model; wherein, the initial skeleton behavior recognition model is constructed based on the initial ST-GCN sub-model and the initial global spatiotemporal coding module.

4. The skeleton behavior recognition method according to claim 3, characterized in that, The training process of the preset animal pose estimation model includes: Preprocess the raw video data representing the daily behavior of the animals to be identified; The preprocessed video data is frame-by-frame labeled with preset key points, and the human pose estimation model is trained using the video data with preset key point labels to obtain a preset animal pose estimation model.

5. A skeleton behavior recognition device, characterized in that, include: The determining unit is used to determine the skeletal data of the animal to be identified; The recognition unit is configured to perform global feature extraction on the skeleton data based on a preset skeleton behavior recognition model, fuse the skeleton data based on the correlation of the first skeleton behavior feature and the second skeleton behavior feature obtained from the global feature extraction, and perform skeleton behavior recognition based on the fused features; the global feature extraction on the skeleton data based on the preset skeleton behavior recognition model includes: Based on the preset ST-GCN sub-model in the preset skeleton behavior recognition model, channel transformation is performed on the skeleton data; Global feature extraction is performed on the skeleton data after channel transformation according to the preset channel dimensions; When the number of the first skeleton behavior features is greater than the number of the second skeleton behavior features, the process of fusing the skeleton data based on the correlation of the first skeleton behavior features obtained from the global feature extraction and the second skeleton behavior features includes: Based on the correlation between the various first skeleton behavior features obtained from the global feature extraction, a correlation matrix is ​​determined; The skeleton data is fused based on the correlation matrix and the second skeleton behavior features; The determination of the correlation matrix based on the correlation of the first skeleton behavior features obtained from the global feature extraction includes: Each first skeleton behavior feature extracted based on the global features is subjected to dimensionality reduction processing to obtain the corresponding first target skeleton behavior feature; Calculate the correlation matrix of the behavioral features of the first target skeleton; The process of fusing the skeleton data based on the correlation matrix and the second skeleton behavior features includes: The second skeleton behavior features are subjected to dimensionality reduction processing to obtain the second target skeleton behavior features; Based on the aforementioned correlation matrix, the behavioral features of the second target skeleton are reweighted; The reweighted features are then subjected to dimensionality-upgrading processing. Based on the features obtained after the dimensionality upgrade, the skeleton data is fused.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the skeleton behavior recognition method as described in any one of claims 1 to 4.

Images