Anti-shielding animal attitude estimation method based on online distillation

Abstract

The embodiment of the invention discloses an anti-occlusion animal attitude estimation method based on online distillation, and the method comprises the steps: obtaining an animal image, inputting the animal image into an anti-occlusion animal attitude estimation model, obtaining the image feature information of the animal image in a student model through a backbone network HRNet, obtaining the image local space information through RFAConv space attention, and carrying out the calculation of the image local space information. Spatial dimensions are extended, a first set of feature vectors and a second set of feature vectors are obtained through two independent full-connection layers, and the first set of feature vectors learn feature vectors reconstructed in a teacher model to obtain learned feature vectors; the relevance between the independent key points of the animal image is obtained according to the learned feature vectors and the first set of feature vectors, the second set of feature vectors and the learned feature vectors are fused, key point information is obtained by using a second decoder formed by an MLP-Mixer architecture, better animal attitude prediction can be achieved in the occlusion environment, and the prediction accuracy is improved. And animal researchers can be better assisted to deeply understand and analyze animal behavior characteristics.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to an anti-occlusion animal posture estimation method based on online distillation. Background Art

[0002] With the rapid development of computer vision technology, tasks based on human posture estimation have gradually been commercialized. Some new research directions have emerged in this field, such as hand gesture estimation and animal posture estimation. In recent years, the field of animal research has become an emerging research hotspot. By estimating the posture of animals, we can deeply understand and analyze the behavioral characteristics of animals based on the relative positional relationship between the key points of animals. At present, obtaining the posture of animals requires the use of expensive motion capture equipment to capture the posture. Using animal posture estimation models to predict the posture of animals has the advantage of lower cost. However, in actual complex scenes, the occlusion of animal posture is very common, and there are disadvantages in animal posture estimation in occluded scenes. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The disclosed embodiment provides an anti-occlusion animal posture estimation method based on online distillation, which can achieve better animal posture prediction in an occluded environment and obtain the correlation between each independent key point, which can better assist animal researchers to deeply understand and analyze the behavioral characteristics of animals.

[0005] An embodiment of the first aspect of the present disclosure is a method for estimating animal poses with anti-occlusion based on online distillation, comprising:

[0006] Obtain animal images;

[0007] Inputting the animal image into an anti-occlusion animal posture estimation model, wherein the anti-occlusion animal posture estimation model includes a teacher model and a student model;

[0008] In the student model, image feature information of the animal image is obtained through the backbone network HRNet;

[0009] According to the image feature information, the local spatial information of the image is obtained using RFAConv spatial attention;

[0010] According to the local spatial information of the image, the spatial dimension is extended and a set of feature vectors is obtained through two independent fully connected layers, and the set of feature vectors is split into two sets of feature vectors, namely a first set of feature vectors and a second set of feature vectors;

[0011] Learning the first set of feature vectors with the feature vectors reconstructed in the teacher model to obtain learned feature vectors;

[0012] Obtaining the correlation between each independent key point of the animal image according to the learned feature vector and the first group of feature vectors;

[0013] In the student model, the second set of feature vectors and the learned feature vectors are fused, and after fusion, key point information is obtained by decoding using a second decoder;

[0014] Wherein, in the teacher model, the key point true value in the animal data set is obtained, the key point true value is converted into a feature vector, the feature vector is input into a first encoder to generate a new feature vector; the new feature vector is quantized by nearest neighbor search using a coding space codebook; the quantized feature vector is input into a first decoder to obtain a reconstructed feature vector;

[0015] Among them, the first encoder, the first decoder, and the second decoder are respectively constructed using the MLP-Mixer architecture.

[0016] According to certain embodiments of the first aspect of the present disclosure, the local spatial information of the image is obtained using RFAConv spatial attention according to the image feature information, which is expressed as:

[0017] F = Softmax(g 1×1 (AvgPool(X)))×ReLU(Norm(g k×k (X)))

[0018] In the formula, g 1×1 is a grouped convolution of size 1×1, k is the size of the convolution kernel, Norm is normalization, and X is the input feature map.

[0019] According to certain embodiments of the first aspect of the present disclosure, the correlation between the independent key points of the animal image obtained according to the learned feature vector and the first group of feature vectors is expressed as:

[0020]

[0021] In the formula, p i is the softened probability distribution of the teacher model p = soft max (z t / T), z t is the learned feature vector, q i is the softened probability distribution of the student model q = soft max (z s / T), z s is the first set of eigenvectors, and T is the temperature.

[0022] According to certain embodiments of the first aspect of the present disclosure, the loss function during the training of the student model is:

[0023]

[0024] Where P θ,φ (x|I) is the probability of the keypoint annotated on the latent distribution, and θ, φ are the parameters of the regression model and the normalized flow model, respectively.

[0025] According to some embodiments of the first aspect of the present disclosure, the coding space codebook is:

[0026] C=(c 1 , …, c v ) T

[0027] Where v is the number of dimensions in the codebook;

[0028] The coding space codebook is used to quantize the new feature vector by nearest neighbor search, which is expressed as:

[0029]

[0030] Where, t i is the new feature vector obtained by the first encoder, c j is the vector in the codebook, q(t i ) is the index of the corresponding codebook, and G is the true value of the key point.

[0031] According to certain embodiments of the first aspect of the present disclosure, the MLP-Mixer architecture is composed of a Token-mixing MLP and a Channel-mixing MLP fully connected;

[0032] The Token-mixing MLP is used to exchange information between feature vectors at different spatial locations, and the Channel-mixing MLP is used to exchange information between feature vectors at the same spatial location but at different channels. The Token-mixing MLP and the Channel-mixing MLP are executed alternately to mix features in both spatial and channel dimensions, which can be expressed as:

[0033] U=X+W 2 σ(W 1 LayerNorm(X)

[0034] Y=U+W 4 σ(W 3LayerNorm(U)

[0035] Where σ is the GELU activation function, W 1 and W 2 is the fully connected layer of Token-mixing MLP, W 3 and W 4 It is the fully connected layer of Channel-mixing MLP.

[0036] According to certain embodiments of the first aspect of the present disclosure, the loss function during training of the teacher model is:

[0037]

[0038] In the formula, The coordinate values ​​of the reconstruction are supervised by the smooth L1 loss function. is the reconstructed feature vector obtained after decoding by the first decoder, is the submission loss, sg is the stop gradient calculation, β is the hyperparameter value, and G is the true value of the key point.

[0039] An embodiment of the second aspect of the present disclosure is an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the anti-occlusion animal posture estimation method based on online distillation as described above.

[0040] An embodiment of the third aspect of the present disclosure is a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the anti-occlusion animal posture estimation method based on online distillation as described above is implemented.

[0041] The disclosed embodiments include at least the following beneficial effects: obtaining an animal image; inputting the animal image into an anti-occlusion animal posture estimation model, the anti-occlusion animal posture estimation model including a teacher model and a student model; in the student model, obtaining the image feature information of the animal image through the backbone network HRNet; according to the image feature information, using RFAConv spatial attention to obtain the local spatial information of the image; according to the local spatial information of the image, extending the spatial dimension and obtaining a set of feature vectors through two independent fully connected layers, splitting a set of feature vectors into two sets of feature vectors, namely a first set of feature vectors and a second set of feature vectors; learning the feature vectors reconstructed in the teacher model with the first set of feature vectors to obtain learned feature vectors; obtaining each independent key point of the animal image according to the learned feature vectors and the first set of feature vectors The correlation between them; in the student model, the second set of feature vectors and the learned feature vectors are fused, and the second decoder is used to decode the fused feature vectors to obtain the key point information; in the teacher model, the true value of the key points in the animal data set is obtained, the true value of the key points is converted into a feature vector, and the feature vector is input into the first encoder to generate a new feature vector; the encoding space codebook is used to quantize the new feature vector through nearest neighbor search; the quantized feature vector is input into the first decoder to obtain a reconstructed feature vector; the first encoder, the first decoder, and the second decoder are respectively composed of MLP-Mixer architectures, which can achieve better animal posture prediction in occluded environments, and obtain the correlation between each independent key point, which can better assist animal researchers in deeply understanding and analyzing the behavioral characteristics of animals.

[0042] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0044] Figure 1 An optional flow chart of an anti-occlusion animal posture estimation method based on online distillation provided in an embodiment of the present disclosure;

[0045] Figure 2 An optional flow chart for obtaining a reconstructed feature vector according to a teacher model provided in an embodiment of the present disclosure;

[0046] Figure 3 A comparison chart of experimental results of an anti-occlusion animal pose estimation method based on online distillation provided in an embodiment of the present disclosure and other methods on the MMPose official website based on the AP-10K validation set;

[0047] Figure 4 A schematic diagram of the overall network framework of the anti-occlusion animal posture estimation model provided in an embodiment of the present disclosure;

[0048] Figure 5 A schematic diagram of the network structure of the backbone network HRNet provided in an embodiment of the present disclosure;

[0049] Figure 6 A schematic diagram of the RFAConv spatial attention structure provided in an embodiment of the present disclosure;

[0050] Figure 7 A schematic diagram of the overall structure of the teacher model provided in the embodiment of the present disclosure;

[0051] Figure 8 A schematic diagram of the MLP-Mixer architecture provided in an embodiment of the present disclosure;

[0052] Fig. 9 Schematic diagram of the RLELoss loss function provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0054] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0056] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0057] Online distillation refers to a single-stage end-to-end training scheme with efficient parallel computing. Online distillation is developed based on the concept of knowledge distillation. Unlike traditional knowledge distillation, the teacher model and the student model participate in training and parameter update at the same time during the online distillation process.

[0058] Teacher model: The teacher model plays the role of a guide, guiding the training of the student model through its feature representation and probability distribution to achieve the goal of improving performance.

[0059] Student model: The student model plays the role of a learner. The student model can obtain richer information from the teacher model, prompting it to achieve better performance.

[0060] Receptive-Field Attention Convolution (RFAConv) is a new attention mechanism for convolutional neural networks, which aims to enhance the spatial characteristics of the receptive field. By combining spatial attention with convolution, RFAConv solves the problem of traditional convolution kernel parameter sharing, especially significantly improving the processing capability of large-size convolution kernels.

[0061] Vector-Quantised Variational AutoEncoder (VQ-VAE) is a generative model that is often used to process high-dimensional data such as images and audio. The encoder of VQ-VAE encodes the input data into a latent representation. This is similar to a standard autoencoder, but the output of the encoder is not a direct latent vector, but a representation symbol (codebook index). The encoder's task is to find the representation symbol that is closest to the input, that is, the closest cluster center. The quantizer of VQ-VAE takes the output of the encoder and maps it to a discrete representation, which is done by finding the closest cluster center, and then outputs the index of the cluster center. This step introduces discreteness, reduces the dimensionality of the representation, and reduces the complexity. The decoder of VQ-VAE takes the discrete representation from the quantizer and tries to generate data that matches the original input. This process is similar to the decoder of a standard autoencoder, but in VQ-VAE, the decoder's task is more difficult because it must map the discrete representation back to continuous data.

[0062] Object Keypoint Similarity (OKS) is an evaluation indicator in key point detection. This indicator is inspired by the IoU indicator in target detection. Its purpose is to calculate the similarity between the true value and the predicted human key points.

[0063] The relative entropy loss function (Kullback-Leibler divergence Loss, referred to as KLDivLoss) refers to the difference in information entropy between two probability distributions, which is used to measure the similarity between two probability distributions.

[0064] The embodiments of the present disclosure are further described below in conjunction with the accompanying drawings.

[0065] Reference Figure 1 As shown, Figure 1 An optional flowchart of an anti-occlusion animal posture estimation method based on online distillation is provided for an embodiment of the present disclosure. The anti-occlusion animal posture estimation method based on online distillation includes but is not limited to steps 101 to 108, and each step is introduced in turn below.

[0066] Step 101: Acquire an animal image.

[0067] Step 102: Input the animal image into an anti-occlusion animal pose estimation model.

[0068] Specifically, the acquired animal image is input into the anti-occlusion animal posture estimation model, referring to Figure 4 As shown, Figure 4 A schematic diagram of the overall network framework of the anti-occlusion animal pose estimation model provided in the embodiment of the present disclosure, wherein the anti-occlusion animal pose estimation model includes a teacher model and a student model. The teacher model uses the VQ-VAE method and the MLP-Mixer architecture.

[0069] Specifically, the anti-occlusion animal pose estimation model uses online distillation and does not require additional training time and computing resources to pre-train a large teacher network.

[0070] Specifically, the student model consists of three parts: backbone, neck and head.

[0071] Specifically, in deep learning, the common backbone, neck, and head refer to different parts of the network structure, each of which has different functions. Backbone: usually refers to the main part of the entire deep neural network, responsible for extracting the features of the input data. Neck: located between the backbone and the head, responsible for further processing and integration of the features extracted by the backbone. The neck usually includes some dimensionality reduction or pooling layers, as well as some feature fusion or attention mechanisms to enhance the expressiveness of the features. Head: responsible for performing specific tasks. The head is usually composed of fully connected layers or convolutional layers, which are used to map the features extracted by the neck to the final output space and generate the final prediction results of the network. In different tasks, the structure of the head will be different to adapt to different task requirements. This modular design makes deep learning networks more versatile and flexible in different tasks and data sets.

[0072] Reference Figure 2 , Figure 7 As shown, Figure 2 An optional flow chart of obtaining a reconstructed feature vector according to a teacher model provided in an embodiment of the present disclosure, Figure 7 This is a schematic diagram of the overall structure of the teacher model provided in the embodiment of the present disclosure. The reconstructed feature vector obtained according to the teacher model includes but is not limited to the following steps 201 to 203.

[0073] Step 201: In the teacher model, obtain the true value of the key points in the animal data set, convert the true value of the key points into a feature vector, and input the feature vector into the first encoder to generate a new feature vector.

[0074] Step 202: Use the encoding space codebook to quantize the new feature vector by nearest neighbor search.

[0075] Step 203: Input the quantized feature vector into the first decoder to obtain a reconstructed feature vector.

[0076] Specifically, the key point true value in the animal data set refers to the key point label value manually marked on the animal image in the animal data set. In the teacher model, the key point true value in the animal data set is obtained, and the key point true value is converted into a feature vector, where it should be noted that the dimensions of these feature vectors must be consistent, the dimension values ​​are all D, and the number of feature vectors obtained is K. Next, the feature vector is input into the first encoder to generate a new feature vector, where the number of new feature vectors generated is M. Then, the encoding space codebook is used to quantize the new feature vector through nearest neighbor search, and the quantized feature vector is input into the first decoder to obtain the reconstructed feature vector.

[0077] In some embodiments, the code space codebook is:

[0078] C=(c 1 , …, c v ) T

[0079] Where v is the number of dimensions in the codebook and T is the temperature.

[0080] Specifically, the nearest neighbor search refers to finding the element with the smallest distance to a given query item in a certain distance metric and a search space. The new feature vector generated by the first encoder is mapped to the encoding space codebook, and the new feature vector is quantized by the nearest neighbor search to obtain a quantized feature vector.

[0081] In some embodiments, the new feature vector is quantized by nearest neighbor search using the encoding space codebook, expressed as:

[0082]

[0083] Where, t i is the new feature vector obtained by the first encoder, c j is the vector in the codebook, q(t i ) is the index of the corresponding codebook, and G is the true value of the key point.

[0084] In some embodiments, the first encoder and the first decoder are respectively constructed using an MLP-Mixer architecture. Specifically, the quantized feature vector is input into the first decoder constructed using the MLP-Mixer architecture, and the reconstructed feature vector is obtained by decoding.

[0085] In some embodiments, reference Figure 8 As shown, Figure 8 Schematic diagram of the MLP-Mixer architecture provided in an embodiment of the present disclosure. The MLP-Mixer architecture is composed of a Token-mixing MLP and a Channel-mixing MLP fully connected; the Token-mixing MLP is used to exchange information between feature vectors at different spatial positions, and the Channel-mixing MLP is used to exchange information between feature vectors at different channels at the same spatial position. The Token-mixing MLP and the Channel-mixing MLP are executed alternately to mix features in both spatial and channel dimensions, which can be expressed as:

[0086] U=X+W 2 σ(W 1 LayerNorm(X)

[0087] Y=U+W 4 σ(W 3 LayerNorm(U)

[0088] Where σ is the GELU activation function, W 1 and W 2 is the fully connected layer of Token-mixing MLP, W 3 and W 4 is the fully connected layer of Channel-mixing MLP. LayerNorm is normalization.

[0089] Specifically, the MLP-Mixer architecture consists of a fully connected Token-mixing MLP and a Channel-mixing MLP, which can mix features in both spatial and channel dimensions, and integrate feature information to make it more comprehensive. The MLP-Mixer architecture has a simple structure, a simple calculation process for each layer, and high computational efficiency. The MLP-Mixer architecture has a flexible structure that can adapt to various input data sizes and dimensions, and can also be expanded to deeper layers to improve the performance of the model.

[0090] Specifically, in the teacher model, the true values ​​of the key points in the animal dataset are obtained, the true values ​​of the key points are converted into feature vectors, and the feature vectors are input into the first encoder composed of the MLP-Mixer architecture to generate new feature vectors; the encoding space codebook is used to quantize the new feature vectors through nearest neighbor search; the quantized feature vectors are input into the first decoder composed of the MLP-Mixer architecture to obtain the reconstructed feature vectors.

[0091] In some embodiments, the loss function during teacher model training is:

[0092]

[0093] In the formula, The coordinate values ​​of the reconstruction are supervised by the smooth L1 loss function. is the reconstructed feature vector obtained after decoding by the first decoder, is the submission loss, sg is the stop gradient calculation, β is the hyperparameter value, and G is the true value of the key point.

[0094] Specifically, the L1 loss function, i.e., the mean absolute error, refers to the average of the absolute differences between the model predictions and the true values. M is the number of new feature vectors.

[0095] Specifically, the loss function is an important concept in machine learning and deep learning, which is used to measure the difference or error between the model's prediction results and the true results. It is a numerical evaluation metric that provides a measure of model performance by comparing the model output and the true label. When training the teacher model, the parameters of the teacher model are adjusted to minimize the loss function and improve the performance of the teacher model. The submission loss is used to constrain the new feature vector output by the first encoder so that the new feature vector is close to the features in the codebook. Stopping the gradient operation is used to prevent the new feature vector output by the first encoder from frequently jumping between the features of each codebook.

[0096] In some embodiments, when training the teacher model, animal images can be obtained from the AP-10K dataset. Specifically, the AP-10K dataset is the first large-scale mammal pose dataset, and AP-10K contains approximately 10,000 mammal images labeled with pose information.

[0097] Step 103: In the student model, the image feature information of the animal image is obtained through the backbone network HRNet.

[0098] Specifically, the image feature information of the animal image is obtained through the backbone network HRNet, which is the backbone part of the student model. In the student model, the animal image is processed by the backbone network HRNet to obtain high-resolution image feature information of the animal image. Figure 5 As shown, Figure 5 The schematic diagram of the network structure of the backbone network HRNet provided for the embodiment of the present disclosure is different from the traditional network structure that gradually reduces the resolution. HRNet starts with a high-resolution convolution flow, gradually adds high-to-low-resolution convolution flows, and maintains the parallel connection of these multi-resolution convolution flows, thereby maintaining high-resolution representation throughout the network. At the same time, upsampling and downsampling are used between different resolutions to achieve cross-resolution information exchange. Among them, upsampling refers to increasing the resolution, and downsampling refers to reducing the resolution.

[0099] Step 104: Based on the image feature information, use RFAConv spatial attention to obtain the local spatial information of the image.

[0100] Specifically, the local spatial information of the image is obtained using RFAConv spatial attention according to the image feature information, which is the neck part of the student model. Figure 6 As shown, Figure 6Schematic diagram of the RFAConv spatial attention structure provided for the embodiment of the present disclosure. The RFAConv spatial attention uses average pooling AvgPool to aggregate the global information of each receptive field feature, and then uses a 1×1 group of convolution operations for information interaction. It uses a softmax activation function to emphasize the importance of each feature within the receptive field feature, and finally multiplies the receptive field spatial feature obtained by using a 3×3 group of convolution operations to obtain the attention weight of each feature within the receptive field, which helps the network to capture local features more effectively and improve performance.

[0101] Specifically, RFAConv spatial attention focuses on the spatial features within the receptive field, which goes beyond the limitations of traditional spatial dimensions, comprehensively solves the parameter sharing problem of convolution kernels, and considers the importance of each feature in the receptive field. RFAConv spatial attention enables the network to more efficiently identify and process local areas in the image, thereby improving the accuracy of feature extraction.

[0102] Specifically, in deep learning and convolutional neural networks, pooling is an important operation used to reduce the dimension of data, reduce the amount of computation, and enhance the robustness of the model. Pooling operations are usually divided into maximum pooling and average pooling. The average pooling layer takes the average of the elements in each window (usually 2x2 in size) of the input data and uses this average as the output. Softmax is an activation function that normalizes a numerical vector into a probability distribution vector with the sum of each probabilities being 1. For multi-classification problems, the softmax activation function can predict the probability of each category. The softmax activation function is generally used as the last layer of a neural network, accepting input values ​​from the previous layer of the network and converting them into probabilities.

[0103] In some embodiments, according to the image feature information, the local spatial information of the image is obtained using RFAConv spatial attention, which is expressed as:

[0104] F = Softmax(g 1×1 (AvgPool(X)))×ReLU(Norm(g k×k (X)))

[0105] In the formula, g 1×1 is a grouped convolution of size 1×1, k is the size of the convolution kernel, Norm is normalization, and X is the input feature map. Specifically, g k×k is a group convolution of size k×k, and ReLU is the activation function.

[0106] Specifically, the animal image is processed by the backbone network HRNet, and the image feature information of the high-resolution animal image is obtained after processing. The high-resolution image feature information can be used as X, or a feature map can be obtained from the high-resolution image feature information and the feature map can be used as X. F is the local spatial information of the image. Since the image features are superimposed with attention weights, the image can focus on the local spatial information and obtain the local spatial information of the image. The feature map obtained by F is the feature map after "reshaping", and its width and height are k times the input, so a k×k convolution operation with a step size of k is required to restore it to the size of the input.

[0107] Step 105: According to the local spatial information of the image, the spatial dimension is extended and a set of feature vectors is obtained through two independent fully connected layers, and the set of feature vectors is split into two sets of feature vectors, namely a first set of feature vectors and a second set of feature vectors.

[0108] Specifically, according to the local spatial information of the image, the spatial dimension is extended and a group of feature vectors are obtained through two independent fully connected layers. The group of feature vectors is split into two groups of feature vectors, namely the first group of feature vectors and the second group of feature vectors. The dimensions of the first group of feature vectors and the second group of feature vectors must be consistent.

[0109] Step 106: The first set of feature vectors are used to learn the feature vectors reconstructed in the teacher model to obtain learned feature vectors.

[0110] Specifically, using KLDivLoss as the loss function, the first set of feature vectors are learned with the feature vectors reconstructed in the teacher model to obtain the learned feature vectors. The dimension of the first set of feature vectors must be consistent with the dimension of the feature vectors reconstructed in the teacher model. Then, the dimensions of the first set of feature vectors, the second set of feature vectors, and the feature vectors reconstructed in the teacher model must be consistent. According to the local spatial information of the image, the spatial dimension is extended to ensure that the dimensions of the first set of feature vectors and the second set of feature vectors are consistent with the dimensions of the feature vectors reconstructed in the teacher model. Then, the first set of feature vectors are learned with the feature vectors reconstructed in the teacher model to obtain the learned feature vectors.

[0111] Step 107: Obtain the correlation between each independent key point of the animal image according to the learned feature vector and the first set of feature vectors.

[0112] In some embodiments, the correlation between each independent key point of the animal image is obtained according to the learned feature vector and the first set of feature vectors, which is expressed as:

[0113]

[0114] In the formula, p iis the softened probability distribution p = softmax(z t / T), z t is the learned feature vector, q i is the softened probability distribution of the student model q = softmax(z s / T), z s is the first set of eigenvectors, T is the temperature. Specifically, L KD is the correlation between each independent key point of the animal image, and n is the batch size.

[0115] Specifically, the correlation between each independent key point can also be understood as the relative position relationship between each independent key point. The correlation between each independent key point obtained by the anti-occlusion animal posture estimation model can better assist animal researchers to deeply understand and analyze the behavioral characteristics of animals. The teacher model in the anti-occlusion animal posture estimation model takes into account the connection between key points, so it can better predict the true posture of animals in an occluded environment.

[0116] Step 108: In the student model, the second set of feature vectors and the learned feature vectors are fused, and after fusion, the second decoder is used to decode to obtain key point information.

[0117] Specifically, according to the local spatial information of the image, the spatial dimension is extended and a group of feature vectors are obtained through two independent fully connected layers. The group of feature vectors is split into two groups of feature vectors, namely the first group of feature vectors and the second group of feature vectors. The first group of feature vectors are used to learn the feature vectors reconstructed in the teacher model to obtain learned feature vectors. According to the learned feature vectors and the first group of feature vectors, the correlation between each independent key point of the animal image is obtained. In the student model, the second group of feature vectors and the learned feature vectors are fused. After fusion, the second decoder is used to decode and obtain the key point information, which is the head part of the student model.

[0118] In some embodiments, the second decoder is constructed using an MLP-Mixer architecture. Specifically, the second set of feature vectors and the learned feature vectors are fused, and after fusion, the second decoder constructed using the MLP-Mixer architecture is used to decode and obtain the key point information. The key point information includes the key point coordinates.

[0119] In some embodiments, the loss function during student model training is:

[0120]

[0121] Where P θ,φ(x|I) is the probability of the key points annotated on the latent distribution, θ, φ are the parameters of the regression model and the normalized flow model respectively. I is the latent distribution, μ g The key points of the annotation.

[0122] Reference Fig. 9 As shown, Fig. 9 Schematic diagram of the RLELoss loss function provided by the embodiment of the present disclosure. Specifically, the normalized flow model is used to learn the potential distribution and the loss function from the perspective of maximum likelihood estimation is calculated from the standardized distribution. By using reparameterization, a simple distribution is converted into a complex distribution through the normalized flow model. And use the formula Rescale and offset. Where, represent the mean and variance of the fully connected layer respectively.

[0123] Specifically, when training the student model, the RLELoss loss function is used to directly decode the key point coordinates, which uses a regression approach. Training based on the regression method speeds up the inference speed.

[0124] In some embodiments, when training the student model, animal images can be obtained from the AP-10K dataset.

[0125] Specifically, the teacher model reconstructs the true value of the key points. The teacher model uses the MLP-Mixer architecture to enable the first encoder and the first decoder to implicitly learn the association of key point information, thereby achieving better animal posture prediction in occluded environments.

[0126] It can be seen that an animal image is obtained; the animal image is input into the anti-occlusion animal posture estimation model, and the anti-occlusion animal posture estimation model includes a teacher model and a student model; in the student model, the image feature information of the animal image is obtained through the backbone network HRNet; according to the image feature information, the local spatial information of the image is obtained using the RFAConv spatial attention; according to the local spatial information of the image, the spatial dimension is extended and a set of feature vectors are obtained through two independent fully connected layers, and a set of feature vectors is split into two sets of feature vectors, namely the first set of feature vectors and the second set of feature vectors; the first set of feature vectors are used to learn the feature vectors reconstructed in the teacher model to obtain the learned feature vectors; according to the learned feature vectors and the first set of feature vectors, the correlation between each independent key point of the animal image is obtained; In the student model, the second set of feature vectors and the learned feature vectors are fused, and after fusion, the second decoder is used to decode and obtain the key point information; in the teacher model, the true value of the key point in the animal data set is obtained, the true value of the key point is converted into a feature vector, and the feature vector is input into the first encoder to generate a new feature vector; the encoding space codebook is used to quantize the new feature vector through nearest neighbor search; the quantized feature vector is input into the first decoder to obtain a reconstructed feature vector; the first encoder, the first decoder, and the second decoder are respectively constructed using the MLP-Mixer architecture, which can achieve better animal posture prediction in an occluded environment, and obtain the correlation between each independent key point, which can better assist animal researchers in deeply understanding and analyzing the behavioral characteristics of animals.

[0127] In order to illustrate the effect of the anti-occlusion animal pose estimation method based on online distillation disclosed in the present invention, in an embodiment of the present invention, experiments are conducted based on the AP-10K validation set, and the anti-occlusion animal pose estimation method based on online distillation disclosed in the present invention is compared with other methods on the MMPose official website.

[0128] The input size is set to 256×256, and the average precision AP (Average Precision) value is calculated based on the pose estimation OKS (Object Keypoint Similarity) formula, where AP represents the accuracy obtained by summing and averaging the values ​​in the range of 0.5 to 0.95 at intervals of 0.05. 50 and AP 75 They represent the accuracy and AP respectively when the OKS threshold is 0.5 and 0.75. M and AP L Denotes the accuracy of OKS threshold on medium and large instances, respectively.

[0129] Reference Figure 3 As shown, Figure 3A comparison chart of experimental results of an anti-occlusion animal pose estimation method based on online distillation provided in an embodiment of the present invention and other methods on the MMPose official website based on the AP-10K validation set is provided. The experimental results show that the anti-occlusion animal pose estimation method based on online distillation provided in the present invention achieves an accuracy of 74.2AP on the AP-10K validation set, which is better than other methods on the MMPose official website, proving that the anti-occlusion animal pose estimation method based on online distillation provided in the present invention can accurately predict animal poses in complex environments.

[0130] The embodiment of the present disclosure also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned anti-occlusion animal posture estimation method based on online distillation when executing the computer program. The electronic device can be any intelligent terminal including a computer.

[0131] The embodiment of the present disclosure also provides a storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned anti-occlusion animal posture estimation method based on online distillation is implemented.

[0132] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0133] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0134] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0135] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0136] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for anti-occlusion animal pose estimation based on online distillation, characterized in that: include: Obtain animal images; Inputting the animal image into an anti-occlusion animal posture estimation model, wherein the anti-occlusion animal posture estimation model includes a teacher model and a student model; In the student model, image feature information of the animal image is obtained through the backbone network HRNet; According to the image feature information, the local spatial information of the image is obtained using RFAConv spatial attention; According to the local spatial information of the image, the spatial dimension is extended and a set of feature vectors is obtained through two independent fully connected layers, and the set of feature vectors is split into two sets of feature vectors, namely a first set of feature vectors and a second set of feature vectors; Learning the first set of feature vectors with the feature vectors reconstructed in the teacher model to obtain learned feature vectors; Obtaining the correlation between each independent key point of the animal image according to the learned feature vector and the first group of feature vectors; In the student model, the second set of feature vectors and the learned feature vectors are fused, and after fusion, key point information is obtained by decoding using a second decoder; Wherein, in the teacher model, the key point true value in the animal data set is obtained, the key point true value is converted into a feature vector, the feature vector is input into a first encoder to generate a new feature vector; the new feature vector is quantized by nearest neighbor search using a coding space codebook; the quantized feature vector is input into a first decoder to obtain a reconstructed feature vector; Among them, the first encoder, the first decoder, and the second decoder are respectively constructed using the MLP-Mixer architecture.

2. The method for anti-occlusion animal posture estimation based on online distillation according to claim 1, characterized in that: According to the image feature information, the local spatial information of the image is obtained using RFAConv spatial attention, which is expressed as: F=Softmax(g 1×1 (AvgPool(X)))×ReLU(Norm(g k×k (X))) In the formula, g 1×1 is a grouped convolution of size 1×1, k is the size of the convolution kernel, Norm is normalization, and X is the input feature map.

3. The method for anti-occlusion animal posture estimation based on online distillation according to claim 1, characterized in that: The correlation between the independent key points of the animal image obtained according to the learned feature vector and the first group of feature vectors is expressed as: In the formula, p i is the softened probability distribution p = softmax(z t / T), z t is the learned feature vector, q i is the softened probability distribution of the student model q = softmax(z s / T), z s is the first set of eigenvectors, T For temperature.

4. The method for anti-occlusion animal posture estimation based on online distillation according to claim 1, characterized in that: The loss function during the student model training is: Where P θ,φ (x|I) is the probability of the keypoint annotated on the latent distribution, and θ, φ are the parameters of the regression model and the normalized flow model, respectively.

5. The method for anti-occlusion animal posture estimation based on online distillation according to claim 1, characterized in that: The code space codebook is: C=(c1,…,c v ) T Where v is the number of dimensions in the codebook; The coding space codebook is used to quantize the new feature vector by nearest neighbor search, which is expressed as: Where, t i is the new feature vector obtained by the first encoder, c j is the vector in the codebook, q(t i ) is the index of the corresponding codebook, and G is the true value of the key point.

6. The method for anti-occlusion animal posture estimation based on online distillation according to claim 1, characterized in that: The MLP-Mixer architecture consists of a fully connected Token-mixing MLP and a Channel-mixing MLP; The Token-mixing MLP is used to exchange information between feature vectors at different spatial locations, and the Channel-mixing MLP is used to exchange information between feature vectors at the same spatial location but at different channels. The Token-mixing MLP and the Channel-mixing MLP are executed alternately to mix features in both spatial and channel dimensions, which can be expressed as: U=X+W2σ(W1LayerNorm(X)) Y=U+W4σ(W3LayerNorm(U)) Where σ is the GELU activation function, W1 and W2 are the fully connected layers of the Token-mixing MLP, and W3 and W4 are the fully connected layers of the Channel-mixing MLP.

7. The method for anti-occlusion animal posture estimation based on online distillation according to claim 1, characterized in that: The loss function during teacher model training is: In the formula, The coordinate values ​​of the reconstruction are supervised by the smooth L1 loss function. is the reconstructed feature vector obtained after decoding by the first decoder, is the submission loss, sg is the stop gradient calculation, β is the hyperparameter value, and G is the true value of the key point.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the anti-occlusion animal posture estimation method based on online distillation according to any one of claims 1 to 7 is implemented.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for estimating animal postures against occlusion based on online distillation according to any one of claims 1 to 7 is implemented.

Images