An anti-occlusion animal pose estimation method based on online distillation

Abstract

The embodiment of the disclosure discloses an anti-occlusion animal pose estimation method based on online distillation. An animal image input anti-occlusion animal pose estimation model is obtained. The student model obtains image feature information of the animal image through the backbone network HRNet. The image local spatial information is obtained by using RFAConv spatial attention. The spatial dimension is extended, and the first group of feature vectors and the second group of feature vectors are obtained through two independent fully connected layers. The first group of feature vectors is learned into the feature vectors reconstructed in the teacher model to obtain learned feature vectors. The correlation between each independent key point of the animal image is obtained according to the learned feature vectors and the first group of feature vectors. The second group of feature vectors and the learned feature vectors are fused. The second decoder formed by using the MLP-Mixer architecture obtains key point information. Better animal pose prediction can be realized in the occlusion environment, and better assistance can be provided for animal researchers to deeply understand and analyze the behavior characteristics of animals.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer vision, and in particular, to an online distillation based anti-occlusion animal pose estimation method. BACKGROUND

[0002] With the rapid development of computer vision technology, the task based on human pose estimation has gradually realized commercial application. In this field, some new research directions have emerged, such as gesture pose estimation, animal pose estimation, etc. In recent years, the field of animal research has become a new research hotspot. By estimating the animal pose, according to the mutual positional relationship between the key points of the animal, the behavior characteristics of the animal can be deeply understood and analyzed. At present, the animal pose needs to be obtained by using expensive motion capture equipment for pose capture. Using an animal pose estimation model to predict the animal pose has the advantage of lower cost. However, in actual complex scenes, the occlusion phenomenon of animal pose is very common, and the animal pose estimation in the occlusion scene has drawbacks. SUMMARY

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of protection of the claims.

[0004] The embodiment of the present disclosure provides an online distillation based anti-occlusion animal pose estimation method, which can realize better animal pose prediction in an occlusion environment, and obtain the correlation between each independent key point, which can better assist animal researchers to deeply understand and analyze the behavior characteristics of animals.

[0005] Embodiments of the first aspect of the present disclosure, an online distillation based anti-occlusion animal pose estimation method, comprising:

[0006] obtaining an animal image;

[0007] inputting the animal image into an anti-occlusion animal pose estimation model, the anti-occlusion animal pose estimation model comprising a teacher model and a student model;

[0008] in the student model, obtaining image feature information of the animal image through a backbone network HRNet;

[0009] using RFAConv spatial attention to obtain image local spatial information according to the image feature information;

[0010] According to the image local spatial information, the spatial dimension is extended and a group of feature vectors are obtained through two independent fully connected layers, and the group of feature vectors are split into two groups of feature vectors, which are a first group of feature vectors and a second group of feature vectors;

[0011] The first group of feature vectors are learned as the reconstructed feature vectors in the teacher model, to obtain learned feature vectors;

[0012] According to the learned feature vectors and the first group of feature vectors, the correlation between each independent key point of the animal image is obtained;

[0013] In the student model, the second group of feature vectors and the learned feature vectors are fused, and after fusion, key point information is obtained by using a second decoder for decoding;

[0014] In the teacher model, the key point true value in the animal data set is obtained, the key point true value is converted into a feature vector, and the feature vector is input into a first encoder to generate a new feature vector; the new feature vector is quantized by using a nearest neighbor search through a codebook; and the quantized feature vector is input into a first decoder to obtain a reconstructed feature vector.

[0015] The first encoder, the first decoder and the second decoder are respectively constituted by using an MLP-Mixer architecture.

[0016] According to certain embodiments of the first aspect of the present disclosure, the image local spatial information is obtained according to the image feature information by using an RFAConv spatial attention, and is represented as:

[0017] F=Softmax(g 1×1 (AvgPool(X)))×ReLU(Norm(g k×k (X)))

[0018] In the formula, g 1×1 is a grouped convolution with a size of 1×1, k is the size of the convolution kernel, Norm is normalization, and X is the input feature map.

[0019] According to certain embodiments of the first aspect of the present disclosure, the correlation between each independent key point of the animal image is obtained according to the learned feature vectors and the first group of feature vectors, and is represented as:

[0020]

[0021] In the formula, p i is a softened probability distribution p=soft max(z t / T) of the teacher model, z t is the learned feature vector, q i is a softened probability distribution q=soft max(z s / T) of the student model, z s is the first group of feature vectors, and T is the temperature.

[0022] According to certain embodiments of the first aspect of the present disclosure, the loss function when training the student model is:

[0023]

[0024] where P θ,φ (x|I) is the probability of the annotated keypoint on the latent distribution, and θ, φ are the parameters of the regression model and the normalization flow model, respectively.

[0025] According to certain embodiments of the first aspect of the present disclosure, the codebook is:

[0026] C=(c1,…,c v ) T

[0027] where v is the dimension number in the codebook;

[0028] The new feature vector is quantized by nearest neighbor search using the codebook, denoted as:

[0029]

[0030] where t i is the new feature vector obtained by the first encoder, c j is the vector in the codebook, q(t i ) is the index of the corresponding codebook, and G is the ground truth of the keypoint.

[0031] According to certain embodiments of the first aspect of the present disclosure, the MLP-Mixer architecture is composed of Token-mixing MLP and Channel-mixing MLP full connections;

[0032] The Token-mixing MLP is used to exchange information between feature vectors of different spatial positions, and the Channel-mixing MLP is used to exchange information between feature vectors of different channels at the same spatial position. The Token-mixing MLP and the Channel-mixing MLP are executed alternately to mix the features in the spatial and channel dimensions, denoted as:

[0033] U=X+W2σ(W1LayerNorm(X))

[0034] Y=U+W4σ(W3LayerNorm(U))

[0035] wherein σ is a GELU activation function, W1 and W2 are fully connected layers of the Token-mixing MLP, and W3 and W4 are fully connected layers of the Channel-mixing MLP.

[0036] According to certain embodiments of the first aspect of the present disclosure, a loss function of the teacher model during training is:

[0037]

[0038] wherein, is a smooth L1 loss function to supervise the reconstructed coordinate values, is a reconstructed feature vector obtained after decoding by the first decoder, is a submission loss, sg is a stop gradient operation, β is a hyperparameter value, and G is a ground truth value of the key points.

[0039] According to embodiments of the second aspect of the present disclosure, an electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the online distillation based anti-occlusion animal pose estimation method as described above when executing the computer program.

[0040] According to embodiments of the third aspect of the present disclosure, a storage medium stores a computer program, and the computer program implements the online distillation based anti-occlusion animal pose estimation method as described above when executed by a processor.

[0041] The embodiments of the present disclosure at least have the following beneficial effects: an animal image is acquired; the animal image is input into an anti-occlusion animal pose estimation model, the anti-occlusion animal pose estimation model comprising a teacher model and a student model; in the student model, image feature information of the animal image is obtained through a backbone network HRNet; according to the image feature information, image local spatial information is obtained using RFAConv spatial attention; according to the image local spatial information, a spatial dimension is extended and a set of feature vectors is obtained through two independent fully connected layers, the set of feature vectors is split into two sets of feature vectors, which are a first set of feature vectors and a second set of feature vectors; the first set of feature vectors is learned to be a reconstructed feature vector in the teacher model, to obtain a learned feature vector; according to the learned feature vector and the first set of feature vectors, the relevance between each independent key point of the animal image is obtained; in the student model, the second set of feature vectors and the learned feature vector are fused, and after fusion, key point information is obtained using a second decoder; wherein, in the teacher model, key point ground truths in an animal dataset are acquired, the key point ground truths are converted into feature vectors, and the feature vectors are input into a first encoder to generate new feature vectors; the new feature vectors are quantized through nearest neighbor search using an encoding space codebook; the quantized feature vectors are input into a first decoder to obtain the reconstructed feature vector; the first encoder, the first decoder and the second decoder are respectively constituted using an MLP-Mixer architecture, which can realize better animal pose prediction in an occlusion environment, and obtain the relevance between each independent key point, which can better assist animal researchers in deeply understanding and analyzing the behavior characteristics of animals.

[0042] Other features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings are included to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used to explain the technical solutions of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions of the present disclosure.

[0044] Figure 1 An optional flowchart of an anti-occlusion animal pose estimation method based on online distillation provided by the embodiments of the present disclosure;

[0045] Figure 2 An optional flowchart of obtaining a reconstructed feature vector according to a teacher model provided by the embodiments of the present disclosure;

[0046] Figure 3 A comparison chart of experimental results of experiments based on AP-10K validation set between the anti-occlusion animal pose estimation method based on online distillation provided by the embodiments of the present disclosure and other methods of MMPose official.

[0047] Figure 4 A schematic diagram of the overall network framework of the anti-occlusion animal pose estimation model provided in the embodiments of the present disclosure is shown in FIG. 1.

[0048] Figure 5 A schematic diagram of the network structure of the backbone network HRNet provided in the embodiments of the present disclosure is shown in FIG. 2.

[0049] Figure 6 A schematic diagram of the RFAConv spatial attention structure provided in the embodiments of the present disclosure is shown in FIG. 3.

[0050] Figure 7 A schematic diagram of the overall structure of the teacher model provided in the embodiments of the present disclosure is shown in FIG. 4.

[0051] Figure 8 A schematic diagram of the MLP-Mixer architecture provided in the embodiments of the present disclosure is shown in FIG. 5.

[0052] Figure 9 A schematic diagram of the RLELoss loss function provided in the embodiments of the present disclosure is shown in FIG. 6. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, further detailed descriptions will be given to the present disclosure in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure, and are not intended to limit the present disclosure.

[0054] In the embodiments of the present disclosure, the term “module” or “unit” refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0056] In order to facilitate the understanding of the technical solutions provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are first explained here:

[0057] Online distillation refers to a single-stage end-to-end training scheme with high-efficiency parallel computing. Online distillation is based on the concept of knowledge distillation, and unlike traditional knowledge distillation, the teacher model and student model participate in training and parameter updating at the same time during online distillation.

[0058] Teacher model: The teacher model plays the role of a guide, guiding the training of the student model through its feature representation and probability distribution, with the goal of improving performance.

[0059] Student model: The student model plays the role of a learner, and it can obtain more abundant information from the teacher model, which helps it achieve better performance.

[0060] Receptive-Field Attention Convolution (RFAConv) refers to a new attention mechanism for convolutional neural networks, aiming to enhance receptive field spatial features. By combining spatial attention with convolution, RFAConv solves the problem of traditional convolution kernel parameter sharing, and significantly improves the processing ability of large-size convolution kernels.

[0061] Vector-Quantised Variational AutoEncoder (VQ-VAE) is a generative model commonly used to handle high-dimensional data such as images and audio. The encoder of VQ-VAE encodes the input data into a latent representation. This is similar to a standard autoencoder, but the output of the encoder is not a direct latent vector, but a codebook index. The task of the encoder is to find the closest representation symbol to the input, i.e., the closest cluster center. The quantizer of VQ-VAE accepts the output of the encoder and maps it to a discrete representation by finding the closest cluster center, and then outputs the index of that cluster center. This step introduces discreteness, reducing the dimensionality of the representation and reducing complexity. The decoder of VQ-VAE accepts the discrete representation from the quantizer and attempts to generate data that matches the original input. This process is similar to the decoder of a standard autoencoder, but in VQ-VAE, the task of the decoder is more difficult because it must map the discrete representation back to continuous data.

[0062] Object Keypoint Similarity (OKS) is an evaluation index in keypoint detection, inspired by the IoU index in object detection, aiming to calculate the similarity between true value and predicted human keypoint.

[0063] The relative entropy loss function (Kullback-Leibler divergence loss, or KLDivLoss for short) refers to the difference in information entropy between two probability distributions, and is used to measure the similarity between the two probability distributions.

[0064] The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0065] Reference Figure 1 As shown, Figure 1 This is an optional flowchart of an anti-occlusion animal pose estimation method based on online distillation provided in this embodiment of the present disclosure. The anti-occlusion animal pose estimation method based on online distillation includes, but is not limited to, steps 101 to 108, which are described in turn below.

[0066] Step 101: Obtain animal images.

[0067] Step 102: Input the animal image into the anti-occlusion animal pose estimation model.

[0068] Specifically, the acquired animal images are input into an anti-occlusion animal pose estimation model, referring to... Figure 4 As shown, Figure 4 This diagram illustrates the overall network framework of the anti-occlusion animal pose estimation model provided in this embodiment of the disclosure. The anti-occlusion animal pose estimation model includes a teacher model and a student model. The teacher model uses the VQ-VAE method and the MLP-Mixer architecture.

[0069] Specifically, the anti-occlusion animal pose estimation model uses online distillation, eliminating the need for additional training time and computational resources to pre-train a large teacher network.

[0070] Specifically, the student model consists of three parts: the backbone, the neck, and the head.

[0071] Specifically, in deep learning, the common backbone, neck, and head refer to different parts of the network structure, each of which has different functions. Backbone: usually refers to the main part of the entire deep neural network, responsible for extracting the features of the input data. Neck: located between the backbone and the head, responsible for further processing and integrating the features extracted by the backbone. The neck usually includes some dimension reduction or pooling layers, as well as some feature fusion or attention mechanisms to enhance the expression ability of the features. Head: responsible for performing specific tasks. The head is usually composed of fully connected layers or convolutional layers, used to map the features extracted by the neck to the final output space, generating the final prediction results of the network. In different tasks, the structure of the head will be different to adapt to different task requirements. This modular design makes the deep learning network more versatile and flexible in different tasks and datasets.

[0072] Referring to Figure 2 , Figure 7 , Figure 2 An optional flowchart for obtaining a reconstructed feature vector according to a teacher model provided by an embodiment of the present disclosure is shown. Figure 7 A schematic diagram of the overall structure of a teacher model provided by an embodiment of the present disclosure is shown. The reconstructed feature vector obtained according to the teacher model includes but is not limited to the following steps 201 to 203.

[0073] Step 201: In the teacher model, the key point ground truth in the animal dataset is obtained, the key point ground truth is converted into a feature vector, and the feature vector is input into a first encoder to generate a new feature vector.

[0074] Step 202: Use the codebook to quantize the new feature vector by nearest neighbor search.

[0075] Step 203: Input the quantized feature vector into a first decoder to obtain a reconstructed feature vector.

[0076] Specifically, the key point ground truth in the animal dataset refers to the key point label value manually marked on the animal image in the animal dataset. In the teacher model, the key point ground truth in the animal dataset is obtained, and the key point ground truth is converted into a feature vector. It should be noted that the dimensions of these feature vectors are consistent, and the dimension value is D, and the number of obtained feature vectors is K. Then, the feature vector is input into a first encoder to generate a new feature vector, wherein the number of generated new feature vectors is M. Then, use the codebook to quantize the new feature vector by nearest neighbor search, and input the quantized feature vector into a first decoder to obtain a reconstructed feature vector.

[0077] In some embodiments, the codebook is:

[0078] C = (c1, …, c v ) T

[0079] where v is the dimension number in the codebook. T is the temperature.

[0080] Specifically, the nearest neighbor search refers to finding the element with the minimum distance to the given query in a certain distance metric and a search space. The new feature vector generated by the first encoder is mapped to the codebook, and the new feature vector is quantized by the nearest neighbor search to obtain the quantized feature vector.

[0081] In some embodiments, the new feature vector is quantized by the nearest neighbor search using the codebook, denoted as:

[0082]

[0083] where t i is the new feature vector obtained by the first encoder, c j is the vector in the codebook, q(t i ) is the index of the corresponding codebook, and G is the ground truth of the key point.

[0084] In some embodiments, the first encoder and the first decoder are respectively constructed using the MLP-Mixer architecture. Specifically, the quantized feature vector is input into the first decoder constructed by the MLP-Mixer architecture, and the reconstructed feature vector is decoded.

[0085] In some embodiments, referring to FIG. 1, Figure 8 Figure 8 is a schematic diagram of the MLP-Mixer architecture provided by the embodiments of the present disclosure, which is composed of Token-mixing MLP and Channel-mixing MLP full connection; the Token-mixing MLP is used for information exchange between feature vectors at different spatial positions, and the Channel-mixing MLP is used for information exchange between feature vectors at different channels at the same spatial position, the Token-mixing MLP and the Channel-mixing MLP are executed alternately, and the features are mixed in the spatial and channel dimensions, denoted as:

[0086] U = X + W2σ(W1LayerNorm(X))

[0087] Y = U + W4σ(W3LayerNorm(U))​

[0088] where σ is the GELU activation function, W1 and W2 are the fully connected layers of the Token-mixing MLP, W3 and W4 are the fully connected layers of the Channel-mixing MLP. LayerNorm is the normalization.

[0089] Specifically, the MLP-Mixer architecture is composed of Token-mixing MLP and Channel-mixing MLP fully connected, which can mix features in two dimensions of space and channel, integrate feature information, and make it more comprehensive. The structure of the MLP-Mixer architecture is simple, the calculation process of each layer is simple, and it has high computational efficiency. The structure of the MLP-Mixer architecture is flexible, which can adapt to the size and dimension of various input data, and can be extended to deeper layers to improve the performance of the model.

[0090] Specifically, in the teacher model, the key point ground truth in the animal data set is obtained, the key point ground truth is converted into a feature vector, and the feature vector is input into the first encoder composed of the MLP-Mixer architecture to generate a new feature vector; the new feature vector is quantized by using the nearest neighbor search of the codebook; the quantized feature vector is input into the first decoder composed of the MLP-Mixer architecture to obtain a reconstructed feature vector.

[0091] In some embodiments, the loss function of the teacher model during training is:

[0092]

[0093] wherein, is a smooth L1 loss function to supervise the reconstructed coordinate value, is the reconstructed feature vector obtained after decoding by the first decoder, is the submission loss, sg is the stop gradient operation, β is the hyperparameter value, and G is the key point ground truth.

[0094] Specifically, the L1 loss function, i.e., the mean absolute error, refers to the average value of the absolute difference between the model prediction value and the true value. M is the number of new feature vectors.

[0095] Specifically, the loss function is an important concept in machine learning and deep learning, which is used to measure the difference or error between the model prediction results and the true results. It is a numerical evaluation index that provides a measure of the performance of the model by comparing the model output and the true label. When training the teacher model, the parameters of the teacher model are adjusted to minimize the loss function and improve the performance of the teacher model. The submission loss is used to constrain the new feature vector output by the first encoder, so that the new feature vector is close to the features in the codebook. The stop gradient operation is used to prevent the new feature vector output by the first encoder from frequently jumping between the features of the codebook.

[0096] In some embodiments, when training the teacher model, animal images can be obtained from the AP-10K dataset. Specifically, the AP-10K dataset is the first large-scale mammal pose dataset, and the AP-10K contains about 10,000 mammal pictures labeled with pose information.

[0097] Step 103: In the student model, the image feature information of the animal image is obtained through the backbone network HRNet.

[0098] Specifically, the image feature information of the animal image is obtained through the backbone network HRNet, which is the backbone part of the student model. In the student model, the animal image is processed by the backbone network HRNet, and the high-resolution image feature information of the animal image is obtained after processing. Referring to Figure 5 , Figure 5 The network structure diagram of the backbone network HRNet provided by the embodiments of the present disclosure is shown in the figure. The backbone network HRNet is different from the traditional network structure of gradually reducing the resolution. The HRNet starts from a high-resolution convolution stream, gradually adds high-to-low resolution convolution streams, and maintains the parallel connection of these multi-resolution convolution streams, thereby maintaining a high-resolution representation in the entire network. At the same time, information exchange across resolutions is achieved through upsampling and downsampling between different resolutions. Among them, upsampling refers to increasing the resolution, and downsampling refers to reducing the resolution.

[0099] Step 104: According to the image feature information, the image local spatial information is obtained using the RFAConv spatial attention.

[0100] Specifically, the image local spatial information is obtained using the RFAConv spatial attention according to the image feature information, which is the neck part of the student model. Referring to Figure 6 , Figure 6The RFAConv spatial attention structure diagram provided by the embodiments of the present disclosure is shown in the figure. The RFAConv spatial attention uses an average pooling AvgPool to aggregate the global information of each receiving domain feature, then uses a 1x1 group convolution operation for information interaction, and uses a softmax activation function to emphasize the importance of each feature within the receptive field feature, and finally multiplies the receptive field spatial feature obtained by using a 3x3 group convolution operation to obtain the attention weight of each feature in the receptive field, which helps the network to more effectively capture local features and improve performance.

[0101] Specifically, the RFAConv spatial attention focuses on the spatial features within the receptive field, which goes beyond the limitations of traditional spatial dimensions, comprehensively solves the parameter sharing problem of the convolution kernel, and considers the importance of each feature in the receptive field. The RFAConv spatial attention enables the network to more efficiently identify and process local regions in the image, thereby improving the accuracy of feature extraction.

[0102] Specifically, in deep learning and convolutional neural networks, pooling is an important operation that reduces the dimensionality of data, reduces computational complexity, and enhances the robustness of the model. Pooling operations are generally divided into max pooling and average pooling. The average pooling layer takes the average value of the elements within each window (usually 2x2 size) of the input data and takes this average value as the output. Softmax is an activation function that can normalize a numerical vector into a probability distribution vector, and the sum of each probability is 1. For multi-classification problems, the softmax activation function can predict the probability of each class. The softmax activation function is generally used as the last layer of the neural network, which accepts the input value from the previous layer of the network and then converts it into a probability.

[0103] In some embodiments, according to the image feature information, the RFAConv spatial attention is used to obtain the local spatial information of the image, which is represented as:

[0104] F = Softmax(g 1×1 (AvgPool(X))) x ReLU(Norm(g k×k (X)))

[0105] In the formula, g 1×1 is a group convolution with a size of 1x1, k is the size of the convolution kernel, Norm is normalization, and X is the input feature map. Specifically, g k×k is a group convolution with a size of kxk, and ReLU is an activation function.

[0106] Specifically, the animal image is processed by the backbone network HRNet, and high-resolution image feature information of the animal image is obtained after processing. The high-resolution image feature information can be taken as X, or a feature map can be obtained from the high-resolution image feature information, and the feature map is taken as X. F is image local spatial information. Because the image feature is superimposed with attention weight, the image can focus on local spatial information, and image local spatial information is obtained. The feature map obtained by F is a feature map after "shape adjustment", and the width and height are k times of the input, so a k*k convolution operation with a step of k is needed to restore the size of the input.

[0107] Step 105: according to the image local spatial information, the spatial dimension is extended, and a group of feature vectors are obtained by passing through two independent fully connected layers. The group of feature vectors are split into two groups of feature vectors, which are a first group of feature vectors and a second group of feature vectors.

[0108] Specifically, according to the image local spatial information, the spatial dimension is extended, and a group of feature vectors are obtained by passing through two independent fully connected layers. The group of feature vectors are split into two groups of feature vectors, which are a first group of feature vectors and a second group of feature vectors. The dimensions of the first group of feature vectors and the second group of feature vectors must be consistent.

[0109] Step 106: the first group of feature vectors are learned from the feature vectors reconstructed in the teacher model to obtain learned feature vectors.

[0110] Specifically, KLDivLoss is used as a loss function, the first group of feature vectors are learned from the feature vectors reconstructed in the teacher model to obtain learned feature vectors. The dimensions of the first group of feature vectors and the feature vectors reconstructed in the teacher model must be consistent. Then, the dimensions of the first group of feature vectors, the second group of feature vectors and the feature vectors reconstructed in the teacher model must be consistent. According to the image local spatial information, the spatial dimension is extended to ensure that the dimensions of the first group of feature vectors and the second group of feature vectors are consistent with the dimensions of the feature vectors reconstructed in the teacher model. Then, the first group of feature vectors are learned from the feature vectors reconstructed in the teacher model to obtain learned feature vectors.

[0111] Step 107: according to the learned feature vectors and the first group of feature vectors, the correlation between each independent key point of the animal image is obtained.

[0112] In some embodiments, according to the learned feature vectors and the first group of feature vectors, the correlation between each independent key point of the animal image is obtained, which is represented as:

[0113]

[0114] In the formula, p ip = softmax(z / T), z t is a softened probability distribution of the teacher model t is a learned feature vector, q i q = softmax(z / T), z s is a softened probability distribution of the student model s is a first group of feature vectors, T is a temperature. Specifically, L KD is a correlation between each independent key point of the animal image, and n is a batch size.

[0115] Specifically, the correlation between each independent key point can also be understood as the mutual positional relationship between each independent key point. According to the anti-occlusion animal pose estimation model, the correlation between each independent key point can better assist animal researchers in deeply understanding and analyzing the behavior characteristics of animals. The teacher model in the anti-occlusion animal pose estimation model considers the correlation between key points, so it can better predict the real pose of the animal in the occluded environment.

[0116] Step 108: In the student model, the second group of feature vectors and the learned feature vectors are fused, and after fusion, the second decoder is used for decoding to obtain key point information.

[0117] Specifically, according to the image local space information, the spatial dimension is extended and a group of feature vectors is obtained through two independent fully connected layers. The group of feature vectors is split into two groups of feature vectors, which are the first group of feature vectors and the second group of feature vectors. The first group of feature vectors is learned from the reconstructed feature vectors in the teacher model to obtain the learned feature vectors. According to the learned feature vectors and the first group of feature vectors, the correlation between each independent key point of the animal image is obtained. In the student model, the second group of feature vectors and the learned feature vectors are fused, and after fusion, the second decoder is used for decoding to obtain key point information, which is the Head part of the student model.

[0118] In some embodiments, the second decoder is constructed using the MLP-Mixer architecture. Specifically, the second group of feature vectors and the learned feature vectors are fused, and after fusion, the second decoder constructed using the MLP-Mixer architecture is used for decoding to obtain key point information. Wherein, the key point information includes key point coordinates.

[0119] In some embodiments, the loss function of the student model during training is:

[0120]

[0121] P θ,φ(x|I) represents the probability of keypoints annotated on the latent distribution, and θ and φ are the parameters of the regression model and the normalized flow model, respectively. I is the latent distribution, μ g The key points are for annotation.

[0122] Reference Figure 9 As shown, Figure 9 This diagram illustrates the RLELoss loss function provided in an embodiment of this disclosure. Specifically, a normalized flow model is used to learn the latent distribution, and the loss function from the perspective of maximum likelihood estimation is calculated from the standardized distribution. By utilizing reparameterization, a simple distribution is transformed into a complex distribution through the normalized flow model. Use formula Rescaling and offsetting. Among them, These represent the mean and variance of the fully connected layer, respectively.

[0123] Specifically, during student model training, the RLELoss loss function is used to directly decode the keypoint coordinates, which employs a regression approach. Training based on regression speeds up inference.

[0124] In some embodiments, animal images can be obtained from the AP-10K dataset when training the student model.

[0125] Specifically, the teacher model reconstructs the ground truth of key points. The teacher model uses an MLP-Mixer architecture, which enables the first encoder and the first decoder to implicitly learn the association of key point information, thereby achieving better animal pose prediction in occluded environments.

[0126] It can be seen that the animal image is acquired; the animal image is input into an anti-occlusion animal pose estimation model, the anti-occlusion animal pose estimation model comprising a teacher model and a student model; in the student model, image feature information of the animal image is obtained through a backbone network HRNet; according to the image feature information, image local spatial information is obtained using RFAConv spatial attention; according to the image local spatial information, a spatial dimension is extended and a set of feature vectors is obtained through two independent fully connected layers, the set of feature vectors is split into two sets of feature vectors, which are a first set of feature vectors and a second set of feature vectors; the first set of feature vectors is learned to be a reconstructed feature vector in the teacher model, to obtain a learned feature vector; according to the learned feature vector and the first set of feature vectors, the correlation between each independent key point of the animal image is obtained; in the student model, the second set of feature vectors and the learned feature vector are fused, and after fusion, key point information is obtained using a second decoder; wherein in the teacher model, key point ground truths in an animal dataset are acquired, the key point ground truths are converted into feature vectors, and the feature vectors are input into a first encoder to generate new feature vectors; the new feature vectors are quantized through nearest neighbor search using an encoding space codebook; the quantized feature vectors are input into a first decoder to obtain reconstructed feature vectors; the first encoder, the first decoder and the second decoder are respectively constituted using an MLP-Mixer architecture, which can achieve better animal pose prediction in an occlusion environment, and obtain the correlation between each independent key point, which can better assist animal researchers in understanding and analyzing the behavior characteristics of animals.

[0127] In order to illustrate the effect of the anti-occlusion animal pose estimation method based on online distillation of the present disclosure, in the embodiment of the present disclosure, experiments are carried out based on the AP-10K validation set, and the anti-occlusion animal pose estimation method based on online distillation of the present disclosure is compared with other methods of the official MMPose.

[0128] The input size is set to 256x256, and the average precision AP (Average Precision) value is calculated based on the pose estimation OKS (Object Keypoint Similarity) formula, wherein AP represents the precision obtained by summing and averaging the precision in the range of 0.5 to 0.95 at intervals of 0.05, AP 50 and AP 75 respectively represent the precision when the OKS threshold is 0.5 and 0.75, and AP M and AP L respectively represent the precision on medium and large instances.

[0129] Referring to FIG. Figure 3 , as shown, Figure 3The experimental results of the method for anti-occlusion animal pose estimation based on online distillation provided by the embodiment of the present disclosure and the experimental results of other methods based on AP-10K verification set of the official MMPose are compared in the comparison chart of experimental results. The experimental results show that the method for anti-occlusion animal pose estimation based on online distillation provided by the present disclosure has an accuracy of 74.2 AP on the AP-10K verification set, which is better than other methods based on the official MMPose, proving that the method for anti-occlusion animal pose estimation based on online distillation can accurately predict the animal pose in a complex environment.

[0130] The embodiment of the present disclosure also provides an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for anti-occlusion animal pose estimation based on online distillation. The electronic device can be any intelligent terminal, such as a computer.

[0131] The embodiment of the present disclosure also provides a storage medium, which stores a computer program. When the computer program is executed by a processor, the method for anti-occlusion animal pose estimation based on online distillation is implemented.

[0132] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances to describe the embodiments of the present disclosure, for example, the embodiments can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0133] It should be understood that in the present disclosure, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0134] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is two or more, greater than, less than, more than, etc. is not included in the number, above, below, within, etc. is understood to include the number.

[0135] It should also be understood that various embodiments provided by the embodiments of the present disclosure can be combined in any manner to achieve different technical effects.

[0136] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above implementation. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit and scope of the present disclosure. These equivalent modifications or replacements are all included in the scope defined by the claims of the present disclosure.

Claims

1. An online distillation based anti-occlusion animal pose estimation method, characterized in that, The method comprises: acquiring an animal image; inputting the animal image into an anti-occlusion animal pose estimation model, the anti-occlusion animal pose estimation model comprising a teacher model and a student model; in the student model, obtaining image feature information of the animal image through a backbone network HRNet; obtaining image local spatial information using RFAConv spatial attention according to the image feature information; according to the image local spatial information, extending in the spatial dimension and obtaining a group of feature vectors through two independent fully connected layers, and splitting the group of feature vectors into two groups of feature vectors, namely a first group of feature vectors and a second group of feature vectors; learning the feature vectors reconstructed in the teacher model using the first group of feature vectors to obtain learned feature vectors; obtaining the correlation between each independent key point of the animal image according to the learned feature vectors and the first group of feature vectors; in the student model, fusing the second group of feature vectors and the learned feature vectors, and then decoding using a second decoder to obtain key point information; wherein, in the teacher model, the key point true value in the animal data set is obtained, the key point true value is converted into a feature vector, and the feature vector is input into a first encoder to generate a new feature vector; the new feature vector is quantized by using a codebook through nearest neighbor search; and the quantized feature vector is input into a first decoder to obtain a reconstructed feature vector; wherein, the first encoder, the first decoder and the second decoder are respectively constructed using an MLP-Mixer architecture.

2. The anti-occlusion animal pose estimation method based on online distillation according to claim 1, wherein, The image local spatial information obtained according to the image feature information using RFAConv spatial attention is represented as: F = Softmax(g 1×1 (AvgPool(X))) x ReLU(Norm(g k×k (X))) In the formula, g 1×1 is a 1x1 grouped convolution, k is the size of the convolution kernel, Norm is normalization, and X is the input feature map.

3. The anti-occlusion animal pose estimation method based on online distillation according to claim 1, wherein, The correlation between each independent key point of the animal image obtained according to the learned feature vectors and the first group of feature vectors is represented as: where p i is the softened probability distribution of the teacher model p = softmax(z t / T), z t is the learned feature vector, q i is the softened probability distribution of the student model q = softmax(z s / T), z s is the first set of feature vectors, and T is the temperature.

4. The anti-occlusion animal pose estimation method based on online distillation according to claim 1, characterized in that, The loss function of the student model during training is: where P θ,φ (x | I) is the probability of the annotated keypoint on the latent distribution, and θ, φ are the parameters of the regression model and the normalization flow model, respectively.

5. The anti-occlusion animal pose estimation method based on online distillation according to claim 1, characterized in that, The codebook is: C = (c1,..., c v ) T wherein, v is the dimension number in the codebook; The new feature vector is quantized by using a codebook through nearest neighbor search, and is represented as: where t i is the new feature vector obtained by the first encoder, c j is the vector in the codebook, q(t i ) is the index of the corresponding codebook, and G is the ground truth of the key point.

6. The anti-occlusion animal pose estimation method based on online distillation according to claim 1, characterized in that, The MLP-Mixer architecture is composed of Token-mixing MLP and Channel-mixing MLP full connections; The Token-mixing MLP is used for information exchange of feature vectors at different spatial positions, the Channel-mixing MLP is used for information exchange of feature vectors at different channels in the same spatial position, and the Token-mixing MLP and the Channel-mixing MLP are alternately executed to mix features in the spatial and channel dimensions, and is represented as: U = X + W2σ(W1LayerNorm(X)) Y = U + W4σ(W3LayerNorm(U)) In the formula, σ is a GELU activation function, W1 and W2 are full connection layers of a Token-mixing MLP, and W3 and W4 are full connection layers of a Channel-mixing MLP.

7. The anti-occlusion animal pose estimation method based on online distillation according to claim 1, characterized in that, The loss function of the teacher model during training is: wherein, is a smooth L1 loss function to supervise the reconstructed coordinate values, is a reconstructed feature vector after decoding by the first decoder, is a submission loss, sg is a stop gradient operation, β is a hyperparameter value, and G is the ground truth of the key points.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the online distillation-based anti-occlusion animal pose estimation method of any one of claims 1 to 7.

9. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the online distillation-based anti-occlusion animal pose estimation method of any one of claims 1 to 7.

Images