Visual text multi-modal animal posture estimation method based on probability representation learning

Abstract

The invention relates to a visual text multi-modal animal posture estimation method based on probability representation learning, and the method comprises the steps: obtaining an image data set, and extracting image features; constructing an attribute description set, and converting the text key point category names and obtaining a prompt template; constructing multivariable Gaussian distribution of each key point based on the prompt template; constructing a key point spatial position score distribution map based on the multivariable Gaussian distribution and the image features; and obtaining a key point thermodynamic diagram based on the key point spatial position score distribution diagram and the image features, and performing animal attitude estimation based on the key point thermodynamic diagram to obtain key point coordinates. Compared with the prior art, the problem of inaccurate attitude estimation caused by a single prompt template and non-uniform data distribution of various animals in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and behavior analysis, and in particular to a visual text multimodal animal posture estimation method based on probability representation learning. Background Art

[0002] Animal pose estimation (APE) is a basic task in behavioral analysis, which aims to detect the coordinates of the key points of the target body in the localized image / video. However, since it takes a lot of manpower and material resources to obtain data by tracking wild animals, and complex scene factors such as occlusion and large changes in illumination in the wild will cause the acquired animal pose data to be very limited. In addition, there are problems of large data variance and uneven distribution for multiple categories of animals, which greatly increases the difficulty of animal pose estimation. In recent years, with the development of vision-and-language model (VLM), especially the research on contrastive learning language-image pre-training (CLIP), the pre-trained image-text pair knowledge is transferred to downstream visual tasks and shows strong generalization ability in zero-shot experiments. As a new paradigm, the multimodal animal pose estimation method with text interaction can use the prior knowledge of the large model to alleviate the problems of small data volume and large variance for multiple categories of animals. However, how to design a prompt learning method for pose estimation of multiple species of animals remains a challenge. There are two main defects in the existing methods: 1) Multi-species animal posture estimation faces huge data variance, but the existing multimodal posture estimation methods all use discriminative prompt templates. The representation learned by this discriminative learning method will be biased towards the distribution of training data and cannot be adapted to animal data of unseen categories. Therefore, the accuracy of multi-species animal posture estimation is often low; 2) Due to the variability of posture changes and complex wild environments, even animals of the same species, and even different species, show significant differences. These differences are reflected in their various attributes, such as color, texture, and position, making it impractical to comprehensively summarize them through a single text description. In addition, multi-species animal data also faces the problem of uneven distribution, namely the long tail phenomenon, which reflects that the natural state of data distribution in the real world cannot be ignored.

[0003] Therefore, image features or a single textual hint template alone cannot provide enough information to overcome the multi-species pose estimation problem. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a visual text multimodal animal posture estimation method based on probabilistic representation learning. The probabilistic prompt learning method is used to model text prompts, which solves the problems of the single discriminant text prompt template and the difficulty in handling the uneven distribution of multiple species of animals in the prior art, thereby improving the generalization of the model.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A visual text multimodal animal posture estimation method based on probabilistic representation learning includes the following steps:

[0007] Acquire an image data set and extract image features, wherein the image data set includes a plurality of text key point category names;

[0008] Constructing an attribute description set, converting the text key point category names, and randomly inserting the converted text key point category names into the attribute description set to obtain multiple prompt templates;

[0009] Constructing a multivariate Gaussian distribution of each key point based on the multiple prompt templates;

[0010] Constructing a key point spatial position score distribution map based on the multivariate Gaussian distribution and image features;

[0011] A key point heat map is obtained based on the key point spatial position score distribution map and image features, and animal posture estimation is performed based on the key point heat map to obtain key point coordinates.

[0012] As a preferred technical solution, the method for constructing the multivariate Gaussian distribution is:

[0013] Based on the prompt template, a prompt feature is obtained, and a difference constraint is imposed on the prompt feature, and the expression is:

[0014]

[0015] Among them, P i represents the prompt feature corresponding to the i-th key point, Represents the transpose of the prompt feature, I represents the dimension is N p The unit diagonal matrix, N p represents the number of prompt templates, and K represents the number of key points;

[0016] Calculate the mean of the multivariate Gaussian distribution based on the cue features after imposing a difference constraint;

[0017] Interacting the image features with the prompt features to obtain the variance of the multivariate Gaussian distribution;

[0018] Construct a multivariate Gaussian distribution based on the variance and mean.

[0019] As a preferred technical solution, the method for calculating the mean is:

[0020] The self-attention mechanism is used to process the prompt features, and its expression is as follows:

[0021]

[0022]

[0023] in, represents the query generated by the t-th hint feature of the ith keypoint, represents the key generated by the t-th hint feature of the ith keypoint, represents the value generated by the tth hint feature of the ith key point, W q , W k and W v represents the learnable parameter matrix, represents the tth prompt feature of the i-th key point, MLP and SA both represent operation, Represents the mean.

[0024] As a preferred technical solution, the method for calculating the variance is:

[0025] The cross attention mechanism is used to interact the image features and the prompt features, and the variance is calculated, which is expressed as:

[0026]

[0027]

[0028] in, represents the query generated by the t-th hint feature of the i-th keypoint, v k represents the key generated by the t-th hint feature of the ith keypoint, v v represents the value generated by the tth hint feature of the ith key point, W q ′,W k ′ and W′ v Both represent the learnable parameter matrix, F vis Represents visual features, represents variance, MLP and CA both represent operations.

[0029] As a preferred technical solution, the method for constructing a key point spatial position score distribution map is:

[0030] The multivariate Gaussian distribution is sampled using a reparameterization method to obtain a sampling prompt feature;

[0031] Acquire a real coordinate mask of key points based on the animal picture, and sample the image features based on the mask to obtain sampled image features;

[0032] The contrast loss of the sampled hint feature and the sampled image feature is calculated, and the expression is:

[0033]

[0034]

[0035] Among them, F keypoint represents the sampled image features, represents the sampling hint feature, M represents the text-visual similarity matrix, and M label is a diagonal matrix, M T is the transposed matrix, CE represents the cross loss entropy calculation;

[0036] Aligning the cue features with the image features based on the contrast loss;

[0037] The image features are mapped and cross-modal information fused with the feature alignment results to obtain multiple key point score maps, and the number of the key point score maps is greater than the number of key points. The multiple key point score maps are spatially aligned to obtain a key point spatial position score distribution map.

[0038] As a preferred technical solution, the method for obtaining the sampling prompt feature is:

[0039] The variable Gaussian distribution is resampled to obtain the sampling hint feature, which is expressed as:

[0040]

[0041] in, represents the sampling hint feature, represents the mean, represents variance, ∈ represents a variable drawn from a normal distribution;

[0042] The divergence value is calculated based on the sampling hint feature and the difference constraint to guide the next sampling, and its expression is:

[0043]

[0044] in, represents the divergence value, represents the difference constraint, N pIndicates the number of prompt templates. represents a multivariate Gaussian distribution, represents the tth hint feature of the ith key point, represents the t-th hint feature of the ith keypoint in the multivariate Gaussian distribution, represents a standard normal distribution.

[0045] As a preferred technical solution, the spatial alignment processing of the multiple key point score maps comprises the following steps:

[0046] Randomly sample N from the multiple key point score maps s Key point score map;

[0047] Construct a 2D heat map based on the multivariate Gaussian distribution, define the mean point as the coordinate point, and the variance as a constant value;

[0048] Based on the key point score map and 2D heat map obtained by sampling, heuristic selection, integrated selection or attention-based selection are used for spatial alignment to obtain the key point spatial position score distribution map.

[0049] The heuristic selection is: calculating the 2D heat map and the N s The similarity of each key point score map in the key point score maps is calculated, and the key point score map with the highest similarity is selected as the key point spatial position score distribution map of the corresponding key point;

[0050] The integration option is: connect the N s The key point score map is then converted into a key point score map and the convolution operation Conv is applied to perform signal modulation to obtain the key point spatial position score distribution map, which is expressed as:

[0051] S = Conv(Concat(S′))

[0052] Among them, S represents the key point spatial position score distribution map, S′ represents N s Key point score graph, Concat means connection;

[0053] The attention-based selection is: based on the N s The key point score map obtains the query vector, and outputs the key point spatial position score distribution map based on the query vector, which is expressed as:

[0054] S i = Flatten(S′),

[0055] Q i =q i W q ,K i =Si W k ,V i =S i W v ,

[0056]

[0057] Among them, S i represents the i-th key point score map, Flatten represents the flattening operation, and W q ,W k ,W v and W a represents the learnable parameter matrix, Q i , K i and V i represents the query, key and value of the i-th key point, H represents the feature height, W represents the feature width, Indicates transpose.

[0058] As a preferred technical solution, the construction of the key point spatial position score distribution map further includes:

[0059] The loss is calculated based on the 2D heat map and the key point spatial position score distribution map, and the spatial loss used to construct the key point spatial position score distribution map is expressed as:

[0060]

[0061] in, represents the spatial loss, S represents the key point spatial position score distribution map, H target Represents a 2D heat map.

[0062] As a preferred technical solution, the method for obtaining the key point heat map is: cascading the key point spatial position score distribution map and the image features, and the expression is:

[0063]

[0064] Among them, F o represents the key point heat map, F vis represents the image features, and S represents the key point spatial position score distribution map.

[0065] As a preferred technical solution, the method for obtaining the key point coordinates is:

[0066] Multiplying the coordinates of the key point heat map by a preset ratio to restore it to the size of the animal image;

[0067] The coordinates of the key points are obtained based on the key point heat map after the size is restored. The expression is:

[0068] Kpt k =s1·argmax 1≤e≤H,1≤f≤W F o (e,f),k=1,…,K,

[0069] Among them, Kpt k Represents the coordinate sequence of K key points, s1 represents the preset ratio, argmax represents the operation, k represents the kth key point, K represents the total number of key points, H represents the height of the feature, W represents the width of the feature, e and f represent the height and width of the key heat map respectively.

[0070] Compared with the prior art, the present invention has the following advantages:

[0071] 1) The present invention constructs a multivariate Gaussian distribution for each key point and adopts a probabilistic prompt learning method to model text prompts, thereby solving the problems in the prior art that the representation learned by the large model is biased towards the distribution of training data and the estimation accuracy is low due to the single discriminant text prompt template and uneven distribution of multiple animal data, thereby improving the generalization and estimation accuracy of the model;

[0072] 2) The present invention introduces a set of learnable attribute description sets to express attributes that are unrelated to key point categories, such as position, size, and color, so as to represent the common attributes of multi-species targets. In order to more effectively describe the diverse and information-rich attributes of the entire key point, the key point category names are randomly inserted into the prompt template to increase the diversity of the prompt template and make the prompt template more consistent with the natural language description of the key point;

[0073] 3) In order to generate key point descriptions of different animals more flexibly, the present invention models the attribute distribution of each key point as a different normal distribution, and then extracts sampled features from multiple normal distributions to achieve generalized multi-species animal posture estimation. Considering that posture estimation is an intensive prediction task, the present application also establishes a spatial connection between the sampled prompt template and the image features to achieve more accurate posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a flowchart of the invention of this application;

[0075] Figure 2 This is a large model structure diagram of this application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0077] This embodiment provides a visual text multimodal animal posture estimation method based on probability representation learning. The method flow of the invention is as follows: Figure 1 As shown in the figure, by probabilistically modeling the hint template of the large model, the problem of inaccurate pose estimation caused by the single hint template is improved, and the hint template and image features are closely linked through feature alignment and spatial alignment, which strengthens the model performance. The improved model structure is shown in Figure 2 As shown, the estimation accuracy is improved. The specific method includes the following steps:

[0078] S1. Image feature extraction:

[0079] S11. Obtain an image dataset, wherein the image dataset includes K text key point category names k i Represents the i-th key point category name, such as k1 for left eye and k2 for right eye.

[0080] S12. Extract image features F using the CLIP pre-trained image encoder θ(·) vis ∈R C×H×W , where H, W, and C represent the height, width, and number of channels of the feature, respectively.

[0081] S2. Prompt template construction:

[0082] S21. Use CLIP’s tokenizer to name key point classes Perform the conversion and randomly initialize N p A learnable Used to represent a set of attribute descriptions for an image.

[0083] S22, randomly inserting the converted text key point category names into the attribute description set to obtain multiple prompt templates Among them, i and y represent the t-th attribute description of the ith key point respectively.

[0084] S3, multivariate Gaussian distribution construction:

[0085] Step S3 is to perform probabilistic modeling on the prompt template of the key points, aiming to model different attribute descriptions of the key points, so as to overcome the problem of large data variance caused by multi-species animal posture estimation, including:

[0086] S31. Use the CLIP pre-trained text encoder φ(·) to obtain prompt features based on the prompt template Where N p is the number of learnable tokens and also the number of prompt templates, C emb is the feature dimension size. To ensure that the prompt template can represent different attributes of key points, that is, to construct independent Gaussian distributions, a difference constraint is imposed on the prompt feature, and its expression is:

[0087]

[0088] Among them, P i represents the prompt feature corresponding to the i-th key point, Represents the transpose of the prompt feature, I represents the dimension is N p The unit diagonal matrix, N p represents the number of learnable tokens, and K represents the number of key points.

[0089] After the difference constraint is passed, the prompt features belonging to the same attribute will be more similar, and conversely, the prompt features of different attributes will be more different.

[0090] S32. Based on the prompt features after applying the difference constraint, the prompt features are processed using the self-attention mechanism to calculate the mean of the multivariate Gaussian distribution Its expression is:

[0091]

[0092]

[0093] in, represents the query generated by the t-th hint feature of the ith keypoint, represents the key generated by the t-th hint feature of the ith keypoint, represents the value generated by the tth hint feature of the ith key point, W q , W k and W v represents the learnable parameter matrix, represents the tth hint feature of the i-th key point, MLP and SA both represent multi-layer perception and layer regularization operations, Represents the mean.

[0094] S33. Use the cross-attention mechanism to interact the image features and the prompt features, introduce uncertainty in the image features, and thus learn the variance of the multivariate Gaussian distribution. Its expression is:

[0095]

[0096]

[0097] in, represents the query generated by the t-th hint feature of the i-th keypoint, v k represents the key generated by the t-th hint feature of the ith keypoint, v v represents the value generated by the tth hint feature of the ith key point, W q ′,W k ′ and W v ′ denotes the learnable parameter matrix, F vis Represents visual features, represents variance, MLP and CA both represent operations.

[0098] S34. Construct a multivariate Gaussian distribution based on variance and mean.

[0099] S4. Construct a key point spatial position score distribution map:

[0100] To ensure that the learning process is differentiable, a reparameterization technique is used to sample N from a multivariate Gaussian distribution. s Key point hint features And obtain key point features based on image feature sampling Compute contrast loss to align visual and text features, and compare with the mapped image features Perform cross-modal information fusion to obtain the score map S∈R of the key point spatial position K ×H×W ,This score map reflects the effect of text prompts on spatial positioning.

[0101] Because direct sampling in a multivariate Gaussian distribution is not differentiable, it is necessary to use a reparameterization operation to sample to obtain the prompt feature, and then achieve text-visual alignment at the feature end, and establish the connection between the text prompt and the spatial position of the key point, including:

[0102] S41. Using the reparameterization method to sample the multivariate Gaussian distribution to obtain a sampling prompt feature. Specifically, reparameterization is performed on the variable Gaussian distribution to obtain a sampling prompt feature, and its expression is:

[0103]

[0104] in, represents the sampling hint feature, represents the mean, represents variance, ∈ represents a variable drawn from a normal distribution;

[0105] In order to prevent the variance of the fitting from tending to zero during the network learning process, this embodiment introduces an additional KL divergence as a regularization term, and calculates the divergence value based on the sampling prompt feature and the difference constraint to guide the next sampling. Its expression is:

[0106]

[0107] in, represents the divergence value, represents the difference constraint, N p Indicates the number of prompt templates. represents a multivariate Gaussian distribution, represents the tth hint feature of the ith key point, represents the t-th hint feature of the ith keypoint in the multivariate Gaussian distribution, represents a standard normal distribution.

[0108] S42, corresponding to the prompt feature, it is hoped that the image feature of a specific key point is close to its corresponding text prompt feature and away from the prompt feature of other key points, and vice versa, the same operation is applied to the text feature, thereby enhancing the discrimination ability of text features and image features and promoting feature alignment. Specifically, the real coordinate mask of the key point is obtained based on the animal picture, and the image feature is sampled based on the mask to obtain the sampled image feature, and the expression is:

[0109]

[0110] Among them, F keypoint represents sampled image features, and GS represents network sampling.

[0111] S43, calculating the contrast loss of the sampled hint feature and the sampled image feature, the expression is:

[0112]

[0113]

[0114] Among them, F keypoint represents the sampled image features, represents the sampling hint feature, M represents the text-visual similarity matrix, and M label is a diagonal matrix, M T is the transposed matrix, and CE represents the cross loss entropy calculation.

[0115] S44, aligning the prompt feature and the image feature based on the contrast loss.

[0116] S45, mapping the image features, and performing cross-modal information fusion with the feature alignment results to obtain multiple key point score maps, wherein the number of the key point score maps is greater than the number of key points, and spatially aligning the multiple key point score maps to obtain a key point spatial position score distribution map, including the following steps:

[0117] S451, randomly sample N from multiple key point score maps s Keypoint score map

[0118] S452. Constructing 2D heat map based on multivariate Gaussian distribution H target ∈R K×H×W , define the mean point as the coordinate point and the variance as a constant value.

[0119] S453. Based on the key point score map and 2D heat map obtained by sampling, spatial alignment processing is performed using heuristic selection, integrated selection or attention-based selection to obtain a key point spatial position score distribution map.

[0120] Among them, the heuristic choice is: calculate the 2D heat map and the N s The similarity of each key point score map in the key point score maps is calculated, and the key point score map with the highest similarity is selected as the key point spatial position score distribution map of the corresponding key point;

[0121] Integrated selection: The integrated selection strategy assumes that all features contain contextual information. In order to prevent the loss of text clues, N s The key point score map is then converted into a key point score map and the convolution operation Conv is applied to perform signal modulation to obtain the key point spatial position score distribution map, which is expressed as:

[0122] S = Conv(Concat(S′))

[0123] Among them, S represents the key point spatial position score distribution map, S′ represents N s Key point score graph, Concat means connection;

[0124] Attention-based selection: Introducing an additional learnable query vector q i ∈R K×(HW) , based on N s The key point score map is used to obtain the query vector, and the attention mechanism is used to output the key point spatial position score distribution map based on the query vector, and its expression is:

[0125] S i= Flatten(S′),

[0126] Q i =q i W q ,K i =S i W k ,V i =S i W v ,

[0127]

[0128] Among them, S i represents the i-th key point score map, flatten represents the flattening operation, W q ,W k ,W v and W a represents the learnable parameter matrix, Q i , K i and V i represents the query, key and value of the i-th key point, H represents the feature height, W represents the feature width, Indicates transpose.

[0129] S454. After obtaining S, S will be reshaped into a 2D feature map.

[0130] S455, based on the 2D heat map and the key point spatial position score distribution map, the loss is calculated as the spatial loss for constructing the key point spatial position score distribution map, and its expression is:

[0131]

[0132] in, represents the spatial loss, S represents the key point spatial position score distribution map, H target Represents a 2D heat map.

[0133] S5. Get the key point coordinates:

[0134] S51, cascading the key point spatial position score distribution map and the image features, the expression is:

[0135]

[0136] F o represents the key point heat map, F vis represents image features, S represents the key point spatial position score distribution map, where HEAD represents a universal key point detection head, which is composed of multiple deconvolution and multi-layer perceptron layers.

[0137] S52, according to the general posture estimation post-processing method, apply a simple argmax operation to each key point heat map, and multiply the coordinates by the scale s1 to restore to the input image size, and obtain the coordinates Kpt of the K key points k , whose expression is:

[0138] Kpt k =s1·argmax 1≤e≤H,1≤f≤W F o (e,f),k=1,…,K,

[0139] Among them, Kpt k Represents the coordinate sequence of K key points, s1 represents the preset ratio, argmax represents the operation, k represents the kth key point, K represents the total number of key points, H represents the height of the feature, W represents the width of the feature, e and f represent the height and width of the key heat map respectively.

[0140] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A visual text multimodal animal posture estimation method based on probabilistic representation learning, characterized in that: The method includes: Acquire an image data set and extract image features, wherein the image data set includes a plurality of text key point category names; Constructing an attribute description set, converting the text key point category names, and randomly inserting the converted text key point category names into the attribute description set to obtain multiple prompt templates; Constructing a multivariate Gaussian distribution of each key point based on the multiple prompt templates; Constructing a key point spatial position score distribution map based on the multivariate Gaussian distribution and image features; A key point heat map is obtained based on the key point spatial position score distribution map and image features, and animal posture estimation is performed based on the key point heat map to obtain key point coordinates.

2. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 1 is characterized in that: The method to construct the multivariate Gaussian distribution is: Based on the prompt template, a prompt feature is obtained, and a difference constraint is imposed on the prompt feature, and the expression is: Among them, P i represents the prompt feature corresponding to the i-th key point, Represents the transpose of the prompt feature, I represents the dimension is N p The unit diagonal matrix, N p represents the number of prompt templates, and K represents the number of key points; Calculate the mean of the multivariate Gaussian distribution based on the cue features after imposing a difference constraint; Interacting the image features with the prompt features to obtain the variance of the multivariate Gaussian distribution; Construct a multivariate Gaussian distribution based on the variance and mean.

3. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 2 is characterized in that: The method for calculating the mean is: The self-attention mechanism is used to process the prompt features, and its expression is as follows: in, represents the query generated by the t-th hint feature of the ith keypoint, represents the key generated by the t-th hint feature of the ith keypoint, represents the value generated by the tth hint feature of the ith key point, W q , W k and W v represents the learnable parameter matrix, represents the tth prompt feature of the i-th key point, MLP and SA both represent operation, Represents the mean.

4. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 2, characterized in that: The method for calculating the variance is: The cross attention mechanism is used to interact the image features and the prompt features, and the variance is calculated, which is expressed as: in, represents the query generated by the t-th hint feature of the i-th keypoint, v k represents the key generated by the t-th hint feature of the ith keypoint, v v represents the value generated by the tth hint feature of the i-th key point, W′ q , W′ k and W′ v Both represent the learnable parameter matrix, F vis Represents visual features, represents variance, MLP and CA both represent operations.

5. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 1, characterized in that: The method for constructing the key point spatial position score distribution map is: The multivariate Gaussian distribution is sampled using a reparameterization method to obtain a sampling prompt feature; Acquire a real coordinate mask of key points based on the animal picture, and sample the image features based on the mask to obtain sampled image features; The contrast loss of the sampled hint feature and the sampled image feature is calculated, and the expression is: Among them, F keypoint represents the sampled image features, represents the sampling hint feature, M represents the text-visual similarity matrix, and M label is a diagonal matrix, M T is the transposed matrix, CE represents the cross loss entropy calculation; Aligning the cue features with the image features based on the contrast loss; The image features are mapped and cross-modal information fused with the feature alignment results to obtain multiple key point score maps, and the number of the key point score maps is greater than the number of key points. The multiple key point score maps are spatially aligned to obtain a key point spatial position score distribution map.

6. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 5, characterized in that: The method for obtaining the sampling prompt feature is: The variable Gaussian distribution is resampled to obtain the sampling hint feature, which is expressed as: in, represents the sampling hint feature, represents the mean, represents variance, ∈ represents a variable drawn from a normal distribution; The divergence value is calculated based on the sampling hint feature and the difference constraint to guide the next sampling, and its expression is: in, represents the divergence value, represents the difference constraint, N p Indicates the number of prompt templates. represents a multivariate Gaussian distribution, represents the tth hint feature of the ith key point, represents the t-th hint feature of the ith keypoint in the multivariate Gaussian distribution, represents a standard normal distribution.

7. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 5, characterized in that: The spatial alignment processing of the multiple key point score maps comprises the following steps: Randomly sample N from the multiple key point score maps s Key point score map; Construct a 2D heat map based on the multivariate Gaussian distribution, define the mean point as the coordinate point, and the variance as a constant value; Based on the key point score map and 2D heat map obtained by sampling, heuristic selection, integrated selection or attention-based selection are used for spatial alignment to obtain the key point spatial position score distribution map. The heuristic selection is as follows: calculating the similarity between the 2D heat map and each of the Ns key point score maps, and selecting the key point score map with the highest similarity as the key point spatial position score distribution map of the corresponding key point; The integration selection is: connecting the Ns key point score maps and applying the convolution operation Conv to perform signal modulation to obtain a key point spatial position score distribution map, which is expressed as: S = Conv(Concat(S′)) Among them, S represents the key point spatial position score distribution map, S′ represents the Ns key point score map, and Concat represents connection; The attention-based selection is: obtaining a queryable vector based on the Ns key point score maps, and outputting a key point spatial position score distribution map based on the queryable vector, and its expression is: S i =Flatten(S′), Q i =q i W q ,K i =S i W k ,V i =S i W v , Among them, S i represents the i-th key point score map, Flatten represents the flattening operation, and W q , W k , W v and W a represents the learnable parameter matrix, Q i , K i and V i represents the query, key and value of the i-th key point, H represents the feature height, W represents the feature width, Indicates transpose.

8. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 7, characterized in that: The construction of the key point spatial position score distribution map also includes: The loss is calculated based on the 2D heat map and the key point spatial position score distribution map, and the spatial loss used to construct the key point spatial position score distribution map is expressed as: in, represents the spatial loss, S represents the key point spatial position score distribution map, H target Represents a 2D heat map.

9. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 1, characterized in that: The method for obtaining the key point heat map is: cascading the key point spatial position score distribution map and the image features, and the expression is: Among them, F o represents the key point heat map, F vis represents the image features, and S represents the key point spatial position score distribution map.

10. The method for multimodal animal posture estimation based on probabilistic representation learning of visual text according to claim 1, characterized in that: The method for obtaining the coordinates of key points is: Multiplying the coordinates of the key point heat map by a preset ratio to restore it to the size of the animal image; The coordinates of the key points are obtained based on the key point heat map after the size is restored. The expression is: Kpt k s1·argmax 1≤e≤H，1≤f≤W F o (e,f),k1,...,K, Among them, Kpt k Represents the coordinate sequence of K key points, s1 represents the preset ratio, argmax represents the operation, k represents the kth key point, K represents the total number of key points, H represents the height of the feature, W represents the width of the feature, e and f represent the height and width of the key heat map respectively.

Images