A Visual Text Multimodal Animal Pose Estimation Method Based on Probabilistic Representation Learning

Abstract

This invention relates to a visual text multimodal animal pose estimation method based on probabilistic representation learning. The method includes: acquiring an image dataset and extracting image features; constructing an attribute description set, converting the category names of the text keypoints, and obtaining a cue template; constructing a multivariate Gaussian distribution for each keypoint based on the cue template; constructing a keypoint spatial location score distribution map based on the multivariate Gaussian distribution and image features; obtaining a keypoint heatmap based on the keypoint spatial location score distribution map and image features; and estimating animal pose based on the keypoint heatmap to obtain keypoint coordinates. Compared with existing technologies, this invention solves the problem of inaccurate pose estimation caused by a single cue template and uneven distribution of data from various animal types in existing technologies.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and behavior analysis, and in particular to a visual text multimodal animal pose estimation method based on probabilistic representation learning. Background Technology

[0002] Animal pose estimation (APE) is a fundamental task in behavior analysis, aiming to detect and locate the coordinates of key points on the body of a target in images / videos. However, acquiring data by tracking wild animals requires significant human and material resources, and complex factors such as varying lighting conditions and occlusion in the wild result in very limited animal pose data. Furthermore, the diverse range of animal species presents problems of high variance and uneven distribution, greatly increasing the difficulty of animal pose estimation. In recent years, with the development of Vision-and-Language Models (VLMs), especially research on Contrastive Learning Language-Image Pre-training (CLIP), the knowledge of pre-trained image-text pairs can be transferred to downstream visual tasks, demonstrating strong generalization ability in zero-shot experiments. Multimodal animal pose estimation methods based on text interaction, as a new paradigm, can leverage the prior knowledge of large models to alleviate the problems of limited data and high variance in multi-species animal pose estimation. However, designing cue learning methods for pose estimation of multiple animal species remains a challenge. Existing methods suffer from two main drawbacks: 1) Multi-species animal pose estimation faces significant data variance, but existing multimodal pose estimation methods all use discriminative cue templates. This discriminative learning approach learns representations biased towards the distribution of the training data and cannot adapt to animal data from unseen categories, thus often resulting in low accuracy in multi-species animal pose estimation; 2) Due to the variability of pose changes and complex wild environments, even animals of the same species, and even different species, exhibit significant differences. These differences are reflected in various attributes, such as color, texture, and location, making it impractical to comprehensively summarize them using a single textual description. Furthermore, multi-species animal data also faces the problem of uneven distribution, i.e., the long-tail phenomenon, which reflects the non-negligible natural state of data distribution in the real world.

[0003] Therefore, relying solely on image features or a single text prompt template cannot provide sufficient information to overcome the multi-species pose estimation problem. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a visual text multimodal animal pose estimation method based on probabilistic representation learning. It adopts a probabilistic cue learning approach to model text cues, which solves the problems of single discriminative text cue templates and difficulty in handling uneven distribution of multiple animal species in the existing technology, thereby improving the generalization of the model.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A visual text-based multimodal animal pose estimation method based on probabilistic representation learning includes the following steps:

[0007] Acquire an image dataset and extract image features, wherein the image dataset includes multiple text keypoint category names;

[0008] Construct an attribute description set, transform the text key point category names, and randomly insert the transformed text key point category names into the attribute description set to obtain multiple prompt templates;

[0009] A multivariate Gaussian distribution for each key point is constructed based on the aforementioned multiple prompt templates;

[0010] A keypoint spatial location score distribution map is constructed based on the aforementioned multivariate Gaussian distribution and image features.

[0011] Based on the key point spatial location score distribution map and image features, a key point heatmap is obtained. Animal posture estimation is performed based on the key point heatmap to obtain key point coordinates.

[0012] As a preferred technical solution, the method for constructing the multivariate Gaussian distribution is as follows:

[0013] Based on the aforementioned prompt template, prompt features are obtained, and differential constraints are applied to these prompt features. The expression for these features is as follows:

[0014]

[0015] Among them, P i This represents the cue feature corresponding to the i-th key point. This represents the transpose of the cue feature, where I indicates that the dimension is N. p The unit diagonal matrix, N p This indicates the number of prompt templates, where K represents the number of key points;

[0016] Calculate the mean of a multivariate Gaussian distribution based on the cue features after applying differential constraints;

[0017] The image features and prompt features are interacted to obtain the variance of the multivariate Gaussian distribution;

[0018] A multivariate Gaussian distribution is constructed based on the variance and mean.

[0019] As a preferred technical solution, the method for calculating the mean is as follows:

[0020] The aforementioned cue features are processed using a self-attention mechanism, and its expression is as follows:

[0021]

[0022]

[0023] in, This represents the query generated by the t-th cue feature of the i-th key point. Let represent the key generated by the t-th cue feature of the i-th keypoint. W represents the value generated by the t-th cue feature of the i-th keypoint. q W k and W v Represents the learnable parameter matrix, This represents the t-th cue feature of the i-th keypoint. Both MLP and SA represent computational operations. This represents the mean.

[0024] As a preferred technical solution, the method for calculating the variance is as follows:

[0025] The image features and cue features are interacted using a cross-attention mechanism, and the variance is calculated. The expression for this variance is:

[0026]

[0027]

[0028] in, v represents the query generated by the t-th cue feature of the i-th key point. k v represents the key generated by the t-th cue feature of the i-th keypoint. v W represents the value generated by the t-th cue feature of the i-th keypoint. q ′,W k ′ and W′ v Both represent the learnable parameter matrix, F vis Indicates visual characteristics, The variance is represented by MLP and CA, which both represent computational operations.

[0029] As a preferred technical solution, the method for constructing the spatial location score distribution map of key points is as follows:

[0030] The sampling cue features are obtained by sampling the multivariate Gaussian distribution using the reparameter method;

[0031] Based on the animal image, the true coordinate mask of the key points is obtained, and the image features are sampled based on the mask to obtain sampled image features.

[0032] The contrast loss between the sampling cue features and the sampling image features is calculated, and its expression is as follows:

[0033]

[0034]

[0035] Among them, F keypoint Represents the features of the sampled image. Let M represent the sampling cue features, and M represent the text-visual similarity matrix. label M is a diagonal matrix. T Let CE be the transpose matrix, and let CE denote the calculation of cross-loss entropy.

[0036] Based on the aforementioned contrast loss, feature alignment is performed between cue features and image features;

[0037] The image features are mapped and fused with the feature alignment results to obtain multiple keypoint score maps, and the number of keypoint score maps is greater than the number of keypoints. The multiple keypoint score maps are then spatially aligned to obtain a keypoint spatial location score distribution map.

[0038] As a preferred technical solution, the method for obtaining the sampling prompt features is as follows:

[0039] The sampling cue feature is obtained by reparameter sampling of the Gaussian distribution of the variable, and its expression is:

[0040]

[0041] in, Indicates sampling prompt features, This represents the mean. denoted by variance, and ∈ denotes a variable extracted from a normal distribution;

[0042] Based on the aforementioned sampling cue features and difference constraints, the divergence value is calculated to guide the next sampling. Its expression is:

[0043]

[0044] in, Represents the divergence value. N represents the difference constraint. pIndicates the number of prompt templates. Represents a multivariate Gaussian distribution. This represents the t-th cue feature of the i-th key point. This represents the t-th cue feature of the i-th keypoint in a multivariate Gaussian distribution. This represents a standard normal distribution.

[0045] As a preferred technical solution, the spatial alignment processing of the multiple key point score maps includes the following steps:

[0046] N samples were randomly selected from the multiple key point score maps. s Key point score chart;

[0047] A 2D heatmap is constructed based on the aforementioned multivariate Gaussian distribution, with the mean point defined as the coordinate point and the variance as a constant.

[0048] Based on the keypoint score map and 2D heatmap obtained from sampling, spatial alignment processing is performed using heuristic selection, ensemble selection, or attention-based selection to obtain a keypoint spatial location score distribution map.

[0049] The heuristic selection method is as follows: calculating the 2D heatmap and the N... s The similarity of each keypoint score map in the keypoint score map is used to select the keypoint score map with the highest similarity as the keypoint spatial location score distribution map of the corresponding keypoint.

[0050] The integration selection is: connecting the N s The keypoint score map is obtained by applying convolution operation (Conv) to the signal modulation, and the expression is as follows:

[0051] S = Conv(Concat(S′))

[0052] Where S represents the spatial location score distribution map of key points, and S′ represents N s A keypoint score graph, where Concat represents a connection;

[0053] The attention-based selection is: based on the N mentioned above. s The keypoint score map is used to obtain queryable vectors, and based on these queryable vectors, a keypoint spatial location score distribution map is output, the expression of which is:

[0054] S i =Flatten(S′),

[0055] Q i =q i W q ,K i =Si W k V i =S i W v ,

[0056]

[0057] Among them, S i Let W represent the score map of the i-th key point, Flatten denotes the flattening operation, and W... q W k W v and W a Let Q represent the learnable parameter matrix. i K i and V i This represents the query, key, and value of the i-th key point, where H represents feature height and W represents feature width. This indicates transpose.

[0058] As a preferred technical solution, the construction of the key point spatial location score distribution map further includes:

[0059] The loss is calculated based on the aforementioned 2D heatmap and keypoint spatial location score distribution map. This loss, used to construct the keypoint spatial location score distribution map, is expressed as follows:

[0060]

[0061] in, S represents the spatial loss, and H represents the spatial location score distribution map of key points. target This represents a 2D heat map.

[0062] As a preferred technical solution, the method for obtaining the key point heatmap is as follows: concatenating the key point spatial location score distribution map and the image features, the expression of which is:

[0063]

[0064] Among them, F o Represents a heatmap of key points, F vis S represents the image features, and S represents the spatial location score distribution map of key points.

[0065] As a preferred technical solution, the method for obtaining the coordinates of key points is as follows:

[0066] Multiply the coordinates of the key point heatmap by a preset ratio to restore it to the size of the animal image;

[0067] The coordinates of key points are obtained from the restored key point heatmap, and the expression is:

[0068] Kpt k =s1·argmax 1≤e≤H,1≤f≤W F o (e,f),k=1,…,K,

[0069] Among them, Kpt k Let s1 represent the coordinate sequence of K key points, argmax represent the operation, k represent the kth key point, K represent the total number of key points, H represent the height of the feature, W represent the width of the feature, and e and f represent the height and width of the key heatmap, respectively.

[0070] Compared with the prior art, the present invention has the following advantages:

[0071] 1) This invention constructs a multivariate Gaussian distribution for each key point and uses a probabilistic cue learning method to model text prompts. This solves the problems in the prior art where the large model learns a representation that is biased towards the distribution of the training data and has low prediction accuracy due to the single discriminative text prompt template and the uneven distribution of data from various types of animals. This improves the generalization and prediction accuracy of the model.

[0072] 2) This invention introduces a set of learnable attribute descriptions to express attributes that are independent of key point categories, such as position, size, and color, in order to represent the common attributes of multi-species targets. In order to more effectively describe the diverse and information-rich attributes of the entire key point, the key point category name is randomly inserted into the prompt template to increase the diversity of the prompt template and make the prompt template more in line with the natural language description of the key point.

[0073] 3) In order to generate key point descriptions of different animals more flexibly, this invention models the attribute distribution of each key point as a different normal distribution, and then extracts sampling features from multiple normal distributions to achieve generalized multi-species animal pose estimation. Considering that pose estimation is a dense prediction task, this application also establishes a spatial relationship between the sampled prompt template and image features to achieve more accurate pose estimation. Attached Figure Description

[0074] Figure 1 This is a flowchart of the invention process of this application;

[0075] Figure 2 This is a structural diagram of the large model of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0077] This embodiment provides a visual text multimodal animal pose estimation method based on probabilistic representation learning. The method flow of this invention is as follows: Figure 1 As shown, by probabilistically modeling the cue templates of a large model, the problem of inaccurate pose estimation caused by a single cue template is improved. Furthermore, feature alignment and spatial alignment are used to tightly link the cue templates and image features, enhancing model performance. The improved model structure is shown below. Figure 2 As shown, this method improves prediction accuracy and includes the following steps:

[0078] S1. Image Feature Extraction:

[0079] S11. Obtain the image dataset, which includes K text keypoint category names. k i This represents the category name of the i-th key point, such as k1 for left eye and k2 for right eye.

[0080] S12. Extract image features F using the CLIP pre-trained image encoder θ(·). vis ∈R C×H×W , where H, W, and C represent the height, width, and number of channels of the feature, respectively.

[0081] S2, Prompt Template Construction:

[0082] S21. Use CLIP's tokenizer to classify key points. Perform the transformation and randomly initialize N. p A learnable array of length L Used to represent a set of attribute descriptions for an image.

[0083] S22. Randomly insert the converted text key point category names into the attribute description set to obtain multiple prompt templates. Where i and y represent the t-th attribute description of the i-th key point, respectively.

[0084] S3. Construction of a multivariate Gaussian distribution:

[0085] Step S3 involves probabilistically modeling the keypoint cue templates. This aims to model the different attribute descriptions of the keypoints to overcome the large data variance problem caused by pose estimation for multiple animal species. This includes:

[0086] S31. Obtain cue features based on cue templates using the CLIP pre-trained text encoder φ(·). Where N p C represents both the number of learnable tokens and the number of prompt templates. emb To determine the feature dimension, and to ensure that the prompt template represents the different attributes of the key points (i.e., constructing mutually independent Gaussian distributions), a difference constraint is imposed on the prompt features, expressed as:

[0087]

[0088] Among them, P i This represents the cue feature corresponding to the i-th key point. This represents the transpose of the cue feature, where I indicates that the dimension is N. p The unit diagonal matrix, N p This represents the number of learnable tokens, and K represents the number of key points.

[0089] After applying the difference constraint, cue features belonging to the same attribute will be more similar, while cue features of different attributes will be more different.

[0090] S32. Based on the cue features after applying differential constraints, the cue features are processed using a self-attention mechanism to calculate the mean of a multivariate Gaussian distribution. Its expression is:

[0091]

[0092]

[0093] in, This represents the query generated by the t-th cue feature of the i-th key point. Let represent the key generated by the t-th cue feature of the i-th keypoint. W represents the value generated by the t-th cue feature of the i-th keypoint. q W k and W v Represents the learnable parameter matrix, This represents the t-th cue feature of the i-th keypoint. Both MLP and SA represent multilayer perceptron and layer regularization operations, respectively. This represents the mean.

[0094] S33. Utilize the cross-attention mechanism to interact image features and cue features, introducing uncertainty into image features, thereby learning the variance of a multivariate Gaussian distribution and obtaining the variance of the multivariate Gaussian distribution. Its expression is:

[0095]

[0096]

[0097] in, v represents the query generated by the t-th cue feature of the i-th key point. k v represents the key generated by the t-th cue feature of the i-th keypoint. v W represents the value generated by the t-th cue feature of the i-th keypoint. q ′,W k ′ and W v ′ represents the learnable parameter matrix, F vis Indicates visual characteristics, The variance is represented by MLP and CA, which both represent computational operations.

[0098] S34. Construct a multivariate Gaussian distribution based on variance and mean.

[0099] S4. Construct a spatial location score distribution map of key points:

[0100] To ensure the differentiability of the learning process, a reparameterization technique is used to sample N from a multivariate Gaussian distribution. s Key features Furthermore, key point features are obtained based on image feature sampling. Contrast loss is calculated to align visual and textual features, and simultaneously with the mapped image features. Cross-modal information fusion is performed to obtain a score map S∈R of the spatial locations of key points. K ×H×W This score chart reflects the role of text prompts in spatial positioning.

[0101] Because direct sampling from a multivariate Gaussian distribution is non-differentiable, a reparameterization operation is needed to obtain cue features. This allows for text-visual alignment at the feature level and the establishment of a connection between the text cue and the spatial location of key points, including:

[0102] S41. Using the reparameterization method to sample a multivariate Gaussian distribution, sampling cue features are obtained. Specifically, the expression for obtaining sampling cue features by reparameterization of the Gaussian distribution is as follows:

[0103]

[0104] in, Indicates sampling prompt features, This represents the mean. denoted by variance, and ∈ denotes a variable extracted from a normal distribution;

[0105] To prevent the variance of the fitted data from approaching zero during network learning, this embodiment introduces an additional KL divergence as a regularization term. The divergence value is calculated based on sampling cue features and difference constraints to guide the next sampling iteration. Its expression is:

[0106]

[0107] in, Represents the divergence value. N represents the difference constraint. p Indicates the number of prompt templates. Represents a multivariate Gaussian distribution. This represents the t-th cue feature of the i-th key point. This represents the t-th cue feature of the i-th keypoint in a multivariate Gaussian distribution. This represents a standard normal distribution.

[0108] S42. Corresponding to the cue features, it is desired that the image features of specific key points are close to their corresponding text cue features and far away from the cue features of other key points, and vice versa. This enhances the discriminative ability of text features and image features and promotes feature alignment. Specifically, the true coordinate mask of key points is obtained based on the animal image, and the image features are sampled based on the mask to obtain sampled image features, expressed as:

[0109]

[0110] Among them, F keypoint GS represents the features of the sampled image.

[0111] S43. Calculate the contrast loss between the sampling cue features and the sampling image features, expressed as follows:

[0112]

[0113]

[0114] Among them, F keypoint Represents the features of the sampled image. Let M represent the sampling cue features, and M represent the text-visual similarity matrix. label M is a diagonal matrix. T Let CE be the transpose matrix, and let CE denote the calculation of cross-loss entropy.

[0115] S44. Based on the contrast loss, perform feature alignment between the cue features and the image features.

[0116] S45. Mapping the image features and fusing them with the feature alignment results across modalities to obtain multiple keypoint score maps, where the number of keypoint score maps is greater than the number of keypoints. Spatial alignment is then performed on the multiple keypoint score maps to obtain a keypoint spatial location score distribution map, including the following steps:

[0117] S451. Randomly sample N from multiple keypoint score maps. s Key Point Score Chart

[0118] S452. Constructing a 2D heatmap H based on a multivariate Gaussian distribution. target ∈R K×H×W The mean point is defined as the coordinate point, and the variance is a constant.

[0119] S453. Based on the key point score map and 2D heat map obtained from sampling, spatial alignment processing is performed using heuristic selection, ensemble selection or attention-based selection to obtain the key point spatial location score distribution map.

[0120] Among them, heuristic selection: calculating the 2D heatmap and the aforementioned N s The similarity of each keypoint score map in the keypoint score map is used to select the keypoint score map with the highest similarity as the keypoint spatial location score distribution map of the corresponding keypoint.

[0121] Ensemble selection: The ensemble selection strategy assumes that all features contain contextual information. To prevent the loss of textual clues, it connects N... s The keypoint score map is obtained by applying convolution operation (Conv) to the signal modulation, and the expression is as follows:

[0122] S = Conv(Concat(S′))

[0123] Where S represents the spatial location score distribution map of key points, and S′ represents N s A keypoint score graph, where Concat represents a connection;

[0124] Attention-based selection: Introducing an additional learnable query vector q i ∈R K×(HW) Based on N s The keypoint score map is used to obtain queryable vectors. An attention mechanism is then used to output a keypoint spatial location score distribution map based on these queryable vectors. The expression for this map is:

[0125] S i=Flatten(S′),

[0126] Q i =q i W q ,K i =S i W k V i =S i W v ,

[0127]

[0128] Among them, S i This represents the score map of the i-th key point, where flatten represents the flattening operation, and W... q W k W v and W a Let Q represent the learnable parameter matrix. i K i and V i This represents the query, key, and value of the i-th key point, where H represents feature height and W represents feature width. This indicates transpose.

[0129] S454. After obtaining S, S will be reshaped into a 2D feature map.

[0130] S455. Calculate the loss based on the 2D heatmap and the keypoint spatial location score distribution map. This loss serves as the spatial loss for constructing the keypoint spatial location score distribution map, and its expression is:

[0131]

[0132] in, S represents the spatial loss, and H represents the spatial location score distribution map of key points. target This represents a 2D heat map.

[0133] S5. Obtain the coordinates of key points:

[0134] S51. Concatenate the key point spatial location score distribution map and the image features, the expression of which is:

[0135]

[0136] F o Represents a heatmap of key points, F vis Let S represent the image features, and let S represent the spatial location score distribution map of key points. Here, HEAD represents a general key point detection head, which consists of multiple deconvolution and multilayer perceptron layers.

[0137] S52. Following the common pose estimation post-processing method, apply a simple argmax operation to the heatmap of each keypoint, and multiply the coordinates by a scale s1 to restore the input image size, obtaining the coordinates Kpt of K keypoints. k Its expression is:

[0138] Kpt k =s1·argmax 1≤e≤H,1≤f≤W F o (e,f),k=1,…,K,

[0139] Among them, Kpt k Let s1 represent the coordinate sequence of K key points, argmax represent the operation, k represent the kth key point, K represent the total number of key points, H represent the height of the feature, W represent the width of the feature, and e and f represent the height and width of the key heatmap, respectively.

[0140] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual text-based multimodal animal pose estimation method based on probabilistic representation learning, characterized in that, The method includes: Acquire an image dataset and extract image features, wherein the image dataset includes multiple text keypoint category names; Construct an attribute description set, transform the text key point category names, and randomly insert the transformed text key point category names into the attribute description set to obtain multiple prompt templates; A multivariate Gaussian distribution is constructed for each key point based on the multiple prompt templates mentioned above. The method for constructing the multivariate Gaussian distribution is as follows: Based on the aforementioned prompt template, prompt features are obtained, and differential constraints are applied to these prompt features. The expression for these features is as follows: Among them, P i This represents the cue feature corresponding to the i-th key point. This represents the transpose of the cue feature, where I indicates that the dimension is N. p The unit diagonal matrix, N p This indicates the number of prompt templates, where K represents the number of key points; Calculate the mean of a multivariate Gaussian distribution based on the cue features after applying differential constraints; The image features and prompt features are interacted to obtain the variance of the multivariate Gaussian distribution; A multivariate Gaussian distribution is constructed based on the variance and mean described above; Based on the aforementioned multivariate Gaussian distribution and image features, a keypoint spatial location score distribution map is constructed. The method for constructing the keypoint spatial location score map is as follows: The sampling cue features are obtained by sampling the multivariate Gaussian distribution using the reparameter method; The true coordinate mask of key points is obtained from animal images, and the image features are sampled based on the mask to obtain sampled image features. The contrast loss between the sampling cue features and the sampling image features is calculated, and its expression is as follows: Among them, F keypoint Represents the features of the sampled image. Let M represent the sampling cue features, and M represent the text-visual similarity matrix. label M is a diagonal matrix. T Let M be the transpose of the text-visual similarity matrix, and CE denote the calculation of cross-loss entropy; Based on the aforementioned contrast loss, feature alignment is performed between cue features and image features; The image features are mapped and fused with the feature alignment results to obtain multiple key point score maps, and the number of key point score maps is greater than the number of key points. The multiple key point score maps are then spatially aligned to obtain a key point spatial location score distribution map. Based on the key point spatial location score distribution map and image features, a key point heatmap is obtained. Animal posture estimation is performed based on the key point heatmap to obtain key point coordinates.

2. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 1, characterized in that, The method for calculating the mean is as follows: The aforementioned cue features are processed using a self-attention mechanism, and its expression is as follows: in, The cue feature representing the i-th keypoint is based on the learnable parameter matrix W. q The generated query, The cue feature representing the i-th keypoint is based on the learnable parameter matrix W. k The generated key, The cue feature representing the i-th keypoint is based on the learnable parameter matrix W. v The generated value, W q W k and W v Represents the learnable parameter matrix, Let represent the t-th cue feature of the i-th keypoint, MLP represent multilayer perceptron operation, and SA represent arithmetic operation. This represents the mean.

3. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 1, characterized in that, The method for calculating the variance is as follows: The image features and cue features are interacted using a cross-attention mechanism, and the variance is calculated. The expression for this variance is: in, The cue feature representing the i-th keypoint is based on the learnable parameter matrix W. q The generated query, v k Indicates based on image features F vis The generated key, v v Indicates based on image features F vis The generated value, W′ q ,W′ k and W′ v Both represent the learnable parameter matrix, F vis Representing image features, Variance is represented by MLP and CA, which both represent computational operations, with MLP representing multilayer sensing operations. Let t represent the cue feature of the i-th key point.

4. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 1, characterized in that, The method for obtaining the sampling prompt features is as follows: The sampling cue feature is obtained by reparameter sampling of the Gaussian distribution of the variable, and its expression is: in, Indicates sampling prompt features, This represents the mean. denoted by variance, and ∈ denotes a variable extracted from a normal distribution; Based on the aforementioned sampling cue features and difference constraints, the divergence value is calculated to guide the next sampling. Its expression is: in, Represents the divergence value. N represents the difference constraint. p Indicates the number of prompt templates. Represents a multivariate Gaussian distribution. This represents the t-th cue feature of the i-th key point. Indicates sampling prompt features, It represents a standard normal distribution; KL represents the divergence calculation.

5. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 1, characterized in that, Spatial alignment of the multiple keypoint score maps includes the following steps: N samples were randomly selected from the multiple key point score maps. s Key point score chart; A 2D heatmap is constructed based on the aforementioned multivariate Gaussian distribution, with the mean point defined as the coordinate point and the variance as a constant. Based on the keypoint score map and 2D heatmap obtained from sampling, spatial alignment processing is performed using heuristic selection, ensemble selection, or attention-based selection to obtain a keypoint spatial location score distribution map. The heuristic selection method is as follows: calculating the 2D heatmap and the N... s The similarity of each keypoint score map in the keypoint score map is used to select the keypoint score map with the highest similarity as the keypoint spatial location score distribution map of the corresponding keypoint. The integration selection is: connecting the N s The keypoint score map is obtained by applying convolution operation (Conv) to the signal modulation, and the expression is as follows: S = Conv(Concat(S′)) Where S represents the spatial location score distribution map of key points, and S′ represents N s A keypoint score graph, where Concat represents a connection; The attention-based selection is: based on the N mentioned above. s The keypoint score map is used to obtain queryable vectors, and based on these queryable vectors, a keypoint spatial location score distribution map is output, the expression of which is: S i =Flatten(S′), Q i =q i W q ,K i =S i W k ,V i =S i W v , Among them, S i Let W represent the score map of the i-th key point, Flatten denotes the flattening operation, and W... q W k W v and W a Let Q represent the learnable parameter matrix. i K i and V i This represents the query, key, and value of the i-th key point, where H represents feature height and W represents feature width. Key K i transpose of q; i This represents a learnable query vector.

6. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 5, characterized in that, The construction of the key point spatial location score distribution map also includes: The loss is calculated based on the aforementioned 2D heatmap and keypoint spatial location score distribution map. This loss, used to construct the keypoint spatial location score distribution map, is expressed as follows: in, S represents the spatial loss, and H represents the spatial location score distribution map of key points. target This represents a 2D heat map.

7. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 1, characterized in that, The method for obtaining the key point heatmap is as follows: the key point spatial location score distribution map and the image features are concatenated, and the expression is: Among them, F o Represents a heatmap of key points, F vis S represents the image features, and HEAD represents the spatial location score distribution map of key points.

8. The visual text multimodal animal pose estimation method based on probabilistic representation learning according to claim 7, characterized in that, The method for obtaining the coordinates of key points is as follows: Multiply the coordinates of the key point heatmap by a preset ratio to restore it to the size of the animal image; The coordinates of key points are obtained from the restored key point heatmap, and the expression is: Kpt k s1·argmax 1≤e≤H,1≤f≤W F o (e,f),k1,...,K, Among them, Kpt k Let s1 represent the coordinate sequence of K key points, argmax represent the operation, k represent the kth key point, K represent the total number of key points, H represent the height of the feature, W represent the width of the feature, and e and f represent the height and width of the key heatmap, respectively.

Images