Human whole-body pose keypoint prediction method based on visual-linguistic hierarchical alignment relationship

Abstract

The application provides a human whole body posture key point prediction method based on visual-language hierarchical alignment relationship. The method comprises the following steps: a pyramid visual Transformer network is used to perform multi-level feature extraction on a target image to obtain a multi-level feature pyramid; a multi-modal text encoder is used to extract a text feature vector from a learned prompt and a predefined category label, and a text semantic feature space is obtained based on the text feature vector; an optimal transport algorithm is used to perform feature vector alignment operation between the feature pyramid and the text semantic feature space in the same level to obtain a visual-language latent space; a feature pyramid prediction network is used to perform feature vector mapping operation on the visual-language latent space to generate a heat map of the human whole body posture key points in the target image, and position coordinate information of each human whole body posture key point in the target image is predicted based on the heat map of the human whole body posture key points.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for predicting key points of human body posture based on visual-linguistic hierarchical alignment, as well as an electronic device and storage medium. Background Technology

[0002] Full-body pose keypoint prediction technology has been widely applied in fields such as human-computer interaction, film industry, action recognition, virtual reality, and augmented reality. Unlike prediction tasks that only target certain key points on the body, full-body pose estimation presents at least the following challenges: First, the scale differences between different body parts: the model experiences performance degradation for both extremely large and extremely small scale targets during prediction (for example, in real life, it is often difficult to clearly distinguish distant objects); second, the semantic ambiguity at small scales stemming from these scale differences: in prediction tasks, for small-scale regions such as hands and faces, the amount of information obtainable is limited due to the small scale, yet many keypoints need to be labeled, making it more difficult for technicians to distinguish specific details, such as identifying which finger is the index finger or the middle finger in a hand; third, the complex hierarchical structure of the human body: requiring precise localization of fine-grained keypoints.

[0003] While significant progress has been made in full-body pose estimation techniques, such as ZoomNAS, which performs high-resolution scaling of cropped hand and facial landmark features to obtain more detailed facial and hand information, this strategy successfully mitigates scale issues and small-scale semantic ambiguity. However, it also presents some potential negative impacts. Specifically, cropping results in the loss of human contextual information, which may disrupt the potential connections between the torso and limbs. Furthermore, super-resolution operations may introduce additional biases, causing the model to learn suboptimal results from noisy data. Therefore, the ZoomNAS approach suffers from the following technical problems: first, cropping images leads to the loss of human contextual information, thereby disrupting the potential connections between the torso and limbs; second, super-resolution operations may introduce additional biases, causing the model to learn suboptimal results from noisy data. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method and system for predicting key points of human body posture based on visual-linguistic hierarchical alignment, an electronic device and a storage medium, in order to solve at least one of the above problems.

[0005] According to a first aspect of the present invention, a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment is provided, comprising:

[0006] The human body full-body posture key point prediction model is trained, and the information used for cue learning is optimized during the model training process to obtain the trained human body full-body posture key point prediction model and the learned cue. The trained human body full-body posture key point prediction model includes a pyramid visual Transformer network, a multimodal text encoder and a feature pyramid prediction network. The learned cue is a vector used to represent the human body full-body posture key points.

[0007] The target image is extracted using a pyramid visual Transformer network to obtain a multi-level feature pyramid. The image features at different levels of the multi-level feature pyramid have different sizes, different resolutions and different image granularities.

[0008] A multimodal text encoder is used to extract text feature vectors from learned prompts and predefined category labels. Based on the extracted text feature vectors, a multi-level text semantic feature space with subordinate relationships is obtained. The text semantic feature space has the same number of levels as the feature pyramid.

[0009] By using a predefined optimal transfer algorithm, the feature pyramid and the text semantic feature space are aligned one-to-one with the feature vectors at the same level to obtain a visual-language latent space that integrates visual feature vectors and text semantic feature vectors.

[0010] The feature pyramid prediction network is used to perform feature vector mapping on the visual-language latent space to generate a heat map of the key points of the human body's full-body posture in the target image. Based on the heat map of the key points of the human body's full-body posture, the position coordinate information of each key point of the human body's full-body posture in the target image is predicted.

[0011] According to an embodiment of the present invention, the above-mentioned training of the human body full-body posture key point prediction model, and the optimization of the information used for cue learning during the model training process, to obtain the trained human body full-body posture key point prediction model and the learned cue, includes:

[0012] A human body full-body posture key point prediction model is constructed based on a visual language model, and a pyramid visual Transformer network is used to extract multi-level features from image samples to obtain a feature sample pyramid.

[0013] Based on the hierarchical attribute information of the multi-level feature sample pyramid and the category label samples, the information used for cue learning is initialized hierarchically to obtain the initial cue.

[0014] A multimodal text encoder is used to extract text semantic features and inject text associations into the initial prompts to obtain a text semantic feature sample space. During the extraction and injection operations, the language loss value is calculated.

[0015] A cost function is defined using the cosine similarity between the feature sample pyramid and the text semantic feature sample space, and an optimization objective of an entropy-constrained Lagrange multiplier is constructed using the cost function.

[0016] Based on the optimization objective of the entropy-constrained Lagrange multipliers, the optimal transfer algorithm is used to perform a one-to-one alignment operation of the feature sample pyramid and the text semantic feature sample space between the same level by minimizing the total distance operation, so as to obtain the visual-language latent space sample.

[0017] The feature pyramid prediction network is used to perform feature vector mapping on visual-language latent space samples to generate heat map samples of key points of human body posture in the target image, and the heat map loss value is calculated during the heat map sample generation process.

[0018] Based on predefined weights, the total training loss is calculated using language loss and heatmap loss. The total training loss is then used to update the parameters of the human body full-body posture key point prediction model and optimize the information used for cue learning. This process is repeated multiple times until the preset training conditions are met, resulting in a trained human body full-body posture key point prediction model and well-learned cues.

[0019] According to an embodiment of the present invention, the above-described method of using a pyramid visual Transformer network to perform multi-level feature extraction on image samples to obtain a feature sample pyramid includes:

[0020] Image samples are segmented using the embedding layer of the pyramid visual Transformer network, the resulting image blocks are arranged into a sequence, and the sequence is transformed into a two-dimensional matrix through a linear mapping operation.

[0021] The two-dimensional matrix and the positional information of the sequence are input into the pyramid visual Transformer network, and the Transformer encoder of the pyramid visual Transformer network is used to extract multi-level feature sample pyramids from image samples.

[0022] According to an embodiment of the present invention, the above-described multimodal text encoder performs text semantic feature extraction and text attachment relation injection operations on the initial prompt to obtain a text semantic feature sample space. During the extraction and injection operations, the calculation of the language loss value includes:

[0023] A multimodal text encoder is used to extract text semantic features from the initial prompt, resulting in multi-level text semantic feature samples.

[0024] Based on prior knowledge of textual relationships, a multimodal text encoder is used to inject textual subordinate relationships by mapping feature vectors between adjacent levels of textual semantic feature samples, resulting in a multi-level textual semantic feature sample space with subordinate relationships.

[0025] The language loss value is calculated in the text semantic feature sample extraction operation and the text affiliation relation injection operation using the language loss function based on the L1 paradigm.

[0026] According to an embodiment of the present invention, the language loss function based on the L1 paradigm is as follows:

[0027]

[0028] Among them, E text (*) indicates the multimode encoder shown, t i This indicates the prompt for the i-th level. This indicates the prompt for the (i+1)th level.

[0029] According to an embodiment of the present invention, the above-described optimization objective of defining a cost function using the cosine similarity between the feature sample pyramid and the text semantic feature sample space, and constructing an entropy-constrained Lagrange multiplier using the cost function, includes:

[0030] The cost function is defined using the cosine similarity between the feature sample pyramid and the text semantic feature sample space.

[0031] Based on the method of mapping the feature vectors of different modes to a common vector space in the optimal transmission problem, the optimization objective of the entropy-constrained Lagrange multiplier is obtained by using the cost function, the predefined transmission plan matrix, and the predefined entropy constraint.

[0032] According to an embodiment of the present invention, the optimization objective of the entropy-constrained Lagrange multipliers described above is expressed by the following equation:

[0033]

[0034] Where C represents the cost function, T represents the predefined transmission plan matrix, λh(T) represents the predefined entropy constraint, u represents the vector component in the feature sample pyramid, and v represents the vector component in the text semantic feature sample space.

[0035] According to a second aspect of the present invention, a prediction system for key points of human body posture based on visual-linguistic hierarchical alignment is provided, comprising:

[0036] The model training module is used to train the human body full-body posture key point prediction model and optimize the information used for cue learning during the model training process to obtain the trained human body full-body posture key point prediction model and the learned cue. The trained human body full-body posture key point prediction model includes a pyramid visual Transformer network, a multimodal text encoder and a feature pyramid prediction network. The learned cue is a vector used to represent the human body full-body posture key points.

[0037] The image feature extraction module is used to perform multi-level feature extraction on the target image using the pyramid visual Transformer network to obtain a multi-level feature pyramid. The image features at different levels in the multi-level feature pyramid have different sizes and resolutions.

[0038] The text semantic feature space acquisition module is used to extract text feature vectors from learned prompts and predefined category labels using a multimodal text encoder, and based on the extracted text feature vectors, obtain a multi-level text semantic feature space with subordinate relationships, wherein the text semantic feature space has the same number of levels as the feature pyramid.

[0039] The visual-language latent space acquisition module is used to perform one-to-one alignment of feature vectors between the feature pyramid and the text semantic feature space at the same level using a predefined optimal transfer algorithm, so as to obtain a visual-language latent space that integrates visual feature vectors and text semantic feature vectors.

[0040] The key point prediction module is used to perform feature vector mapping on the visual-language latent space using the feature pyramid prediction network, generate a heat map of the key points of the human body's full-body posture in the target image, and predict the position coordinate information of each key point of the human body's full-body posture in the target image based on the heat map of the key points of the human body's full-body posture.

[0041] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0042] One or more processors;

[0043] Storage device for storing one or more programs.

[0044] Specifically, when one or more programs are executed by one or more processors, the one or more processors execute a method for predicting key points of the human body's full-body posture based on visual-language hierarchical alignment.

[0045] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform a method for predicting key points of human body posture based on a visual-linguistic hierarchical alignment relationship.

[0046] The method for predicting key points of human body posture based on visual-linguistic hierarchical alignment provided by this invention can predict the positions of key points of various parts of the human body from a single image, solving the difficulty of estimating the whole body posture in the prior art. The method for predicting key points of human body posture based on visual-linguistic hierarchical alignment provided by this invention utilizes the excellent algebraic properties of text information to construct the subordinate relationships of text features at different levels, and constructs a text language space based on this. Through the operation of aligning text subordinate relationships, the prior knowledge of text relationships is integrated into the distribution of language features in the form of an algebraic sum, thereby constructing a hierarchically distinct visual-text latent space with excellent properties, which makes the prediction of key points of the whole body, especially key points of small-scale parts, more accurate. Attached Figure Description

[0047] Figure 1 This is a flowchart of a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment according to an embodiment of the present invention.

[0048] Figure 2 This is an architecture diagram of a method for predicting key points of human body posture based on visual-language hierarchical alignment according to an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of a multi-level semantic representation space according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the process of performing multi-level feature extraction using the visual / image branch according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the structure of a PVT (Pyramid Vision Transformer) network according to an embodiment of the present invention;

[0052] Figure 6 This is an illustration of the effect of a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment relationship according to an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of a human body full-body posture key point prediction system based on visual-language hierarchical alignment relationship according to an embodiment of the present invention.

[0054] Figure 8A block diagram of an electronic device adapted to implement a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment is illustrated according to an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0056] Compared to local body keypoint prediction, full-body posture keypoint prediction technology faces numerous technical challenges. Furthermore, existing full-body posture keypoint prediction technologies suffer from low prediction accuracy, slow prediction speed, low efficiency, and poor correlation with predicted body postures. This invention provides a method for predicting full-body posture keypoints based on visual-linguistic hierarchical alignment. This method effectively addresses the scale differences between different body parts and the problem of small-scale semantic ambiguity. By constructing a hierarchical visual-textual latent space with hierarchical relationships, it can effectively model the potential connections between different body parts, achieving more accurate keypoint localization. This solves the problems of scale differences between different body parts, addresses small-scale semantic ambiguity, and ensures the accuracy of the model's prediction task while making the algorithm more efficient and faster.

[0057] To better understand the full-body pose key point prediction technology provided by this invention, the following technical elements need to be clarified: Encoder-Decoder, Transformer model, Prompt Learning, Optimal Transport (OT), PVT (Pyramid Vision Transformer), FPN (Feature Pyramid Networks), and CLIP (Contrastive Language Image Pre-training).

[0058] The encoder-decoder network model is typically used for feature extraction from images and other data. These features can be certain attributes, patterns, or representations of the data. By extracting feature vectors, the model can better learn and understand the data, improve model performance, and reduce computational costs.

[0059] Transformer models excel at handling sequential data, such as in Natural Language Processing (NLP). Unlike traditional Recurrent Neural Networks (RNNs), they utilize self-attention mechanisms to capture long-range dependencies in the input sequence. Composed of an encoder and decoder, they are implemented through multiple layers of self-attention mechanisms and feedforward neural networks. By leveraging self-attention, they can better capture long-range dependencies in sequential data, making them outstanding in sequence prediction tasks (such as language modeling and machine translation). Simultaneously, they take into account information from all positions in the input sequence, contributing to a better understanding and prediction of the context.

[0060] Prompt learning transforms the downstream task into a text generation task by adding "cues" to the input without significantly altering the structure and parameters of the pre-trained language model. Its advantages lie in simplifying complex tasks, allowing the model to focus more on specific domains or tasks, and reducing data requirements and computational costs. Prompt learning can quickly adapt to new domains, is particularly powerful for zero-shot learning and few-shot learning, and provides stronger interpretability.

[0061] Optimal Transport (OT) can be understood as an algorithm that maps two feature representations to a common space, making them comparable or similar in this common space.

[0062] PVT (Pyramid Vision Transformer): This network extracts the feature pyramid of an image. It consists of four Transformer Encoder modules to extract the feature representation of each layer. There are four layers in total, corresponding to four different resolutions: 1 / 4, 1 / 8, 1 / 16, and 1 / 32. This multi-scale feature extraction process alleviates the problems caused by scale differences in full-body pose keypoint prediction tasks.

[0063] FPN (Feature Pyramid Networks): Prediction tasks are performed using feature pyramids extracted by this network. Compared to traditional feature extraction (such as single feature map, Featured image pyramid), it has higher accuracy and efficiency.

[0064] CLIP (Contrastive Language Image Pre-training) is a text-image pre-training model that uses contrastive learning. This model is pre-trained directly using a large amount of internet data and has achieved state-of-the-art performance on many tasks.

[0065] The traditional paradigm for solving multi-scale problems is to construct multi-scale feature representations. However, multi-scale features extracted from visual images do not eliminate the semantic ambiguity of small-scale parts. This invention proposes Affiliation Alignment Networks, called A2Net, which addresses this problem by aligning visual-linguistic hierarchical relationships. Specifically, this invention first inputs a target image and uses a PVT (Pyramid Vision Transformer) network structure to output a feature pyramid with four levels. From the bottom to the top, the extracted features exhibit a transformation from concrete to abstract and from local to global. Because the text modality has the good property of being unaffected by image scale issues and small-scale semantic ambiguity, this invention further constructs a hierarchical linguistic latent space with implicit subordinate relationships through text relation injection, hoping to supplement ambiguous feature information through text cues.

[0066] Finally, this invention employs the Optimal Transport algorithm to align image features at different scales with corresponding text features at different levels, thereby constructing a visual-linguistic latent space free from image scale issues and small-scale semantic ambiguity. Extensive experimental results on two full-body pose estimation datasets demonstrate that the model of this invention achieves convincing performance compared to current state-of-the-art methods.

[0067] The method for predicting key points of human body posture provided by the present invention will be further described in detail below through specific embodiments and accompanying drawings.

[0068] Figure 1 This is a flowchart of a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment according to an embodiment of the present invention.

[0069] like Figure 1 As shown, the above-mentioned method for predicting key points of human body posture based on visual-linguistic hierarchical alignment includes operations S110 to S150.

[0070] In operation S110, the human body full-body posture key point prediction model is trained, and the information used for cue learning is optimized during the model training process to obtain the trained human body full-body posture key point prediction model and the learned cue.

[0071] The trained human body full-body posture key point prediction model includes a pyramid vision Transformer network, a multimodal text encoder, and a feature pyramid prediction network. During the training of the human body full-body posture key point prediction model, the pre-trained multimodal text encoder is called. That is, the parameters of the multimodal text encoder remain unchanged during the training of the human body full-body posture key point prediction model. The learned cue is a vector used to represent the human body full-body posture key points.

[0072] In operation S120, the pyramid vision Transformer network is used to extract multi-level features from the target image, resulting in a multi-level feature pyramid. The image features at different levels in the multi-level feature pyramid have different sizes, resolutions, and granularities.

[0073] The aforementioned Pyramid Vision Transformer network borrows a variant of VisionTransformer, Vit-Adapter.

[0074] The aforementioned multi-level feature pyramid, optionally with four levels, represents the process of change from fine-grained to coarse-grained.

[0075] Different network layers focus on different levels of features. Lower-level network layers are more sensitive to local structures and textures in images. By aligning low-level features with fine-grained text features, the detailed features of the image can be better preserved and emphasized. Higher-level network layers can learn more abstract and advanced semantic information. By imposing coarse-grained semantic constraints in higher-level network layers, the model can better understand the overall scene, semantic relationships, and semantic structure in the image.

[0076] In operation S130, a multimodal text encoder is used to extract text feature vectors from learned prompts and predefined category labels. Based on the extracted text feature vectors, a multi-level text semantic feature space with subordinate relationships is obtained, wherein the text semantic feature space has the same number of levels as the feature pyramid.

[0077] The prior knowledge of the above textual relationships, for example, the composition of the human body includes the head, limbs, and trunk.

[0078] In operation S140, a predefined optimal transfer algorithm is used to perform one-to-one alignment of feature vectors between the feature pyramid and the text semantic feature space at the same level, resulting in a visual-linguistic latent space that fuses visual feature vectors and text semantic feature vectors.

[0079] The aforementioned visual-linguistic latent space aims to enable the extraction of image features to also possess the desirable properties of textual association.

[0080] In operation S150, the feature pyramid prediction network is used to perform feature vector mapping on the visual-language latent space to generate a heat map of the key points of the human body's full-body posture in the target image, and the position coordinate information of each key point of the human body's full-body posture in the target image is predicted based on the heat map of the key points of the human body's full-body posture.

[0081] The feature vector mapping operation of the visual-language latent space using the feature pyramid prediction network includes the feature vector mapping operation using the aligned visual feature vectors (i.e., visual feature representations or visual feature pyramids) in the visual-language latent space of the feature pyramid prediction network.

[0082] The method for predicting key points of human body posture based on visual-linguistic hierarchical alignment provided by this invention can predict the positions of key points of various parts of the human body from a single image, solving the difficulty of estimating the whole body posture in the prior art. The method for predicting key points of human body posture based on visual-linguistic hierarchical alignment provided by this invention utilizes the excellent algebraic properties of text information to construct the subordinate relationships of text features at different levels, and constructs a text language space based on this. Through the operation of aligning text subordinate relationships, the prior knowledge of text relationships is integrated into the distribution of language features in the form of an algebraic sum, thereby constructing a hierarchically distinct visual-text latent space with excellent properties, which makes the prediction of key points of the whole body, especially key points of small-scale parts, more accurate.

[0083] The following detailed description, in conjunction with specific embodiments and accompanying drawings, further illustrates the method for predicting key points of human body posture based on visual-linguistic hierarchical alignment provided by the present invention.

[0084] Figure 2 This is an architecture diagram of a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment according to an embodiment of the present invention.

[0085] like Figure 2As shown, this invention constructs a multi-level visual-language latent space, effectively enabling the features extracted from images to simultaneously possess some desirable textual semantic properties. Here, Text encoder represents a multimodal text encoder, Image encoder represents a visual / image encoder, path embedding represents path embedding, language latent space represents the language latent space, feature pyramid represents the feature pyramid, vision-language latent space represents the visual-language latent space, Multi-level text feature represents multi-level text features, multi-scale image feature represents multi-scale visual / image features, position embedding represents position embedding, element-wise addition represents element addition, language branch represents the language branch, and vision branch represents the vision branch. The architecture diagram of the above method mainly includes three parts: image feature processing, text processing, Optimal Transport (OT) application, and human pose keypoint prediction.

[0086] The image feature processing section (see...) Figure 2 The lower left section borrows a variant of the Vision Transformer module, Vit-Adapter (PVT), to obtain a four-level feature pyramid, showing the process of change from low-level to high-level. The features extracted at each level correspond to different scales and resolutions. Combined with the visualized Feature Pyramid, it is easy to see that the feature resolution decreases from left to right, and the extracted features also reflect the process of change from detailed to abstract.

[0087] Text processing section (see) Figure 2 In the upper left section, learnable prompts and category labels are input into the CLIP Text Encoder to extract text feature vectors. Furthermore, this invention utilizes a text dependency injection operation, incorporating prior knowledge of text relationships into the distribution of language features in the form of an algebraic sum. This allows for the creation of a hierarchical text semantic space with dependency relationships.

[0088] Figure 3 This is a schematic diagram of a multi-level semantic representation space according to an embodiment of the present invention.

[0089] like Figure 3As shown, this invention aims to construct a four-level semantic representation space, where the change from fine-grained to coarse-grained is reflected from left to right, corresponding to the Keypoint level, Region level, Group level, and the final Whole level, respectively. The keypoints in these four levels have certain subordinate relationships, and these four text spaces are also related to the feature pyramid extracted from the image. Figure 3 The lower half is visualized, showing the change from low-level to high-level from left to right, with each layer corresponding to the previous one from left to right.

[0090] In the application of the Optimal Transport (OT) algorithm, the solution to the optimal transport problem is borrowed. The four-layer feature pyramid extracted from the image is aligned with the four-level text feature space with good properties. This constructs a visual-language latent space with distinct levels and good properties, so that the features extracted from the image also have some of the good properties of the text features.

[0091] In the human pose keypoint prediction part, the aligned features are output and the FPN (Feature Pyramid Network) is used to output the heatmap of each keypoint in the whole body, thereby obtaining the position coordinates of each keypoint.

[0092] According to an embodiment of the present invention, the above-mentioned training of the human body full-body posture key point prediction model, and the optimization of the information used for cue learning during the model training process, to obtain the trained human body full-body posture key point prediction model and the learned cue, includes: constructing the human body full-body posture key point prediction model based on a visual language model, and using a pyramid visual Transformer network to perform multi-level feature extraction on image samples to obtain a feature sample pyramid; performing hierarchical initialization of the information used for cue learning according to the hierarchical attribute information and category label samples of the multi-level feature sample pyramid to obtain an initial cue; using a multimodal text encoder to perform text semantic feature extraction and text attachment relation injection operations on the initial cue to obtain a text semantic feature sample space, and calculating the language loss value during the extraction and injection operations, wherein the multimodal text encoder is a pre-trained model; and utilizing the feature sample pyramid and the text semantic feature sample space. The cosine similarity between the two is used to define a cost function, and an optimization objective with entropy constraints and Lagrange multipliers is constructed using the cost function. Based on the optimization objective with entropy constraints and Lagrange multipliers, the optimal transport algorithm is used to perform a one-to-one alignment operation of the feature vectors between the feature sample pyramid and the text semantic feature sample space at the same level by minimizing the total distance, thus obtaining visual-linguistic latent space samples. The feature pyramid prediction network is used to perform feature vector mapping operation on the visual-linguistic latent space samples to generate heat map samples of human body full-body posture key points in the target image, and the heat map loss value is calculated during the heat map sample generation process. According to the predefined weights, the language loss value and the heat map loss value are used to calculate the total training loss value, and the total training loss value is used to update the parameters of the human body full-body posture key point prediction model and optimize the information used for cue learning. The training is iterated for multiple rounds until the preset training conditions are met, thus obtaining the trained human body full-body posture key point prediction model and the learned cue.

[0093] The training process of the human body posture key point prediction model will be further explained in detail below through an example.

[0094] First, the text semantic space is divided into four layers, each corresponding to a different semantic level. Prompt vectors are set for each keypoint in the four layers of text space at different levels. The four layers have 133, 15, 2, and 1 keypoints respectively, corresponding to the change from fine-grained to coarse-grained.

[0095] Secondly, in each iteration, an image is input into the Image Encoder module and outputs a four-level image feature pyramid.

[0096] Furthermore, the Optimal Transport (OT) algorithm is employed to establish an algebraic expression (loss function) for the semantic relationships of the text. This expression embeds the inherent algebraic properties of language itself, along with the superior relationships acquired through iterative training. Combined with an image feature pyramid, this aligns the image and text spaces. A set of learnable text prompts is trained to supplement ambiguous feature information, aiming to align the four layers of the feature pyramid with the trained text semantic space, thereby obtaining a well-defined and high-quality visual-linguistic latent space. The goal is to ensure that the trained four-layer image features also possess this type of relationship, while the four levels of the feature space reflect a progression from fine-grained to coarse-grained granularity.

[0097] Finally, the input is fed into the FPN network, which outputs a heatmap of 133 fine-grained key points across the entire body and marks the key points on the image for visualization.

[0098] According to an embodiment of the present invention, the above-mentioned multi-level feature extraction of image samples using a pyramid visual Transformer network to obtain a feature sample pyramid includes: cutting the image samples using the embedding layer of the pyramid visual Transformer network, arranging the cut image blocks into a sequence, and transforming the sequence into a two-dimensional matrix through a linear mapping operation; inputting the two-dimensional matrix and the position information of the sequence into the pyramid visual Transformer network, and extracting a multi-level feature sample pyramid from the image samples using the Transformer encoder of the pyramid visual Transformer network.

[0099] The process of obtaining the above-mentioned feature sample pyramid through the visual / image branch provided by the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0100] Figure 4 This is a schematic diagram of the process of performing multi-level feature extraction using the visual / image branch according to an embodiment of the present invention.

[0101] Figure 5 This is a schematic diagram of the structure of a PVT (Pyramid Vision Transformer) network according to an embodiment of the present invention.

[0102] like Figure 4As shown, to address the problems caused by scale differences, this invention utilizes the Vit-Adapter module (PVT network) to extract the feature pyramid of an image. Furthermore, the visualized four-layer Feature Pyramid diagram clearly shows that the four layers of the pyramid correspond to different resolutions, demonstrating a shift from local to global, and from fine-grained to coarse-grained detail. It is also an inherent fact that the semantic levels of features extracted from different layers of the network are different.

[0103] like Figure 5 As shown, the entire PVT (Pyramid Vision Transformer) network can be simply viewed as an Encoder module of four transformers.

[0104] Among them, for Figure 4 In the standard Transformer module, the Patch Embedding operation takes a sequence of tokens (vectors), i.e., a two-dimensional matrix, as input. However, image data is multi-channel data, so it needs to be transformed through the Embedding layer. First, the image is cut into multiple blocks and arranged in a sequence. Then, it is mapped to a two-dimensional matrix using Linear Projection, and the positional information of each element in the sequence is added as input. Finally, the Transformer Encoder extracts the feature representation of each layer. There are four layers in total, corresponding to four resolutions: 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Extracting features at multiple scales mitigates the problems caused by scale differences.

[0105] According to an embodiment of the present invention, the above-described text semantic feature extraction and text dependency injection operations performed on the initial prompt using a multimodal text encoder to obtain a text semantic feature sample space, and the calculation of language loss value during the extraction and injection operations, includes: performing text semantic feature extraction on the initial prompt using a multimodal text encoder to obtain multi-level text semantic feature samples; based on prior knowledge of text relationships, using a multimodal text encoder to achieve text dependency injection through feature vector mapping between adjacent levels of text semantic feature samples to obtain a multi-level text semantic feature sample space with dependency relationships; and calculating language loss value during the text semantic feature sample extraction and text dependency injection operations using a language loss function based on the L1 paradigm.

[0106] The language loss function based on the L1 paradigm is shown in the following equation:

[0107]

[0108] Among them, E text (*) indicates the multimode encoder shown, t i This indicates the prompt for the i-th level. This indicates the prompt for the (i+1)th level.

[0109] The process of obtaining the above-mentioned text semantic feature sample space will be further explained in detail below through specific embodiments.

[0110] To address the issue of semantic ambiguity at small scales, this invention aims to "guide" the prediction task using textual information. Leveraging the favorable properties of textual information and employing text relation injection, prior knowledge of textual relations is incorporated into the distribution of linguistic features in the form of an algebraic sum. Furthermore, the feature pyramid extracted from the image is aligned with each layer of the text space to obtain a well-defined and high-quality visual-linguistic latent space. In summary, a set of learnable textual cues is trained to supplement the ambiguous feature information; the text encoder used here is CLIP Text Encoder.

[0111] Specifically, for example, at the first level, or Keypoint level, this invention establishes a Prompt vector t for each of the 133 keypoints in the body, as shown in formula (1):

[0112] t=[v1][v2][v3]…[vm][CLASS](1).

[0113] Here, v1...vm are learnable prefixes, and CLASS indicates the group. Figure 3To explain, in the first keypoint level on the upper left side of the diagram (i.e., the irregular circles marked "Three Left Foot Keypoints"), there are three corresponding prompt vectors. In the next region level, the authors set 15 prompt vectors representing regions, one of which represents the left foot. This invention aims to find a mapping that allows certain keypoints from the previous level to semantically correspond to a specific region in the next level. Here, the goal is to find a vector whose encoding (using a text encoder) of the left foot's three-toe prompt vector at the keypoint level is algebraically equal to the encoded prompt vector of the left foot at the next region level. This algebraic sum allows for an implicit mapping, enabling the feature representation learned by this invention to imply this affiliation—that the three left toes belong to the left foot region. In the third layer, two prompts are set up: an upper body and a lower body. The present invention aims to ensure that the algebraic sum of the vectors encoded by the TextEncoder is equal to the feature representation corresponding to the whole body at the final whole level.

[0114] The loss function (objective function) for the language branch is shown in formula (2):

[0115]

[0116] in, E represents a specific Prompt vector at the next level (i.e., a relatively coarse-grained feature), and t represents all corresponding keypoint Prompt vectors at the previous level (a relatively fine-grained feature). text (*) indicates operations encoded using CLIP Text Encoder. This invention aims to include all operations at the next higher level related to... The feature representation obtained after encoding the key points related to the level is equal to the algebraic sum of the key points. Encoded feature representation.

[0117] This learning process allows the learned text feature representations to embody this hierarchical relationship and also reflects the process of change from fine-grained to coarse-grained. It's worth noting that the model parameters in the CLIP Text Encoder used here are fixed; the learnable parameters in this invention are v1...vm, and the representation of keypoint features is optimized by adjusting this set of v1 to vm.

[0118] This textual dependency injection operation allows the construction of four-level domains, finding a mapping that enables features extracted from the previous level of text to be mapped to the corresponding next level. This implicitly contains dependency relationships and reflects a shift from fine-grained to coarse-grained detail, corresponding to the four layers of the image feature pyramid from concrete to abstract. Figure 3 The lower half is shown.

[0119] Thus, the present invention can obtain a well-structured text feature space with four levels (not only possessing the excellent algebraic properties of language itself: King-Man + Woman = Queen, but also containing a mapping, a potential subordinate relationship, such as Whole body = upper body + lower body, and the four levels correspond to the process of change from fine-grained to coarse-grained).

[0120] Finally, this invention aims to ensure that the feature representation extracted from the image can contain the excellent properties of the hierarchical text space. Therefore, the OT algorithm is used to align the two feature spaces.

[0121] According to an embodiment of the present invention, the above-described optimization objective of defining a cost function using the cosine similarity between the feature sample pyramid and the text semantic feature sample space, and constructing an entropy-constrained Lagrange multiplier using the cost function, includes: defining a cost function using the cosine similarity between the feature sample pyramid and the text semantic feature sample space; and obtaining the optimization objective of the entropy-constrained Lagrange multiplier using the cost function, a predefined transmission plan matrix, and a predefined entropy constraint, based on the method of mapping feature vectors of different modalities to a common vector space in the optimal transmission problem.

[0122] The optimization objective of the entropy-constrained Lagrange multipliers described above is expressed by the following equation:

[0123]

[0124] Where C represents the cost function, T represents the predefined transmission plan matrix, λh(T) represents the predefined entropy constraint, u represents the vector component in the feature sample pyramid, and v represents the vector component in the text semantic feature sample space.

[0125] The following specific examples will provide a more detailed explanation of the above-mentioned text attachment alignment (OT algorithm).

[0126] This invention draws upon the Optimal Transport algorithm. Through this operation, the feature representation extracted from the image can contain the excellent properties of hierarchical text space (from concrete to abstract, multi-scale, and implicit subordinate relationships).

[0127] OT (Optimal Transport) can be understood as mapping two feature representations to a common space, making them comparable or similar in this common space.

[0128] When this invention considers two sets of points (feature sets), as shown in formulas (3) and (4), their discrete distribution can be formalized into the following mathematical expression:

[0129]

[0130]

[0131] Where U and V represent discrete probability vectors summing to 1, and δf represents the Dirac delta function placed at support point f in the embedding space. Next, as shown in equation (5), this invention expresses this total distance as:

[0132]

[0133] Where C represents the cost matrix, as shown in formula (6):

[0134] C = 1 - F T G(6),

[0135] As shown in formula (6), the present invention defines the cosine similarity between visual features and text feature spaces as a cost function.

[0136]

[0137] Q = H × W (8),

[0138] As shown in formulas (7) and (8), matrix F represents the visual features extracted by the visual encoder.

[0139]

[0140] As shown in formula (9), matrix G represents the text features extracted by the text encoder.

[0141] C q,i =1-sim(f q ,g i (10),

[0142] As shown in formula (10), in the cost matrix C, each element C q,i fq and g i The costs between them.

[0143] Simultaneously, this invention introduces a T matrix called the Transport Plan, which is learned to minimize the total distance. Finally, the optimization problem of the entropy-constrained Lagrange multipliers can be expressed as equation (11):

[0144]

[0145] Where h(*) represents entropy, and λ≥0 is a hyperparameter. Subsequently, according to formula (12), this invention rapidly optimizes the solution through a small number of iterations:

[0146] T * =diag(u (r) )exp(-C / λ)diag(v (r) (12)

[0147] Where r represents the number of iterations. As shown in formulas (13) and (14), in each iteration,

[0148]

[0149] v (r) =v / (exp(-C / λ) T u (r) (14),

[0150] In the first iteration, v (0) =1. To learn the Transport Plan, this invention pushes G towards F (two feature spaces) by minimizing the OT distance:

[0151] This invention obtains T in the Transport Plan using formulas (12), (13), and (14). * The solution and the final OT distance d OT As shown in formula (15):

[0152] d OT =d OT (u,v|1-F T G) (15).

[0153] The OT algorithm can align the feature pyramid extracted from the image with the text space, allowing the extracted features to contain the excellent properties of text information.

[0154] Through this operation, a visual-textual latent space is constructed that is aligned layer by layer, has clear hierarchy, and possesses excellent properties. Its main advantages are:

[0155] (1) Different network layers pay different attention to features at different levels. Low-level network layers are more sensitive to local structures and textures in images. By aligning low-level features with fine-grained text features, the detailed features of the image can be better preserved and emphasized. High-level network layers are able to learn more abstract and advanced semantic information. By imposing coarse-grained semantic constraints on high-level network layers, the model can better understand the overall scene, semantic relationships, and semantic structures in the image.

[0156] (2) By constructing hierarchical relationships between different levels of text features and aligning them with image features, a fine-grained to coarse-grained image understanding process can be formed. The evolution of hierarchical relationships from low-level features to high-level features is explicitly modeled. This helps the model focus on both details and the overall picture, improving its understanding of different levels of image content.

[0157] Combining these two advantages makes the network model more accurate in predicting key points throughout the body, especially key points in small-scale areas.

[0158] Final prediction and network objective function: Finally, the aligned new feature representation is input into the FPN network to obtain the heatmap of each key point, that is, the 133 key points, and finally output to the image, as shown in formula (16):

[0159] L local =α·L heatmap +β·L cal (16),

[0160] As shown in formula (16), L is the objective function of the entire network. heatmap This represents the loss value between the keypoint heatmap and the keypoint heatmap of the ground truth image. The model parameters are tuned by minimizing the objective function.

[0161] In the training process of the above-mentioned human body full-body pose key point prediction model, the Text Encoder used in the text branch (or language branch) is borrowed from the TextEncoder of CLIP (Contrastive Language Image Pre-training), where the parameters are frozen, i.e., cannot be modified. The parameters that the entire model can learn / optimize through training are: (1) the prefixes of the learnable vectors [v1][v2][v3]...[vm]; (2) the network parameters in the Vit-Adapter module; (3) the matrix T in the OT algorithm.

[0162] The human full-body pose keypoint prediction model provided by this invention constructs a hierarchically aligned, high-quality visual-textual latent space, allowing different network layers to focus on different levels of features. Lower-level network layers are more sensitive to local structures and textures in images. By aligning low-level features with fine-grained text features, detailed image features can be better preserved and emphasized. Higher-level network layers can learn more abstract and advanced semantic information. By imposing coarse-grained semantic constraints in higher-level network layers, the model can better understand the overall scene, semantic relationships, and semantic structure in images. Simultaneously, by constructing hierarchical relationships between different levels of text features and aligning them with image features, a fine-grained to coarse-grained image understanding process can be formed. The model explicitly models the evolution of hierarchical relationships from low-level features to high-level features. This helps the model focus on both details and the whole picture, improving its understanding of different levels of image content.

[0163] To better illustrate the advantages of the method provided by this invention, a comparison is made below with other methods for predicting key points of human body posture.

[0164] Table 1 - Experimental results of A2Net and SOTA whole-body pose estimation models on the COCO-WholeBody dataset.

[0165]

[0166] Table 1 shows the experimental results of A2Net and state-of-the-art (SOTA) full-body pose estimation models on the COCO-WholeBody dataset. As shown in Table 1, our model achieves an AP of 61. Note that our vision branch did not use any pre-trained model. The results demonstrate that A2Net achieves excellent accuracy with the lowest GFLOPs (4.1). Specifically, compared to the state-of-the-art method ViTPose+-H, our model achieves comparable performance with only 1 / 30th of its GFLOPs. Compared to ViTPose+-s, which has similar GFLOPS, A2Net outperforms by 15.8 AP and 6.4 AP in the face and hand regions, respectively.

[0167] Table 2 - Comparison of the model of this invention with state-of-the-art models on the Halpe-FullBody dataset

[0168]

[0169] As shown in Table 2, we compared our model with state-of-the-art models on the Halpe-FullBody dataset. With the same experimental settings as the COCO-WholeBody dataset, our method achieved state-of-the-art performance among all models except FastPose50-dcn-si*, but their models were trained through multi-domain knowledge distillation. Maintaining the same experimental settings, our model achieved state-of-the-art performance among all methods. Compared to the state-of-the-art model FastPose152-si, our model achieved only about 31% of its GFLOPs, yet outperformed it by 2 AP (approximately 4.4%). The results, as shown in Table 2, are similar to our observations in the COCO-WholeBody experiments, explicitly modeling the evolution of hierarchical relationships where low-level features converge into high-level features. This helps the model focus on both details and the overall picture simultaneously, improving its understanding of different levels of image content.

[0170] Table 3 - Comparison of the model of this invention with the state-of-the-art model on the Halpe-FullBody dataset for various parts of the body.

[0171]

[0172] As shown in Table 3, the same method as in Table 2 was used. Comparative experiments were conducted on various parts of the body, and it was found that our model, especially in small-scale areas, has superior accuracy. Compared to the state-of-the-art model FastPose152-si, our model has only about 31% of its GFLOPs, but it is 4 AP higher in the face area.

[0173] In addition, we conducted ablation experiments on the COCO-Wholebod dataset.

[0174] Table 4 - Ablation experiments on the COCO-Wholebod dataset

[0175]

[0176] This invention mainly focuses on comparative experiments on the following two points.

[0177] (1) Whether to use visual-text alignment.

[0178] (2) Extract image features by trying other network structures.

[0179] As shown in Table 4, it was found that the operation using the Vit-Adapter module in conjunction with Visual Text Alignment (VLA) yielded the best results.

[0180] Figure 6This is an illustration of the effect of a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment, according to an embodiment of the present invention.

[0181] like Figure 6 As shown in the visualization results, the prediction method provided by this invention has good prediction effect on key points in small-scale parts of the whole body.

[0182] Figure 7 This is a schematic diagram of the structure of a human body full-body posture key point prediction system based on visual-linguistic hierarchical alignment relationship according to an embodiment of the present invention.

[0183] like Figure 7 As shown, the prediction system includes a model training module 710, an image feature extraction module 720, a text semantic feature space acquisition module 730, a visual-language latent space acquisition module 740, and a key point prediction module 750.

[0184] The model training module 710 is used to train the human body full-body posture key point prediction model and optimize the information used for cue learning during the model training process to obtain the trained human body full-body posture key point prediction model and the learned cue. The trained human body full-body posture key point prediction model includes a pyramid visual Transformer network, a multimodal text encoder and a feature pyramid prediction network. The learned cue is a vector used to represent the human body full-body posture key points.

[0185] The image feature extraction module 720 is used to perform multi-level feature extraction on the target image using a pyramid visual Transformer network to obtain a multi-level feature pyramid. The image features at different levels in the multi-level feature pyramid have different sizes and resolutions.

[0186] The text semantic feature space acquisition module 730 is used to extract text feature vectors from learned prompts and predefined category labels using a multimodal text encoder, and to obtain a multi-level text semantic feature space with subordinate relationships based on the extracted text feature vectors, wherein the text semantic feature space has the same number of levels as the feature pyramid.

[0187] The visual-language latent space acquisition module 740 is used to perform a one-to-one alignment operation of feature vectors between the feature pyramid and the text semantic feature space at the same level using a predefined optimal transfer algorithm, so as to obtain a visual-language latent space that integrates visual feature vectors and text semantic feature vectors.

[0188] The key point prediction module 750 is used to perform feature vector mapping operation on the visual-language latent space using the feature pyramid prediction network, generate a heat map of the key points of the human body's whole body posture in the target image, and predict the position coordinate information of each key point of the human body's whole body posture in the target image based on the heat map of the key points of the human body's whole body posture.

[0189] Figure 8 A block diagram of an electronic device adapted to implement a method for predicting key points of human body posture based on visual-linguistic hierarchical alignment is illustrated according to an embodiment of the present invention.

[0190] like Figure 8 As shown, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0191] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 802 and / or RAM 803. It should be noted that programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0192] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 904. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0193] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0194] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0196] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting key points of human full-body posture based on visual-linguistic hierarchical alignment, characterized in that, include: A full-body posture keypoint prediction model is trained, and the information used for cue learning is optimized during the model training process to obtain a trained full-body posture keypoint prediction model and learned cues. This includes: constructing the full-body posture keypoint prediction model based on a visual language model, and using a pyramid visual Transformer network to extract multi-level features from image samples to obtain a feature sample pyramid; performing hierarchical initialization of the information used for cue learning based on the hierarchical attribute information and category label samples of the multi-level feature sample pyramid to obtain initial cues; using a multimodal text encoder to perform text semantic feature extraction and text affiliation injection operations on the initial cues to obtain a text semantic feature sample space, and calculating the language loss value during the extraction and injection operations; defining a cost function using the cosine similarity between the feature sample pyramid and the text semantic feature sample space, and using the cost function to construct an optimization objective with entropy-constrained Lagrange multipliers; based on the... The optimization objective of the entropy-constrained Lagrange multipliers is to use the optimal transport algorithm to perform a one-to-one alignment operation of the feature sample pyramid and the text semantic feature sample space between the same level by minimizing the total distance, thereby obtaining visual-language latent space samples. A feature pyramid prediction network is then used to perform feature vector mapping on the visual-language latent space samples to generate heatmap samples of key points of the human body's full-body posture in the target image. During the heatmap sample generation process, a heatmap loss value is calculated. Based on predefined weights, the language loss value and the heatmap loss value are used to calculate the total training loss value. This total training loss value is then used to update the parameters of the human body's full-body posture key point prediction model and optimize the information used for cue learning. Multiple rounds of training are iteratively performed until preset training conditions are met, resulting in the trained human body full-body posture key point prediction model and the learned cue. The image features at different levels in the multi-level feature pyramid have different sizes, resolutions, and image granularities. The multimodal text encoder extracts text feature vectors from the learned prompts and predefined category labels, and based on the extracted text feature vectors, obtains a multi-level text semantic feature space with subordinate relationships, wherein the text semantic feature space has the same number of levels as the feature pyramid; Using a predefined optimal transmission algorithm, the feature pyramid and the text semantic feature space are aligned one-to-one between feature vectors at the same level to obtain a visual-language latent space that integrates visual feature vectors and text semantic feature vectors. The feature pyramid prediction network is used to perform feature vector mapping on the visual-language latent space to generate a heatmap of key points of human body posture in the target image, and the position coordinate information of each key point of human body posture in the target image is predicted based on the heatmap of key points of human body posture.

2. The method according to claim 1, characterized in that, The image sample pyramid is obtained by using the pyramid visual Transformer network to perform multi-level feature extraction on the image samples, including: The image samples are segmented using the embedding layer of the pyramid visual Transformer network, the resulting image blocks are arranged into a sequence, and the sequence is transformed into a two-dimensional matrix through a linear mapping operation. The two-dimensional matrix and the position information of the sequence are input into the pyramid visual Transformer network, and the Transformer encoder of the pyramid visual Transformer network is used to extract multi-level feature sample pyramids from the image samples.

3. The method according to claim 1, characterized in that, The multimodal text encoder is used to perform text semantic feature extraction and text association relation injection operations on the initial prompt, resulting in a text semantic feature sample space. During the extraction and injection operations, the language loss value is calculated, including: The multimodal text encoder is used to extract text semantic features from the initial prompt to obtain multi-level text semantic feature samples. Based on prior knowledge of textual relationships, the multimodal text encoder uses the feature vector mapping between adjacent levels of textual semantic feature samples to inject textual subordinate relationships, thereby obtaining a multi-level textual semantic feature sample space with subordinate relationships. The language loss value is calculated in the text semantic feature sample extraction operation and the text affiliation relation injection operation using the language loss function based on the L1 paradigm.

4. The method according to claim 3, characterized in that, The language loss function based on L1 paradigm is shown in the following formula: ， in, The multimode encoder shown is illustrated. Indicates the first Layer prompt, Indicates the first The layer's prompt.

5. The method according to claim 1, characterized in that, The cost function is defined using the cosine similarity between the feature sample pyramid and the text semantic feature sample space, and the optimization objective of the entropy-constrained Lagrange multiplier is constructed using the cost function, including: A cost function is defined using the cosine similarity between the feature sample pyramid and the text semantic feature sample space. Based on the method of mapping the feature vectors of different modes to a common vector space in the optimal transmission problem, the optimization objective of the Lagrange multiplier with the entropy constraint is obtained by using the cost function, the predefined transmission plan matrix, and the predefined entropy constraint.

6. The method according to claim 5, characterized in that, The optimization objective of the entropy-constrained Lagrange multipliers is expressed by the following equation: ， in, Represents the cost function, This represents the predefined transmission plan matrix. This represents the predefined entropy constraint. This represents the vector components in the feature sample pyramid. This represents the vector components in the text semantic feature sample space.

7. A system based on the prediction method of claim 1, characterized in that, include: The model training module is used to train the human body full-body posture key point prediction model and optimize the information used for cue learning during the model training process to obtain the trained human body full-body posture key point prediction model and the learned cue. The trained human body full-body posture key point prediction model includes a pyramid visual Transformer network, a multimodal text encoder and a feature pyramid prediction network. The learned cue is a vector used to represent the human body full-body posture key points. The image feature extraction module is used to extract multi-level features from the target image using the pyramid visual Transformer network to obtain a multi-level feature pyramid, wherein the image features at different levels in the multi-level feature pyramid have different sizes and resolutions. The text semantic feature space acquisition module is used to extract text feature vectors from the learned prompts and predefined category labels using the multimodal text encoder, and to obtain a multi-level text semantic feature space with subordinate relationships based on the extracted text feature vectors, wherein the text semantic feature space has the same number of levels as the feature pyramid; The visual-language latent space acquisition module is used to perform a one-to-one alignment operation of the feature pyramid and the text semantic feature space at the same level using a predefined optimal transfer algorithm, so as to obtain a visual-language latent space that integrates visual feature vectors and text semantic feature vectors. The key point prediction module is used to perform feature vector mapping operation on the visual-language latent space using the feature pyramid prediction network, generate a heat map of the key points of the human body's full-body posture in the target image, and predict the position coordinate information of each key point of the human body's full-body posture in the target image based on the heat map of the key points of the human body's full-body posture.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images