Attitude estimation method and system based on deep learning

Abstract

The present invention provides a posture estimation method and system based on deep learning, which are applied to the field of image recognition, wherein the method comprises: obtaining a target image, a natural language instruction and a template image; encoding the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features; performing multimodal feature fusion on the text features and the visual features through a large language model of the prompt generator to obtain multimodal feature information; generating a posture prompt vector based on the multimodal feature information through a visual embedding projection layer of the prompt generator; inputting the target image and the posture prompt vector into a preset posture estimator to obtain a key point heat map output by the posture estimator; performing key point position analysis on the key point heat map to obtain a posture estimation result of the target image. The present invention can flexibly adapt to different types of posture estimation requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a posture estimation method and system based on deep learning. Background Art

[0002] Pose estimation is the process of inferring the pose of an object or person from an image or video, and involves identifying and localizing key points on the body.

[0003] Existing posture estimation techniques usually target specific types of objects, such as humans or animals, and rely on a predefined set of key points, such as human joints or specific parts of animals. Although this method can provide accurate posture recognition in specific scenarios, it is difficult to directly apply to undefined objects, such as different types of furniture, vehicles, or daily necessities, when faced with diverse targets and complex real-world environments.

[0004] It can be seen that the posture estimation method in the related art has a technical problem of poor versatility due to the fact that it can only be used for specific types of objects. Summary of the invention

[0005] The present invention provides a posture estimation method and system based on deep learning, so as to solve the defect of poor versatility caused by the posture estimation method in the prior art that it can only be used for specific types of objects, and realize flexible adaptation to posture estimation requirements of different categories.

[0006] The present invention provides a posture estimation method based on deep learning, comprising the following steps.

[0007] A target image, a natural language instruction and a template image are obtained, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; the natural language instruction and the template image are respectively encoded by a preset prompt generator to obtain text features and visual features; the text features and the visual features are multimodally fused by a large language model of the prompt generator to obtain multimodal feature information; a posture prompt vector is generated based on the multimodal feature information by a visual embedding projection layer of the prompt generator, wherein the posture prompt vector is used to represent a target focus position in the template image; the target image and the posture prompt vector are input into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; the key point position is parsed on the key point heat map to obtain a posture estimation result of the target image.

[0008] According to a deep learning-based posture estimation method provided by the present invention, the target image and the posture prompt vector are input into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator, including: extracting features of the target image through a preset posture estimator to obtain a target image feature code, wherein the target image feature code has the same dimension as the posture prompt vector; and aligning the target image feature code with the posture prompt vector through the posture estimator to obtain a key point heat map.

[0009] According to a deep learning-based posture estimation method provided by the present invention, after extracting features from the target image to obtain the target image feature encoding, the method further includes: performing feature fusion on the target image feature encoding and the posture prompt vector through the attention mechanism of the posture estimator to learn the weights between different modalities.

[0010] According to a deep learning-based posture estimation method provided by the present invention, the target image feature encoding is aligned with the posture prompt vector to obtain a key point heat map, including: aligning based on the similarity between the target image feature encoding and the posture prompt vector to obtain a similarity map; using the similarity map as a key point heat map, wherein the number of channels of the key point heat map is the same as the number of channels of the posture prompt vector.

[0011] According to a deep learning-based posture estimation method provided by the present invention, before obtaining the target image, natural language instructions and template image, the method also includes: obtaining a posture estimation sample data set and an instruction question and answer sample data set for targets of different categories, wherein each posture estimation sample in the posture estimation sample data set includes a target point annotation; converting the target point annotation of each posture estimation sample in the posture estimation sample data set into a dialogue annotation to obtain a posture estimation instruction question and answer sample data set; and jointly training the prompt generator and the posture estimator based on the posture estimation instruction question and answer sample data set and the instruction question and answer sample data set.

[0012] According to a deep learning-based posture estimation method provided by the present invention, the method also includes: for each posture estimation sample in the posture estimation sample data set, randomly generating different types of questions and answers for instruction fine-tuning, wherein the type of questions and answers includes at least one of the following: a natural language instruction sample, wherein the natural language instruction sample includes an image category; the natural language instruction sample and a template image sample, wherein the natural language instruction sample does not include the image category, and the template image sample includes a visual prompt; the natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category, and the template image sample does not include the visual prompt; the natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category, and the template image sample does not include the visual prompt; the natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category, and the template image sample includes the visual prompt.

[0013] The present invention also provides a posture estimation system based on deep learning, comprising the following modules:

[0014] An acquisition module is used to acquire a target image, a natural language instruction and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; an encoding module is used to encode the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features; a fusion module is used to perform multimodal feature fusion on the text features and the visual features through a large language model of the prompt generator to obtain multimodal feature information; a generation module is used to generate a posture prompt vector based on the multimodal feature information through a visual embedding projection layer of the prompt generator, wherein the posture prompt vector is used to represent a target focus position in the template image; an output module is used to input the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; a parsing module is used to perform key point position parsing on the key point heat map to obtain a posture estimation result of the target image.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a posture estimation method based on deep learning as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the deep learning-based posture estimation methods described above.

[0017] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned deep learning-based posture estimation methods.

[0018] The deep learning-based posture estimation method and system provided by the present invention obtain a natural language instruction including an image category of a target image, and a template image for visually prompting the target image. Therefore, the introduction of the natural language instruction and the template image can more accurately judge the image category of the target image, thereby improving the versatility of posture estimation; the prompt generator encodes the natural language instruction and the template image into text features and visual features, and fuses multimodal information in the text features and the visual features through a large language model. Therefore, the combination of text information and visual information can improve the accuracy of posture estimation, and generates a posture prompt vector for representing the target focus position in the template image through a visual embedding projection layer. The target image and the posture prompt vector are input into a posture estimator based on a Transformer structure to obtain a key point heat map, and the key point position analysis is performed on the key point heat map to obtain the posture estimation result of the target image. Therefore, the position in the target image that is most similar to the posture prompt vector can be predicted as the position of the target key point, thereby realizing accurate reconstruction of the posture estimation result, thereby solving the technical problem that the posture estimation method in the related art has poor versatility due to only being able to target specific types of objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of the posture estimation method based on deep learning provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the overall flow of the posture estimation method based on deep learning provided by the present invention.

[0022] Figure 3 It is a structural schematic diagram of the attitude estimator provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the prompt generator provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the deep learning-based posture estimation system provided by the present invention.

[0025] Figure 6 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In the field of modern computer vision, general pose estimation is a challenging key technology, which involves accurately identifying and predicting the poses of a variety of targets such as humans, animals, vehicles, furniture, clothing, etc. This technology has broad application prospects in many fields such as video surveillance, human-computer interaction, virtual reality, motion analysis and autonomous driving.

[0028] The emergence of deep learning technology has brought revolutionary changes to pose estimation. Previous pose estimation methods usually rely on hand-crafted features and complex image processing techniques, which are often ineffective when faced with complex scenes and diverse poses. However, deep learning can automatically learn and extract complex features in images by building deep neural networks, thereby achieving high-precision estimation of the target pose. Deep learning models, especially convolutional neural networks (CNNs) and transformers, have achieved high accuracy in single-category pose estimation tasks. CNNs, with their powerful feature extraction capabilities, can capture key information about human poses from images, while transformers, through the self-attention mechanism, can better handle long-distance dependencies in images, further improving the accuracy of pose estimation. These deep learning models have achieved high accuracy in single-category pose estimation tasks, can adapt to different lighting conditions, background interference, and various pose changes, and provide a robust solution for pose estimation.

[0029] However, existing pose estimation techniques usually target specific types of objects, such as humans or animals, and rely on a predefined set of key points, such as joints of humans or specific parts of animals. Although this approach can provide accurate pose recognition in specific scenarios, its limitations become apparent when faced with diverse targets and complex real-world environments. First, the limitation of specific target categories means that these methods are difficult to directly apply to undefined objects, such as different types of furniture, vehicles, or daily necessities. Whenever a new target category is encountered, the model may need to be redesigned or adjusted, which not only increases the cost of development and deployment, but also reduces the universality of pose estimation technology. Second, the preset set of key points also limits the flexibility of pose estimation methods. In the real world, the same type of object may present a variety of forms due to factors such as pose, perspective, or occlusion, and the preset key points may not cover all situations.

[0030] In recent years, the emergence of multimodal large models has not only expanded the boundaries of deep learning models' ability to process information, but also promoted the development of deep learning technology in a more intelligent and universal direction. By integrating and processing multiple data modalities such as text, images, audio and video, this type of model demonstrates the great potential of deep learning in understanding and generating cross-modal content, which not only improves the model's understanding ability, but also enhances the model's generalization performance. For the specific task of posture estimation, the introduction of multimodal large models provides new possibilities for promoting its universality. For example, multimodal large models can locate the target of interest through language instructions and other means, and generate corresponding posture prompt words, thereby guiding the posture estimation module to locate key points and flexibly adapt to different categories of posture estimation needs.

[0031] In view of this, the present invention provides a posture estimation method based on deep learning. The method includes two core components: a prompt generator based on a multimodal large model and a general posture estimator based on a prompt vector. The prompt generator is designed to dynamically generate a prompt vector through language instructions and input images, thereby guiding the posture estimator to generate corresponding postures for different categories of targets. Specifically, the prompt generator first encodes the natural language instructions and the input image into text features and visual features respectively. Subsequently, the attention mechanism in the large language model is used to effectively fuse the multimodal information in the text and visual features. This process not only enhances the model's ability to understand language instructions, but also improves the visual perception of image content. Finally, the prompt generator outputs the position of the target of interest in the image and generates a posture prompt vector related to it. After receiving the target position and posture prompt words, the posture estimator encodes the pixels in the target area in the image as visual features. Then, the attention mechanism is used to fuse the semantic information in the visual features and the prompt vector. This strategy helps the model better understand the semantic meaning of the target posture. Finally, by aligning the image features and the hint vector, the pose estimator is able to predict the location in the image that is most semantically similar to the hint vector as the location of the target keypoint.

[0032] Optionally, the deep learning-based posture estimation method of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the deep learning-based posture estimation method in this embodiment executed by a server as an example.

[0033] Figure 1 is a flow chart of a posture estimation method based on deep learning provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0034] Step 101 , obtaining a target image, a natural language instruction, and a template image, wherein the natural language instruction includes an image category of the target image, and the template image is used to provide a visual prompt for the target image.

[0035] In an embodiment of the present invention, the target image is the image for which posture estimation is required, the natural language instruction may be a descriptive word input by a user according to the image category of the target image, and the template image is an image of the same or similar image category as the target image.

[0036] For example, when the image content of the target image is "polar bear", the natural language instruction may be "extract the posture and action of the polar bear", and the template image may be other "polar bear" images with the same or similar actions.

[0037] Through the embodiments of the present invention, when acquiring a target image that needs to be estimated in terms of posture, introducing natural language instructions and template images can more accurately determine the image category of the target image, thereby improving the versatility of posture estimation.

[0038] Step 102, encoding the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features.

[0039] In the embodiment of the present invention, the natural language instruction is encoded by the natural language encoder of the prompt generator to obtain text features.

[0040] For example, through natural language processing (NLP) technology, such as word embedding or sentence embedding, natural language instructions are converted into text features in a high-dimensional vector space. Text features can capture the semantic information in the instructions, such as image categories, action types, etc., providing a basis for subsequent multimodal feature fusion.

[0041] In the embodiment of the present invention, the template image is encoded by the template visual encoder of the prompt generator to obtain visual features.

[0042] For example, using convolutional neural networks (CNN) or other image encoding techniques in deep learning, the template image is converted into visual features, which usually contain visual information such as texture, shape, color, and possible spatial layout and object relationships. By encoding the template image, the key visual elements in the template image can be learned, which are instructive for subsequent pose estimation.

[0043] Step 103, through the large language model of the prompt generator, multimodal feature fusion is performed on the text features and the visual features to obtain multimodal feature information.

[0044] In an embodiment of the present invention, multimodal information in text and visual features is fused by using an attention mechanism in a large language model of a prompt generator.

[0045] In an embodiment of the present invention, the encoded text features and visual features provide necessary input for multimodal feature fusion. Multimodal fusion refers to combining information from different modalities (such as text and image) to obtain multimodal information. In the subsequent posture estimation task, multimodal fusion information helps the model to more accurately understand the relationship between natural language instructions and template images, thereby more accurately estimating the posture of the target image.

[0046] Step 104 , generating a posture hint vector based on the multimodal feature information through the visual embedding projection layer of the hint generator, wherein the posture hint vector is used to represent the target focus position in the template image.

[0047] In an embodiment of the present invention, the prompt generator includes a large language model , a visual embedding projection layer ,The hint generator takes text and images as input and produces text output. In addition, when the user requests the target pose information, it also returns the corresponding pose hint vector.

[0048] Given a text string and images As input, the model produces a textual response, i.e., a pose cue vector or in the absence of an image , corresponding to the pose cue vector The language embedding of If the text output includes gesture prompt words, then Retrieve its corresponding hint embedding from .

[0049] In some embodiments, when a key point that cannot be described in words appears, it can be Adding visual cues to generate corresponding cue vectors.

[0050] Step 105, input the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator.

[0051] In the embodiment of the present invention, the target image is feature extracted and vectorized by the posture estimator to obtain the target image feature code, and the target image feature code and the posture prompt vector are feature fused by the posture estimator based on the Transformer structure, and the spatial position of the target image feature code is restored and upsampled to align it with the posture prompt vector. By calculating the similarity between the target image feature code and the posture prompt vector, a similarity map is generated as a key point heat map , the number of channels The number of is the same as the number of pose cue vectors of the target.

[0052] Step 106, performing key point position analysis on the key point heat map to obtain a pose estimation result of the target image.

[0053] Here, key point heatmap (or key point confidence map) is a common form of pose estimation network output. Each key point heatmap corresponds to a specific key point (such as a joint of the human body), and each pixel value in the key point heatmap represents the confidence (or probability) that the position is the corresponding key point. Usually, the size of the key point heatmap is the same as the target image or scaled proportionally to keep it consistent in spatial position.

[0054] In an embodiment of the present invention, a local maximum is searched on each channel (i.e., each key point) of the heat map, for example, by non-maximum suppression (NMS) or other methods to find the most likely position of each key point.

[0055] By parsing the positions of all key points from the heat map, we can generate pose estimates for the target image based on these positions. For example, we can map the key point positions back to the original image space (if they were scaled or transformed before) and output the pose estimates in some format (such as a list of key point coordinates, a skeleton model, etc.).

[0056] Through the above steps of the embodiment of the present invention, a target image, a natural language instruction and a template image are obtained, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; through a preset prompt generator, the natural language instruction and the template image are respectively encoded to obtain text features and visual features; through the large language model of the prompt generator, multimodal feature fusion is performed on the text features and the visual features to obtain multimodal feature information; through the visual embedding projection layer of the prompt generator, a posture prompt vector is generated based on the multimodal feature information, wherein the posture prompt vector is used to represent the target focus position in the template image; the target image and the posture prompt vector are input into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; the key point heat map is analyzed for key point positions to obtain a posture estimation result of the target image; thereby, the position in the target image that is most similar to the posture prompt vector can be predicted as the position of the target key point, thereby achieving accurate reconstruction of the posture estimation result, thereby solving the technical problem that the posture estimation method in the related art has poor versatility due to only being able to target specific types of objects.

[0057] According to a deep learning-based posture estimation method provided by the present invention, a target image and a posture prompt vector are input into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator, including:

[0058] By using a preset posture estimator, feature extraction is performed on the target image to obtain a target image feature code, wherein the target image feature code has the same dimension as the posture prompt vector;

[0059] Through the pose estimator, the target image feature encoding is aligned with the pose prompt vector to obtain the key point heat map.

[0060] In an embodiment of the present invention, the posture estimator adopts a Transformer structure, whose input includes an instance image (target image) and a corresponding posture prompt vector. The image features and the prompt vector are fused using an attention mechanism, and the semantics of the image features and the prompt vector are aligned in space to achieve the purpose of locating key points.

[0061] In the embodiment of the present invention, in order to convert the target image into a serialized vector in the same form as the posture prompt vector, the target image is tokenized, that is, a convolutional neural network is used to extract features from image blocks and break them up into sequences to obtain image tokens, that is, target image feature encodings. In addition, the posture prompt vector is directly used as a prompt token, wherein the image token and the prompt token have the same dimension.

[0062] In order to effectively fuse the features of image tokens (target image feature encoding) and prompt tokens (pose prompt vectors), the attention mechanism in Transformer is used to learn the weights between different modalities to achieve information complementarity between modalities. In order to align image tokens and prompt tokens, the similarity between image tokens and prompt tokens is calculated, and the similarity map is obtained as the key point heat map.

[0063] In an embodiment of the present invention, the pose estimator has the following advantages: through an effective cross-modal interaction mechanism, the image token and the corresponding prompt token can be deeply aligned in a high-dimensional embedding space. During the alignment process, the model not only focuses on the modal interaction in the global context, but also deeply explores the correlation between fine-grained key point features and prompt tokens. In this way, the model can obtain key feature indications about the target object from the prompt token, thereby guiding it to adaptively predict the key points of any category of targets, rather than being limited to a pre-defined fixed category set.

[0064] Furthermore, the present invention embeds the poses of different categories of targets (pose hint vectors) as different hint tokens, and prompts the large language model to output these tokens when querying pose-related questions. These tokens are then used as hint vectors and input into the pose estimator to predict the target pose, i.e., the key point position.

[0065] According to a posture estimation method based on deep learning provided by the present invention, after extracting features from a target image to obtain feature encoding of the target image, the method further includes:

[0066] Through the attention mechanism of the pose estimator, the target image feature encoding and the pose cue vector are feature fused to learn the weights between different modalities.

[0067] In the embodiment of the present invention, the target image feature is encoded as an image token, and the posture hint vector is used as a hint token. The image token and the hint token have the same dimension. In order to effectively fuse the features of the image token and the hint token, the attention mechanism in the Transformer is used to learn the weights between different modalities to achieve information complementarity between the modalities.

[0068] According to a deep learning-based posture estimation method provided by the present invention, the target image feature encoding is aligned with the posture prompt vector to obtain a key point heat map, including:

[0069] Align based on the similarity between the target image feature encoding and the posture prompt vector to obtain a similarity graph;

[0070] The similarity map is used as a keypoint heatmap, where the number of channels of the keypoint heatmap is the same as the number of channels of the pose hint vector.

[0071] In this embodiment of the present invention, the image token (target image feature encoding) is spatially restored and upsampled, and aligned with the prompt token (pose prompt vector). By calculating the similarity between the two tokens, a similarity map is generated as a key point heat map , the number of channels The number of is the same as the number of hint tokens of the target.

[0072] In an embodiment of the present invention, the similarity graph represents the degree of match between each position in the target image feature encoding and the posture prompt vector. The similarity calculation can use dot product, cosine similarity or other measurement methods to calculate the similarity between each position in the feature graph and the posture prompt vector. The result of the similarity graph is a two-dimensional array (or tensor), in which each element represents the similarity between the corresponding position in the feature graph and the posture prompt vector.

[0073] The similarity map is used as a keypoint heatmap, where the number of channels in the heatmap is the same as the number of channels in the pose cue vector. Thus, each channel corresponds to a specific pose keypoint, and the value in the heatmap represents the possible location of the keypoint in the image.

[0074] According to a posture estimation method based on deep learning provided by the present invention, before acquiring the target image, the natural language instruction and the template image, the method further includes:

[0075] Obtaining a posture estimation sample data set and a command question and answer sample data set for targets of different categories, wherein each posture estimation sample in the posture estimation sample data set includes a target point annotation;

[0076] The target point annotation of each posture estimation sample in the posture estimation sample dataset is converted into a dialogue annotation to obtain a posture estimation command question-answering sample dataset;

[0077] Based on the pose estimation instruction question and answer sample dataset and the instruction question and answer sample dataset, the prompt generator and pose estimator are jointly trained.

[0078] In an embodiment of the present invention, pose estimation (sample) datasets of targets of different categories are collected, and key points of some rare categories of targets are supplemented with annotations to form a pose estimation dataset; a pose estimator based on prompt vectors is designed to meet the pose estimation requirements of different target categories; a multimodal large model is used as a prompt generator; in order to enable the collected data to be used for the joint training of the prompt generator and the pose estimator, the key point annotations are converted into annotations in the form of dialogues; and a command question and answer (sample) dataset and a pose estimation command question and answer (sample) dataset are used to jointly train the prompt generator and the pose estimator.

[0079] In this embodiment of the present invention, when training the model, the template visual encoder is kept Freeze, train the general pose estimator , for large language models Fine-tune the instructions and optimize the model using the following objective function:

[0080]

[0081] in, represents the loss function, A large language model representing the prompt generator The weight of represents the cross entropy loss, represents the text response produced by the model (i.e., the pose hint vector), Indicates the generated instruction label, represents the weight of the pose estimator, Indicates that it corresponds to The language embedding representation of Represents the heat map labels generated by using key point position annotations.

[0082] According to a deep learning-based posture estimation method provided by the present invention, the method further includes:

[0083] For each posture estimation sample in the posture estimation sample data set, randomly generate different types of questions and answers for instruction fine-tuning, wherein the types of questions and answers include at least one of the following:

[0084] A natural language instruction sample, wherein the natural language instruction sample includes an image category;

[0085] The natural language instruction sample and the template image sample, wherein the natural language instruction sample does not include the image category and the template image sample includes a visual prompt;

[0086] The natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category and the template image sample does not include the visual prompt;

[0087] The natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category, and the template image sample includes the visual prompt.

[0088] In this embodiment of the present invention, for each sample, four different types of questions and answers are randomly generated for instruction fine-tuning, namely:

[0089] Only textual input is used to guide the generation of gesture hint vectors, including the target category in the language; text and images are used as input to guide the generation of gesture hint vectors, excluding the target category in the language and including visual hints in the image; text and images are used as input to guide the generation of gesture hint vectors, including the target category in the language and excluding visual hints in the image; text and images are used as input to guide the generation of gesture hint vectors, including the target category in the language and also including visual hints in the image.

[0090] The following describes an example of a posture estimation method based on deep learning provided by an embodiment of the present invention in an actual application scenario.

[0091] refer to Figure 2 , Figure 2 It is a schematic diagram of the overall flow of the posture estimation method based on deep learning provided by the present invention.

[0092] like Figure 2 As shown, an image and a language instruction are input, and the input image and the language instruction are encoded respectively to obtain a template image feature encoding and a language feature encoding; then the template image feature encoding and the language feature encoding are multimodally fused to generate a posture prompt vector; the target image feature encoding is obtained; the image features and the prompt vector are aligned to resolve the key point position; and then the target posture is output according to the key point position. Specifically, the following steps are included:

[0093] Step 1: Encode the features of the template image and input text to tokenize them.

[0094] Step 2: Fuse multimodal information such as template image tokens and input text tokens through a large language model and output a posture hint vector.

[0095] Step 3: Extract and vectorize the target image and fuse its features with the prompt vector through Transformer.

[0096] Step 4: Restore and upsample the target image token and align it with the prompt token. By calculating the similarity between the two tokens, a similarity map is generated as the key point heat map. , the number of its channels c is the same as the number of prompt tokens of the target.

[0097] refer to Figure 3 , Figure 3 It is a structural schematic diagram of the posture estimator provided by the present invention, where the target image is tokenized to obtain an image token; and a prompt token (a posture prompt vector output by a prompt generator) is obtained, and the image token and the prompt token are respectively input into the VIT encoder (Vision Transformer), and the VIT encoder utilizes an attention mechanism to fuse the visual features of the image token and the semantic features in the prompt token, restores the spatial arrangement of the visual features and upsamples them, and performs similarity calculation with the semantic features to obtain a key point heat map.

[0098] refer to Figure 4 , Figure 4 It is a structural schematic diagram of the prompt generator provided by the present invention, where input text and a template image (the template image includes visual prompts) are obtained, the template image is encoded by a (template) visual encoder to obtain template visual features, the input text and the template visual features are input into a large language model, and output text and prompt tokens (i.e., posture prompt vectors) are obtained, and then the prompt tokens and the target image are input into a (universal) posture estimator to obtain a posture estimation result.

[0099] In order to enable the pose estimation dataset to be used for joint training of the prompt generator and the general pose estimator, the key point annotations are converted into instruction annotations in the form of dialogue. For each sample, four different types of questions and answers are randomly generated for instruction fine-tuning, namely: 1) only text input is used to guide the generation of the pose prompt vector, including the target category in the language; 2) text and images are used as input to guide the generation of the pose prompt vector, excluding the target category in the language and including visual cues in the image; 3) text and images are used as input to guide the generation of the pose prompt vector, including the target category in the language and excluding visual cues in the image; 4) text and images are used as input to guide the generation of the pose prompt vector, including the target category in the language and also including visual cues in the image. The instruction question and answer dataset constructed by the literature "Visual Instruction Tuning" and the pose estimation instruction question and answer dataset are used to jointly train the prompt generator and the general pose estimator.

[0100] Keep the template visual encoder while training the model Freeze, train the general pose estimator , for large language models Fine-tune the instructions and optimize the model using the following objective function:

[0101]

[0102] in, represents the loss function, Large Language Model for Representation Prompt Generator The weight of represents the cross entropy loss, represents the text response produced by the model (i.e., the pose hint vector), Indicates the generated instruction label, represents the weight of the pose estimator, Indicates that it corresponds to The language embedding representation of Represents the heat map labels generated by using key point position annotations.

[0103] The deep learning-based posture estimation method provided by the present invention aims to provide a flexible and efficient technical solution to achieve posture estimation tasks for various targets in two-dimensional images, including but not limited to human bodies, animals, vehicles, furniture, clothing, etc. The innovation of this method lies in its unique system architecture, which consists of two core components: one is a prompt generator based on a multimodal large model, and the other is a posture estimator. The prompt generator encodes natural language instructions and input images into text features and visual features, and fuses multimodal information in text and visual features through the attention mechanism in the large language model, and generates relevant posture prompt vectors. After receiving the posture prompt word, the posture estimator encodes the pixels in the target area of ​​the target image as visual features, and uses the attention mechanism to fuse the semantics in the visual features and the prompt vector. Finally, by aligning the image features and the prompt vector, the model can predict the position in the image that is most similar to the prompt vector semantics as the position of the target key point, thereby realizing accurate reconstruction of the target posture.

[0104] The core advantage of the present invention lies in its high versatility and flexibility in the posture estimation process. Compared with traditional posture estimation methods, the method of the present invention is not limited to fixed target categories and key point settings, and can flexibly adapt to various scenarios and needs according to the instructions of language instructions. This feature greatly expands the application scope of posture estimation technology and provides basic technical support for various complex visual analysis tasks.

[0105] The deep learning-based posture estimation system provided by the present invention is described below. The deep learning-based posture estimation system described below and the deep learning-based posture estimation method described above can refer to each other.

[0106] refer to Figure 5 , Figure 5 It is a structural diagram of the deep learning-based posture estimation system provided by the present invention, which includes: an acquisition module 501, an encoding module 502, a fusion module 503, a generation module 504, an output module 505, and a parsing module 506.

[0107] An acquisition module 501 is used to acquire a target image, a natural language instruction, and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image;

[0108] An encoding module 502 is used to encode the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features;

[0109] A fusion module 503 is used to perform multimodal feature fusion on the text feature and the visual feature through the large language model of the prompt generator to obtain multimodal feature information;

[0110] A generating module 504, configured to generate a posture prompt vector based on the multimodal feature information through the visual embedding projection layer of the prompt generator, wherein the posture prompt vector is used to represent the target focus position in the template image;

[0111] An output module 505 is used to input the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator;

[0112] The analysis module 506 is used to perform key point position analysis on the key point heat map to obtain a pose estimation result of the target image.

[0113] Specifically, the above-mentioned deep learning-based posture estimation system provided by the present invention can implement all the method steps implemented by the above-mentioned deep learning-based posture estimation method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0114] refer to Figure 6 , Figure 6 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a posture estimation method based on deep learning, which includes: obtaining a target image, a natural language instruction and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; encoding the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features; performing multimodal feature fusion on the text features and the visual features through a large language model of the prompt generator to obtain multimodal feature information; generating a posture prompt vector based on the multimodal feature information through a visual embedding projection layer of the prompt generator, wherein the posture prompt vector is used to represent the target focus position in the template image; inputting the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; performing key point position analysis on the key point heat map to obtain a posture estimation result of the target image.

[0115] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the deep learning-based posture estimation method provided by the above methods, the method including: obtaining a target image, a natural language instruction and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; encoding the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features; performing multimodal feature fusion on the text features and the visual features through a large language model of the prompt generator to obtain multimodal feature information; generating a posture prompt vector based on the multimodal feature information through the visual embedding projection layer of the prompt generator, wherein the posture prompt vector is used to represent the target focus position in the template image; inputting the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; performing key point position analysis on the key point heat map to obtain a posture estimation result of the target image.

[0117] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the deep learning-based posture estimation method provided by the above-mentioned methods, the method comprising: obtaining a target image, a natural language instruction and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; encoding the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features; performing multimodal feature fusion on the text features and the visual features through a large language model of the prompt generator to obtain multimodal feature information; generating a posture prompt vector based on the multimodal feature information through a visual embedding projection layer of the prompt generator, wherein the posture prompt vector is used to represent the target focus position in the template image; inputting the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; performing key point position analysis on the key point heat map to obtain a posture estimation result of the target image.

[0118] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A posture estimation method based on deep learning, characterized in that: include: Acquire a target image, a natural language instruction, and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; By using a preset prompt generator, the natural language instruction and the template image are respectively encoded to obtain text features and visual features; Performing multimodal feature fusion on the text features and the visual features through the large language model of the prompt generator to obtain multimodal feature information; Generate a gesture cue vector based on the multimodal feature information through a visual embedding projection layer of the cue generator, wherein the gesture cue vector is used to represent a target focus position in the template image; Inputting the target image and the posture prompt vector into a preset Transformer-based posture estimator to obtain a key point heat map output by the posture estimator; Performing key point position analysis on the key point heat map to obtain a pose estimation result of the target image; Before acquiring the target image, the natural language instruction and the template image, the method further includes: Acquire a posture estimation sample data set and a command question and answer sample data set of targets of different categories, wherein each posture estimation sample in the posture estimation sample data set includes a target point annotation; Converting the target point annotation of each posture estimation sample in the posture estimation sample data set into a dialogue annotation to obtain a posture estimation instruction question and answer sample data set; Based on the posture estimation instruction question and answer sample data set and the instruction question and answer sample data set, jointly training the prompt generator and the posture estimator; For each posture estimation sample in the posture estimation sample data set, randomly generate different types of questions and answers for instruction fine-tuning, wherein the types of questions and answers include at least one of the following: A natural language instruction sample, wherein the natural language instruction sample includes an image category; The natural language instruction sample and the template image sample, wherein the natural language instruction sample does not include the image category and the template image sample includes a visual prompt; The natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category and the template image sample does not include the visual prompt; The natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category, and the template image sample includes the visual prompt.

2. The posture estimation method based on deep learning according to claim 1, characterized in that: The step of inputting the target image and the posture prompt vector into a preset posture estimator based on a Transformer structure to obtain a key point heat map output by the posture estimator includes: By using a preset posture estimator, feature extraction is performed on the target image to obtain a target image feature code, wherein the target image feature code has the same dimension as the posture prompt vector; The target image feature encoding is aligned with the posture prompt vector through the posture estimator to obtain a key point heat map.

3. The posture estimation method based on deep learning according to claim 2, characterized in that: After extracting features from the target image to obtain the target image feature code, the method further includes: The target image feature encoding and the pose prompt vector are feature fused through the attention mechanism of the pose estimator to learn the weights between different modalities.

4. The posture estimation method based on deep learning according to claim 2, characterized in that: The step of aligning the target image feature encoding with the posture prompt vector to obtain a key point heat map includes: Aligning based on the similarity between the target image feature encoding and the posture prompt vector to obtain a similarity graph; The similarity map is used as a key point heat map, wherein the number of channels of the key point heat map is the same as the number of channels of the posture prompt vector.

5. A posture estimation system based on deep learning, characterized in that: include: An acquisition module, used to acquire a target image, a natural language instruction and a template image, wherein the natural language instruction includes the image category of the target image, and the template image is used to provide a visual prompt for the target image; An encoding module, used to encode the natural language instruction and the template image respectively through a preset prompt generator to obtain text features and visual features; A fusion module, used for performing multimodal feature fusion on the text features and the visual features through the large language model of the prompt generator to obtain multimodal feature information; A generating module, configured to generate a posture hint vector based on the multimodal feature information through a visual embedding projection layer of the hint generator, wherein the posture hint vector is used to represent a target focus position in the template image; An output module, used for inputting the target image and the posture prompt vector into a preset posture estimator based on a Transformer structure to obtain a key point heat map output by the posture estimator; A parsing module, used to perform key point position parsing on the key point heat map to obtain a pose estimation result of the target image; Before acquiring the target image, the natural language instruction and the template image, the system is further used for: Acquire a posture estimation sample data set and a command question and answer sample data set of targets of different categories, wherein each posture estimation sample in the posture estimation sample data set includes a target point annotation; Converting the target point annotation of each posture estimation sample in the posture estimation sample data set into a dialogue annotation to obtain a posture estimation instruction question and answer sample data set; Based on the posture estimation instruction question and answer sample data set and the instruction question and answer sample data set, jointly training the prompt generator and the posture estimator; For each posture estimation sample in the posture estimation sample data set, randomly generate different types of questions and answers for instruction fine-tuning, wherein the types of questions and answers include at least one of the following: A natural language instruction sample, wherein the natural language instruction sample includes an image category; The natural language instruction sample and the template image sample, wherein the natural language instruction sample does not include the image category and the template image sample includes a visual prompt; The natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category and the template image sample does not include the visual prompt; The natural language instruction sample and the template image sample, wherein the natural language instruction sample includes the image category, and the template image sample includes the visual prompt.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the deep learning-based posture estimation method as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the deep learning-based posture estimation method as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the deep learning-based posture estimation method as described in any one of claims 1 to 4 is implemented.

Images