High-precision attitude estimation method for processing limited data based on pre-training diffusion model

Abstract

The invention provides a high-precision attitude estimation method for processing limited data based on a pre-training diffusion model, and the method comprises the steps: carrying out the preprocessing of an input image, and calling a function to combine with noise to generate a key point prompt; mapping the input image through an encoder to obtain a potential code, and increasing Gaussian noise of different amplitudes on the potential code to obtain a noise image; performing attention mechanism processing on the noise image to obtain a semantic corresponding point between the input image and the reference image; training the image pair, integrating training results of the basic class image and the new class image, and inputting an integration result into an Adam optimizer for optimization to obtain an optimization result; evaluating the image pair; according to the method, the key points of the unlabeled image data can be learned in a cross-category mode, a good attitude estimation effect is achieved under the condition that the data is limited, and the actual application value of attitude estimation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and attitude estimation, and in particular to a high-precision attitude estimation method based on a pre-trained diffusion model to process restricted data. Background Art

[0002] Pose estimation is a fundamental computer vision task and has a wide range of applications in the real world, including intelligent interaction, behavior analysis, and augmented reality. Most existing methods focus on fully supervised learning of specific categories, including humans, animals, and vehicles. Technically, these methods can be roughly divided into regression-based, heatmap-based, and transformer-based methods. However, these methods require rich annotations, and the learned key points cannot be transferred to novel classes. Few-shot learning and unsupervised learning are two effective and common paradigms for saving annotation costs. However, few-shot key point estimation still requires rich and non-negligible annotations. Although unsupervised key point detection can learn key points from unannotated data, it is still difficult to achieve the expected key point prediction and only supports single-category key point prediction.

[0003] Recently, large image diffusion models have achieved impressive image generation quality. These models learn priors of real images within the diffusion model latent space and provide useful initialization for many downstream tasks such as image correspondence, object detection, semantic segmentation, and image classification. Interestingly, without any retraining, these models show an innate ability to understand 3D spatial configurations. Recent work in each of these areas shows the emerging power of these large models, most of which use the model without any modification or additional supervision. More importantly, the cross-attention maps of pre-trained stable diffusion models are found to connect textual tags to semantically related regions in the image.

[0004] Semantic consistency is a basic requirement for pose estimation. Recent work has successfully extracted accurate semantic correspondences from diffusion models. Semantic correspondences allow the learned keypoints prompts to represent common semantics in each image, such as eyes, mouth, etc.

[0005] Although existing pose estimation methods have achieved great success, these methods usually require large-scale labeled data of all categories for fully supervised learning. The expensive labeled data greatly limits the category expansion and wide application of pose estimation. Summary of the invention

[0006] In view of the above situation, the main purpose of the present invention is to propose a high-precision posture estimation method based on a pre-trained diffusion model to process limited data, so as to solve the above technical problems.

[0007] The present invention proposes a high-precision posture estimation method based on a pre-trained diffusion model to process limited data, the method comprising the following steps: Step 1: Obtain an input image and a reference image of an image, perform image classification on the input image and the reference image, and obtain a basic category image and a new category image, wherein the basic category image is a labeled picture class; Performing image scaling on the base category image and the new category image to obtain an image scaling result, and performing a normalization mechanism on the image scaling result to obtain a preprocessed image pair; Call the function to generate a random embedding vector, initialize the random embedding vector with noise, and get the key point prompt; Step 2: Map the preprocessed image pair through the encoder to obtain a latent code, and add Gaussian noise to the latent code to obtain a noisy image; Step 3: Input the noisy image into the encoder and combine it with the key point prompts to obtain the query representation of the attention head and the key representation of the attention head respectively; The query representation of the attention head and the key representation of the attention head are processed by a normalization mechanism to obtain a comprehensive cross-attention graph; Input the preprocessed image pair into the decoder in U-Net, and take the output of the residual block in the decoder as the feature map; The feature map is weighted by using an aggregation network with mixed weights to obtain an aggregated feature map; The operation of taking the highest point of cosine similarity on the aggregated feature map is performed to obtain the semantic corresponding point between the input image and the reference image in the preprocessed image pair; Step 4: According to the category to which the image belongs, for the annotated image class, the key point prompt and aggregation network are learned respectively through the annotated matching loss and the annotated semantic loss; For unlabeled images, effective key points are learned through unlabeled unsupervised loss and unlabeled consistency loss, and back propagation training is performed using unlabeled unsupervised loss and unlabeled consistency loss to improve the semantic understanding function of the model; The weights and parameters of the model are updated according to the learning of the labeled image class and the back propagation training of the unlabeled image class to obtain the trained model; Step 5: Input the new category image into the trained model, and make predictions based on the key point prompts to obtain the key point heat map of the predicted new category image test image. Upsample the key point heat map of the predicted new category image test image to obtain the coordinates of the maximum value of the key point heat map.

[0008] Compared with the prior art, the present invention has the following beneficial effects: 1. This paper uses a compact framework to use a pre-trained diffusion model for data-constrained pose estimation. It explores the case of data constraints. With the support of limited labeled data of the base category, key points can be learned from some unlabeled data. The key point hints and correspondences learned from the base category can be transferred to the new category to learn effective key points. 2. The present invention achieves a better attitude estimation effect while using only a small amount of labeled data and some unlabeled data, which is more economical and improves the practicality of the attitude estimation method; 3. The present invention uses a matching algorithm to match the predicted heat map and the real label, so that the learned knowledge is embedded in the same position of the key point prompt, thereby generating transferability; 4. The present invention combines key point heat maps with aggregated image features, designs multi-task objectives to optimize key point prompts and aggregate networks respectively, and uses learned semantic correspondence functions to enhance key point prediction performance. This improves the accuracy and generalization ability of the model's key point prediction; 5. The present invention sets key point prompts and aggregation networks as updateable parameters to reduce computing costs and accelerate the training process. It utilizes pre-trained model knowledge through fine-tuning to adapt to tasks faster and improve task performance.

[0009] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a high-precision posture estimation method based on a pre-trained diffusion model for processing restricted data proposed by the present invention; Figure 2 This is an overall framework diagram of the training phase of the high-precision posture estimation method based on the pre-trained diffusion model to process restricted data proposed in the present invention. DETAILED DESCRIPTION

[0011] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0012] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0013] See also Figure 1 The embodiment of the present invention provides a high-precision posture estimation method based on a pre-trained diffusion model to process limited data, the method comprising the following steps: Step 1: Obtain an input image and a reference image of an image, perform image classification on the input image and the reference image, and obtain a basic category image and a new category image, wherein the basic category image is a labeled picture class; Performing image scaling on the base category image and the new category image to obtain an image scaling result, and performing a normalization mechanism on the image scaling result to obtain a preprocessed image pair; Call the function to generate a random embedding vector, initialize the random embedding vector with noise, and get the key point prompt.

[0014] Furthermore, the image category is determined. The basic category image is the labeled image, and the new category image is the unlabeled image. In the experiment, the default configuration is 70 categories with 3 basic category images per category and 20 categories with 30 new category images as the training set, and 20 categories with 40 new category images per category as the test set. Each time a pair of images of the same category is input into the model, such as a pair of basic category images or a pair of new category images.

[0015] Scale the image to a uniform size with a resolution of 256 256; Normalize the scaled image.

[0016] In this step, the basic category images and new category images are set to learn the effective key points of the new category unlabeled images with the support of the labeled images of the basic category. Using a small number of basic category images and some new category images in the training phase is more economical and practical than supervised learning, few-sample learning, and unsupervised learning. In the test phase, new category images that were not used in the training phase are used to predict key points to ensure the generalization of the model.

[0017] Step 2: Map the preprocessed image pair through the encoder to obtain a latent code, and add Gaussian noise to the latent code to obtain a noisy image.

[0018] See also Figure 2 In step 2, the preprocessed image is mapped through the encoder to obtain the potential code. The corresponding relationship of the process is as follows: ; in, represents the potential code, represents the encoder, Represents the input image.

[0019] It should be noted that the encoder comes from a vector quantization generative adversarial network, which facilitates the transformation between pixel space and latent space.

[0020] In the step of adding Gaussian noise to the latent code to obtain a noisy image, the corresponding relationship of the process is as follows: ; in, represents a noisy image, represents a predefined noise schedule, represents randomly sampled noise.

[0021] It should be noted that during the diffusion process, the amount of noise added to the image gradually increases as time goes by. The amount of noise is determined by a predefined noise plan, which specifies how much noise should be added at each time point.

[0022] Step 3: Input the noisy image into the encoder and combine it with the key point prompts to obtain the query representation of the attention head and the key representation of the attention head respectively; The query representation of the attention head and the key representation of the attention head are processed by a normalization mechanism to obtain a comprehensive cross-attention graph; Input the preprocessed image pair into the decoder in U-Net, and take the output of the residual block in the decoder as the feature map; The feature map is weighted by using an aggregation network with mixed weights to obtain an aggregated feature map; The operation of taking the highest point of cosine similarity on the aggregated feature map is performed to obtain the semantic corresponding points between the input image and the reference image in the preprocessed image pair.

[0023] In step 3, the noise image is input into the encoder, and combined with the key point prompt, the query representation of the attention head and the key representation of the attention head are obtained respectively. The calculation process of the query representation of the attention head corresponds to the following relationship: ; in, Represents the first Transformer in U-Net The attention head and The query representation calculated by the linear layer is Represents the first A head of attention, represents the shape of the data tensor, Indicates the image height of the specified layer of U-Net. Indicates the image width of the specified layer of U-Net, Indicates the dimension of the specified layer of U-Net; In the steps of inputting the noisy image into the encoder and combining the key point prompts to obtain the query representation of the attention head and the key representation of the attention head, the calculation process of the key representation of the attention head corresponds to the following relationship: ; in, Represents the first Transformer in U-Net The attention head and The key representation calculated by the linear layer is Indicates the first Linear layers, represents the shape of the data tensor, Indicates the number of tokens. Indicates key point prompts; In the step of normalizing the query representation of the attention head and the key representation of the attention head to obtain a comprehensive cross-attention graph, the corresponding relationship between the process is as follows: ; in, represents the comprehensive cross-attention map, Indicated in arrive The average value between layers is taken. Indicated in The average value of the attention heads, represents the dimension of the attention map, Indicated in The attention heads are normalized; In the step of performing mixed weight processing on the feature map using the aggregation network to obtain the aggregated feature map, the corresponding relationship between the process is as follows: ; in, represents the aggregate feature map, represents the time step, represents the total number of linear layers, represents the bottleneck layer, Indicates The linear layer The feature map of the time step, Indicates The linear layer The mixing weights of the time steps; In the step of obtaining the semantic corresponding points between the input image and the reference image in the preprocessed image pair by taking the highest cosine similarity point of the aggregated feature map, the corresponding relationship between the process is as follows: ; in, Represents the semantic correspondence operation given an input image point Find the reference image The corresponding point , represents one of the aggregate features extracted from two images in a semantic correspondence, represents another aggregate feature extracted from two images in the semantic correspondence, It represents the operation of taking the highest point of cosine similarity. Indicates the operation of calculating cosine similarity, Indicates the processing of the aggregation network mechanism.

[0024] Furthermore, during the model training process, in order to extract key point cues and areas associated with the semantic content of the image, the upper cross attention maps of specific layers in the denoiser were collected. Specifically, a total of nine layers of upper cross attention maps were collected to facilitate further analysis and identify key areas that are closely connected to the semantic content of the image.

[0025] Step 4: According to the category to which the image belongs, for the annotated image class, the key point prompt and aggregation network are learned respectively through the annotated matching loss and the annotated semantic loss; For unlabeled images, effective key points are learned through unlabeled unsupervised loss and unlabeled consistency loss, and back propagation training is performed using unlabeled unsupervised loss and unlabeled consistency loss to improve the semantic understanding function of the model; The weights and parameters of the model are updated according to the learning of the labeled image classes and the back-propagation training of the unlabeled image classes to obtain the trained model.

[0026] In step 4, according to the category to which the image belongs, for the labeled picture class, the key point prompt and the aggregation network are learned respectively through the labeled matching loss and the labeled semantic loss. The specific steps are as follows: Perform image classification on the preprocessed image pairs, judge the classification categories according to the image pairs, and obtain the judgment results; When the judgment result is a base category image, obtain the true label of the base category image; Repeat steps 1 to 3 in an iterative manner using the training set to obtain a key point heat map of the base category image prediction and an aggregated image feature of the base category image prediction; The key point heat map predicted by the base category image is matched with the comprehensive cross attention map, combined with the semantic corresponding points between the input image and the reference image in the preprocessed image pair, to obtain the matched predicted key point heat map and the matched real key point heat map; Based on the matched predicted key point heat map and the matched real key point heat map, a mean square error loss is constructed to obtain the matching loss of the base category image, and the key point prompts are learned using the matching loss of the base category image to obtain the learned key point prompts; The aggregated image features predicted by the base category image are combined with the true label to perform a symmetric cross entropy loss mechanism to obtain the symmetric cross entropy loss calculation result of the base category image, and the aggregate network is learned through the symmetric cross entropy loss calculation result of the base category image to obtain the learned aggregate network; Among them, for the unlabeled image class, the effective key points are learned through the unlabeled unsupervised loss and the unlabeled consistency loss, and the back propagation training is performed using the unlabeled unsupervised loss and the unlabeled consistency loss to improve the semantic understanding function of the model. The specific steps are as follows: When the judgment result is a new category image, the key point coordinates and visibility prediction are performed on the comprehensive cross-attention map, and the aggregation network is used for mechanism processing to obtain the corresponding points of the input image in the reference image, and the pseudo label points are obtained using the corresponding points of the input image in the reference image; The operation of taking the highest point of cosine similarity of the aggregated feature map is performed, and combined with the comprehensive cross-attention map to obtain the key point heat map of the image pair prediction and the pseudo label points of the image pair prediction, and the pseudo label points of the image pair prediction are transformed to obtain the transformed Gaussian heat map; Based on the key point heat map predicted by the image pair and the transformed Gaussian heat map, the mean square error loss is constructed to obtain the consistency loss calculation result of the new category image, and the key point heat map predicted by the image pair is learned through the consistency loss calculation result of the new category image to obtain a more concentrated key point heat map predicted by the image pair; The positioning loss is calculated between the predicted key point heat map and the Gaussian heat map to obtain the positioning loss calculation result of the key point heat map predicted by the new category image. The positioning loss calculation result of the key point heat map predicted by the new category image and the consistency loss calculation result of the new category image are used to perform back propagation training to obtain a model with enhanced semantic understanding function. Among them, the weights and parameters of the model are updated according to the learning of the labeled image class and the back propagation training of the unlabeled image class to obtain the trained model. The specific steps are as follows: Calculate the positioning loss calculation result of the key point heat map predicted by the new category image and the consistency loss calculation result of the new category image to obtain the overall loss calculation result of the image pair; Dynamic weight allocation is performed on the matching loss of the base category image, the symmetric cross entropy loss calculation result of the base category image, the consistency loss calculation result of the new category image, and the positioning loss calculation result of the key point heat map predicted by the new category image to obtain the dynamic weight allocation result of multiple tasks; The matching loss of the base category image, the symmetric cross entropy loss calculation results of the base category image, the consistency loss calculation results of the new category image, and the positioning loss calculation results of the key point heat map predicted by the new category image are integrated and input into the Adam optimizer to update the weights and parameters of the model to obtain the trained model.

[0027] In the step of constructing the mean square error loss based on the matched predicted key point heat map and the matched real key point heat map to obtain the matching loss of the base category image, the corresponding relationship of the matching loss calculation process of the base category image is as follows: ; in, represents the basic category image matching loss function, represents the total number of key points, Indicates that the matching function The input image found Previous Prediction key point heat map, Represents the input image Previous A real key point heat map, Indicates that the matching function Reference images found Previous Prediction key point heat map, Represents the reference image Previous A heat map of real key points.

[0028] In the step of combining the aggregated image features predicted by the base category image with the true label to perform a symmetric cross entropy loss mechanism to obtain the symmetric cross entropy loss calculation result of the base category image, and learning the aggregated network through the symmetric cross entropy loss calculation result of the base category image to obtain the learned aggregated network, the relationship corresponding to the calculation process of the learned aggregated network is as follows: ; in, represents the aggregation network function optimized for base class images, represents the aggregated feature map of the input image, represents the aggregated feature map of the reference image, represents the cross entropy function, Represents the input image The index set of the real key points on the feature map scale after coordinate mapping, Represents the index set of the real key points of the reference image at the feature map scale after coordinate mapping.

[0029] When the result is judged to be a new category image, the key point coordinates and visibility prediction are performed on the comprehensive cross-attention map, and the aggregation network is used for mechanism processing to obtain the corresponding points of the input image in the reference image. In the step of obtaining the pseudo label points using the corresponding points of the input image in the reference image, the corresponding relationship of the process is as follows: ; in, Pseudo-label points representing new category images.

[0030] In the step of taking the highest point of cosine similarity from the aggregated feature map, combining the comprehensive cross-attention map to obtain the key point heat map predicted by the image pair and the pseudo-label points predicted by the image pair, transforming the pseudo-label points predicted by the image pair to obtain the transformed Gaussian heat map, the corresponding relationship between the process is as follows: ; in, The reference image The Gaussian heat map generated by the maximum coordinates of the pseudo key point coordinates corresponding to the predicted key points, Represents the input image The Gaussian heat map generated by the maximum coordinates of the pseudo key point coordinates corresponding to the predicted key points, represents the generated coordinate tensor, The reference image The pseudo keypoint coordinates corresponding to the predicted keypoints, Represents the input image The pseudo keypoint coordinates corresponding to the predicted keypoints, represents the exponential function, Represents the parameter sigma required to generate the Gaussian plot.

[0031] It should be noted that Used to calculate the value of Gaussian distribution; The Gaussian image size is generated according to the need The operation and dimension adjustment generate image coordinates of dimension [64, 64, 2]. The height and width of the Gaussian image are set to 64 to be compatible with the size of the extracted key point heat map.

[0032] In the step of constructing the mean square error loss based on the predicted key point heat map of the image pair and the transformed Gaussian heat map to obtain the consistency loss calculation result of the new category image, the corresponding relationship between the process is as follows: ; in, represents the consistency loss function of the new category image, Indicates Key point heat map The visibility of is taken as the confidence level, Indicates Key point heat map The visibility of is taken as the confidence level, Represents the input image No. Prediction heatmap of key points, Represents the reference image No. Prediction heatmap of key points, Represents the operation of converting keypoints to Gaussian maps.

[0033] In the step of calculating the positioning loss calculation result of the key point heat map predicted by the new category image and the consistency loss calculation result of the new category image to obtain the overall loss calculation result of the image pair, the corresponding relationship between the process is as follows: ; in, represents the overall loss function, Indicates that the equivariant loss is calculated for each key point and the average value is taken. represents a reversible image geometric transformation operation, Indicates Normalized predicted key point heat map generated by key points, Represents the key point heat map after the image has been reversibly transformed. represents the key point heat map, Indicates The Gaussian heat map generated by the maximum position of the pseudo keypoint coordinates corresponding to the predicted keypoints.

[0034] It should be noted that Used to restore the transformed feature heat map to the original coordinate space. The specific setting is an affine transformation within a certain range, such as randomly rotating the image angle. Spend.

[0035] In the step of dynamically allocating weights to the matching loss of the base category image, the symmetric cross entropy loss calculation result of the base category image, the consistency loss calculation result of the new category image, and the positioning loss calculation result of the key point heat map predicted by the new category image, and obtaining the dynamic weight allocation result of multiple tasks, the corresponding relationship between the process is as follows: ; in, Represents the weight distribution result of the overall loss, represents the indicator function, Representation and Different indicator functions, , and represents three different hyperparameters that make trade-offs.

[0036] It should be noted that As an exponential function, it can be used to determine whether an image belongs to a basic category or a new category. If it is a basic category image pair, matching loss and cross entropy loss are used to optimize the key point prompt and aggregation network respectively. If it is a new category image pair, consistency loss and unsupervised loss are used to learn the key point heat map. , and The parameters in the experiments are set to 0.1, 0.1, and 0.5 respectively.

[0037] Furthermore, because the new category images are unlabeled data, consistency loss and positioning loss are used as the loss function for predicting the key point heat map of the new category images to supplement the unsupervised learning of the new category images. The positioning loss is to make the predicted key point heat map approximate the Gaussian distribution, and the consistency loss is to ensure the geometric invariance of the model's attention mechanism. The loss is minimized by updating the weights and learning parameters to improve the accuracy of the predicted key point heat map and semantic correspondence.

[0038] Step 5: Input the new category image into the trained model, and make predictions based on the key point prompts to obtain the key point heat map of the predicted new category image test image, and perform upsampling operation on the key point heat map of the predicted new category image test image to obtain the coordinates of the maximum value of the key point heat map; The accuracy mean is calculated based on the coordinates of the maximum value of the key point heat map and the true key point heat map to obtain the accuracy mean calculation result, and the prediction result is evaluated based on the accuracy mean calculation result.

[0039] In step 5, the new category image is input into the trained model, and predicted in combination with the key point prompts to obtain the key point heat map of the predicted new category image test image, and the key point heat map of the predicted new category image test image is upsampled to obtain the coordinates of the maximum value of the key point heat map. The corresponding relationship of the process is as follows: ; in, Indicates that it is The coordinates of the predicted key points, Represents finding the key point heat map The index operation of the pixel point with the largest value in , Represents the pixel coordinates of the key point heat map in the width direction, It represents the pixel coordinates of the key point heat map in the height direction.

[0040] Furthermore, the accuracy mean is calculated based on the coordinates of the maximum value of the key point heat map and the true key point heat map to obtain the accuracy mean calculation result. The prediction result is evaluated by the accuracy mean calculation result. The corresponding relationship of the process is as follows: ; in, represents the sum of the accuracy scores of all test images, Indicates the test image The visibility of the real label key points, Indicates the test image The coordinates of the predicted key points, Indicates the test image The coordinates of the real label key points, Indicates the test image The visibility of the predicted key points.

[0041] Furthermore, considering that the true labels of new category images have never been used in the experimental setting, an evaluation method based on the Hungarian matching algorithm is adopted for evaluation. Since unmatched key points will reduce the recall rate, the mean precision (AP) is used as the evaluation indicator.

[0042] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0043] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0044] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A high-precision attitude estimation method based on a pre-trained diffusion model to process restricted data, characterized in that: The method comprises the following steps: Step 1: Obtain an input image and a reference image of an image, perform image classification on the input image and the reference image, and obtain a basic category image and a new category image, wherein the basic category image is a labeled picture class; Performing image scaling on the base category image and the new category image to obtain an image scaling result, and performing a normalization mechanism on the image scaling result to obtain a preprocessed image pair; Call the function to generate a random embedding vector, initialize the random embedding vector with noise, and get the key point prompt; Step 2: Map the preprocessed image pair through the encoder to obtain a potential code, and add Gaussian noise to the potential code to obtain a noisy image; Step 3: Input the noisy image into the encoder and combine it with the key point prompts to obtain the query representation of the attention head and the key representation of the attention head respectively; The query representation of the attention head and the key representation of the attention head are processed by a normalization mechanism to obtain a comprehensive cross-attention graph; Input the preprocessed image pair into the decoder in U-Net, and take the output of the residual block in the decoder as the feature map; The feature map is weighted by using an aggregation network with mixed weights to obtain an aggregated feature map; The operation of taking the highest point of cosine similarity on the aggregated feature map is performed to obtain the semantic corresponding point between the input image and the reference image in the preprocessed image pair; Step 4: According to the category to which the image belongs, for the annotated image class, the key point prompt and aggregation network are learned respectively through the annotated matching loss and the annotated semantic loss; For unlabeled images, effective key points are learned through unlabeled unsupervised loss and unlabeled consistency loss, and back propagation training is performed using unlabeled unsupervised loss and unlabeled consistency loss to improve the semantic understanding function of the model; The weights and parameters of the model are updated according to the learning of the labeled image class and the back propagation training of the unlabeled image class to obtain the trained model; Step 5: Input the new category image into the trained model, and make predictions based on the key point prompts to obtain the key point heat map of the predicted new category image test image. Upsample the key point heat map of the predicted new category image test image to obtain the coordinates of the maximum value of the key point heat map.

2. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 1 is characterized in that: In step 3, the noise image is input into the encoder, and combined with the key point prompt, the query representation of the attention head and the key representation of the attention head are obtained respectively. The calculation process of the query representation of the attention head corresponds to the following relationship: ； in, Represents the first Transformer in U-Net The attention head and The query representation calculated by the linear layer is Represents the first A head of attention, represents the shape of the data tensor, Indicates the image height of the specified layer of U-Net. Indicates the image width of the specified layer of U-Net, Indicates the dimension of the specified layer of U-Net, represents a noisy image; In the steps of inputting the noisy image into the encoder and combining the key point prompts to obtain the query representation of the attention head and the key representation of the attention head, the calculation process of the key representation of the attention head corresponds to the following relationship: ； in, Represents the first Transformer in U-Net The attention head and The key representation calculated by the linear layer is Indicates the first Linear layers, represents the shape of the data tensor, Indicates the number of tokens. Indicates key point prompts; In the step of normalizing the query representation of the attention head and the key representation of the attention head to obtain a comprehensive cross-attention graph, the corresponding relationship between the process is as follows: ； in, represents the comprehensive cross-attention map, Indicated in arrive The average value between layers is taken. Indicated in The average value of the attention heads, represents the dimension of the attention map, Indicated in The attention heads are normalized; In the step of performing mixed weight processing on the feature map using the aggregation network to obtain the aggregated feature map, the corresponding relationship between the process is as follows: ； in, represents the aggregate feature map, represents the time step, represents the total number of linear layers, represents the bottleneck layer, Indicates The linear layer The feature map of the time step, Indicates The linear layer The mixing weights of the time steps; In the step of obtaining the semantic corresponding points between the input image and the reference image in the preprocessed image pair by taking the highest cosine similarity point of the aggregated feature map, the corresponding relationship between the process is as follows: ； in, Represents the semantic correspondence operation given an input image point Find the reference image The corresponding point , represents one of the aggregate features extracted from two images in a semantic correspondence, represents another aggregate feature extracted from two images in the semantic correspondence, It represents the operation of taking the highest point of cosine similarity. Indicates the operation of calculating cosine similarity, Indicates the processing of the aggregation network mechanism.

3. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 2 is characterized in that: In step 4, according to the category to which the image belongs, for the labeled picture class, the key point prompt and the aggregation network are learned respectively through the labeled matching loss and the labeled semantic loss. The specific steps are as follows: Perform image classification on the preprocessed image pairs, judge the classification categories according to the image pairs, and obtain the judgment results; When the judgment result is a base category image, obtain the true label of the base category image; Repeat steps 1 to 3 in an iterative manner using the training set to obtain a key point heat map of the base category image prediction and an aggregated image feature of the base category image prediction; The predicted key point heat map of the base category image is matched with the comprehensive cross attention map, combined with the semantic corresponding points between the input image and the reference image in the preprocessed image pair, to obtain the matched predicted key point heat map and the matched real key point heat map; Based on the matched predicted key point heat map and the matched real key point heat map, a mean square error loss is constructed to obtain the matching loss of the base category image, and the key point prompts are learned using the matching loss of the base category image to obtain the learned key point prompts; The aggregated image features predicted by the base category image are combined with the true label to perform a symmetric cross entropy loss mechanism to obtain the symmetric cross entropy loss calculation result of the base category image, and the aggregate network is learned through the symmetric cross entropy loss calculation result of the base category image to obtain the learned aggregate network; Among them, for the unlabeled image class, the effective key points are learned through the unlabeled unsupervised loss and the unlabeled consistency loss, and the back propagation training is performed using the unlabeled unsupervised loss and the unlabeled consistency loss to improve the semantic understanding function of the model. The specific steps are as follows: According to the judgment results, the new category images are trained to obtain the training results of the new category images: When the judgment result is a new category image, the key point coordinates and visibility prediction are performed on the comprehensive cross-attention map, and the aggregation network is used for mechanism processing to obtain the corresponding points of the input image in the reference image, and the pseudo label points are obtained using the corresponding points of the input image in the reference image; The operation of taking the highest point of cosine similarity of the aggregated feature map is performed, and combined with the comprehensive cross-attention map to obtain the key point heat map of the image pair prediction and the pseudo label points of the image pair prediction, and the pseudo label points of the image pair prediction are transformed to obtain the transformed Gaussian heat map; Based on the key point heat map predicted by the image pair and the transformed Gaussian heat map, the mean square error loss is constructed to obtain the consistency loss calculation result of the new category image, and the key point heat map predicted by the image pair is learned through the consistency loss calculation result of the new category image to obtain a more concentrated key point heat map predicted by the image pair; The positioning loss is calculated between the predicted key point heat map and the Gaussian heat map to obtain the positioning loss calculation result of the key point heat map predicted by the new category image. The positioning loss calculation result of the key point heat map predicted by the new category image and the consistency loss calculation result of the new category image are used to perform back propagation training to obtain a model with enhanced semantic understanding function. Among them, the weights and parameters of the model are updated according to the learning of the labeled image class and the back propagation training of the unlabeled image class to obtain the trained model. The specific steps are as follows: Calculate the positioning loss calculation result of the key point heat map predicted by the new category image and the consistency loss calculation result of the new category image to obtain the overall loss calculation result of the image pair; Dynamic weight allocation is performed on the matching loss of the base category image, the symmetric cross entropy loss calculation result of the base category image, the consistency loss calculation result of the new category image, and the positioning loss calculation result of the key point heat map predicted by the new category image to obtain the dynamic weight allocation result of multiple tasks; The matching loss of the base category image, the symmetric cross entropy loss calculation results of the base category image, the consistency loss calculation results of the new category image, and the positioning loss calculation results of the key point heat map predicted by the new category image are integrated and input into the Adam optimizer to update the weights and parameters of the model to obtain the trained model.

4. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 3 is characterized in that: In the step of constructing the mean square error loss based on the matched predicted key point heat map and the matched real key point heat map to obtain the matching loss of the base category image, the corresponding relationship of the matching loss calculation process of the base category image is as follows: ； in, represents the basic category image matching loss function, represents the total number of key points, Indicates that the matching function The input image found Previous Prediction key point heat map, Represents the input image Previous A real key point heat map, Indicates that the matching function Reference images found Previous Prediction key point heat map, Represents the reference image Previous A heat map of real key points.

5. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 4 is characterized in that: In the step of combining the aggregated image features predicted by the base category image with the true label to perform a symmetric cross entropy loss mechanism to obtain the symmetric cross entropy loss calculation result of the base category image, and learning the aggregated network through the symmetric cross entropy loss calculation result of the base category image to obtain the learned aggregated network, the relationship corresponding to the calculation process of the learned aggregated network is as follows: ； in, represents the aggregation network function optimized for base class images, represents the aggregated feature map of the input image, represents the aggregated feature map of the reference image, represents the cross entropy function, Represents the input image The index set of the real key points on the feature map scale after coordinate mapping, Represents the index set of the real key points of the reference image at the feature map scale after coordinate mapping.

6. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 5 is characterized in that: In the step of taking the highest point of cosine similarity from the aggregated feature map, combining the comprehensive cross-attention map to obtain the key point heat map predicted by the image pair and the pseudo-label points predicted by the image pair, transforming the pseudo-label points predicted by the image pair to obtain the transformed Gaussian heat map, the corresponding relationship between the process is as follows: ； in, The reference image The Gaussian heat map generated by the maximum coordinates of the pseudo key point coordinates corresponding to the predicted key points, Represents the input image The Gaussian heat map generated by the maximum coordinates of the pseudo key point coordinates corresponding to the predicted key points, represents the generated image coordinate tensor, The reference image The pseudo keypoint coordinates corresponding to the predicted keypoints, Represents the input image The pseudo keypoint coordinates corresponding to the predicted keypoints, represents the exponential function, Represents the parameter sigma required to generate the Gaussian plot.

7. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 6 is characterized in that: In the step of constructing the mean square error loss based on the predicted key point heat map of the image pair and the transformed Gaussian heat map to obtain the consistency loss calculation result of the new category image, the corresponding relationship between the process is as follows: ； in, represents the consistency loss function of the new category image, Indicates Key point heat map The visibility of is taken as the confidence level, Indicates Key point heat map The visibility of is taken as the confidence level, Represents the input image No. Prediction heatmap of key points, Represents the reference image No. Prediction heatmap of key points, Represents the operation of converting keypoints to Gaussian maps.

8. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 7 is characterized in that: In the step of calculating the positioning loss calculation result of the key point heat map predicted by the new category image and the consistency loss calculation result of the new category image to obtain the overall loss calculation result of the image pair, the corresponding relationship between the process is as follows: ； in, represents the overall loss function, Indicates that the equivariant loss is calculated for each key point and the average value is taken. represents a reversible image geometric transformation operation, Indicates Normalized predicted key point heat map generated by key points, Represents the key point heat map after the image has been reversibly transformed. represents the key point heat map, Indicates The Gaussian heat map generated by the maximum position of the pseudo keypoint coordinates corresponding to the predicted keypoints.

9. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 8, characterized in that: In the step of dynamically allocating weights to the matching loss of the base category image, the symmetric cross entropy loss calculation result of the base category image, the consistency loss calculation result of the new category image, and the positioning loss calculation result of the key point heat map predicted by the new category image, and obtaining the dynamic weight allocation result of multiple tasks, the corresponding relationship between the process is as follows: ； in, Represents the weight distribution result of the overall loss, represents the indicator function, Representation and Different indicator functions, , and represents three different hyperparameters that make trade-offs.

10. The high-precision posture estimation method based on pre-trained diffusion model processing limited data according to claim 9, characterized in that: In step 5, the new category image is input into the trained model, and predicted in combination with the key point prompts to obtain the key point heat map of the predicted new category image test image, and the key point heat map of the predicted new category image test image is upsampled to obtain the coordinates of the maximum value of the key point heat map. The corresponding relationship of the process is as follows: ； in, Indicates The coordinates of the predicted key points, Represents finding the key point heat map The index operation of the pixel point with the largest value in , Represents the pixel coordinates of the key point heat map in the width direction, Represents the pixel coordinates of the key point heat map in the height direction.

Images