Global-to-local key point positioning method and device based on fusion attention

Abstract

The invention discloses a global-to-local key point positioning method and device based on fusion attention, and the method comprises the following steps: 1, collecting a picture through a camera, marking key points of the picture, and carrying out the preprocessing of picture data, and constructing a key point positioning data set; 2, constructing a global regression model and carrying out supervised training; 3, constructing a local regression model on the basis of the global regression model, and supervising and training the local regression model through joint loss; and 4, inputting the preprocessed picture into the global regression model to obtain a key point prediction coordinate. The global regression model provided by the invention extracts the deep features of the image, fuses the image features related to the key points and maps the image features to the heat map space, thereby achieving the high-precision positioning of the key points on the whole image. Moreover, the fusion attention module can effectively capture the relation between the global features and the local features, guides the global features of the image to approach the local features, and achieves the gradual and accurate positioning of the coordinates of the key points in the local blocks.

Description

Technical Field

[0001] The present invention relates to the intersection of image key point positioning and deep learning technology, and in particular to a global to local key point positioning method and device based on fused attention. The method uses a deep neural network model to calculate the fused attention of global features and local block features, and realizes accurate global to local positioning of key points in stages, so as to improve the accuracy and robustness of image key point positioning. Background Art

[0002] The key point localization task refers to identifying some specific and significant points in a given image. These points are usually feature points that have important information about the image content. For example, in facial recognition, key points may be the locations of eyes, nose, and mouth; in object detection, key points may be the edges and corners of an object. The deep learning method of the key point localization task uses deep neural networks to extract structural and semantic information from images, analyze and process its complex features, and accurately predict the coordinates of each key point. It is a key application branch in the field of computer vision that integrates image processing and machine learning. It is widely used in scenarios such as human posture estimation and object recognition, and provides important technical support for the automation and intelligence of image-related applications.

[0003] Among the existing key point automatic localization methods, methods using image gradients and second-order derivative matrices are sensitive to image illumination changes and noise. Methods based on feature point descriptors and matching algorithms cannot flexibly cope with large-scale or dynamically changing scenes. Deep learning methods usually regress key point coordinates directly from image features through heat map regression or coordinate regression models. However, direct regression is too dependent on the texture features of the area around the key point, resulting in large key point localization errors in areas with weak texture features. In addition, when using the attention mechanism to learn the relative relationship between different key points, the key role of global features in localizing areas with weak textures is often ignored, thereby reducing the accuracy of key point localization.

[0004] Therefore, in order to reduce the key point positioning error, taking into account the problem that the local visual features of weak texture areas are not sufficient to support the direct and accurate positioning of the key point coordinates of the image, the present invention proposes a global to local key point positioning method and device based on fused attention. Summary of the invention

[0005] Aiming at the problem in the prior art that the local visual features of weak texture areas are insufficient to support the direct and accurate positioning of key point coordinates, the present invention provides a global to local key point positioning method and device based on fused attention.

[0006] In order to achieve the above-mentioned invention object, the present invention provides the following technical solutions:

[0007] Step 1: Use a camera to capture images and mark their key points, pre-process the image data to build a key point positioning data set;

[0008] Step 2: Construct a global regression model. The global regression model mainly includes a backbone network and a heat map decoding module. The global regression model is constructed by connecting the backbone network and the heat map decoding module in series. The backbone network extracts image structure information and deep semantic information in image representation. The heat map decoding module generates a probability distribution heat map of each key point position, completing the regression from image representation to heat map.

[0009] Step 3: Perform end-to-end supervised training on the global regression model;

[0010] Step 4: Construct a local regression model based on the global regression model; the local regression model mainly includes a local block feature encoding module, a global feature encoding module and a coordinate decoding module; the local block feature encoding module encodes local block features through multi-scale image representation according to the two-dimensional spatial position of the forward key point coordinates; the global feature encoding module fuses the multi-scale image representation to extract the global feature representation of the image; the coordinate decoding module uses the local block features to guide the global features to generate the key point encoding vector, and regresses the encoding vector into a local offset;

[0011] Step 5: Perform end-to-end joint supervised training on the local regression model and the global regression model;

[0012] Step 6: Input the preprocessed positioning image into the trained global regression model. The output of the local regression model is the predicted key point coordinates.

[0013] As a preferred embodiment of the present invention, a global to local key point positioning method based on fusion attention is characterized in that step 1 includes the following processes:

[0014] S11: resize the image to a size of n×n and normalize the pixel values ​​of the image;

[0015] S12: With the help of the pre-trained public image border detection model, the image is roughly detected and cropped to exclude most of the irrelevant background information, while retaining the background margin of k pixels outside the detection frame to ensure that the image information is not lost due to cropping;

[0016] S13: Calculate new key point coordinates according to the cropped border of step S12, and use a two-dimensional Gaussian function to generate a Gaussian heat map with each key point coordinate of each image as the center;

[0017] S14: Divide the processed dataset into training set, test set and validation set.

[0018] As a preferred solution of the present invention, a global to local key point positioning method based on fusion attention is characterized in that the backbone network in step 2 includes a multi-scale feature extraction module and a feature alignment module; the multi-scale feature extraction module is a feature extraction part of an HRNet network with multiple output branches of different scales, and is composed of a two-dimensional convolutional layer and a ReLu activation layer; the feature alignment module completes feature alignment through interpolation operation;

[0019] Specifically, the image is preprocessed to obtain three-channel image data, and the multi-scale feature extraction module extracts multi-scale image information from the image data by loading the pre-trained HRNet model; the feature alignment module takes the multi-scale image information as input, generates interpolated supplementary pixel values ​​by utilizing neighboring information, and generates image structure information and deep semantic information aligned in the spatial dimension to better support subsequent heat map decoding and coordinate decoding tasks.

[0020] As a preferred embodiment of the present invention, a global to local key point positioning method based on fusion attention is characterized in that the heat map decoding module in step 2 includes a feature mapping module and a heat map prediction head module; the feature mapping module is composed of a two-dimensional transposed convolution layer, a batch normalization layer, a ReLu activation function and a Dropout layer in series; the heat map prediction head module is composed of a two-dimensional convolution layer;

[0021] Specifically, the feature mapping module fuses image features of different scales and maps them into a unified initial heat map representation, and also provides basic features for the subsequent global feature extraction module; the heat map prediction head module fuses the initial heat map representation, and maps the position of each key point in the global image into a separate probability distribution map. The position with the largest activation value in each probability distribution map is the predicted key point, which serves as the initial local block center of the subsequent local regression model.

[0022] As a preferred embodiment of the present invention, a global to local key point positioning method based on fused attention is characterized in that the supervised training in step 3 includes: using the key point heat map in the key point positioning data set as label information and the processed image as a sample; performing n rounds of iterative training and saving the model parameters with the smallest loss function value on the test set; the backbone network and heat map decoding module after training are used as the initial parameter values ​​for subsequent joint training; the loss function used in the supervised training is the mean square error loss.

[0023] As a preferred embodiment of the present invention, a global to local key point positioning method based on fusion attention is characterized in that the global feature encoding module in step 4 includes a feature extraction module and a feature compression module; the global feature extraction module is composed of a large-core two-dimensional convolution layer, a batch normalization layer and a ReLu activation function connected in series; the feature compression module is composed of an adaptive pooling layer and a linear mapping layer connected in series;

[0024] Specifically, the global feature extraction module uses large kernel convolution to cover a larger area of ​​the image and aggregate features over a larger range, thereby enhancing the understanding of the overall structure and long-distance dependencies, capturing more comprehensive overall image information, and extracting a global representation of the image; the feature compression module integrates the global representation and performs dimensionality reduction to ensure that the most important global information in the image is retained, thereby enhancing the model's understanding of the global context and improving its ability to focus on specific areas or features, effectively reducing the size of the feature map, compressing the two-dimensional feature map into a one-dimensional global feature vector, and providing global features for subsequent fusion attention calculations.

[0025] As a preferred embodiment of the present invention, a global to local key point positioning method based on fusion attention is characterized in that the local block feature encoding module in step 4 includes a multi-scale feature fusion module and a block encoding module; the multi-scale feature fusion module is composed of a two-dimensional convolution layer, a batch normalization layer and a ReLu function; the block encoding module samples the local block pixel values ​​and performs weighted fusion to generate local block features;

[0026] Specifically, the multi-scale feature fusion module fuses the multi-scale image features aligned by channel attention weighted fusion as the initial feature map of the local block feature; the block coding module uniformly takes n×n point pixels on the fused initial feature map in a square frame with a side length of d pixels centered on the local block to form a local block LP, and calculates the weighted sum of the pixel values ​​of each local block as the current local block feature Code, specifically:

[0027]

[0028] Where i represents the channel number of the local block; LP i represents the i-th channel feature map of the local block; C i Represents the encoded value of the i-th channel feature map of the local block, and each channel of the local block is encoded with a real number; ω i,j,k is the pixel value weight at the i-th feature map coordinate (j, k) of the local block.

[0029] As a preferred embodiment of the present invention, a global to local key point positioning method based on fusion attention is characterized in that the coordinate decoding module in step 4 includes a fusion attention module and an offset prediction module; the coordinate decoding module is formed by alternating stacking of multiple fusion attention modules and offset prediction modules; the fusion attention module is composed of a self-attention layer, a cross attention layer, a forward propagation layer and three two-dimensional position encoding vectors; the offset prediction module is composed of a single linear mapping layer;

[0030] Specifically, the fusion attention module calculates the autocorrelation of the global features of the image as the self-attention weight through the self-attention layer and aggregates the global feature representation of the image. The specific operations are as follows:

[0031]

[0032] Among them, self-attention represents the self-attention weight of the global feature; Q t-1 Represents the image global feature input of the current coordinate decoding module; is the query weight matrix of the self-attention module; is the key weight matrix of the self-attention module; is the value weight matrix of the self-attention module; is the feature dimension of the self-attention module; the Softmax function maps the element value to between 0 and 1; P is the global feature position encoding vector; Q S is the aggregation feature; LayerNorm is the layer normalization layer;

[0033] Then, by calculating the correlation between the aggregated features and the local block features as the cross attention weight, the target block features are weighted and fused to guide the global features to move closer to the local block features. The specific operations are as follows:

[0034]

[0035] Among them, M t-1 is the local block feature input of the current coordinate decoding module; [·] represents the splicing operation; is the value weight matrix of the self-attention module; is the query weight matrix of the cross-attention module; is the key weight matrix of the criss-cross attention module; is the value weight matrix of the cross attention module; is the feature dimension of the cross-attention module; cross-attention represents the cross-attention weight; P′ is the local block feature position encoding vector; R M is the coordinate encoding embedding vector; R is the coordinate encoding vector; Q M The weighted output of the cross-attention module;

[0036] The global weighted features and local weighted features share the same forward propagation network, which reduces the number of parameters while transmitting and updating the global features and local block features. The specific operations are as follows:

[0037]

[0038] Among them, FFN is the forward propagation layer; Dropuout is the Dropuout layer; Q tis the global feature input of the current coordinate decoding module; M t It is the local block feature output of the current coordinate decoding module;

[0039] The offset prediction module maps the global feature to the key point coordinate offset Offset through linear transformation t , so as to obtain the coordinates of the key points. The specific operations are as follows:

[0040]

[0041] Among them, OffsetPredictor is the offset prediction module; A t-1 A is the key point coordinates input previously; t The key point coordinates output by the coordinate decoding module.

[0042] As a preferred solution of the present invention, a global to local key point positioning method based on fusion attention is characterized in that the end-to-end joint supervision training in step 5 is divided into three stages, and the training process is as follows:

[0043] S91: In the first stage, the location with the largest activation of the key point heat map generated by the pre-trained global regression model is selected as the initial local block center;

[0044] S92: Generate local block features from the aligned multi-scale image feature sampling and encoding according to the local block center;

[0045] S93: inputting the local block feature and the global feature vector into a coordinate decoding module to obtain the coordinates of the key points predicted at this stage;

[0046] S94: while the side length d of the local block in the second and third stages is halved, the predicted coordinates of the local regression model in the previous stage are used as the initial local block center in this stage and step S92 is repeated;

[0047] S95: Collect the heat map output by the global regression model and the key point coordinates output by the local regression model at each stage, use the joint loss function L to optimize the model parameters, and save the model parameters that minimize the joint loss function L value on the test set. The joint loss function L used is:

[0048]

[0049] Among them, stage i represents the i-th stage of joint supervised training; L H is the global regression model loss; is the loss of the local regression model in the i-th stage, using the standardized mean error; H is the weight coefficient of the global regression model loss function, is the weight coefficient of the loss in the i-th stage of the local regression model.

[0050] A global to local key point positioning device based on fused attention, characterized in that the device includes: an input device, an output device, a power supply, at least one processor, and a memory communicatively connected to the processor; the memory also corresponds to instructions specified by at least one processor, and the instructions are executed by the at least one executor so that the at least one processor can execute the method described in any one of claims 1 to 9.

[0051] (1) The present invention uses the HRNet network to extract the multi-scale visual features contained in the image, which greatly reduces the loss of original image information. At the same time, the proposed fusion attention module deeply fuses the global features and local block features through the attention weight, maps the global features to the local block feature space, and then uses the fused global features to predict the offset of the key point in the local block, which greatly enhances the positioning accuracy of the key point in the local block.

[0052] (2) The present invention uses a global regression model to globally locate key points in the entire image, and on this basis builds a local regression model to achieve more accurate positioning of the key point coordinates in the local block. The local block size is gradually reduced in three stages to achieve accurate positioning at each stage, which greatly enhances the accuracy and robustness of positioning. By pre-training the global regression model, it is jointly trained with the local regression model on this basis. Compared with separate training and direct joint training, this method speeds up the convergence speed of the local regression module and further enhances the accuracy of the model.

[0053] (3) The global feature extraction module proposed in the present invention acts on the basic features in the middle of the heat map decoding module. Under the guidance of the heat map, the basic features retain the global information while focusing on the key point positioning task. Compared with directly extracting global features from multi-scale image information, the number of parameters is greatly reduced and the feature extraction capability is enhanced. At the same time, the global features participate in the prediction of the local block offset, ensuring the consistency of the heat map global positioning and the local block offset regression task objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a global to local key point positioning method based on fused attention described in Example 1 of the present invention.

[0055] Figure 2 This is a schematic diagram of a global regression model in a global to local key point positioning method based on fused attention described in Example 1 of the present invention.

[0056] Figure 3This is a schematic diagram of a global feature encoding module of a global to local key point positioning method based on fused attention described in Example 1 of the present invention.

[0057] Figure 4 Schematic diagram of a local block feature encoding module of a global to local key point positioning method based on fusion attention described in Example 1 of the present invention.

[0058] Figure 5 This is a schematic diagram of a coordinate decoding module in a global to local key point positioning method based on fused attention described in Example 1 of the present invention.

[0059] Figure 6 Schematic diagram of a local regression model in a global to local key point positioning method based on fused attention described in Example 1 of the present invention.

[0060] Figure 7 This is a schematic diagram of a global to local key point positioning device based on fusion attention described in Example 2 of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] Example 1

[0063] like Figure 1 As shown, a global to local key point positioning method based on fusion attention includes the following steps:

[0064] S1: Use a camera to collect images and mark the acupoints;

[0065] S2: Preprocess the image data to construct an acupoint location dataset;

[0066] S3: Build a global regression model, which includes a backbone network and a heat map decoding module. The global regression model is built by connecting the backbone network and the heat map decoding module in series.

[0067] S4: Input the preprocessed image data into the global regression model for forward propagation to obtain the predicted heat map, train and optimize the weights of the global regression model through the mean square error loss function, and finally obtain the optimal weights;

[0068] S5: constructing a local regression model based on the global regression model, wherein the local regression model includes a local block feature encoding module, a global feature extraction module and a coordinate decoding module;

[0069] S6: Train and optimize the local regression model parameter weights through the joint loss function, and fine-tune the global regression model parameter weights to save the optimal weights;

[0070] S7: Load the optimal weight into the global regression model and pass it into the preprocessed image to be located to obtain the corresponding coordinates of the acupoints.

[0071] Furthermore, the preprocessing in step S2 includes the following specific steps:

[0072] S21: resize the image to a size of 256×256 and normalize the pixel values ​​of the image;

[0073] S22: With the help of the FaceBoxesV2 face bounding box detection model pre-trained with 12,880 face images from the Wider Face subset, the face images are roughly detected and cropped to exclude most irrelevant background information, while retaining the background margin of 5 pixels outside the detection box to ensure that the face information is not lost due to cropping;

[0074] S23: Calculate new acupoint coordinates according to the cropped border of step S22, and use a two-dimensional Gaussian function to generate a Gaussian heat map with each acupoint coordinate of each face image as the center;

[0075] S24: Divide the processed data set into training set, test set and validation set.

[0076] Further, such as Figure 2 As shown in the figure, the backbone network includes a multi-scale feature extraction module and a feature alignment module; the multi-scale feature extraction module is the feature extraction part of the HRNetV2-18 network with four output branches of different scales, which is mainly composed of a two-dimensional convolutional layer and a ReLu activation layer; the feature alignment module completes feature alignment through interpolation operations;

[0077] Specifically, the image is preprocessed to obtain three-channel image data. The multi-scale feature extraction module extracts multi-scale image information from the image data by loading the HRNetV2-18 model pre-trained by the COCO 2017 dataset; the feature alignment module takes the multi-scale image information as input, generates interpolated supplementary pixel values ​​by utilizing neighboring information, and generates image structure information and deep semantic information aligned in the spatial dimension to better support subsequent heat map decoding and coordinate decoding tasks.

[0078] Further, such as Figure 2 As shown, the heat map decoding module includes a feature mapping module and a heat map prediction head module; the feature mapping module is composed of a two-dimensional transposed convolution layer, a batch normalization layer, a ReLu activation function and a Dropout layer in series; the heat map regression module is composed of a two-dimensional convolution layer;

[0079] Specifically, the feature mapping module fuses image features of different scales and maps them to a unified initial heat map representation, so that the pixels at each position of the heat map can reflect the feature intensity and position of the relevant area in the image, which not only accurately reproduces the initial image features, but also provides basic features for subsequent global feature extraction; the heat map prediction head module fuses the initial heat map representation and maps the position of each acupoint in the global image into a separate probability distribution map. The position with the largest activation value in each probability distribution map is the predicted coordinate of each acupoint, which serves as the initial local block center of the subsequent local regression model.

[0080] Further, such as Figure 2 As shown, the supervised training in step 3 includes: using the acupoint heat map in the facial acupoint positioning data set as label information and the processed pictures as samples; performing n rounds of iterative training and saving the model parameters with the smallest loss function value on the test set; the backbone network and heat map decoding module after training are used as the initial parameter values ​​for subsequent joint training; the supervised training uses the mean square error loss as the loss function.

[0081] Further, such as Figure 3 As shown, the global feature encoding module includes a feature extraction module and a feature compression module; the global feature extraction module is composed of a large-core two-dimensional convolution layer, a batch normalization layer and a ReLu activation function in series; the feature compression module is composed of an adaptive pooling layer and a linear mapping layer in series;

[0082] Specifically, the global feature extraction module uses large kernel convolution to cover a larger area of ​​the image and aggregate features over a larger range, thereby enhancing the understanding of the overall structure and long-term dependencies, capturing more comprehensive overall image information, and extracting a global representation of the image; the feature compression module integrates the global representation and performs dimensionality reduction to ensure that the most important global information in the image is retained, thereby enhancing the model's understanding of the global context and improving its ability to focus on specific areas or features, effectively reducing the size of the feature map, compressing the two-dimensional feature map into a one-dimensional global feature vector, and providing global features for subsequent fusion attention calculations.

[0083] Further, such as Figure 4 As shown, the local block feature encoding module includes a multi-scale feature fusion module and a block encoding module; the multi-scale feature fusion module is composed of a two-dimensional convolution layer, a batch normalization layer and a ReLu function; the block encoding module samples the local block pixel values ​​and performs weighted fusion to generate local block features;

[0084] Specifically, the multi-scale feature fusion module fuses the multi-scale image features aligned by channel attention weighted fusion as the initial feature map of the local block feature; the block coding module uniformly takes n×n point pixels on the fused initial feature map in a square frame with a side length of d pixels centered on the local block to form a local block LP, and calculates the weighted sum of the pixel values ​​of each local block as the current local block feature Code, specifically:

[0085]

[0086] Where i represents the channel number of the local block; LP i represents the i-th channel feature map of the local block; C i Represents the encoded value of the i-th channel feature map of the local block, and each channel of the local block is encoded with a real number; ω i,j,k is the pixel value weight at the i-th feature map coordinate (j, k) of the local block.

[0087] Further, such as Figure 5 As shown, the coordinate decoding module includes a fusion attention module and an offset prediction module; the coordinate decoding module is composed of multiple fusion attention modules and offset prediction modules stacked alternately; the fusion attention module is composed of a self-attention layer, a cross-attention layer, a forward propagation layer and three two-dimensional position encoding vectors; the offset prediction module is composed of a single linear mapping layer;

[0088] Specifically, the fusion attention module calculates the autocorrelation of the global features of the image as the self-attention weight through the self-attention layer and aggregates the global feature representation of the image. The specific operations are as follows:

[0089]

[0090] Among them, self-attention represents the self-attention weight of the global feature; Q t-1 Represents the image global feature input of the current coordinate decoding module; is the query weight matrix of the self-attention module; is the key weight matrix of the self-attention module; is the value weight matrix of the self-attention module; is the feature dimension of the self-attention module; the Softmax function maps the element value to between 0 and 1; P is the global feature position encoding vector; Q S is the aggregation feature; LayerNorm is the layer normalization layer;

[0091] Then, by calculating the correlation between the aggregated features and the local block features as the cross attention weight, the target block features are weighted and fused to guide the global features to move closer to the local block features. The specific operations are as follows:

[0092]

[0093] Among them, M t-1 is the local block feature input of the current coordinate decoding module; [·] represents the splicing operation; is the query weight matrix of the cross-attention module; is the key weight matrix of the criss-cross attention module; is the value weight matrix of the cross attention module; is the feature dimension of the cross-attention module; cross-attention represents the cross-attention weight; P′ is the local block feature position encoding vector; R M is the coordinate encoding embedding vector; R is the coordinate encoding vector; Q M The weighted output of the cross-attention module;

[0094] The global weighted features and local weighted features share the same forward propagation network, which reduces the number of parameters while transmitting and updating the global features and local block features. The specific operations are as follows:

[0095]

[0096] Among them, FFN is the forward propagation layer; Dropuout is the Dropuout layer; Q t is the global feature input of the current coordinate decoding module; M t It is the local block feature output of the current coordinate decoding module;

[0097] The offset prediction module maps the global feature to the key point coordinate offset Offset through linear transformation t , so as to obtain the coordinates of the key points. The specific operations are as follows:

[0098]

[0099] Among them, OffsetPredictor is the offset prediction module; A t-1 A is the key point coordinates input previously; t The key point coordinates output by the coordinate decoding module.

[0100] Further, such as Figure 6 As shown in the figure, the global regression model and the local regression model adopt an end-to-end joint supervision training method. The training is divided into three stages. The training process is as follows:

[0101] S91: In the first stage, the location with the largest activation in the acupoint heat map generated by the pre-trained global regression model is selected as the initial local block center;

[0102] S92: Generate local block features from the aligned multi-scale image feature sampling and encoding according to the local block center;

[0103] S93: inputting the local block features and the global feature vector into a coordinate decoding module to obtain the predicted acupoint coordinates at this stage;

[0104] S94: while the side length d of the local block in the second and third stages is halved, the predicted coordinates of the local regression model in the previous stage are used as the initial local block center in this stage and step S92 is repeated;

[0105] S95: Collect the heat map output by the global regression model and the coordinates of the acupoints output by the local regression model in each stage, use the joint loss function L to optimize the model parameters, and save the model parameters that minimize the joint loss function L value on the test set. The joint loss function L used is:

[0106]

[0107] Among them, stage i represents the i-th stage of joint supervised training; L H is the global regression model loss; is the loss of the local regression model in the i-th stage, using the standardized mean error; H is the weight coefficient of the global regression model loss function, is the weight coefficient of the loss in the i-th stage of the local regression model.

[0108] Example 2

[0109] like Figure 7 As shown, a global to local key point positioning device based on fused attention is characterized in that it includes an input device, an output device, a power supply, at least one processor, and a memory connected to the processor in communication; the memory also corresponds to instructions specified by at least one processor, and the instructions are executed by the at least one executor, and the input and output interfaces include a display, a keyboard, a mouse, and a USB interface to complete data interaction operations; the power supply can be an external power supply or a rechargeable battery to provide power to the electronic device.

[0110] Those skilled in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), disks or optical disks, etc. Various media that can store program codes.

[0111] When the above-mentioned integrated unit of the present invention is implemented in the form of a software functional unit and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A global to local key point localization method based on fusion attention, characterized in that: The following steps are involved: Step 1: Use a camera to capture images and mark their key points, pre-process the image data, and build a key point positioning data set; Step 2: Construct a global regression model. The global regression model mainly includes a backbone network and a heat map decoding module. The global regression model is constructed by connecting the backbone network and the heat map decoding module in series. The backbone network extracts image structure information and deep semantic information in image representation. The heat map decoding module generates a probability distribution heat map of each key point position, completing the regression from image representation to heat map. Step 3: Perform end-to-end supervised training on the global regression model; Step 4: Construct a local regression model based on the global regression model; the local regression model mainly includes a local block feature encoding module, a global feature encoding module and a coordinate decoding module; the local block feature encoding module encodes the local block features through multi-scale image representation according to the two-dimensional spatial position of the forward key point coordinates; The global feature encoding module fuses multi-scale image representations to extract the global feature representation of the image; The coordinate decoding module uses local block features to guide global features to generate key point encoding vectors and regress the encoding vectors into local offsets; Step 5: Perform end-to-end joint supervised training on the local regression model and the global regression model; Step 6: Input the preprocessed positioning image into the trained global regression model. The output of the local regression model is the predicted key point coordinates.

2. A global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The step 1 pre-treatment includes the following processes: S11: resize the image to a size of n×n and normalize the pixel values ​​of the image; S12: With the help of the pre-trained public image border detection model, the image is roughly detected and cropped to exclude most of the irrelevant background information, while retaining the background margin of k pixels outside the detection frame to ensure that the image information is not lost due to cropping; S13: Calculate new key point coordinates according to the cropped border of step S12, and use a two-dimensional Gaussian function to generate a Gaussian heat map with each key point coordinate of each image as the center; S14: Divide the processed dataset into training set, test set and validation set.

3. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: In step 2, the backbone network includes a multi-scale feature extraction module and a feature alignment module; The multi-scale feature extraction module is the feature extraction part of the HRNet network with multiple output branches of different scales, which consists of a two-dimensional convolution layer and a ReLu activation function; the feature alignment module completes feature alignment through interpolation operations; Specifically, the image is preprocessed to obtain three-channel image data, and the multi-scale feature extraction module extracts multi-scale image information from the image data by loading the pre-trained HRNet model; The feature alignment module takes multi-scale image information as input, generates interpolated supplementary pixel values ​​by utilizing neighboring information, and generates image structure information and deep semantic information aligned in the spatial dimension to better support subsequent heat map decoding and coordinate decoding tasks.

4. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The heat map decoding module in step 2 includes a feature mapping module and a heat map prediction head module; the feature mapping module is composed of a two-dimensional transposed convolution layer, a batch normalization layer, a ReLu activation function and a Dropout layer in series; the heat map prediction head module is composed of a two-dimensional convolution layer; Specifically, the feature mapping module fuses image features of different scales and maps them into a unified initial heat map representation, and also provides basic features for the subsequent global feature extraction module; the heat map prediction head module fuses the initial heat map representation, and maps the position of each key point in the global image into a separate probability distribution map. The position with the largest activation value in each probability distribution map is the predicted key point, which serves as the initial local block center of the subsequent local regression model.

5. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The supervised training in step 3 includes: using the key point heat map in the key point positioning data set as label information, The processed images are used as samples; n rounds of iterative training are performed, and the model parameters with the smallest loss function value on the test set are saved; the backbone network and heat map decoding module after training are used as initial parameter values ​​for subsequent joint training; the loss function used in the supervised training is mean square error loss.

6. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The global feature encoding module in step 4 includes a feature extraction module and a feature compression module; The global feature extraction module is composed of a large-core two-dimensional convolution layer, a batch normalization layer, and a ReLu activation function in series; the feature compression module is composed of an adaptive pooling layer and a linear mapping layer in series; Specifically, the global feature extraction module uses large kernel convolution to cover a larger area of ​​the image, aggregates features in a larger range, and extracts the global representation of the image; the feature compression module integrates the global representation and performs dimensionality reduction to ensure that the most important global information in the image is retained, effectively reducing the size of the feature map, compressing the two-dimensional feature map into a one-dimensional feature vector, and providing global features for subsequent fusion attention calculations.

7. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The local block feature encoding module in step 4 includes a multi-scale feature fusion module and a block encoding module; the multi-scale feature fusion module is composed of a two-dimensional convolution layer, a batch normalization layer and a ReLu function; the block encoding module samples the local block pixel values ​​and performs weighted fusion to generate local block features; Specifically, the multi-scale feature fusion module fuses the aligned multi-scale image features through channel attention weighted fusion as the initial feature map of local block features; The block coding module uniformly selects n×n point pixels on the fused initial feature map in a square frame with a side length of d pixels centered on the local block to form a local block LP, and calculates the weighted sum of the pixel values ​​of each local block as the current local block feature Code, specifically: Where i represents the channel number of the local block; LP i represents the i-th channel feature map of the local block; C i Represents the encoded value of the i-th channel feature map of the local block, and each channel of the local block is encoded with a real number; ω i,j,k is the pixel value weight at the i-th feature map coordinate (j, k) of the local block.

8. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The coordinate decoding module in step 4 includes a fusion attention module and an offset prediction module; The coordinate decoding module is composed of multiple fusion attention modules and offset prediction modules stacked alternately; the fusion attention module is composed of a self-attention layer, a cross-attention layer, a forward propagation layer and three two-dimensional position encoding vectors; the offset prediction module is composed of a single linear mapping layer; Specifically, the fusion attention module calculates the autocorrelation of the global features of the image as the self-attention weight through the self-attention layer and aggregates the global feature representation of the image. The specific operations are as follows: Among them, self-attention represents the self-attention weight of the global feature; Q t-1 Represents the image global feature input of the current coordinate decoding module; is the query weight matrix of the self-attention module; is the key weight matrix of the self-attention module; is the value weight matrix of the self-attention module; is the feature dimension of the self-attention module; the Softmax function maps the element value to between 0 and 1; P is the global feature position encoding vector; Q S is the aggregation feature; LayerNorm is the layer normalization layer; Then, by calculating the correlation between the aggregated features and the local block features as the cross attention weight, the target block features are weighted and fused to guide the global features to move closer to the local block features. The specific operations are as follows: Among them, M t-1 is the local block feature input of the current coordinate decoding module; [·] represents the splicing operation; is the value weight matrix of the self-attention module; is the query weight matrix of the cross-attention module; is the key weight matrix of the criss-cross attention module; is the value weight matrix of the cross attention module; is the feature dimension of the cross-attention module; cross-attention represents the cross-attention weight; P′ is the local block feature position encoding vector; R M is the coordinate encoding embedding vector; R is the coordinate encoding vector; Q M The weighted output of the cross-attention module; The global weighted features and local weighted features share the same forward propagation network, which reduces the number of parameters while transmitting and updating the global features and local block features. The specific operations are as follows: Among them, FFN is the forward propagation layer; Dropuout is the Dropuout layer; Q t is the global feature input of the current coordinate decoding module; M t It is the local block feature output of the current coordinate decoding module; The offset prediction module maps the global feature to the key point coordinate offset Offset through linear transformation t , so as to obtain the coordinates of the key points. The specific operations are as follows: Among them, OffsetPredictor is the offset prediction module; A t-1 A is the key point coordinates input previously; t The key point coordinates output by the coordinate decoding module.

9. The global to local key point positioning method based on fusion attention according to claim 1, characterized in that: The end-to-end joint supervision training in step 5 is divided into three stages, and the training process is as follows: S91: In the first stage, the location with the largest activation of the key point heat map generated by the pre-trained global regression model is selected as the initial local block center; S92: Generate local block features from the aligned multi-scale image feature sampling and encoding according to the local block center; S93: inputting the local block features and the global features into a coordinate decoding module to obtain the coordinates of the key points predicted at this stage; S94: while the side length d of the local block in the second and third stages is halved, the predicted coordinates of the local regression model in the previous stage are used as the initial local block center in this stage and step S92 is repeated; S95: Collect the heat map output by the global regression model and the key point coordinates output by the local regression model at each stage, use the joint loss function L to optimize the model parameters, and save the model parameters that minimize the joint loss function L value on the test set. The joint loss function L used is: Among them, stage i represents the i-th stage of joint supervised training; L H is the global regression model loss; is the loss of the local regression model in the i-th stage, using the standardized mean error; H is the weight coefficient of the global regression model loss function, is the weight coefficient of the loss in the i-th stage of the local regression model.

10. A device using the global to local key point positioning method based on fusion attention as claimed in claim 1, characterized in that: The apparatus comprises: an input device, an output device, a power supply, at least one processor, and a memory communicatively connected to the processor; the memory also corresponds to instructions specified by at least one processor, and the instructions are executed by the at least one executor so that the at least one processor can execute the method described in any one of claims 1 to 9.

Images