Key point detection method and system based on hybrid CNN-Transform model

Abstract

The invention discloses a key point detection method and a key point detection system based on a hybrid CNN-Transform model. The model is composed of a CNN network, a Transformer network and a multi-resolution heat map learning strategy training network. The method comprises the steps of obtaining a to-be-detected image and inputting the to-be-detected image into a hybrid CNN-Transformer model; reducing the size of the to-be-detected image through down-sampling, and inputting the to-be-detected image to a CNN network and a Transform network to extract the features of the to-be-detected image; inputting the extracted local features, global features, original image features and low-resolution feature predicted key points into a feature fusion correction network for feature fusion and correction; and applying the corrected features to a multi-resolution heat map learning strategy training network, outputting feature prediction key points under different resolutions, then performing equalization processing, and outputting final prediction key point coordinates. According to the method, unique advantages of CNN and Transform are fully utilized, a feature fusion correction module and a multi-resolution heat map learning strategy training network are introduced, and the accuracy of key point detection is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging, and in particular relates to a key point detection method and system based on a hybrid CNN-Transformer model. Background Art

[0002] Convolutional neural networks (CNNs) based on deep learning are widely used in key point detection in medical images due to their powerful feature learning capabilities and high accuracy. CNN effectively extracts local features through the local receptive field mechanism, and gradually extracts features from low-level to high-level through multiple layers of convolution and pooling layers, which is suitable for recognizing complex shapes. Parameter sharing significantly reduces model parameters and improves training efficiency, while transfer learning uses pre-trained models to enhance performance, especially when the amount of data is small. In addition, ensemble learning methods further improve detection accuracy by combining multiple CNN models. Common architectures such as VGG, ResNet, and Inception perform well in medical image analysis; methods such as U-Net and Mask R-CNN can accurately identify and locate key points while considering the contextual information of the image.

[0003] Transformer network: Transformer is a deep learning architecture based on the self-attention mechanism. It was originally used for natural language processing and has now achieved remarkable results in fields such as computer vision. Its main features include the self-attention mechanism, which can capture long-distance dependencies and improve feature expression capabilities; parallel processing capabilities to improve training speed; multi-head attention, which allows the model to focus on different feature subspaces at the same time and enrich information expression. In addition, Transformer retains the sequential information of the input data through positional encoding, and usually adopts an encoder-decoder structure, where the encoder processes the input information and the decoder generates the output. This architecture has shown excellent performance in fields such as image processing, speech recognition, and bioinformatics, and has promoted the development of deep learning. Overall, Transformer methods have shown good performance in medical image key point detection, such as VIT, Swin-transformer, and Biformer, especially when faced with complex structures and diverse data.

[0004] Although there are some medical image X-Ray key point detection technologies, these technologies still have some shortcomings. Some methods use CNN networks to detect key points in medical images. Through multi-layer convolution and pooling operations, CNN can effectively capture local features, especially good at identifying edges and complex backgrounds in images. However, CNN is insufficient in capturing global information. In response to this, there is a method to introduce the Transformer network for key point detection. The self-attention mechanism of the Transformer significantly improves the ability to capture global information, allowing the model to locate key points more accurately. Despite this, due to the lack of detailed extraction of local features, the Vision Transformer (ViT) is still not as good as the optimized CNN model in processing image details. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention solves the limitations of the convolutional neural network (CNN) in global information extraction and the problem that the Transformer network is not detailed in extracting local feature information. CNN and Transformer are integrated to design a new hierarchical hybrid transformer and CNN architecture. At the same time, in order to further improve the key point detection accuracy of the network, a feature fusion correction network and a multi-resolution heat map learning strategy are introduced to further fuse the features at multiple scales, and a key point detection method and system based on a hybrid CNN-Transformer is specifically proposed.

[0006] In order to achieve the aforementioned object of the invention, the present invention adopts the following scheme:

[0007] One aspect of the present invention provides a key point detection method based on a hybrid CNN-Transformer model, characterized in that it includes:

[0008] The hybrid CNN-Transformer model consists of a CNN network, a Transformer network, and a multi-resolution heat map learning strategy training network. The method specifically includes:

[0009] Acquire an image to be detected, and input the image to be detected into a hybrid CNN-Transformer model;

[0010] Reducing the size of the image to be detected by downsampling;

[0011] The reduced image to be detected is input into the CNN network and the Transformer network respectively to extract the features of the image to be detected, including:

[0012] Extract local features of the image to be detected based on the CNN network;

[0013] Extract global features of the image to be detected based on the Transformer network;

[0014] The extracted local features, global features, original image features and key points predicted by low-resolution features are input into the feature fusion correction network for feature fusion, and the fused features are corrected;

[0015] Apply the corrected features to the multi-resolution heat map learning strategy training network to output feature prediction key points at different resolutions;

[0016] The key points of features at different resolutions are predicted and averaged, and the final predicted key point coordinates are output.

[0017] In one embodiment, the CNN network consists of a 3x3, dilation=1 dilated convolution layer, a normalization and activation function layer, a 3x3, dilation=2 dilated convolution layer, a normalization layer, an attention mechanism module CBAM, and a residual connection layer;

[0018] The method of extracting local features of an image to be detected based on a CNN network includes: the image to be detected first passes through a 3x3, dilation=1 hole convolution layer and performs normalization and activation function processing, then passes through a 3x3, dilation=2 hole convolution layer and normalization processing, then performs feature selection through an attention mechanism module CBAM, and finally outputs the local features of the image to be detected through a residual connection layer.

[0019] In one embodiment, the Transformer network is a Biformer network; the Biformer network captures global features through a double-layer routing attention mechanism BRA;

[0020] The global features of the image to be detected are extracted based on the Transformer network, specifically including:

[0021] The image to be detected first passes through 3x3 DWConv to implicitly encode the relative position relationship, and then passes through LayerNorm normalization processing, and then passes through the BRA module to learn the cross-position relationship of the image to be detected. Finally, LayerNorm and MLP are used to represent the features of each position to obtain the global features of the image to be detected.

[0022] In one of the embodiments, the feature fusion correction network concatenates the extracted features with the upsampled feature map and the heat map, and obtains the predicted heat map after 3x3 and 1x1 convolution operations.

[0023] In one embodiment, in a multi-resolution heat map learning strategy training network, the corrected features are deeply supervised to generate multi-scale heat maps H1, H2, and H3, and the difference between the predicted heat map and the true heat map is calculated using the MSE loss. The loss function is defined as follows:

[0024] L all =L H1 +α1L H2 +α2L H3

[0025] Among them, L all Represents the total loss function of model training, L H1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively, and α1 and α2 are the loss weights.

[0026] Another aspect of the present invention provides a key point detection system based on a hybrid CNN-Transformer model, comprising:

[0027] The hybrid CNN-Transformer model consists of a CNN network, a Tranformer network, and a multi-resolution heat map learning strategy training network. The system specifically includes:

[0028] An input module, used for acquiring an image to be detected and inputting the image to be detected into the hybrid CNN-Transformer model;

[0029] A preprocessing module, used for reducing the size of the image to be detected by downsampling;

[0030] The feature extraction module is used to input the reduced image to be detected into the CNN network and the Transformer network to extract the features of the image to be detected, including:

[0031] Extract local features of the image to be detected based on the CNN network;

[0032] Extract global features of the image to be detected based on the Transformer network;

[0033] The feature fusion correction module is used to input the extracted local features, global features, original image features and key points predicted by low-resolution features into the feature fusion correction network for feature fusion, and correct the fused features;

[0034] The multi-resolution heat map learning module is used to apply the modified features to the multi-resolution heat map learning strategy training network and output the feature prediction key points at different resolutions;

[0035] The output module is used to predict key points of features at different resolutions and perform averaging processing, and output the final predicted key point coordinates.

[0036] In one embodiment, the CNN network consists of a 3x3, dilation=1 dilated convolution layer, a normalization and activation function layer, a 3x3, dilation=2 dilated convolution layer, a normalization layer, an attention mechanism module CBAM, and a residual connection layer;

[0037] The method of extracting local features of an image to be detected based on a CNN network includes: the image to be detected first passes through a 3x3, dilation=1 hole convolution layer and performs normalization and activation function processing, then passes through a 3x3, dilation=2 hole convolution layer and normalization processing, then performs feature selection through an attention mechanism module CBAM, and finally outputs the local features of the image to be detected through a residual connection layer.

[0038] In one embodiment, the Transformer network is a Biformer network; the Biformer network captures global features through a double-layer routing attention mechanism BRA;

[0039] The global features of the image to be detected are extracted based on the Transformer network, specifically including:

[0040] The image to be detected first passes through 3x3 DWConv to implicitly encode the relative position relationship, and then passes through LayerNorm normalization processing, and then passes through the BRA module to learn the cross-position relationship of the image to be detected. Finally, LayerNorm and MLP are used to represent the features of each position to obtain the global features of the image to be detected.

[0041] In one of the embodiments, the feature fusion correction module concatenates the extracted features with the upsampled feature map and the thermal map, and obtains the predicted thermal map after 3x3 and 1x1 convolution operations.

[0042] In one embodiment, in a multi-resolution heat map learning strategy training network, the corrected features are deeply supervised to generate multi-scale heat maps H1, H2, and H3, and the difference between the predicted heat map and the true heat map is calculated using the MSE loss. The loss function is defined as follows:

[0043] L all =L H1 +α1L H2 +α2L H3

[0044] Among them, L all Represents the total loss function of model training, LH1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively, and α1 and α2 are the loss weights.

[0045] Compared with the prior art, the present invention has at least the following advantages: the present invention combines the local feature extraction of CNN and the global feature integration capability of Transformer, introduces an attention mechanism module to enhance the convolutional neural network, and improves the model's ability to express feature maps by combining channel attention and spatial attention mechanisms, further enhancing the performance of CNN. By introducing a high-performance Transformer network, Swin-transformer and Biformer are more powerful in extracting image context information. The convolutional neural network CNN and the visual transformer network Transformer are integrated to design a new hierarchical hybrid transformer and CNN architecture. The hybrid architecture performs well in key point detection, especially in processing multi-scale features and reducing detection errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a schematic diagram of the design of a hybrid CNN-Transformer model provided by a typical implementation case of the present invention;

[0048] Figure 2 It is a schematic diagram of CNN module design provided by a typical implementation case of the present invention;

[0049] Figure 3 It is a schematic diagram of the BRA calculation attention matrix provided by a typical implementation case of the present invention;

[0050] Figure 4 It is a schematic diagram of Biformer module design provided by a typical implementation case of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present invention clearer, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Examples of these preferred embodiments are illustrated in the accompanying drawings. The embodiments of the present invention shown in the accompanying drawings and described according to the accompanying drawings are merely exemplary, and the present invention is not limited to these embodiments.

[0052] The convolutional neural network CNN and the visual transformer network Transformer are integrated to design a new hierarchical hybrid transformer and CNN architecture, which combines the local feature extraction of CNN and the global feature integration capability of Transformer. The attention mechanism module of the enhanced convolutional neural network is introduced. By combining the channel attention and spatial attention mechanisms, the model's ability to express feature maps is improved, further enhancing the performance of CNN. The high-performance Transformer network is introduced, and Swin-transformer and Biformer are used to extract richer image context information.

[0053] One aspect of the present invention provides a key point detection method based on a hybrid CNN-Transformer model, characterized in that it includes:

[0054] The hybrid CNN-Transformer model consists of a CNN network, a Transformer network, and a multi-resolution heat map learning strategy training network. The method specifically includes:

[0055] Acquire an image to be detected, and input the image to be detected into a hybrid CNN-Transformer model;

[0056] Reducing the size of the image to be detected by downsampling;

[0057] The reduced image to be detected is input into the CNN network and the Transformer network respectively to extract the features of the image to be detected, including:

[0058] Extract local features of the image to be detected based on the CNN network;

[0059] Extract global features of the image to be detected based on the Transformer network;

[0060] The extracted local features, global features, original image features and key points predicted by low-resolution features are input into the feature fusion correction network for feature fusion, and the fused features are corrected;

[0061] Apply the corrected features to the multi-resolution heat map learning strategy training network to output feature prediction key points at different resolutions;

[0062] The key points of features at different resolutions are predicted and averaged, and the final predicted key point coordinates are output.

[0063] In one embodiment, the CNN network consists of a 3x3, dilation=1 dilated convolution layer, a normalization and activation function layer, a 3x3, dilation=2 dilated convolution layer, a normalization layer, an attention mechanism module CBAM, and a residual connection layer;

[0064] The method of extracting local features of an image to be detected based on a CNN network includes: the image to be detected first passes through a 3x3, dilation=1 hole convolution layer and performs normalization and activation function processing, then passes through a 3x3, dilation=2 hole convolution layer and normalization processing, then performs feature selection through an attention mechanism module CBAM, and finally outputs the local features of the image to be detected through a residual connection layer.

[0065] In one embodiment, the Transformer network is a Biformer network; the Biformer network captures global features through a double-layer routing attention mechanism BRA;

[0066] The global features of the image to be detected are extracted based on the Transformer network, specifically including:

[0067] The image to be detected first passes through 3x3 DWConv to implicitly encode the relative position relationship, and then passes through LayerNorm normalization processing, and then passes through the BRA module to learn the cross-position relationship of the image to be detected. Finally, LayerNorm and MLP are used to represent the features of each position to obtain the global features of the image to be detected.

[0068] In one of the embodiments, the feature fusion correction network concatenates the extracted features with the upsampled feature map and the heat map, and obtains the predicted heat map after 3x3 and 1x1 convolution operations.

[0069] In one embodiment, in a multi-resolution heat map learning strategy training network, the corrected features are deeply supervised to generate multi-scale heat maps H1, H2, and H3, and the difference between the predicted heat map and the true heat map is calculated using the MSE loss. The loss function is defined as follows:

[0070] L all =L H1 +α1L H2 +α2L H3

[0071] Among them, L all Represents the total loss function of model training, L H1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively, and α1 and α2 are the loss weights.

[0072] Another aspect of the present invention provides a key point detection system based on a hybrid CNN-Transformer model, comprising:

[0073] The hybrid CNN-Transformer model consists of a CNN network, a Tranformer network, and a multi-resolution heat map learning strategy training network. The system specifically includes:

[0074] An input module, used for acquiring an image to be detected and inputting the image to be detected into the hybrid CNN-Transformer model;

[0075] A preprocessing module, used for reducing the size of the image to be detected by downsampling;

[0076] The feature extraction module is used to input the reduced image to be detected into the CNN network and the Transformer network to extract the features of the image to be detected, including:

[0077] Extract local features of the image to be detected based on the CNN network;

[0078] Extract global features of the image to be detected based on the Transformer network;

[0079] The feature fusion correction module is used to input the extracted local features, global features, original image features and key points predicted by low-resolution features into the feature fusion correction network for feature fusion, and correct the fused features;

[0080] The multi-resolution heat map learning module is used to apply the modified features to the multi-resolution heat map learning strategy training network and output the feature prediction key points at different resolutions;

[0081] The output module is used to predict key points of features at different resolutions and perform averaging processing, and output the final predicted key point coordinates.

[0082] In one embodiment, the CNN network consists of a 3x3, dilation=1 dilated convolution layer, a normalization and activation function layer, a 3x3, dilation=2 dilated convolution layer, a normalization layer, an attention mechanism module CBAM, and a residual connection layer;

[0083] The method of extracting local features of an image to be detected based on a CNN network includes: the image to be detected first passes through a 3x3, dilation=1 hole convolution layer and performs normalization and activation function processing, then passes through a 3x3, dilation=2 hole convolution layer and normalization processing, then performs feature selection through an attention mechanism module CBAM, and finally outputs the local features of the image to be detected through a residual connection layer.

[0084] In one embodiment, the Transformer network is a Biformer network; the Biformer network captures global features through a double-layer routing attention mechanism BRA;

[0085] The global features of the image to be detected are extracted based on the Transformer network, specifically including:

[0086] The image to be detected first passes through 3x3 DWConv to implicitly encode the relative position relationship, and then passes through LayerNorm normalization processing, and then passes through the BRA module to learn the cross-position relationship of the image to be detected. Finally, LayerNorm and MLP are used to represent the features of each position to obtain the global features of the image to be detected.

[0087] In one of the embodiments, the feature fusion correction module concatenates the extracted features with the upsampled feature map and the thermal map, and obtains the predicted thermal map after 3x3 and 1x1 convolution operations.

[0088] In one embodiment, in a multi-resolution heat map learning strategy training network, the corrected features are deeply supervised to generate multi-scale heat maps H1, H2, and H3, and the difference between the predicted heat map and the true heat map is calculated using the MSE loss. The loss function is defined as follows:

[0089] L all =L H1 +α1L H2 +α2L H3

[0090] Among them, L all Represents the total loss function of model training, L H1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively, and α1 and α2 are the loss weights.

[0091] like Figure 1As shown in the figure, the CNN-Transformer hybrid network architecture diagram shows the input and output of each stage of X-Ray medical images, and the modules included in each stage. The downsampling part of the model consists of a five-layer hybrid Trans-CNN module. For the input image, Patch Embedding is first performed, then input into the BiFormer module to capture global features, and then the local features are extracted through the CAB module. Multi-scale features {D1, D2, D3, D4, D5} are generated to enhance the feature representation ability of the model, and these features are then uniformly mapped to a feature space with 256 channels. During the upsampling process, we upsample the features through deconvolution operations and retain the original detail information through residual connections to maintain the accuracy of key points, and obtain {U1, U2, U3, U4, U5}. Features U4 and U5 are subjected to 3x3 and 1x1 convolution operations to obtain predicted heat maps H1 and H2. Feature U5, heat map H2 and image feature U6 are fused and corrected, and the predicted heat map H3 is obtained after 3x3 and 1x1 convolution operations. Finally, the key points of the heat maps H1, H2 and H3 predicted by features of different scales are output, and the average value of the key points is calculated to obtain the final output.

[0092] like Figure 2 As shown in the figure, in the original UNet network, only 3x3 convolution is used for feature extraction. Here, dilated convolution is used to increase the convolution receptive field, which is suitable for high-resolution input images. Residual connection is added to retain the original feature information. At the same time, the attention mechanism module CBAM (Convolutional Block Attention Module) of the enhanced convolutional neural network is added for feature selection to improve the performance of the model in complex visual tasks. The input feature first passes through a 3x3, dilation = 1 dilated convolution and is normalized and activated. Then it passes through a 3x3, dilation = 2 dilated convolution, and after normalization, it passes through the CBAM attention block, and finally the final output is obtained through the residual connection.

[0093] By comparing the existing networks with superior performance such as Swin-Transformer, Deformable Transformer and Bi-Level RoutingTransformer, and integrating them with the CNN network, we finally select the Biformer network with the best performance.

[0094] Biformer improves the performance of traditional ViT by introducing a bi-level routing attention mechanism (BRA). The core idea of ​​BRA is to remove the most irrelevant key-value regions at the coarse-grained region level, retain only the most relevant parts for fine-grained token-level attention calculation, and perform token-to-token attention in these regions. Figure 3 As shown, for a region i, collect its top k most relevant region index matrix I r , the gather algorithm is used to calculate the k most relevant Kg and Vg values ​​corresponding to the region, and then the attention matrix A of the region i and the k most relevant regions is calculated. Finally, the local context enhancement LCE is introduced to obtain the final output 0 for the region.

[0095] Compared with other hand-designed static sparse attention mechanisms, BRA can model long-range dependencies more flexibly, especially for dense prediction tasks that need to capture global features. Through the adaptive attention mechanism, the model can dynamically adjust the feature selection strategy according to the image content.

[0096] BiFormer can effectively capture global features through the BRA mechanism, especially the long-distance dependencies and overall geometric relationships between key points. Figure 4 As shown in the figure, the input features first pass through the 3x3 DWConv to implicitly encode the relative position relationship, and then pass through the LayerNorm normalization process, and then learn the cross-position relationship of the input through the BRA module, and finally further process the feature representation of each position through LayerNorm and MLP. Through efficient feature selection and fusion strategies, Biformer reduces the computational cost of traditional ViT and shows better computational efficiency when processing large-size images.

[0097] Since the first layer of the hybrid CNN-Transformer network is not used in the upsampling stage, the final generated heat map is a downsampled version of the original image, resulting in the loss of part of the image's original resolution information. To solve this problem, we introduced a feature fusion correction module, which is responsible for extracting the overall contextual information of the input image, generating image features using a global pooling layer and a fully connected network, and helping the model better capture the relative relationship between anatomical structures. The extracted features are concatenated with the upsampled feature map and the heat map, and the predicted heat map is obtained after 3x3 and 1x1 convolution operations. Through the feature fusion correction module, the model can not only effectively restore the resolution loss caused by downsampling, but also better integrate global and local information, ensuring that the model performs more stably and accurately in complex anatomical structures. This makes the model useful in various medical image application scenarios.

[0098] When detecting and segmenting abnormal areas such as tumors in medical images (such as MRI and CT), heat maps of different resolutions can help identify the size, shape and location of the tumor. Multi-resolution heat maps are generated from multi-scale features extracted from the network to improve the accuracy of key point positioning. Through multi-resolution heat map learning, multi-scale information is comprehensively utilized to perform deep supervision on the model, enhancing the robustness and generalization of the model.

[0099] We perform deep supervision on the output features of the last three layers of the network, generate multi-scale heat maps H1, H2, and H3 for learning, and use MSE loss to measure the difference between the predicted heat map and the real heat map. Through layer-by-layer supervision of the multi-scale heat map, the model can maintain a high prediction accuracy at different scales. The loss function is defined as follows:

[0100] L all =L G1 +α1L H2 +α2L H3

[0101] Among them, L all Represents the total loss function of model training, L H1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively. α1=3 and α2=3 are the weights of the loss. They can be adjusted according to different experimental requirements. Here we apply heat maps with large to small sigma from low resolution to high resolution. The sigma of H1 is 2, the sigma of H2 is 2, and the sigma of H3 is 3, so as to achieve a coarse-to-fine key point detection.

[0102] Experiments were conducted on multiple public datasets to verify the effectiveness and superiority of the designed model. Multiple commonly used public medical image datasets (such as ISBI2015, ISBI2023, hand X-ray Dataset, etc.) were selected to cover different types of medical images and key point detection tasks. X-ray image data was collected and preprocessed, and data enhancement was performed to improve the generalization ability of the model. Standardization processing techniques were used to eliminate noise and bias in the image. Different data enhancement techniques (such as rotation, scaling, translation, etc.) were used to expand the training data and improve the generalization ability of the model. Advanced optimization algorithms (such as Adam, SGD, etc.) were used to train the hybrid CNN-Transformer model. One or more of the feature fusion correction module, multi-resolution heat map learning strategy, and attention module were removed to observe their impact on the model performance. The contribution of each module to the overall model performance improvement was verified to ensure the effectiveness and necessity of each module.

[0103] After building the network with a hybrid CNN-Transformer architecture, the network performance was improved by fusing different transformer networks. After fusing CNN and Biformer, the MRE on the three data sets decreased. First, the three superior Transformer networks were compared at the same time. The MRE of the model introduced by the swin-transformer network was reduced by 0.0239. The MRE of the model was reduced by 0.0158 by combining the swin-transformer network with the Deformable Transformer network. Finally, the MRE of the model was reduced by 0.0394 by introducing the Biformer network, and the network performance was improved after deepening the model depth.

[0104] Table 1 CNN-Transformer hybrid network results

[0105]

[0106]

[0107] An ablation experiment was conducted on the model performance after adding the attention mechanism and the fusion feature correction module to the convolutional neural network. The experiment was conducted on the ISBI2015 head dataset. After adding the attention block, the model MRE was reduced by 0.0064. After fusing the global features, the model MRE decreased by 0.0053, which shows its effectiveness.

[0108] Table 2 Ablation experiment of attention mechanism and feature fusion correction module

[0109]

[0110] Compared with the current optimal SOTA network, our method reduces MRE by 0.05 on the ISBI2015 dataset, 0.03 on the ISBI2023 dataset, and 0.03 on the hand dataset, verifying the superiority of our method.

[0111] Table 3 Comparison of current network and SOTA results

[0112]

[0113] It should be understood that although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions of each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A key point detection method based on a hybrid CNN-Transformer model, characterized in that: include: The hybrid CNN-Transformer model consists of a CNN network, a Transformer network, and a multi-resolution heat map learning strategy training network. The method specifically includes: Acquire an image to be detected, and input the image to be detected into a hybrid CNN-Transformer model; Reducing the size of the image to be detected by downsampling; The reduced image to be detected is input into the CNN network and the Transformer network respectively to extract the features of the image to be detected, including: Extract local features of the image to be detected based on the CNN network; Extract global features of the image to be detected based on the Transformer network; The extracted local features, global features, original image features and key points predicted by low-resolution features are input into the feature fusion correction network for feature fusion, and the fused features are corrected; Apply the corrected features to the multi-resolution heat map learning strategy training network to output feature prediction key points at different resolutions; The key points of features at different resolutions are predicted and averaged, and the final predicted key point coordinates are output.

2. The key point detection method based on hybrid CNN-Transformer according to claim 1, characterized in that: The CNN network consists of a 3x3, dilation=1 hole convolution layer, a normalization and activation function layer, a 3x3, dilation=2 hole convolution layer, a normalization layer, an attention mechanism module CBAM and a residual connection layer; The method of extracting local features of an image to be detected based on a CNN network includes: the image to be detected first passes through a 3x3, dilation=1 hole convolution layer and performs normalization and activation function processing, then passes through a 3x3, dilation=2 hole convolution layer and normalization processing, then performs feature selection through an attention mechanism module CBAM, and finally outputs the local features of the image to be detected through a residual connection layer.

3. The key point detection method based on hybrid CNN-Transformer according to claim 1, characterized in that: The Transformer network is a Biformer network; the Biformer network captures global features through a double-layer routing attention mechanism BRA; The global features of the image to be detected are extracted based on the Transformer network, specifically including: The image to be detected first passes through 3x3 DWConv to implicitly encode the relative position relationship, and then passes through LayerNorm normalization processing, and then passes through the BRA module to learn the cross-position relationship of the image to be detected. Finally, LayerNorm and MLP are used to represent the features of each position to obtain the global features of the image to be detected.

4. The key point detection method based on hybrid CNN-Transformer according to claim 1, characterized in that: The feature fusion correction network concatenates the extracted features with the upsampled feature map and the heat map, and obtains the predicted heat map after 3x3 and 1x1 convolution operations.

5. The key point detection method based on hybrid CNN-Transformer according to claim 1, characterized in that: In the multi-resolution heat map learning strategy training network, the corrected features are deeply supervised to generate multi-scale heat maps H1, H2, and H3. The difference between the predicted heat map and the real heat map is calculated using the MSE loss. The loss function is defined as follows: L all =L H1 +α1L H2 +α2L H3 Among them, L all Represents the total loss function of model training, L H1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively, and α1 and α2 are the loss weights.

6. A key point detection system based on a hybrid CNN-Transformer model, characterized in that: include: The hybrid CNN-Transformer model consists of a CNN network, a Tranformer network, and a multi-resolution heat map learning strategy training network. The system specifically includes: An input module, used for acquiring an image to be detected and inputting the image to be detected into the hybrid CNN-Transformer model; A preprocessing module, used for reducing the size of the image to be detected by downsampling; The feature extraction module is used to input the reduced image to be detected into the CNN network and the Transformer network to extract the features of the image to be detected, including: Extract local features of the image to be detected based on the CNN network; Extract global features of the image to be detected based on the Transformer network; The feature fusion correction module is used to input the extracted local features, global features, original image features and key points predicted by low-resolution features into the feature fusion correction network for feature fusion, and correct the fused features; The multi-resolution heat map learning module is used to apply the modified features to the multi-resolution heat map learning strategy training network and output the feature prediction key points at different resolutions; The output module is used to predict key points of features at different resolutions and perform averaging processing, and output the final predicted key point coordinates.

7. The hybrid CNN-Transformer based key point detection system according to claim 6, characterized in that: The CNN network consists of a 3x3, dilation=1 hole convolution layer, a normalization and activation function layer, a 3x3, dilation=2 hole convolution layer, a normalization layer, an attention mechanism module CBAM and a residual connection layer; The method of extracting local features of an image to be detected based on a CNN network includes: the image to be detected first passes through a 3x3, dilation=1 hole convolution layer and performs normalization and activation function processing, then passes through a 3x3, dilation=2 hole convolution layer and normalization processing, then performs feature selection through an attention mechanism module CBAM, and finally outputs the local features of the image to be detected through a residual connection layer.

8. The hybrid CNN-Transformer based key point detection system according to claim 6, characterized in that: The Transformer network is a Biformer network; the Biformer network captures global features through a double-layer routing attention mechanism BRA; The global features of the image to be detected are extracted based on the Transformer network, specifically including: The image to be detected first passes through 3x3 DWConv to implicitly encode the relative position relationship, and then passes through LayerNorm normalization processing, and then passes through the BRA module to learn the cross-position relationship of the image to be detected. Finally, LayerNorm and MLP are used to represent the features of each position to obtain the global features of the image to be detected.

9. The hybrid CNN-Transformer based key point detection system according to claim 6, characterized in that: The feature fusion correction module concatenates the extracted features with the upsampled feature map and the heat map, and obtains the predicted heat map after 3x3 and 1x1 convolution operations.

10. The hybrid CNN-Transformer based key point detection system according to claim 6, characterized in that: In the multi-resolution heat map learning strategy training network, the corrected features are deeply supervised to generate multi-scale heat maps H1, H2, and H3. The difference between the predicted heat map and the real heat map is calculated using the MSE loss. The loss function is defined as follows: L all =L H1 +α1L H2 +α2L H3 Among them, L all Represents the total loss function of model training, L H1 , L H2 , L H3 They represent the key point loss functions in the H1, H2, and H3 heat maps in the deep supervision layer, respectively, and α1 and α2 are the loss weights.

Images