A two-stage 3D image key point detection method based on multi-scale and multi-dimensional feature fusion

By employing a two-stage approach that integrates multi-scale and multi-dimensional features, the problem of insufficient accuracy in detecting key points of targets in 3D images is solved, achieving higher detection accuracy and robustness.

CN118429667BActive Publication Date: 2026-11-17SHENYANG AEROSPACE UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410638282.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-22
Publication Date
2026-11-17
Estimated Expiration
2044-05-22

AI Technical Summary

Technical Problem

Existing keypoint detection methods for 3D images have insufficient detection accuracy when the proportion of target keypoints is small and their strength is similar to that of adjacent structures, and existing attention methods do not perform well in 3D images.

Method used

A two-stage approach based on multi-scale and multi-dimensional feature fusion is adopted. The region of interest is extracted through a coarse localization network, and the semantic and location information are enhanced by combining multi-dimensional feature enhancement attention and multi-scale feature fusion optimization modules to achieve accurate localization of key points.

Benefits of technology

It improves the accuracy and robustness of key point detection in 3D images, overcoming the problem of insufficient detection precision.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118429667B_ABST
    Figure CN118429667B_ABST
Patent Text Reader

Abstract

This invention proposes a two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion, including: S1: performing coarse localization on the input image, extracting the region of interest (ROI), and cropping the image to obtain the ROI image; S2: feeding the depth features into a multi-dimensional feature enhancement attention module to obtain deep fusion features; S3: simultaneously feeding features at four different scales into a multi-scale feature fusion optimization module to enhance semantic information and obtain multi-scale fusion features; S4: feeding the upsampled largest scale feature into a multi-dimensional feature enhancement attention module to obtain multi-dimensional fusion features; S5: multiplying the multi-scale fusion features with the multi-dimensional fusion features to obtain the final detection result. This invention can fully utilize the feature information generated in the network, enhance and refine semantic and positional information, and improve keypoint detection performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a two-stage method for detecting key points in three-dimensional images based on multi-scale and multi-dimensional feature fusion. Background Technology

[0002] Keypoint detection is a crucial step in image analysis, aiming to identify and locate important keypoints and their coordinates in an image. Convolutional neural networks (CNNs), with their ability to automatically extract both shallow and deep features, have proven to have excellent keypoint detection performance. Researchers have proposed a series of keypoint detection models for 2D and 3D images, mainly divided into end-to-end and multi-stage methods, most of which are based on encoder-decoder CNN architectures and exhibit excellent detection performance.

[0003] Because key points of a target occupy only a small portion of a 3D image, they are difficult to identify accurately. Furthermore, compared to 2D images, 3D images contain significantly more information, meaning that a large portion of the data is useless. This not only places a heavy burden on the computer during training but also hinders the network's ability to recognize and learn key features.

[0004] To reduce interference from structures of similar intensity surrounding keypoints, previous researchers have attempted to address this issue by introducing feature fusion methods from image processing and attention methods from natural language processing. Feature fusion methods can enhance the information interaction of features within the network, improve semantic information, and reduce false localization. Attention methods can focus the model on important information while ignoring unimportant information, effectively improving detection performance. However, most attention methods in images are designed only for two-dimensional images and do not consider the added depth information in three-dimensional images, often resulting in poor performance in three-dimensional models. Summary of the Invention

[0005] The purpose of this invention is to provide a two-stage 3D image key point detection method based on multi-scale and multi-dimensional feature fusion, which makes full use of the feature information generated in the network, enhances semantic and positional information, and solves the problem of insufficient detection accuracy caused by the small proportion of target key points in the image and their similar strength to adjacent structures.

[0006] To achieve the above objectives, the present invention includes the following steps:

[0007] A two-stage keypoint detection method for 3D images based on multi-scale and multi-dimensional feature fusion is characterized by using a two-stage strategy and a multi-scale and multi-dimensional feature fusion method to detect keypoints in 3D images; the method includes the following steps:

[0008] S1: Perform coarse localization on the input image, extract the region of interest, and crop the image to obtain the region of interest image;

[0009] S2: The image of the region of interest is fed into a downsampling network to obtain depth features, and then the depth features are fed into a multi-dimensional feature enhancement attention network to obtain deep fusion features;

[0010] S3: The deep fusion features are fed into the upsampling network to obtain four features at different scales in sequence. Then, the four features at different scales are fed into the multi-scale feature fusion optimization module to enhance semantic information and obtain multi-scale fusion features.

[0011] S4: Input the maximum scale feature obtained by upsampling into the multi-dimensional feature enhancement attention to obtain multi-dimensional fused features;

[0012] S5: Multiply the multi-scale fusion feature with the multi-dimensional fusion feature to obtain the final detection result.

[0013] Specifically, in S1, the input image is first fed into a coarse localization network to obtain the coordinates of the three-dimensional center points of the region containing all target key points. Then, a rectangular block representing the region of interest is generated based on the center coordinates. Finally, the rectangular block is used to crop the original image to obtain the region of interest image.

[0014] Specifically, in S2, deep features are fed into multi-dimensional feature enhancement attention to obtain deep fusion features:

[0015] S21: Input deep features By decomposing the data using three pooling kernels of sizes (H, 1, 1), (1, W, 1), and (1, 1, D), three high-dimensional features with directional information are obtained. Width dimension features and depth dimension features Where H, W, and D represent the height, width, and depth dimensions of the three-dimensional image, respectively. The decomposition process is shown in formula (1).

[0016]

[0017] S22: First, transform the three-dimensional features to 1×1×H, 1×1×W, and 1×1×D scales through a dimensionality transformation operation. Then, concatenate them along the depth dimension. Next, generate new weights through a convolutional processing layer consisting of two sets of 3×3×3 three-dimensional convolutions with alternating ReLU activation functions to obtain multi-dimensional intermediate features. As shown in formula (2),

[0018]

[0019] in[,] DF represents stitching along the depth dimension. conv This is a convolutional processing layer, where p represents the dimensionality transformation operation and δ is the ReLU non-linear activation function.

[0020] S23: Multi-dimensional intermediate features The features are split and transformed back into three dimensions, and then multiplied pairwise to obtain multi-dimensional intermediate features. and The scales are H×W×1, H×1×D, and 1×W×D, respectively;

[0021] Multidimensional intermediate features and First, the features are transformed to 1×W×D and H×1×D scales respectively. Then, the two sets of features with the same scale are concatenated and new weights are generated through a convolutional layer to obtain multi-dimensional intermediate features. and As shown in formulas (3) and (4):

[0022]

[0023]

[0024] in[,] W and[,] H These represent splicing along the width and height dimensions, respectively;

[0025] S23: Multidimensional intermediate features and After splitting and transforming, four intermediate features were obtained. and Two pairs of features with the same scale are fused along the channel dimension, and then processed by a 3D convolution with a kernel size of 1×1×1 and expanded to a size of H×W×D to obtain multi-dimensional intermediate features. and As shown in formulas (5) and (6):

[0026]

[0027]

[0028] in[,] C To stitch along the channel dimension, F1 is a 3D convolution, and E is a dilation operation; simultaneously, intermediate features... Multi-dimensional intermediate features are obtained by sequentially processing convolutional layers, ReLU, and dilation operations. The process is given in formula (7),

[0029]

[0030] S24: As shown in formula (8), the multi-dimensional intermediate features are... and The data is stitched along the channel dimension, processed by 3D convolution, and then combined with multi-dimensional intermediate features. Multiply and fuse the features, then multiply them again with the input features to obtain the deep fused features;

[0031]

[0032] Specifically, the multi-scale feature fusion optimization module first processes the four input multi-scale features through multiple upsampling and downsampling layers to obtain four groups of 16 features at different scales. Then, it adds and fuses the features at the same scale in parallel to obtain the multi-scale intermediate features. and

[0033] Next, the four multi-scale intermediate features are refined to enhance semantic information, resulting in intermediate features. and As in the formula As shown in {i∈(1,4)}, Up represents the upsampling convolution operation, T represents the activation function TanH, F2 represents a 3D convolution with a kernel size of 1×1×1, and DSC represents a 3D depthwise separable convolution with a kernel size of 7×7×7. Finally, the four refined intermediate features are added and fused, and then activated by the TanH function to obtain the final multi-scale fused feature F. s .

[0034] Specifically, S3 includes:

[0035] S31: Input four multi-scale features Sixteen feature depths at four different scales were obtained through multiple upsampling and downsampling layers. The implementation methods of the upsampling and downsampling layers used in this process are shown in formulas (9) and (10).

[0036]

[0037]

[0038] in and These represent the output features of the upsampling layer and the downsampling layer, respectively.

[0039] Up represents upsampling convolution, BN represents batch normalization, DSC represents 3D depthwise separable convolution with a kernel size of 7×7×7, and δ represents the non-linear activation function ReLU.

[0040] S32: Features of each group at the same scale are added and fused in parallel to obtain multi-scale intermediate features. and Furthermore, by optimizing the four multi-scale intermediate features and refining the location information, the intermediate features are obtained. and As shown in formula (11),

[0041]

[0042] Where Up represents upsampling convolution operation, T represents activation function TanH, F2 represents 3D convolution with kernel size of 1×1×1, and DSC represents 3D depthwise separable convolution with kernel size of 7×7×7.

[0043] S33: Intermediate features and After addition and fusion, and activation by the TanH function, the final multi-scale fusion feature is obtained, as shown in formula (12):

[0044]

[0045] Specifically, in S4, the maximum scale feature f obtained from upsampling is... m 8 s Inputting multi-dimensional features enhances attention, resulting in multi-dimensional fused features F. d .

[0046] Specifically, the multi-scale fusion feature F obtained in steps S3 and S4 s and multi-dimensional fusion features F d The results are multiplied and combined to obtain the final test result.

[0047] This invention proposes a two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion, which effectively overcomes the problem of insufficient detection accuracy caused by the small proportion of target keypoints in 3D images and their similar strength to adjacent structures. The first stage of this method uses a 3D U-shaped network to locate the center of the target region and then extracts the region of maximum interest to reduce interference from surrounding irrelevant structures, providing input for the second stage. The second stage adds multi-dimensional feature enhancement attention and multi-scale feature fusion optimization modules to the 3D encoder-decoder network architecture to achieve accurate identification and localization of keypoints. The multi-dimensional feature enhancement attention fused the height, width, and depth features of the feature map multiple times and placed after the output of the encoder and decoder to enhance feature location information. The multi-scale feature fusion optimization module fuses and refines the semantic information of each layer in the decoder. The resulting multi-scale fusion result is multiplied by the multi-dimensional fusion result applied to the decoder output to generate the final detection result. This invention can fully utilize the feature information generated in the network, enhance and refine semantic and location information, and improve keypoint detection performance.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the present invention. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a structural diagram of a two-stage method based on multi-scale and multi-dimensional feature fusion provided in the embodiments of the present invention;

[0052] Figure 2 A structural diagram of a multi-scale feature fusion optimization module is provided for the embodiments disclosed in this invention;

[0053] Figure 3 This invention discloses a multi-dimensional feature-enhanced attention structure diagram for its embodiments. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without innovative effort are within the scope of protection of the present invention.

[0055] The purpose of this invention is to provide a two-stage three-dimensional image key point detection method based on multi-scale and multi-dimensional feature fusion, which can make full use of the feature information generated in the network, enhance the ability to express semantic and positional information, and solve the problem of insufficient detection accuracy caused by the small proportion of target key points in the image and their similar strength to adjacent structures.

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Figure 1 This is a structural diagram of a two-stage method based on multi-scale and multi-dimensional feature fusion provided in an embodiment of the present invention. Figure 2 A structural diagram of a multi-scale feature fusion optimization module is provided for the embodiments disclosed in this invention. Figure 3 A multi-dimensional feature-enhanced attention structure diagram is provided for embodiments of the present invention. See also Figure 1 , Figure 2 , Figure 3 The specific steps of the two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion provided by this invention are as follows:

[0058] S1: Feed the input image into the coarse localization network to obtain the center coordinates of the target region. Extract the largest region of interest (ROI) rectangle based on the center point and use it to crop the original image.

[0059] S2: First, the region of interest image obtained in step S1 is fed into a downsampling network to obtain depth features. Then, the depth features are fed into a multi-dimensional feature enhancement attention network to obtain deep fusion features.

[0060] The multi-dimensional feature-enhanced attention first incorporates the input deep features High-dimensional features with directional information are obtained by decomposing the data using three pooling kernels of sizes (H, 1, 1), (1, W, 1), and (1, 1, D). Width dimension features and depth dimension features The decomposition process is shown in formula (1).

[0061]

[0062]

[0063]

[0064] To better utilize the generated feature information, the three-dimensional features are first transformed into 1×1×H, 1×1×W, and 1×1×D scales through a dimensionality transformation operation. Then, they are concatenated along the depth dimension. Finally, new weights are generated through a convolutional processing layer consisting of two sets of 3×3×3 three-dimensional convolutions and ReLU activation functions, resulting in multi-dimensional intermediate features. As shown in formula (2),

[0065]

[0066] in[,] D F represents stitching along the depth dimension. conv This is a convolutional processing layer, where p represents the dimension transformation operation and δ is the ReLU non-linear activation function.

[0067] Next, the multi-dimensional intermediate features The features are split and transformed back into three dimensions, and then multiplied pairwise to obtain multi-dimensional intermediate features. and The sizes are H×W×1, H×1×D, and 1×W×D, respectively. Multi-dimensional intermediate features. and First, the features are transformed to 1×W×D and H×1×D scales respectively. Then, the two sets of features with the same scale are concatenated and new weights are generated through a convolutional layer to obtain multi-dimensional intermediate features. and As shown in formulas (3) and (4),

[0068]

[0069]

[0070] in[,] W and[,] H These represent splicing along the width and height dimensions, respectively.

[0071] Then multi-dimensional intermediate features and After splitting and transforming, four intermediate features were obtained. and Two pairs of features with the same scale are fused along the channel dimension, and then processed by a 3D convolution with a kernel size of 1×1×1 and expanded to a size of H×W×D to obtain multi-dimensional intermediate features. and As shown in formulas (5) and (6),

[0072]

[0073]

[0074] in[,] C To stitch along the channel dimension, F1 is a 3D convolution, and E is a dilation operation. Meanwhile, intermediate features... Multi-dimensional intermediate features are obtained by sequentially processing convolutional layers, ReLU, and dilation operations. The process is given in formula (7),

[0075]

[0076] Finally, as shown in formula (8), multi-dimensional intermediate features and The data is stitched along the channel dimension, processed by 3D convolution, and then combined with multi-dimensional intermediate features. The features are multiplied and fused, and then multiplied again with the input features to obtain multi-dimensional fused features.

[0077]

[0078] Step S3: The deep fusion features obtained in Step S2 are fed into the upsampling network to obtain four features at different scales. Finally, the four features are fed into the multi-scale feature fusion optimization module to obtain the multi-scale fusion feature F. s .

[0079] The multi-scale feature fusion optimization module first processes the four input multi-scale features... Sixteen feature depths at four different scales were obtained through multiple upsampling and downsampling layers. The implementation methods of the upsampling and downsampling layers used in this process are shown in formulas (9) and (10).

[0080]

[0081]

[0082] in and These represent the output features of the upsampling and downsampling layers, respectively. In this invention, the scale ratio of each output to the input feature is 2, 4, or 8, depending on the different requirements of the input and target features. Up represents the upsampling convolution operation, BN represents batch normalization, DSC represents a 3D depthwise separable convolution with a kernel size of 7×7×7, and δ represents the nonlinear activation function ReLU.

[0083] Then, features from each group at the same scale are added and fused in parallel to obtain multi-scale intermediate features. and Furthermore, by optimizing the four multi-scale intermediate features and refining the location information, the intermediate features are obtained. and As shown in formula (11),

[0084]

[0085] Where Up represents upsampling convolution, T represents the activation function TanH, F2 represents a 3D convolution with a kernel size of 1×1×1, and DSC represents a 3D depthwise separable convolution with a kernel size of 7×7×7.

[0086] Finally, as shown in formula (12), the four optimized intermediate features are added together and fused. After activation by the TanH function, the final multi-scale fused features are obtained.

[0087]

[0088] Step S4: The maximum scale feature obtained by upsampling in step S3 Inputting multi-dimensional features enhances attention, resulting in multi-dimensional fused features F. d .

[0089] Step S5: The multi-scale fusion feature F obtained from steps S3 and S4 s and multi-dimensional fusion features F d The results are multiplied and combined to obtain the final test result.

[0090] This invention proposes a two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion, overcoming the problem of insufficient detection accuracy caused by the small proportion of target keypoints in the image and their similar strength to adjacent structures. It has the advantages of high accuracy and strong robustness.

[0091] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

Claims

1. A two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion, characterized in that, Keypoint detection in 3D images is performed using a two-stage strategy and a multi-scale, multi-dimensional feature fusion method; the method includes the following steps: S1: Perform coarse localization on the input image, extract the region of interest, and crop the image to obtain the region of interest image; S2: The image of the region of interest is fed into a downsampling network to obtain depth features, and then the depth features are fed into a multi-dimensional feature enhancement attention network to obtain deep fusion features; S3: The deep fusion features are fed into the upsampling network to obtain four features at different scales in sequence. Then, the four features at different scales are fed into the multi-scale feature fusion optimization module to enhance semantic information and obtain multi-scale fusion features. S4: Input the maximum scale feature obtained by upsampling into the multi-dimensional feature enhancement attention to obtain multi-dimensional fused features; S5: Multiply the multi-scale fusion feature with the multi-dimensional fusion feature to obtain the final detection result; In S2, deep features are fed into multi-dimensional features to enhance attention and obtain deep fused features: S21: Input deep features By decomposing the data using three pooling kernels of sizes (H, 1, 1), (1, W, 1), and (1, 1, D), respectively, three high-dimensional features with directional information are obtained. Width dimension features and depth dimension features H, W, and D represent the height, width, and depth dimensions of the three-dimensional image, respectively. The decomposition process is shown in formula (1). S22: First transform the three-dimensional features through a dimension transformation operation. , and The scale is then calculated, and the data is concatenated along the depth dimension, followed by two sets of convolutional kernels of size 1. The convolutional processing layer, composed of alternating 3D convolutions and ReLU activation functions, generates new weights and obtains multi-dimensional intermediate features. As shown in formula (2), in This represents stitching along the depth dimension. For convolutional processing layers, Represents a dimensional transformation operation. ReLU is a non-linear activation function. S23: Multi-dimensional intermediate features The features are split and transformed back into three dimensions, and then multiplied pairwise to obtain multi-dimensional intermediate features. , and The scales are respectively , and ; Multidimensional intermediate features and First, transform them into dimensional transformation operations respectively. and The scale is then used to concatenate two sets of features with the same scale, and new weights are generated through a convolutional layer to obtain multi-dimensional intermediate features. and As shown in formulas (3) and (4): in and These represent splicing along the width and height dimensions, respectively; S23: Multidimensional intermediate features and After splitting and transforming, four intermediate features were obtained. , , and Two pairs of features of the same scale are fused along the channel dimension and then processed by a convolution kernel of size [missing information]. 3D convolution processing and dilation to Size, to obtain multi-dimensional intermediate features and As shown in formulas (5) and (6): in To stitch along the channel dimension, For 3D convolution, For expansion operations; simultaneously, intermediate features Multi-dimensional intermediate features are obtained by sequentially processing convolutional layers, ReLU, and dilation operations. The process is given in formula (7). S24: As shown in formula (8), the multi-dimensional intermediate features are... and The data is stitched along the channel dimension, processed by 3D convolution, and then combined with multi-dimensional intermediate features. Multiply and fuse the features, then multiply them again with the input features to obtain the deep fused features; 。 2. The two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion according to claim 1, characterized in that, In S1, the input image is first fed into a coarse localization network to obtain the coordinates of the three-dimensional center points of the region containing all target key points. Then, a rectangular block representing the region of interest is generated based on the center coordinates. Finally, the rectangular block is used to crop the original image to obtain the region of interest image.

3. The two-stage 3D image keypoint detection method based on multi-scale and multi-dimensional feature fusion according to claim 1, characterized in that, The multi-scale feature fusion optimization module first processes the four input multi-scale features through multiple upsampling and downsampling layers to obtain four groups of 16 features at different scales. Then, it adds and fuses the features at the same scale in parallel to obtain the multi-scale intermediate features. , , and ; Next, the four multi-scale intermediate features are refined to enhance semantic information, resulting in intermediate features. , , and , as in the formula As shown, where For upsampling convolution operations, Represents the activation function TanH. The kernel size represents the convolution kernel size. 3D convolution, The kernel size is The three-dimensional depthwise separable convolution is performed; finally, the four refined intermediate features are added and fused, and then activated by the TanH function to obtain the final multi-scale fused features. .

4. The two-stage three-dimensional image key point detection method based on multi-scale and multi-dimensional feature fusion according to claim 1, characterized in that, S3 include: S31: Input four multi-scale features Sixteen feature depths at four different scales were obtained through multiple upsampling and downsampling layers. The implementation methods of the upsampling and downsampling layers used in this process are shown in formulas (9) and (10). in and These represent the output features of the upsampling layer and the downsampling layer, respectively. For upsampling convolution operations, For batch normalization, The kernel size is 3D depthwise separable convolution, ReLU represents a non-linear activation function. S32: Features of the same scale in each group are added and fused in parallel to obtain multi-scale intermediate features. , , and Furthermore, by optimizing the four multi-scale intermediate features and refining the location information, intermediate features are obtained. , , and As shown in formula (11), in For upsampling convolution operations, Represents the activation function TanH. The kernel size represents the convolution kernel size. 3D convolution, The kernel size is 3D depthwise separable convolution; S33: Intermediate features , , and After addition and fusion, and activation by the TanH function, the final multi-scale fusion feature is obtained, as shown in formula (12): 。 5. The two-stage three-dimensional image key point detection method based on multi-scale and multi-dimensional feature fusion according to claim 1, characterized in that, The maximum scale feature obtained by upsampling in S4 Inputting multi-dimensional features enhances attention, resulting in multi-dimensional fused features. .

6. The two-stage three-dimensional image key point detection method based on multi-scale and multi-dimensional feature fusion according to claim 1, characterized in that, The multi-scale fusion features obtained in steps S3 and S4 and multi-dimensional fusion features The results are multiplied and combined to obtain the final test result.