A hip keypoint detection system and method with adaptive feature learning

Abstract

The application relates to a hip joint key point detection system and method based on adaptive feature learning, and in the system part: an encoder, a decoder and a multi-scale feature fusion module are arranged, the encoder is used for carrying out feature coding on an input overall image through a series of convolution and down-sampling operations, and feature information of different scales of the hip joint is extracted; meanwhile, the decoder is matched with the multi-scale feature fusion module, different-scale hip joint feature information obtained in the encoder is fused, and the deficiency that in the traditional UNet, the complementary information between multi-scale features under different receptive fields is ignored because the same-scale features are spliced according to channels through the skip connection is made up.

Description

Technical Field

[0001] This invention relates to the field of imaging medicine technology, and more specifically, to a hip joint key point detection system and method with adaptive feature learning. Background Technology

[0002] In hip joint imaging, the hip joint's peripheral contour is relatively broad, with key points scattered and small targets. Furthermore, the morphology of the hip joint structure exhibits complexity and variability due to individual factors such as age, sex, and pathological abnormalities. Traditional medical imaging key point detection methods typically rely on manually designed features and machine learning algorithms. However, methods based on manual feature extraction depend on specific shape and structural features, failing to adapt to the morphological and structural differences between individuals, and thus exhibiting significant limitations in accuracy and robustness.

[0003] Deep learning methods can learn deep feature representations that reflect the characteristics of images, achieving better detection results by utilizing the extracted high-level semantic information. Currently, deep learning-based hip joint keypoint detection methods are mainly divided into two categories: two-stage detection methods and single-stage detection methods. The final detection accuracy of two-stage detection methods depends on the selection of candidate regions for keypoints or the segmentation results of their neighborhoods in the first stage. In single-stage detection methods, the UNet network has been widely used and has achieved significant breakthroughs due to its powerful semantic feature extraction capabilities.

[0004] UNet mainly consists of three parts: encoder, decoder, and skip connections. In this network, the encoder is used to extract feature information under different receptive fields, the decoder gradually restores the feature map size, and the skip connections of each layer introduce shallow encoder features of the same scale into the decoder to compensate for the information loss in the decoding stage.

[0005] However, although UNet and its augmented networks improve feature utilization to some extent, skip connections in the network only introduce encoded features of the same scale into the decoder, ignoring the complementary information between multi-scale features of different receptive fields. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to make up for the deficiency of ignoring the complementary information between multi-scale features under different receptive fields in single-stage detection methods. In order to overcome the above-mentioned defects of the prior art, this invention provides an adaptive feature learning hip joint key point detection system and method, including an adaptive feature learning hip joint key point detection system and an adaptive feature learning hip joint key point detection method.

[0007] This invention provides an adaptive feature learning-based hip joint key point detection system, the hip joint key point detection system comprising:

[0008] The encoder performs n feature encodings on the hip joint image of the hip joint to be tested using convolution and downsampling operations, and outputs the hip joint feature information formed after each feature encoding to output n hip joint feature information with different scales.

[0009] The decoder performs n decodings starting from the feature information formed by the nth feature encoding output by the encoder. During each decoding except the nth decoding, it issues channel splicing requirement information and converts the fused feature information corresponding to the channel splicing requirement information into a corresponding feature map for the next decoding. During the nth decoding, it obtains the key point prediction result of the hip joint to be tested based on the feature map obtained from the previous decoding.

[0010] The multi-scale feature fusion module receives the channel splicing requirement information sent by the decoder, adjusts the scale and number of channels of several hip joint feature information output by the encoder according to the channel splicing requirement information, splices them into their respective corresponding fused feature information, and sends them back to the decoder.

[0011] in,

[0012] The encoder is communicatively connected to the decoder, and the multi-scale feature fusion module is communicatively connected to both the encoder and the decoder.

[0013] Compared with existing technologies, the adaptive feature learning hip joint keypoint detection system of this application has the following advantages: By setting up an encoder, a decoder, and a multi-scale feature fusion module, the encoder performs feature encoding on the input overall image through a series of convolution and downsampling operations to extract feature information of the hip joint at different scales. On this basis, the decoder is combined with the multi-scale feature fusion module. Since the multi-scale feature fusion module can fuse the hip joint feature information of different scales obtained from the encoder according to the needs of the decoder, it makes up for the shortcomings of the skip connections in the traditional UNet, which can only stitch together features of the same scale by channel and ignore the complementary information between multi-scale features under different receptive fields. It not only adapts to the morphological and structural differences between different individuals, but also has advanced and completeness in terms of accuracy and robustness.

[0014] In one possible implementation, the encoder consists of n sub-encoders connected in series, which sequentially encode the hip joint image from beginning to end. All sub-encoders except the last one are communicatively connected to the multi-scale feature fusion module, and the last sub-encoder is communicatively connected to the decoder.

[0015] Compared with existing technologies, the above technical solution can use different sub-encoders to correspond to the various encodings in n encodings, realize n feature encodings of the hip joint image, and send the hip joint feature information formed after each feature encoding to a multi-scale feature fusion module and decoder to achieve accurate encoding-decoding.

[0016] In one possible implementation, the first sub-encoder includes a first Basic module, and the remaining sub-encoders each include a downsampling module and a second Basic module. The first Basic module is communicatively connected to the multi-scale feature fusion module and the downsampling module in the next sub-encoder.

[0017] In the sub-encoder at the end: the second Basic module is communicatively connected to the downsampling module and the decoder in this sub-encoder;

[0018] In the remaining sub-encoders: the downsampling module is communicatively connected to the second Basic module in this sub-encoder, and the second Basic module is communicatively connected to the multi-scale feature fusion module and the downsampling module in the next sub-encoder;

[0019] This allows for the use of the first Basic module and the second Basic module as convolution operation executors, and the downsampling module as the max pooling downsampling executor, ultimately enabling the successive encoding of the hip joint image to be tested and extracting feature information at different scales.

[0020] In one possible implementation, the decoder is composed of n sub-decoders connected in series, the last sub-decoder is communicatively connected to the second Basic module in the last sub-encoder, and all sub-decoders except the first sub-decoder are communicatively connected to the multi-scale feature fusion module.

[0021] The last sub-decoder is used to upsample and convolve the feature information output by the last sub-encoder to convert it into corresponding channel stitching requirement information and send it to the multi-scale feature fusion module. It also converts the fused feature information returned by the multi-scale feature fusion module into a corresponding feature map and sends it to the previous sub-decoder. The first sub-decoder is used to convolve and sigmoid activate the feature map output by the next sub-decoder to obtain the key point prediction result of the hip joint image. The remaining sub-decoders are used to upsample and convolve the feature map output by the next sub-decoder to convert it into corresponding channel stitching requirement information and send it to the multi-scale feature fusion module. They also convert the fused feature information returned by the multi-scale feature fusion module into a corresponding feature map and send it to the previous sub-decoder. All the feature maps contain the configuration features of the hip joint.

[0022] This allows for the step-by-step decoding of the encoded feature information to restore the feature map size to the original resolution of the network input image, and the final hip joint key point prediction results are performed at the end of the decoder to ensure reliable detection results.

[0023] In one possible implementation, the first sub-decoder includes a third Basic module, and the remaining sub-decoders each include an upsampling module, a fourth Basic module, and a multi-scale feature channel attention module for adaptively extracting key feature information from each hip joint feature information.

[0024] In the final sub-decoder: the upsampling module is communicatively connected to the second Basic module in the final sub-encoder and the fourth Basic module in this sub-decoder; the fourth Basic module in this sub-decoder is communicatively connected to the multi-scale feature channel attention module in this sub-decoder; and the multi-scale feature channel attention module in this sub-decoder is communicatively connected to the multi-scale feature fusion module and the upsampling module in the previous sub-decoder.

[0025] In the first sub-decoder: the third Basic module is communicatively connected to the multi-scale feature channel attention module in the next sub-decoder;

[0026] In the remaining sub-decoders: the upsampling module is communicatively connected to the multi-scale feature channel attention module in the next sub-decoder and the fourth Basic module in this sub-decoder; the fourth Basic module in this sub-decoder is communicatively connected to the multi-scale feature channel attention module in this sub-decoder.

[0027] Compared to existing technologies, deploying the multi-scale feature channel attention module within the system's decoder enables each sub-decoder's decoding layer to combine the features processed by the multi-scale feature fusion module with the high-level semantic features obtained from the previous sub-decoder's decoding layer, resulting in a feature set containing richer information. The multi-scale feature channel attention module guides the network to adaptively learn discriminative and important feature information. Through progressive upsampling operations involving convolution and interpolation in the decoder, the feature map size is restored, and the network's prediction results are output.

[0028] In one possible implementation, the third Basic module includes a first convolutional unit and a sigmoid activation unit arranged sequentially along the running direction;

[0029] This enables the use of pre-defined convolutional kernels to convolve the output features of the last layer of the decoder and perform sigmoid activation at the end of the system network structure, thereby mapping the output features of the decoder module to the desired keypoint detection results.

[0030] In one possible implementation, each of the multi-scale feature channel attention modules includes a first splicing unit, a max pooling unit, a first fully connected unit, an average pooling unit, a second fully connected unit, a weight generation unit, a dot product unit, a second splicing unit, and a Basic submodule.

[0031] All the first stitching units are communicatively connected to the multi-scale feature fusion module, and each first stitching unit is also communicatively connected to the fourth Basic module in its sub-decoder;

[0032] In each of the multi-scale feature channel attention modules:

[0033] The first splicing unit, the max pooling unit, the first fully connected unit, and the weight generation unit are connected via communication to form a network branch. The first splicing unit, the average pooling unit, the second fully connected unit, and the weight generation unit are connected via communication to form another network branch. The dot product unit is connected via communication with the weight generation unit. The second splicing unit is connected via communication with the dot product unit. The Basic submodule is connected via communication with the second splicing unit and the upsampling module of the next sub-decoder in the sub-decoder (21) where this Basic submodule is located.

[0034] The first fully connected unit and the second fully connected unit have the same operational parameters;

[0035] The first splicing unit is used to receive the output of the fourth Basic module connected to it, and use it as the channel splicing requirement information. Then, it calls the multi-scale feature fusion module to output the corresponding fusion feature information, and splices the fusion feature information with the channel splicing requirement information to obtain the first splicing result.

[0036] The weight generation unit is used to fuse the outputs of the first fully connected unit and the second fully connected unit, and generate weights using sigmoid activation.

[0037] The dot product unit is used to perform a dot product operation between the weights generated by the weight generation unit and the fused feature information to obtain a dot product result;

[0038] The second splicing unit is used to perform channel splicing between the dot product result output by the dot product unit and the channel splicing requirement information to obtain the second splicing result;

[0039] The Basic submodule is used to perform convolution, normalization, and ReLU activation on the second concatenation result to obtain the feature map;

[0040] By setting up a multi-scale feature channel attention module with the above structure in the decoder, and through a series of fusion and splicing operations, the corresponding feature map is finally obtained, which improves the detection performance and robustness of the network.

[0041] In one possible implementation, the multi-scale feature fusion module is composed of (n-1) fifth Basic modules connected in parallel. The first Basic module and all the second Basic modules except for the second Basic modules in the last sub-encoder are each communicatively connected to one of the fifth Basic modules. Each of the first splicing units is communicatively connected to all the fifth Basic modules.

[0042] This enables the fusion of features at different scales according to the splicing requirements of each sub-decoder, improving feature utilization and ensuring accurate and reliable prediction of key point features.

[0043] In one possible implementation, all of the fifth Basic modules include a second convolution unit, a normalization unit, and a ReLU activation unit arranged sequentially along the running direction;

[0044] This allows it to adapt to the inconsistent image resolution of features at each level of the decoder, and to meet the different operational requirements for feature fusion at each level, in order to perform convolution, normalization, and activation processing.

[0045] Another technical solution of the present invention is to provide an adaptive feature learning method for hip joint key point detection. The hip joint key point detection system based on the adaptive feature learning method of the present invention includes the following steps:

[0046] S1: The hip joint image of the hip joint to be tested is encoded n times by the encoder, and the hip joint feature information formed after each feature encoding is output to output n hip joint feature information with different scales.

[0047] S2: Receive several hip joint feature information pieces with different scales obtained from step S1 through a multi-scale feature fusion module;

[0048] S3; Based on the hip joint feature information obtained in step S1, the decoder sequentially decodes the feature information formed by the nth feature encoding. In each decoding except the nth decoding, the decoder sends out the channel splicing requirement information for this decoding. The multi-scale feature fusion module receives the channel splicing requirement information for this decoding and adjusts the scale and number of channels of the several hip joint feature information received in step S2 accordingly. Then, it splices the fused feature information for this decoding and sends it back to the decoder. The decoder converts it into the feature map for the next decoding. In the nth decoding, the key point prediction result of the hip joint to be tested is obtained based on the feature map obtained in the previous decoding.

[0049] This invention discloses an adaptive feature learning method for hip joint keypoint detection. By introducing a multi-scale feature fusion module, features containing information at different levels from the encoder are fused. The generated fused features include superficial detailed features and deep global high-level semantic features of the hip joint, effectively guiding the decoding process of each subsequent layer. Therefore, based on the acquired hip joint feature information, the method effectively detects the corresponding anatomical keypoints, exhibiting excellent detection performance and making it suitable for clinical auxiliary diagnosis. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the structure of a hip joint key point detection system with adaptive feature learning in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure and operation of an adaptive feature learning hip joint key point detection system according to an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the multi-scale feature fusion module and the multi-scale feature channel attention module in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the operation of the multi-scale feature fusion module and the multi-scale feature channel attention module in an embodiment of the present invention;

[0054] Figure 5 This is a flowchart of the method in an embodiment of the present invention.

[0055] Explanation of reference numerals in the attached figures:

[0056] 1. Encoder; 11. Sub-encoder; 111. First Basic Module; 112. Downsampling Module; 113. Second Basic Module; 2. Decoder; 21. Sub-decoder; 211. Third Basic Module; 212. Upsampling Module; 213. Fourth Basic Module; 214. Multi-scale Feature Channel Attention Module; 2141. First Concatenation Unit; 2142. Max Pooling Unit; 2143. First Fully Connected Unit; 2144. Average Pooling Unit; 2145. Second Fully Connected Unit; 2146. Weight Generation Unit; 2147. Dot Multiplication Unit; 2148. Second Concatenation Unit; 2149. Basic Sub-Module; 3. Multi-scale Feature Fusion Module; 31. Fifth Basic Module. Detailed Implementation

[0057] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0058] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0059] In the embodiments of this application, unless otherwise explicitly specified and limited, a communication connection between the first feature and the second feature means that there is information transmission between the first feature and the second feature. This information transmission can be unidirectional or bidirectional, and the communication connection can be achieved by means of electrical connection of wires, radio connection, electrical connection of electromagnetic medium (such as semiconductor), communication realized by channel, etc.

[0060] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0061] See Figures 1-4As shown in the figure, this application discloses an adaptive feature learning-based hip joint key point detection system. See also... Figure 1 As shown, the hip joint key point detection system includes an encoder 1, a decoder 2, and a multi-scale feature fusion module 3. The encoder 1 is communicatively connected to the decoder 2, and the multi-scale feature fusion module 3 is communicatively connected to both the encoder 1 and the decoder 2. The functions of each component are listed below:

[0062] First, encoder 1 is set to perform n feature encodings on the hip joint image of the hip joint to be tested using convolution and downsampling operations, and output the hip joint feature information formed after each feature encoding to output n hip joint feature information with different scales.

[0063] Second, decoder 2 is set to perform n decodings starting from the feature information formed by the nth feature encoding output by encoder 1. During each decoding except the nth decoding, channel splicing requirement information is issued, and the fused feature information corresponding to the channel splicing requirement information is converted into a corresponding feature map for the next decoding. During the nth decoding, the key point prediction result of the hip joint to be tested is obtained based on the feature map obtained from the previous decoding.

[0064] Third, the multi-scale feature fusion module 3 is configured to receive the splicing requirements information of each channel sent by the decoder 2, adjust the scale and number of channels of several hip joint feature information output by the encoder 1 according to the splicing requirements information of each channel, splice them into their respective corresponding fused feature information, and send them back to the decoder 2.

[0065] As one implementation method, see Figure 1 , Figure 2 As shown, in this embodiment, n=5, so encoder 1 is composed of 5 sub-encoders 11 connected in series, so that the sub-encoders 11 encode the hip joint image successively in the order from the beginning to the end. All sub-encoders 11 except the last sub-encoder 11 are connected to the multi-scale feature fusion module 3. The last sub-encoder 11 is connected to the decoder 2.

[0066] The functions of each sub-encoder 11 are listed below: The first sub-encoder 11 is used to encode the hip joint image for the first time to form a hip joint image of a predetermined size, which is used as the hip joint feature information after the first feature encoding. It is also used to output the first sub-encoder 11 and input it to the multi-scale feature fusion module 3. The last sub-encoder 11 is used to encode the hip joint feature information output by the previous sub-encoder 11, and input the fifth feature information to the decoder 2. The remaining sub-encoders 11 are used to encode the output information of the previous sub-encoder 11, obtain the corresponding feature information, and output it to the next sub-encoder 11 and input it to the multi-scale feature fusion module 3.

[0067] Please continue reading Figure 1 As shown, in this embodiment, the first sub-encoder 11 includes a first Basic module 111, and the remaining sub-encoders 11 each include a downsampling module 112 and a second Basic module 113. The first Basic module 111 is communicatively connected to the multi-scale feature fusion module 3 and the downsampling module 112 in the next sub-encoder 11.

[0068] Please continue reading Figure 1 As shown, in the last sub-encoder 11: the second Basic module 113 is communicatively connected to the downsampling module 112 and the decoder 2 in this sub-encoder 11. In the remaining sub-encoders 11: the downsampling module 112 is communicatively connected to the second Basic module 113 in this sub-encoder 11, and the second Basic module 113 is communicatively connected to the multi-scale feature fusion module 3 and the downsampling module 112 in the next sub-encoder 11.

[0069] Please continue reading Figure 1 As shown, corresponding to encoder 1, in this embodiment, decoder 2 is composed of 5 sub-decoders 21 connected in series. Their operating order is from the end to the beginning. The sub-decoder 21 at the end is connected to the second Basic module 113 in the sub-encoder 11 at the end. All sub-decoders 21 except the sub-decoder 21 at the beginning are connected to the multi-scale feature fusion module 3.

[0070] Corresponding to the structure and function of decoder 1, in this embodiment, the last sub-decoder 21 is used to upsample and convolve the feature information output by the last sub-encoder 11 to convert it into the corresponding channel stitching requirement information and send it to the multi-scale feature fusion module 3. The fused feature information returned by the multi-scale feature fusion module 3 is then converted into the corresponding feature map and sent to the previous sub-decoder 21. The first sub-decoder 21 is used to convolve and sigmoid activate the feature map output by the next sub-decoder 21 to obtain the key point prediction result of the hip joint image. The remaining sub-decoders 21 are used to upsample and convolve the feature map output by the next sub-decoder 21 to convert it into the corresponding channel stitching requirement information and send it to the multi-scale feature fusion module 3. The fused feature information returned by the multi-scale feature fusion module 3 is then converted into the corresponding feature map and sent to the previous sub-decoder 21. All feature maps contain the configuration features of the hip joint.

[0071] Please continue reading Figure 1 As shown, in this embodiment, the first sub-decoder 21 includes a third Basic module 211, and the remaining sub-decoders 21 each include an upsampling module 212, a fourth Basic module 213, and a multi-scale feature channel attention module 214 for adaptively extracting key feature information from each hip joint feature information. In the last sub-decoder 21: the upsampling module 212 is communicatively connected to the second Basic module 113 in the last sub-encoder 11 and the fourth Basic module 213 in this sub-decoder 21; the fourth Basic module 213 in this sub-decoder 21 is communicatively connected to the multi-scale feature channel attention module 214 in this sub-decoder 21; and the multi-scale feature channel attention module 214 in this sub-decoder 21 is communicatively connected to the multi-scale feature fusion module 3 and the upsampling module 212 in the previous sub-decoder 21.

[0072] Please continue reading Figure 1 As shown, in the first sub-decoder 21, the third Basic module 211 is communicatively connected to the multi-scale feature channel attention module 214 in the next sub-decoder 21 to obtain the final prediction result and output the key point information corresponding to the prediction result. The third Basic module 211 contains a first convolutional unit and a sigmoid activation unit arranged sequentially along the running direction.

[0073] Please continue reading Figure 1As shown, in the remaining sub-decoders 21: the upsampling module 212 is communicatively connected to the multi-scale feature channel attention module 214 in the next sub-decoder 21 and the fourth Basic module 213 in this sub-decoder 21. The fourth Basic module 213 in this sub-decoder 21 is communicatively connected to the multi-scale feature channel attention module 214 in this sub-decoder 21.

[0074] See Figure 2 , Figure 3 and Figure 4 As shown, in this embodiment, each multi-scale feature channel attention module 214 includes a first splicing unit 2141, a max pooling unit 2142, a first fully connected unit 2143, an average pooling unit 2144, a second fully connected unit 2145, a weight generation unit 2146, a dot product unit 2147, a second splicing unit 2148, and a Basic sub-module 2149. All first splicing units 2141 are communicatively connected to the multi-scale feature fusion module 3, and each first splicing unit 2141 is also communicatively connected to the fourth Basic module 213 in its respective sub-decoder 21.

[0075] Please continue reading Figure 3As shown, in each multi-scale feature channel attention module 214: the first splicing unit 2141, the max pooling unit 2142, the first fully connected unit 2143, and the weight generation unit 2146 are connected via communication to form one network branch; the first splicing unit 2141, the average pooling unit 2144, the second fully connected unit 2145, and the weight generation unit 2146 are connected via communication to form another network branch; the dot product unit 2147 is connected to the weight generation unit 2146; the second splicing unit 2148 is connected to the dot product unit 2147; and the Basic submodule 2149 is connected to the second splicing unit 2148 and the upsampling module 212 of the next sub-decoder 21 of the sub-decoder 21 where the Basic submodule 2149 is located. The first fully connected unit 2143 and the second fully connected unit 2145 are both fully connected operation layer execution units, and their operation parameters are the same. The first splicing unit 2141 receives the output of the fourth Basic module 213 connected to it, uses this as the channel splicing requirement information, and then calls the multi-scale feature fusion module 3 to output the corresponding fused feature information. This fused feature information is then spliced ​​with the channel splicing requirement information to obtain the first splicing result. The weight generation unit 2146 fuses the outputs of the first fully connected unit 2143 and the second fully connected unit 2145, and generates weights using sigmoid activation. The dot product unit 2147 performs a dot product operation between the weights generated by the weight generation unit 2146 and the fused feature information to obtain the dot product result. The second splicing unit 2148 splices the dot product result output by the dot product unit 2147 with the channel splicing requirement information to obtain the second splicing result. The Basic submodule 2149 performs convolution, normalization, and ReLU activation on the second splicing result to obtain a feature map.

[0076] Corresponding to the structure of encoder 1, see [link / reference]. Figure 2 , Figure 3 and Figure 4 As shown, in this embodiment, the multi-scale feature fusion module 3 is composed of four fifth Basic modules 31 connected in parallel. The first Basic module 111 and all the second Basic modules 113 except for the second Basic module 113 in the last sub-encoder 11 are each communicatively connected to one fifth Basic module 31. Each first stitching unit 2141 is communicatively connected to all the fifth Basic modules 31. All the fifth Basic modules 31 include a second convolution unit, a normalization unit, and a ReLU activation unit arranged sequentially along the running direction.

[0077] As a concrete example, all fifth Basic modules 31 consist of a second convolutional unit, a normalization unit, and a ReLU activation unit arranged sequentially along the running direction. With this structure, several more max-pooling units and downsampling units can be added. See [link to documentation]. Figure 3 and Figure 4 As shown in the figure, in the order from top to bottom, this specific example further adds a max pooling unit to the first and second fifth Basic modules 31, and an upsampling unit to the fourth fifth Basic module 31, so that each first splicing unit 2141 can call each fifth Basic module 31 to operate as needed and obtain the first splicing result.

[0078] The adaptive feature learning hip joint keypoint detection system disclosed in this embodiment introduces a multi-scale feature fusion module 3 into the UNet network. This module fuses features from encoder 1 containing information at different levels, generating fused features that include both superficial detailed features and deep global high-level semantic features of the hip joint. This effectively guides the decoding process of each subsequent layer. Furthermore, in the decoding stage corresponding to each sub-decoder 21 of decoder 2, a multi-scale feature channel attention module 214 is introduced. This effectively aggregates feature information at various scales while helping the network adaptively learn discriminative and important feature information, strengthening the network's selection of features at different scales. In addition, this hip joint keypoint detection system can fully learn the hip joint feature information contained in X-ray images and effectively detect the corresponding anatomical keypoints. Experimental results demonstrate the excellent detection performance of the proposed hip joint keypoint detection system, making it effectively applicable to clinical auxiliary diagnosis.

[0079] Next, this embodiment will disclose an adaptive feature learning method for hip joint key point detection corresponding to the hip joint key point detection system, see [link to relevant documentation]. Figure 5 As shown, the detection method includes the following steps:

[0080] S1: The hip joint image of the hip joint to be tested is encoded 5 times by encoder 1, and the hip joint feature information formed after each feature encoding is output to output 5 hip joint feature information with different scales.

[0081] See Figure 2 and Figure 5 As shown, this embodiment uses the X-ray image of the hip joint to be tested as input. The encoder 1 performs feature encoding on the X-ray image of the hip joint to be tested input to the hip joint key point detection system through a series of convolution and downsampling operations, and extracts the feature information of the hip joint at different scales.

[0082] S2: The multi-scale feature fusion module 3 receives four hip joint feature pieces from the five hip joint feature pieces with different scales obtained in step S1. These four hip joint feature pieces are the results output from the first four encoding steps. See [link to module 3]. Figure 2 , Figure 4 As shown.

[0083] S3; Based on the hip joint feature information obtained in step S1, decoder 2 decodes the feature information formed by the 5th feature encoding step by step. During each of the 1st to 4th decoding steps, decoder 2 sends out the channel splicing requirement information for this decoding step. The multi-scale feature fusion module 3 receives the channel splicing requirement information for this decoding step and adjusts the scale and number of channels of several hip joint feature information received in step S2 accordingly. Then, it splices the fused feature information for this decoding step and sends it back to decoder 2. Decoder 2 converts it into the feature map for this decoding step for the next decoding step. During the 5th decoding step, the key point prediction result of the hip joint to be tested is obtained based on the feature map obtained in the previous decoding step.

[0084] The following is a specific example illustrating this detection method: See Figure 2 As shown, the X-ray image of the hip joint to be tested is first received by encoder 1. The first Basic module 111 in the first sub-encoder 11 performs a convolution operation to unify the size of the input image to 512×512. Then, subsequent sub-encoders 11 extract feature information of the hip joint at different scales through a series of convolution and downsampling operations. During each downsampling process, max pooling is used in each downsampling module 112 to reduce the size of the feature map. Max pooling can accelerate the training speed of the network and improve its operating efficiency. The convolution operation consists of two 3×3 convolutional layers with a stride of 1, performed by the convolutional units in the second Basic module 113. At the output of each downsampling module 112 of encoder 1, the size of the feature map is reduced by half, while the number of channels is doubled. We define the resulting feature map as {e1,e2,e3,e4,e5}.

[0085] See Figure 2 , Figure 4As shown, in decoder 2, five sub-decoders 21 progressively employ convolution and interpolation upsampling operations from the end to the beginning to restore the feature map size to the original resolution of the network input image. The first sub-decoder 21 in decoder 2 performs the final keypoint prediction for the hip joint X-ray image at the end. Similar to encoder 1, decoder 2 has four upsampling modules 212, each doubling the feature map size while reducing the number of channels to half the input. We define the feature map output by decoder 2 as {d1, d2, d3, d4}. Notably, slightly different from the operations used in encoder 1, the decoding process in decoder 2 is designed to be progressive. Each sub-decoder 21 introduces a multi-scale feature channel attention module 214 as a comprehensive feature attention guidance module, combining the features processed by the multi-scale feature fusion module 3 with the high-level semantic features obtained from the previous sub-decoder 21. Each layer of decoder 2 outputs d... i It includes information such as the position and shape of the hip joint.

[0086] At the end of decoder 2, the third Basic module 211 uses a 1×1 convolution to convolve the output features of the last layer of decoder 2 and performs sigmoid activation to map the output features of decoder 2 to the desired key point detection result.

[0087] See Figure 2 and Figure 4 As shown, after a series of convolution and downsampling operations by encoder 1, feature maps of different scales are obtained. Furthermore, as the encoding deepens, the receptive field of the feature maps gradually increases. The features of the shallow sub-encoder 11 capture local detail discrimination features such as the texture of the hip joint, while the features of the deep sub-encoder 11 capture global high-level semantic discrimination features of the hip joint. Therefore, this embodiment uses the multi-scale feature fusion module 3 to achieve the fusion of features at different scales, thereby improving feature utilization.

[0088] First, the multi-scale feature fusion module 3 needs to unify the size of the feature maps of different scales generated by encoder 1. Considering that the image resolution of each level of features in decoder 2 is inconsistent, feature fusion at each level requires a different operation method. Therefore, the fused features are defined as {ff1,ff2,ff3,ff4} in descending order of size.

[0089] The following example illustrates the feature fusion method of the multi-scale feature fusion module 3, using the feature fusion process of the second decoding level after the deepest decoding feature d5 as an example. Figure 4The gray area in the upper left corner is shown. In this part, feature maps e1 and e2 generated by the encoder are downsampled to the same scale as d3 by the max pooling unit, and then the number of channels is unified to one-quarter of d3 by the convolutional unit of the fifth Basic module 31. Feature map e3, which is the same scale as d2, only needs to be processed by the convolutional module of the fifth Basic module 31 to adjust the number of channels, while feature map e4 is first processed by the convolutional unit of the fifth Basic module 31, and then upsampled by bilinear interpolation. By concatenating these feature maps, a feature map sequence ff3 containing feature information at different levels is obtained. Mathematically, the multi-scale feature fusion module generates the feature map sequence ff. i The process can be formalized as follows:

[0090]

[0091] Where k is the index of the feature map obtained by the encoder, Conv(·) is a 3×3 Basic convolutional module with BN and ReLU, U(·) represents bilinear interpolation upsampling, D(·) represents a max pooling layer, and [·] represents the concatenation operation. Finally, the fused features ff i Send it to the next module.

[0092] The fused feature map sequence ff i It contains feature information from each level to guide each step of the decoding process. However, ff... i The features contained in the code are poorly distinguishable and may contain redundant information. Furthermore, the importance of features at different levels varies for each decoding level. Therefore, an attention mechanism is introduced, and a multi-scale feature channel attention module 214 after feature fusion is proposed to guide the selection of features at each coding level.

[0093] Figure 4 The specific implementation process of the multi-scale feature channel attention module 214 is shown: First, the high-level semantic features d (output by the next sub-decoder 21) of the previous decoding level are processed by the upsampling module 212 of its sub-decoder 21. i+1 Upsampling is performed to obtain u i Then, the fused features are combined using the first splicing unit 2141. i with u iThe features are concatenated according to their channels. Based on this, the average pooling unit 2144 aggregates the spatial information of all features using average pooling, fully preserving feature information at each scale and improving the network's expressive power. Simultaneously, the max pooling unit 2142 extracts the primary information from the input features and suppresses secondary information, guiding the network to learn more discriminative feature information. Subsequently, the results of average pooling and max pooling are passed through a parameter-sharing fully connected layer by the first fully connected unit 2143 and the second fully connected unit 2145, and then summed. The weight generation unit 2146 uses a sigmoid activation function to map the channel descriptors of the feature map to a set of channel attention weights W. i Mathematically, the attention weights W of the generative channels can be... i The process can be formalized as follows:

[0094] W i =S(FC(AvgPool([ff i u i ]))+FC(MaxPool([ff i u i ]))) (2)

[0095] Where S represents the sigmoid activation function, [·] represents the channel concatenation operation, and FC(·) represents a fully connected layer.

[0096] Next, the generated attention weights W are processed by the dot product unit 2147. i With fusion features ff i After performing element-wise dot product, we obtain a sequence of feature maps f containing complete and discriminative feature information. i Mathematically, the sequence of generated feature maps f can be... i The process can be formalized as follows:

[0097] f i =W i ☉ff i (3)

[0098] Here, ⊙ represents element-wise dot product, which enables the MFCA module to adaptively learn important discriminative feature information.

[0099] At the end of the multi-scale feature channel attention module 3 operation, f is processed by the second splicing unit 2148. i and u i The data is concatenated and processed by the Basic submodule 2149 to obtain the decoding result d for this level. i Mathematically, each decoding level i can generate d. i The process can be formalized as follows:

[0100] d i =Conv([f i u i (4)

[0101] Where [·] represents the concatenation operation, and Conv(·) is a 3×3 convolution Basic convolution module with BN and ReLU.

[0102] Compared to the original encoder features, the guided encoder features contain richer hip joint feature information and are more discriminative, thus improving the network's detection performance and robustness.

[0103] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hip joint key point detection system with adaptive feature learning, characterized in that, The hip joint key point detection system includes: The encoder (1) performs n feature encodings on the hip joint image of the hip joint to be tested using convolution and downsampling operations, and outputs the hip joint feature information formed after each feature encoding, so as to output n hip joint feature information with different scales. The decoder (2) starts from the feature information formed by the nth feature encoding output by the encoder (1) and performs n decodings. In each decoding except the nth decoding, it sends out channel splicing requirement information and converts the fusion feature information corresponding to the channel splicing requirement information into a corresponding feature map for the next decoding. In the nth decoding, it obtains the key point prediction result of the hip joint to be tested based on the feature map obtained in the previous decoding. The multi-scale feature fusion module (3) receives the channel splicing requirement information sent by the decoder (2) respectively, adjusts the scale and number of channels of several hip joint feature information output by the encoder (1) according to the channel splicing requirement information, splices them into their respective corresponding fused feature information, and sends them back to the decoder (2). in, The encoder (1) is communicatively connected to the decoder (2), and the multi-scale feature fusion module (3) is communicatively connected to the encoder (1) and the decoder (2) respectively; The decoder (2) is composed of n sub-decoders (21) connected in series. The last sub-decoder (21) is connected in communication with the second Basic module (113) in the last sub-encoder (11). All sub-decoders (21) except the first sub-decoder (21) are connected in communication with the multi-scale feature fusion module (3). The first sub-decoder (21) includes a third Basic module (211), and the remaining sub-decoders (21) each include an upsampling module (212), a fourth Basic module (213), and a multi-scale feature channel attention module (214) for adaptively extracting key feature information from each hip joint feature information. Each of the multi-scale feature channel attention modules (214) includes a first splicing unit (2141), a max pooling unit (2142), a first fully connected unit (2143), an average pooling unit (2144), a second fully connected unit (2145), a weight generation unit (2146), a dot product unit (2147), a second splicing unit (2148), and a Basic submodule (2149).

2. The hip joint key point detection system based on adaptive feature learning according to claim 1, characterized in that, The encoder (1) is composed of n sub-encoders (11) connected in series, so that the hip joint image is successively feature encoded by the sub-encoders (11) in the order from beginning to end. All the sub-encoders (11) except the last sub-encoder (11) are connected to the multi-scale feature fusion module (3) and the last sub-encoder (11) is connected to the decoder (2).

3. The hip joint key point detection system based on adaptive feature learning according to claim 2, characterized in that, The first sub-encoder (11) includes a first Basic module (111), and the remaining sub-encoders (11) each include a downsampling module (112) and a second Basic module (113). The first Basic module (111) is communicatively connected to the multi-scale feature fusion module (3) and the downsampling module (112) in the next sub-encoder (11). In the sub-encoder (11) at the end: the second Basic module (113) is communicatively connected to the downsampling module (112) and the decoder (2) in the sub-encoder (11); In the remaining sub-encoders (11): the downsampling module (112) is communicatively connected to the second Basic module (113) in the sub-encoder (11), and the second Basic module (113) is communicatively connected to the multi-scale feature fusion module (3) and the downsampling module (112) in the next sub-encoder (11).

4. The hip joint key point detection system based on adaptive feature learning according to claim 3, characterized in that, The last sub-decoder (21) is used to upsample and convolve the feature information output by the last sub-encoder (11) to convert it into the corresponding channel stitching requirement information and send it to the multi-scale feature fusion module (3). The fusion feature information returned by the multi-scale feature fusion module (3) is converted into the corresponding feature map and sent to the previous sub-decoder (21). The first sub-decoder (21) is used to convolve and sigmoid activate the feature map output by the next sub-decoder (21) to obtain the key point prediction result of the hip joint image. The remaining sub-decoders (21) are used to upsample and convolve the feature map output by the next sub-decoder (21) to convert it into the corresponding channel stitching requirement information and send it to the multi-scale feature fusion module (3). The fusion feature information returned by the multi-scale feature fusion module (3) is converted into the corresponding feature map and sent to the previous sub-decoder (21). All the feature maps contain the configuration features of the hip joint.

5. The hip joint key point detection system based on adaptive feature learning according to claim 4, characterized in that, In the last sub-decoder (21): the upsampling module (212) is communicatively connected to the second Basic module (113) in the last sub-encoder (11) and the fourth Basic module (213) in this sub-decoder (21); the fourth Basic module (213) in this sub-decoder (21) is communicatively connected to the multi-scale feature channel attention module (214) in this sub-decoder (21); the multi-scale feature channel attention module (214) in this sub-decoder (21) is communicatively connected to the multi-scale feature fusion module (3) and the upsampling module (212) in the previous sub-decoder (21). In the first sub-decoder (21): the third Basic module (211) is communicatively connected to the multi-scale feature channel attention module (214) in the next sub-decoder (21); In the remaining sub-decoders (21): the upsampling module (212) is communicatively connected to the multi-scale feature channel attention module (214) in the next sub-decoder (21) and the fourth Basic module (213) in this sub-decoder (21). The fourth Basic module (213) in this sub-decoder (21) is communicatively connected to the multi-scale feature channel attention module (214) in this sub-decoder (21).

6. The hip joint key point detection system based on adaptive feature learning according to claim 5, characterized in that, The third Basic module (211) includes a first convolutional unit and a sigmoid activation unit arranged sequentially along the running direction.

7. The hip joint key point detection system based on adaptive feature learning according to claim 5 or 6, characterized in that, All of the first splicing units (2141) are communicatively connected to the multi-scale feature fusion module (3), and each of the first splicing units (2141) is also communicatively connected to the fourth Basic module (213) in its sub-decoder (21); In each of the multi-scale feature channel attention modules (214): The first splicing unit (2141), the max pooling unit (2142), the first fully connected unit (2143), and the weight generation unit (2146) are connected via communication to form a network branch. The first splicing unit (2141), the average pooling unit (2144), the second fully connected unit (2145), and the weight generation unit (2146) are connected via communication to form another network branch. The dot product unit (2147) is connected via communication with the weight generation unit (2146). The second splicing unit (2148) is connected via communication with the dot product unit (2147). The Basic submodule (2149) is connected via communication with the second splicing unit (2148) and the upsampling module (212) of the next sub-decoder (21) in which the Basic submodule (2149) is located. The first fully connected unit (2143) and the second fully connected unit (2145) have the same operational parameters; The first splicing unit (2141) is used to receive the output of the fourth Basic module (213) connected to it, and use it as the channel splicing requirement information. Then, it calls the multi-scale feature fusion module (3) to output the corresponding fusion feature information, and splices the fusion feature information with the channel splicing requirement information to obtain the first splicing result. The weight generation unit (2146) is used to fuse the outputs of the first fully connected unit (2143) and the second fully connected unit (2145), and generate weights by activating sigmoid. The dot product unit (2147) is used to perform a dot product operation between the weight generated by the weight generation unit (2146) and the fused feature information to obtain the dot product result; The second splicing unit (2148) is used to perform channel splicing with the dot product result output by the dot product unit (2147) and the channel splicing requirement information to obtain the second splicing result; The Basic submodule (2149) is used to perform convolution, normalization and ReLU activation on the second concatenation result to obtain the feature map.

8. The hip joint key point detection system based on adaptive feature learning according to claim 7, characterized in that, The multi-scale feature fusion module (3) is composed of (n-1) fifth Basic modules (31) connected in parallel. The first Basic module (111) and all the second Basic modules (113) except for the second Basic module (113) in the last sub-encoder (11) are respectively connected to one of the fifth Basic modules (31). Each first splicing unit (2141) is connected to all the fifth Basic modules (31).

9. The hip joint key point detection system based on adaptive feature learning according to claim 8, characterized in that, All of the fifth Basic modules (31) contain a second convolution unit, a normalization unit, and a ReLU activation unit arranged sequentially along the running direction.

10. A method for hip joint keypoint detection using adaptive feature learning, characterized in that, The hip joint key point detection system based on adaptive feature learning according to any one of claims 1-8 includes the following steps: S1: The hip joint image of the hip joint to be tested is encoded n times by encoder (1), and the hip joint feature information formed after each feature encoding is output to output n hip joint feature information with different scales. S2: Receive several hip joint feature information with different scales obtained from step S1 by the multi-scale feature fusion module (3); S3; Based on the hip joint feature information obtained in step S1, the decoder (2) sequentially decodes the feature information formed by the nth feature encoding. In each decoding except the nth decoding, the decoder (2) sends out the channel splicing requirement information for this decoding. The multi-scale feature fusion module (3) receives the channel splicing requirement information for this decoding and adjusts the scale and number of channels of the several hip joint feature information received in step S2 accordingly. Then, it splices the fused feature information for this decoding and sends it back to the decoder (2). The decoder (2) converts it into the feature map for this decoding for the next decoding. In the nth decoding, the key point prediction result of the hip joint to be tested is obtained based on the feature map obtained in the previous decoding.

Images