Knee joint image segmentation method
By adding a multi-scale memory storage module at the bottleneck of the U-Net network, the problems of forgetting and insufficient low-level feature memory in knee joint image processing are solved, and the ability to recognize small organs and learn and remember heterogeneous data is improved.
Patent Information
- Application Number
- CN202511775570.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-01-13
AI Technical Summary
Current medical image processing for the knee joint suffers from amnesia and insufficient memory of low-level features, especially insufficient memory of features of small organs in knee joint images.
A multi-scale memory storage module is added to the bottleneck of the U-Net network. Through the multi-scale feature fusion unit and the memory storage and optimization unit, the multi-scale feature storage and optimization of the output of each layer of the network is realized, thereby improving the learning ability of small organs.
It improves the ability to identify small organs in knee joint images, reduces the problem of information forgetting caused by data differences, and enhances the network's ability to learn and remember heterogeneous data.
Smart Images
Figure CN121329995A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of medical image processing and artificial intelligence, and particularly relates to a knee joint image segmentation method. BACKGROUND
[0002] In knee joint medical image processing, neural networks with structures such as U-Net, Attention U-Net and Deeplab are mainly used; and models such as 3D U-Net three-dimensional argument V-Net and TransU-Net hybrid architecture introducing visual Transformer have also appeared.
[0003] Knee joint data has strong individual differences, and the existing technology has forgetting when applied to knee joint data learning of new data, and the current memory mechanism-based method only stores and optimizes high-level semantic information, and there is a memory deficiency for small organs such as meniscus and fine-grained ligament types in the knee joint image. SUMMARY
[0004] The application proposes a knee joint image segmentation method to solve the problems of forgetting and insufficient bottom feature memory in knee joint medical image segmentation technology, introduces a multi-scale memory storage module at the bottleneck of the U-Net network, stores and optimizes multi-scale features for each network layer, reduces the dependence of learning of different data on original data, and improves the learning ability for small organs.
[0005] To solve the above technical problems, the application adopts the following technical solutions:
[0006] A knee joint image segmentation method is proposed, comprising:
[0007] A knee joint image dataset is constructed;
[0008] A U-Net backbone network is constructed, and a multi-scale memory storage module is added to the bottleneck of each level; the multi-scale memory storage module of each level performs multi-scale fusion on the output of the corresponding level encoder, and stores and optimizes the fused features based on an update rule;
[0009] The improved U-Net segmentation network is trained;
[0010] The knee joint tissue segmentation result is obtained by using the trained U-Net segmentation network.
[0011] In some embodiments of the application, the multi-scale memory storage module comprises:
[0012] Multi-scale feature fusion unit: composed of two parallel convolutional layers, the output Fi of the corresponding level encoder is acted on by two convolutional layers to output Yi and Yj respectively, Yi and Yj are spliced and then a convolutional neural network with a convolution kernel size of 1x1 is used for feature fusion to obtain Z Fi ;
[0013] Memory storage and optimization unit: the output Z of the multi-scale fusion unit is input into the memory storage and optimization unit Fi According to the update rule, the feature is optimized and stored; wherein the memory bank stores the feature in the form of key-value pair, the key is the index value, and the value val is the feature S of the corresponding input feature under the update rule Fi ; The update rule is: for the memory storage and optimization unit corresponding to the output of the i-th level of the encoder, the cosine similarity between the key of the current feature and each stored key is calculated, the similarity score is normalized by Softmax, and the similarity matrix E between the key of the current feature and the stored key is aggregated i ; Based on E in x M n The similarity matrix E in is weighted and aggregated in the value feature M in the memory bank[i] n , and then all weighted val features are summed to obtain the final weighted feature S Fi ; n is the nth val; finally, the weighted feature S Fi is fused with the output Z Fi of the multi-scale fusion unit of the corresponding level to obtain X Fi =S Fi +Z Fi .
[0014] In some embodiments of the application, the construction of the memory storage and optimization unit includes:
[0015] Initialization: determine the queue capacity and feature dimension, configure an independent feature storage space for each feature, and use normal distribution to randomly generate an initial feature vector;
[0016] Configure the feature storage structure: configure a memory bank corresponding to each level for the features of each level, use a feature vector storage structure to store multiple feature vectors, and each feature vector represents a typical feature representation of the features at this level;
[0017] The configuration query encoder and the key encoder are designed as two-layer MLPs, the first layer maps the features to a first dimension, and after a ReLU layer, a second MLP is connected to map the features in the first dimension to a low-dimensional space, and the generated features are subjected to L2 regularization processing; the query encoder directly updates the parameters through standard back propagation and an optimizer; the key encoder has the same structure as the query encoder, but in the update design, the key encoder adopts a momentum update mechanism;
[0018] The configuration feature storage and update rule is: the scale fusion feature Z Fi The corresponding representations are obtained by projecting through the key encoder and the value encoder respectively.
[0019] Feature standardization.
[0020] In some embodiments of the application, the knee joint tissue segmentation result is obtained by using the trained U-Net segmentation network, including:
[0021] Input the knee MRI image;
[0022] Feature extraction: the encoder adopts a shrinkage path for down-sampling, and the input knee MRI image is subjected to feature extraction, and each level outputs a scale feature Fi;
[0023] Multi-scale fusion and memory storage optimization: the encoder output Fi of each level i is subjected to multi-scale processing and splicing by a multi-scale fusion unit to obtain a fusion feature Z Fi The memory storage and optimization unit stores and optimizes the output Z Fi of the multi-scale fusion unit according to the update rule to output a feature S Fi Finally, the output Z Fi of the multi-scale fusion unit and the output S Fi of the memory storage and optimization unit are fused to obtain X Fi；
[0024] Feature fusion: the decoder adopts an expansion path for up-sampling, and the output of each level is connected to the output of the multi-scale memory module of the corresponding layer through a jump connection to guide the final tissue segmentation data.
[0025] Compared with the prior art, the advantages and positive effects of the application are: the multi-scale memory storage module is added in the bottleneck of the U-Net network, the features output by each layer of the encoder are subjected to multi-scale feature fusion, and the fused features are subjected to feature storage, optimization and update based on the update rule, which improves the extraction capability of the network for the underlying features, realizes the learning and recognition capability of small organs, improves the learning and memory capability for heterogeneous data, and reduces the information forgetting problem caused by data differences.
[0026] Other features and advantages of the present application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS
[0027] Figure 1 The knee joint image segmentation method proposed by the present application performs steps;
[0028] Figure 2 The improved N-Net segmentation network structure in the embodiment of the present application;
[0029] Figure 3 The structure of the multi-scale memory storage module in the embodiment of the present application;
[0030] Figure 4 The effect comparison of the diversified features processed by the method of the present application and the diversified features processed by the traditional U-Net segmentation network;
[0031] Figure 5 The MAP index comparison of the method of the present application and the traditional U-Net segmentation network. DETAILED DESCRIPTION
[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0033] In the description of the present application, it should be understood that the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.
[0034] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0035] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.
[0036] The knee joint image segmentation method proposed in this invention modifies the existing U-Net network model by adding a multi-scale memory storage module to the bottleneck at each feature scale. This enhances the ability to recognize small objects and remember diverse data. The specific implementation is as follows: Figure 1 The following are included:
[0037] S1: Construct a knee joint image dataset.
[0038] We built a thin-section MRI dataset of 14 healthy men and women aged 25-40, with a total of 6160 image slices. The images were annotated by professional clinicians, and the annotations included nine categories: tibia, femur, fibula, anterior and posterior cruciate ligaments, medial and lateral menisci, and medial and lateral collateral ligaments.
[0039] The dataset consists of original images divided into validation, test, and training sets, with a ratio of 1:1:4. All images are interpolated to 512x512 pixels. Mask images, also 512x512 pixels in size, are generated from the annotations in the training, validation, and test images. A dataset generation script is used to transform the annotations to fit the U-Net semantic segmentation model's annotation format.
[0040] S2: Construct the U-Net backbone network and add a multi-scale memory storage module to the Bottleneck at each level; the multi-scale memory storage module at each level performs multi-scale fusion of the output of the corresponding level encoder and performs memory storage and optimization of the fused features based on the update rules.
[0041] The embodiments of this invention adopt a general U-Net structure design, such as... Figure 2 As shown, it mainly includes an encoder, a decoder, and a bottleneck module that links the encoder and decoder.
[0042] The encoder uses the shrinking path of U-Net for downsampling. It extracts features from the input knee joint image and outputs scale features Fi at each level, where i represents the i-th level.
[0043] The scale feature Fi output from each level is input into the multi-scale memory module, which stores the scale feature of the corresponding level through an update mechanism.
[0044] The decoder uses the extended path of U-Net for upsampling, and the outputs of each layer and the corresponding multi-scale memory storage module are fused through jumpers.
[0045] like Figure 3 As shown, the multi-scale memory module consists of a multi-scale feature fusion unit and a memory storage and optimization unit. The multi-scale feature fusion unit processes and stitches the encoder output Fi of the corresponding layer at multiple scales to obtain fused features. The memory storage and optimization unit updates and optimizes the fused features output by the multi-scale feature fusion unit and outputs optimized features. Finally, the optimized features are fused with the previous fused features, and the fused features are then fused with the features of the decoder of the corresponding layer through a jump connection.
[0046] The multi-scale feature fusion unit consists of two parallel convolutional layers with kernel sizes of 1×1 and 5×5, respectively. The encoder output Fi (feature dimension W) of the corresponding layer... Fi W Fi C Fi After passing through two convolutional layers, the outputs Yi and Yj are obtained, with feature dimensions (W) respectively. Fi W Fi C Yi ) and (W Fi W Fi C Yj The two features Yi and Yj are concatenated along dimension C to obtain the concatenated feature (W). Fi W Fi C Yi +C Yj Then, a convolutional neural network with a kernel size of 1×1 is used to fuse features to obtain Z. Fi Dimension (W) Fi W Fi C Fi ).
[0047] The memory storage and optimization unit will store the output Z of the multi-scale fusion unit. Fi Feature optimization and storage are performed according to the update rules; the memory stores features in key-value pairs, where the key is the index and the value is the corresponding input feature S under the update rules. Fi S Fi The size of the feature Z at the corresponding scale Fi The sizes are consistent, and the size of the memory bank is n, where n is a variable, meaning that n memory features can be stored according to the update rules.
[0048] The update rule for the memory storage and optimization unit is as follows: For the memory storage and optimization unit memory_banks[i] corresponding to the output of the i-th level of the encoder, calculate the cosine similarity between the key of the current feature and each of the stored keys, normalize the similarity scores using Softmax, and aggregate to obtain the similarity matrix E between the key of the current feature and the stored keys. i Based on E in ×M n The similarity matrix E in The value features M in the weighted aggregation of memory_bank[i] n Then, summing all the weighted val features yields the final weighted feature S. Fi n is the nth value. Finally, the weighted feature S Fi The output Z of the corresponding level multi-scale fusion unit Fi X is obtained by fusion Fi =S Fi +Z Fi Dimension (W) Fi ×W Fi C Fi The feature transformation dimension is (W) Fi W Fi C Fi ), input the corresponding decoder layer.
[0049] The construction of this memory storage and optimization unit includes:
[0050] (1) Initialization: Determine the queue capacity and feature dimension (e.g., 128 dimensions), and configure an independent feature storage space for each feature. Use a normal distribution to randomly generate the initial feature vector.
[0051] (2) Configure feature storage structure: Each level of features is configured with a corresponding memory bank. The feature vector storage structure is adopted, which can store multiple feature vectors. Each feature vector represents a typical feature representation under the feature of that level.
[0052] (3) Configure the query encoder and key encoder: The prototype feature map is a three-dimensional feature map of H×W×C, and a memory bank is designed to store the feature vectors of the corresponding levels. The query encoder is designed as a two-layer MLP (Multilayer Perceptron). The first layer maps the features to 2048 dimensions, and after a ReLU layer, another MLP layer maps the 2048-dimensional features to a lower-dimensional space of 128 dimensions, and performs L2 regularization on the generated features. The query encoder updates parameters directly through standard backpropagation and optimizers (such as SGD, Adam). The key encoder has the same structural design as the query encoder, but in its update design, the key encoder adopts a momentum update mechanism, with the following formula:
[0053]
[0054] in, This is the momentum coefficient, set to 0.999. and These are the parameters for the query encoder and the key encoder, respectively.
[0055] (4) Configure feature storage and update rules: Scale fusion feature Z Fi Convert to (W) Fi ×W Fi C Fi Then, these are projected through a key encoder and a value encoder respectively to obtain the corresponding representations. For example, key(W) Fi ×W Fi ,64),val(W Fi ×W Fi C Fi The first level of memory is denoted as memory_banks[0], which stores 64 key vectors. The dimension of the key is [64, 64], and the dimension of the value is (W). Fi ×W Fi C Fi The second-level memory bank is denoted as memory_banks[1], which stores 64 key vectors. The dimension of the key is [64, 64], and the dimension of the value is (W). Fi ×W Fi C Fi ( ); and so on. The update rules are as described above and will not be repeated here.
[0056] (5) Feature standardization: In order to enable the effective comparison and updating of features of different samples and different scales, a feature standardization method is designed. Before inputting the features into the memory, the features are normalized. L2 normalization is used to eliminate the differences in numerical dimensions between features and normalize them to between 0 and 1.
[0057] S3: Training the improved U-Net segmentation network.
[0058] 1. Determine the loss function: The loss function includes the mask loss function. and memory loss function The mask loss function consists of Dice Loss and cross-entropy loss, and the final loss function is:
[0059] Ltotal = L Dice + L Cross-Entropy + L contrast
[0060] in, =1, =1, =0.5.
[0061] 2. Training process: The training set, the corresponding label file, and the pre-trained model on the ImageNet network are input into the improved U-Net segmentation network for training. During the training process, the learning rate of the improved U-Net segmentation network is set to 0.0001, the optimizer is set to the Adam estimation optimizer, the iteration is M times, and the number of images used in each iteration is B, where M and B are finite positive integers. The training continues until the loss function of the network converges.
[0062] 3. Weight saving: Weights are saved once every F iterations. If the optimal performance weight is obtained and the optimal performance is not obtained again after more than 50 iterations, the process is terminated.
[0063] 4. Validation: Input the validation set and the corresponding label file into the instance segmentation network for validation. The network performs validation once every F iterations.
[0064] S4: The knee joint tissue segmentation results are obtained using the trained U-Net segmentation network.
[0065] (1) Input knee joint MRI image.
[0066] (2) Feature extraction: The encoder uses a shrinking path for downsampling to extract features from the input knee MRI image, and outputs scale features Fi at each level.
[0067] (3 Multi-scale fusion and memory storage optimization: The encoder output Fi of each level i is processed and stitched together by the multi-scale fusion unit to obtain the fused feature Z) Fi The memory storage and optimization unit will store the output Z of the multi-scale fusion unit. Fi Based on the update rules, optimize and store the features, and output the feature S. Fi Finally, the output Z of the multi-scale fusion unit is... Fi With the output S of the memory storage and optimization unit Fi Fusion yields X Fi .
[0068] (4) Feature fusion: The decoder uses an extended path for upsampling, and the output of each level is fused with the output of the corresponding multi-scale memory module through a jump connection to guide the final organization and segmentation of data.
[0069] like Figure 4The improved U-Net segmentation network proposed in this invention has significant advantages in handling various alienation features: the first column in the figure is the original image, the second column shows the detection effect for each alienation region C1, where the U-Net segmentation network misses detections, while the improved U-Net segmentation network can detect them; the third column shows the detection effect for each alienation region C6, where the U-Net segmentation network misses detections, while the improved U-Net segmentation network can detect them.
[0070] like Figure 5 The figures show the MAP (Maximum Aspect Ratio) of the U-Net segmentation network and the improved U-Net segmentation network of this invention, demonstrating the superior performance of the improved U-Net segmentation network from a quantitative perspective.
[0071] It should be noted that the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.
Claims
1. A method for segmenting knee joint images, characterized in that, include: Construct a dataset of knee joint images; A U-Net backbone network is constructed, and a multi-scale memory storage module is added to the Bottleneck of each layer. The multi-scale memory storage module of each layer performs multi-scale fusion of the output of the corresponding layer encoder, and performs memory storage and optimization of the fused features based on the update rules. Train the improved U-Net segmentation network; The trained U-Net segmentation network was used to obtain the knee joint tissue segmentation results.
2. The knee joint image segmentation method according to claim 1, characterized in that, The multi-scale memory storage module includes: Multi-scale feature fusion unit: Consists of two parallel convolutional layers. The encoder output Fi of the corresponding layer is processed by the two convolutional layers to output Yi and Yj respectively. After concatenating Yi and Yj, a convolutional neural network with a kernel size of 1×1 is used to fuse features to obtain Z. Fi ; Memory storage and optimization unit: This unit stores and optimizes the output Z of the multi-scale fusion unit. Fi Feature optimization and storage are performed according to the update rules; the memory stores features in key-value pairs, where the key is the index value and the value is the corresponding input feature S under the update rules. Fi The update rule is as follows: For the memory storage and optimization unit corresponding to the output of the i-th level of the encoder, calculate the cosine similarity between the key of the current feature and each of the stored keys, normalize the similarity scores using Softmax, and aggregate to obtain the similarity matrix E between the key of the current feature and the stored keys. i Based on E in ×M n The similarity matrix E in The value features M in the weighted aggregation of memory_bank[i] n Then, summing all the weighted val features yields the final weighted feature S. Fi ; n is the nth val; finally, the weighted feature S Fi The output Z of the corresponding level multi-scale fusion unit Fi X is obtained by fusion Fi =S Fi +Z Fi .
3. The knee joint image segmentation method according to claim 2, characterized in that, The construction of the memory storage and optimization unit includes: Initialization: Determine the queue capacity and feature dimensions, configure an independent feature storage space for each feature, and randomly generate initial feature vectors using a normal distribution; Configure feature storage structure: Each level of features is configured with a corresponding memory bank, and a feature vector storage structure is used to store multiple feature vectors. Each feature vector represents a typical feature representation under that level of features; Configure the query encoder and key encoder: The query encoder is designed as a two-layer MLP. The first layer maps the features to the first dimension. After passing through a ReLU layer, another MLP layer is connected to map the features of the first dimension to a low-dimensional space and perform L2 regularization on the generated features. The query encoder updates the parameters directly through standard backpropagation and the optimizer. The key encoder has the same structural design as the query encoder, but in terms of update design, the key encoder adopts a momentum update mechanism. Configure feature storage and update rules: Scale-fused feature Z Fi The corresponding representations are obtained by projection through both the key encoder and the value encoder; Feature standardization.
4. The knee joint image segmentation method according to claim 1, characterized in that, The trained U-Net segmentation network was used to obtain the knee joint tissue segmentation results, including: Input knee MRI image; Feature extraction: The encoder uses a shrinking path for downsampling to extract features from the input knee MRI image, and outputs scale features Fi at each level; Multi-scale fusion and memory storage optimization: The encoder output Fi of each level i is processed and stitched together by a multi-scale fusion unit to obtain the fused feature Z. Fi The memory storage and optimization unit will store the output Z of the multi-scale fusion unit. Fi Based on the update rules, optimize and store the features, and output the feature S. Fi Finally, the output Z of the multi-scale fusion unit is... Fi With the output S of the memory storage and optimization unit Fi Fusion yields X Fi； Feature fusion: The decoder uses an extended path for upsampling, and the outputs of each level and the corresponding multi-scale memory module are fused through jump connections to guide the final organization and segmentation of data.