Knee osteoarthritis key part segmentation method and device based on medical nuclear magnetic imaging
By combining convolutional neural networks, diffusion networks and large language models, the automation accuracy of key structures and substructure segmentation in knee osteoarthritis is solved, achieving efficient diagnosis and time-saving effects.
Patent Information
- Application Number
- CN202411444778.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to automatically and accurately complete the segmentation of multiple key structures and substructures in knee osteoarthritis, resulting in low diagnostic efficiency and long diagnostic time.
The method based on convolutional neural network, diffusion network and pre-trained large language model is adopted to obtain medical nuclear magnetic images, extract higher-order features, and fuse text features to achieve accurate segmentation of knee bone joints.
It realizes accurate segmentation on images with high noise or structures with unclear edges, improves diagnostic efficiency, saves diagnosis time, and optimizes the automatic segmentation of key structures of knee osteoarthritis based on MRI.
Smart Images
Figure CN119600039A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to a method and device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging. Background Art
[0002] Knee osteoarthritis (KOA) is a common degenerative joint disease that primarily affects the cartilage of the knee joint, causing joint pain, stiffness, and limited mobility.
[0003] With the aging of the population and the increase in obesity rates, the incidence of knee osteoarthritis is also rising, which has a serious impact on the quality of life of patients and places a heavy burden on the medical system.
[0004] In KOA, quantitative analysis and measurement of multiple key structures based on MRI plays an important auxiliary role in doctors' diagnosis. There is an urgent need for a segmentation technology to automatically and accurately complete the segmentation of multiple main structures and substructures to obtain quantitative indicators of key structures. Summary of the invention
[0005] The purpose of this application is to provide a method and device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging to achieve accurate segmentation of key structures and substructures of the knee joint.
[0006] According to a first aspect of an embodiment of the present application, a method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging is provided, which is applied to a device for segmenting key parts of knee osteoarthritis. The device for segmenting key parts of knee osteoarthritis includes: a convolutional neural network, a diffusion network, and a pre-trained large language model. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging includes:
[0007] Obtain medical MRI images of the knee joint;
[0008] Input the medical MRI image into the convolutional neural network to extract the first high-order features of the medical MRI image;
[0009] Inputting 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or the 3D segmentation label of the medical nuclear magnetic resonance image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain the second high-order feature;
[0010] Input the position description and substructure relationship description text of the key structures of the knee joint into the pre-trained large language model to obtain a text feature vector;
[0011] The first high-order feature, the second high-order feature and the text feature vector are fused and aligned to obtain a fused feature;
[0012] The fused features are input into the diffusion network to obtain the segmentation label of the knee joint.
[0013] In some optional embodiments of the present application, the convolutional neural network includes multiple layers;
[0014] The medical MRI images are input into the convolutional neural network to extract the first high-order features of the medical MRI images, including:
[0015] Input medical MRI images into the convolutional neural network;
[0016] The spatial attention and channel attention of the convolutional neural network are used to strengthen the feature attention, and each layer is fused independently to obtain the first high-order features of the medical MRI images.
[0017] In some optional embodiments of the present application, the convolutional neural network includes a first layer, a second layer, a third layer, and a fourth layer;
[0018] The medical MRI images are input into the convolutional neural network to extract the first high-order features of the medical MRI images, including:
[0019] The 3D image of the medical MRI is input into the first layer, and the first feature map is obtained through convolution and pooling. The first feature map is subjected to spatial attention and channel attention to obtain the first spliced feature map with attention, and the channel splicing is performed with the feature map obtained by upsampling;
[0020] The first feature map is downsampled and convolved to obtain a feature map, which is then convolved and pooled to obtain a second feature map. The second feature map is subjected to spatial attention and channel attention to obtain a second concatenated feature map with attention, which is then channel-concatenated with the feature map obtained by upsampling.
[0021] The second feature map is downsampled and convolved to obtain the third feature map of the third layer. The third feature map is subjected to spatial attention and channel attention to obtain a third concatenated feature map with attention, which is then channel-concatenated with the feature map obtained by upsampling.
[0022] The third feature map is downsampled and convolved to obtain the fourth feature map of the fourth layer. The fourth feature map is upsampled to obtain the opposite features. The features of each layer in the opposite direction after channel splicing are convolved and pooled, and then upsampled to obtain the features to be spliced of the previous layer to obtain the first high-order features.
[0023] In some optional embodiments of the present application, the diffusion network includes a first layer and a second layer;
[0024] The 3D Gaussian noise of the same spatial size as the medical nuclear magnetic image or the 3D segmentation label of the medical nuclear magnetic image containing Gaussian noise of a preset ratio and the first high-order feature are input into the diffusion network to obtain the second high-order feature, including:
[0025] The 3D Gaussian noise of the same spatial size as the medical MRI image or the 3D segmentation label of the medical MRI image containing a certain proportion of Gaussian noise is input into the first layer, and the image is convolved and pooled to obtain the first encoding feature of the first layer;
[0026] The first encoding feature and the first high-order feature are compressed with spatial attention and channel attention, and input into the second layer to obtain the second encoding feature.
[0027] In some optional embodiments of the present application, the diffusion network further includes a third layer and a fourth layer;
[0028] The 3D Gaussian noise of the same spatial size as the medical nuclear magnetic image or the 3D segmentation label of the medical nuclear magnetic image containing a certain proportion of Gaussian noise and the first high-order feature are input into the diffusion network to obtain the second high-order feature, which also includes:
[0029] The second encoding feature is concatenated with the opposite same-layer feature using spatial attention and channel attention to obtain the second concatenated feature;
[0030] The second concatenated feature is input into the third layer, and the third coded feature is obtained through convolution and pooling. The third coded feature is then subjected to channel attention and spatial attention to obtain the opposite feature with attention and perform channel concatenation, the fourth concatenated feature;
[0031] The fourth concatenated feature is downsampled to obtain the fourth encoded feature, and the second high-order feature is finally obtained after convolution pooling.
[0032] In some optional embodiments of the present application, the text describing the position of the key structure of the knee joint and the substructure relationship is a priori knowledge text;
[0033] The position description of the key structures of the knee joint and the substructure relationship description text are input into the pre-trained large language model to obtain the text feature vector, which is specifically:
[0034] The position description of the key structure and the substructure relationship description text are used as the input of the large language model, and the text feature vector is obtained after encoding by the large language model.
[0035] In some optional embodiments of the present application, the first high-order feature, the second high-order feature and the text feature vector are fused and aligned to obtain a fused feature, including:
[0036] Transforming the first high-order feature and the second high-order feature into a first frequency domain feature and a second frequency domain feature;
[0037] Align the first frequency domain feature and the second frequency domain feature to obtain an aligned feature;
[0038] Transform the alignment features to obtain time domain features;
[0039] The time domain features are fused and aligned with the text feature vector to obtain the fused features.
[0040] According to a second aspect of an embodiment of the present application, a device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging is provided. The device for segmenting key parts of knee osteoarthritis comprises: a convolutional neural network, a diffusion network and a pre-trained large language model, including:
[0041] An acquisition module, used for acquiring medical magnetic resonance imaging of the knee joint;
[0042] A first input module is used to input the medical nuclear magnetic resonance image into the convolutional neural network to extract the first high-order features of the medical nuclear magnetic resonance image;
[0043] The second input module is used to input the 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or the 3D segmentation label of the medical nuclear magnetic resonance image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain the second high-order feature;
[0044] The third input module is used to input the position description of the key structure of the knee joint and the substructure relationship description text into the pre-trained large language model to obtain a text feature vector;
[0045] A fusion module is used to fuse and align the first high-order feature, the second high-order feature and the text feature vector to obtain a fused feature;
[0046] The fourth input module is used to input the fused features into the diffusion network to obtain the segmentation label of the knee joint.
[0047] According to a third aspect of an embodiment of the present application, an electronic device is provided, which may include:
[0048] processor;
[0049] a memory for storing processor-executable instructions;
[0050] The processor is configured to execute instructions to implement the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging as shown in any one of the embodiments of the first aspect.
[0051] According to a fourth aspect of an embodiment of the present application, a storage medium is provided. When instructions in the storage medium are executed by a processor of an information processing device or a server, the information processing device or the server implements a method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging as shown in any one of the embodiments of the first aspect.
[0052] The above technical solution of the present application has the following beneficial technical effects:
[0053] The method of the embodiment of the present application integrates a diffusion network, a convolutional neural network and a large language model, so that the network has the doctor's prior knowledge and can achieve accurate segmentation even on images with high noise or structures with unclear edges, which will save diagnosis time for patients and improve the doctor's diagnostic efficiency. In addition, this method specifically optimizes the automatic segmentation of key structures in the MRI-based KOA problem, which can assist clinicians in quickly and accurately making disease judgments, saving a lot of manpower and material costs for subsequent rapid disposal and treatment. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flowchart of a method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging in an exemplary embodiment of the present application;
[0055] Figure 2 is a diagram of a segmented network architecture in an exemplary embodiment of the present application;
[0056] Figure 3 This is a schematic diagram of the structure of a device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging in an exemplary embodiment of the present application.
[0057] Figure 4 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present application;
[0058] Figure 5 It is a schematic diagram of the hardware structure of an electronic device in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with specific implementations and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concepts of the present application.
[0060] The accompanying drawings show schematic diagrams of layer structures according to embodiments of the present application. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clarity. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.
[0061] Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0062] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.
[0063] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.
[0064] In the following, in conjunction with the accompanying drawings, the method and device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging provided by the embodiment of the present application are described in detail through specific embodiments and their application scenarios.
[0065] like Figure 1 As shown, in a first aspect of the embodiment of the present application, a method for segmenting key parts of knee osteoarthritis based on medical nuclear magnetic resonance images is provided, which is applied to a device for segmenting key parts of knee osteoarthritis. The device for segmenting key parts of knee osteoarthritis includes: a convolutional neural network, a diffusion network and a pre-trained large language model. The method for segmenting key parts of knee osteoarthritis based on medical nuclear magnetic resonance images includes:
[0066] S110: Obtain medical MRI images of the knee joint;
[0067] S120: inputting the medical nuclear magnetic resonance image into the convolutional neural network to extract the first high-order features of the medical nuclear magnetic resonance image;
[0068] S130: Inputting 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or the 3D segmentation label of the medical nuclear magnetic resonance image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain a second high-order feature;
[0069] S140: inputting the position description and substructure relationship description text of the key structure of the knee joint into the pre-trained large language model to obtain a text feature vector;
[0070] S150: fusing and aligning the first high-order feature, the second high-order feature, and the text feature vector to obtain a fused feature;
[0071] S160: Input the fused features into the diffusion network to obtain the segmentation label of the knee joint.
[0072] The method of this embodiment integrates the diffusion network, convolutional neural network and large language model, so that the network has the doctor's prior knowledge and can achieve accurate segmentation even on noisy images or structures with unclear edges, which will save diagnosis time for patients and improve the doctor's diagnosis efficiency. In addition, this method specifically optimizes the automatic segmentation of key structures in the MRI-based KOA problem, which can assist clinicians in quickly and accurately making disease judgments, saving a lot of manpower and material costs for subsequent rapid disposal and treatment.
[0073] In some embodiments, the convolutional neural network includes multiple layers;
[0074] The medical MRI images are input into the convolutional neural network to extract the first high-order features of the medical MRI images, including:
[0075] Input medical MRI images into the convolutional neural network;
[0076] The spatial attention and channel attention of the convolutional neural network are used to strengthen the feature attention, and each layer is fused independently to obtain the first high-order features of the medical MRI images.
[0077] In this embodiment, the convolutional neural network can be a convolutional neural network with a U-shaped architecture. In order to enable the network to learn the essential abstract features of MRI, the input end of this embodiment is set to MRI, and the output end is also a U-shaped network of the MRI. The fusion method of features between the same layers in the U-shaped network is improved, that is, spatial attention and channel attention are used to strengthen feature attention, and each layer is independently fused. By strengthening attention, more meaningful features can be extracted during the network learning process.
[0078] like Figure 2 As shown, in some embodiments, the convolutional neural network includes a first layer, a second layer, a third layer, and a fourth layer;
[0079] The medical MRI images are input into the convolutional neural network to extract the first high-order features of the medical MRI images, including:
[0080] The 3D image of the medical MRI is input into the first layer, and the first feature map is obtained through convolution and pooling. The first feature map is subjected to spatial attention and channel attention to obtain the first spliced feature map with attention, and the channel splicing is performed with the feature map obtained by upsampling;
[0081] The first feature map is downsampled and convolved to obtain a feature map, which is then convolved and pooled to obtain a second feature map. The second feature map is subjected to spatial attention and channel attention to obtain a second concatenated feature map with attention, which is then channel-concatenated with the feature map obtained by upsampling.
[0082] The second feature map is downsampled and convolved to obtain the third feature map of the third layer. The third feature map is subjected to spatial attention and channel attention to obtain a third concatenated feature map with attention, which is then channel-concatenated with the feature map obtained by upsampling.
[0083] The third feature map is downsampled and convolved to obtain the fourth feature map of the fourth layer. The fourth feature map is upsampled to obtain the opposite features. The features of each layer in the opposite direction after channel splicing are convolved and pooled, and then upsampled to obtain the features to be spliced of the previous layer to obtain the first high-order features.
[0084] After the MRI 3D image of this embodiment passes through the U-shaped architecture, high-order feature extraction of the original MRI is achieved.
[0085] In some embodiments, the diffusion network includes a first layer and a second layer;
[0086] The 3D Gaussian noise of the same spatial size as the medical nuclear magnetic image or the 3D segmentation label of the medical nuclear magnetic image containing Gaussian noise of a preset ratio and the first high-order feature are input into the diffusion network to obtain the second high-order feature, including:
[0087] The 3D Gaussian noise of the same spatial size as the medical MRI image or the 3D segmentation label of the medical MRI image containing a certain proportion of Gaussian noise is input into the first layer, and the image is convolved and pooled to obtain the first encoding feature of the first layer;
[0088] The first encoding feature and the first high-order feature are compressed with spatial attention and channel attention, and input into the second layer to obtain the second encoding feature.
[0089] The diffusion network of this embodiment is a diffusion network with a U-shaped architecture, and the first layer is a 3D Gaussian noise of the same spatial size as the original MRI or an MRI 3D segmentation label containing a certain proportion of Gaussian noise. The input image is convolved and pooled to obtain the encoding features of the first layer. First, the encoding features of the first layer are compressed with the first high-order features of the above embodiment for spatial attention and channel attention, and used as the input features of the second layer of the diffusion network. Secondly, the encoding features of the first layer use spatial attention and channel attention to obtain the splicing features of the same layer, which are used to perform channel splicing with the features obtained by upsampling the third layer of the diffusion network to perform feature fusion.
[0090] In some embodiments, the diffusion network further includes a third layer and a fourth layer;
[0091] The 3D Gaussian noise of the same spatial size as the medical nuclear magnetic image or the 3D segmentation label of the medical nuclear magnetic image containing a certain proportion of Gaussian noise and the first high-order feature are input into the diffusion network to obtain the second high-order feature, which also includes:
[0092] The second encoding feature is concatenated with the opposite same-layer feature using spatial attention and channel attention to obtain the second concatenated feature;
[0093] The second concatenated feature is input into the third layer, and the third coded feature is obtained through convolution and pooling. The third coded feature is then subjected to channel attention and spatial attention to obtain the opposite feature with attention and perform channel concatenation, the fourth concatenated feature;
[0094] The fourth concatenated feature is downsampled to obtain the fourth encoded feature, and the second high-order feature is finally obtained after convolution pooling.
[0095] In this embodiment, the input features of the second layer of the diffusion network are convolved and pooled to obtain corresponding features, which are spliced with the opposite same-layer features using spatial attention and channel attention, and downsampled to obtain the input features of the third layer. The input features of the third layer are convolved and pooled to obtain corresponding features, which are then passed through channel attention and spatial attention to obtain opposite features with attention, and channel splicing is performed. At the same time, the features are downsampled to obtain the fourth-layer features, which are finally obtained through convolution and pooling to obtain the second high-order features. In the diffusion network with a U-shaped architecture, spatial attention and channel attention of the same layer are introduced to strengthen the feature connection of the same layer, so that meaningful features can be extracted and strengthened.
[0096] In some embodiments, the text describing the position of the key structures of the knee joint and the substructure relationship is a priori knowledge text;
[0097] The position description of the key structures of the knee joint and the substructure relationship description text are input into the pre-trained large language model to obtain the text feature vector, which is specifically:
[0098] The position description of the key structure and the substructure relationship description text are used as the input of the large language model, and the text feature vector is obtained after encoding by the large language model.
[0099] This embodiment introduces the doctor's prior knowledge and aligns the high-order features of the large language model and the image model to achieve accurate segmentation under the guidance of prior knowledge. The doctor's prior knowledge, i.e., the position description of the key structure and the substructure relationship description text, is used as the input of the large language model, and the text feature vector is obtained after being encoded by the large language model.
[0100] In some embodiments, the first high-order feature, the second high-order feature, and the text feature vector are fused and aligned to obtain a fused feature, including:
[0101] Transforming the first high-order feature and the second high-order feature into a first frequency domain feature and a second frequency domain feature;
[0102] Align the first frequency domain feature and the second frequency domain feature to obtain an aligned feature;
[0103] Transform the alignment features to obtain time domain features;
[0104] The time domain features are fused and aligned with the text feature vector to obtain the fused features.
[0105] In order to strengthen the feature interaction between the diffusion network and the convolutional neural network and achieve alignment in this embodiment, a time-frequency-time feature processing method is designed in this embodiment. Specifically, the first high-order feature and the second high-order feature are first Fourier transformed to obtain the corresponding frequency domain features. Since the features of the diffusion network and the convolutional neural network come to a relatively consistent space in the frequency domain space, the features are fused and aligned in this space. The frequency domain features from the first high-order feature and the second high-order feature are then transformed in the frequency domain using a cross-attention mechanism.
[0106] After cross attention, a set of new features in the frequency domain will be obtained, which will represent the most meaningful features from the diffusion network and the convolutional neural network in the frequency domain. Finally, the frequency domain features obtained by cross attention are subjected to inverse Fourier transform to obtain features in the time domain. Thus, the time-frequency-time feature conversion process is completed. The time domain features obtained by inverse Fourier transform are then fused and aligned with the features from the large language model, that is, the fusion alignment of the two sets of features is achieved by using cross attention.
[0107] It should be noted that the features from the large language model here are digital vector features, which are multi-dimensional digital vectors with values between 0 and 1 formed by normalizing the text feature vector. The digital vector and the time domain features obtained by the inverse Fourier transform are cross-attentioned to strengthen the fusion and alignment of the features, and output a new feature vector after the cross-attention operation. The feature vector obtained after the cross-attention is finally used as the feature input of the third layer of the diffusion model. After the channel splicing of the features, the output features of the third layer are obtained by convolution and pooling. The features are upsampled to obtain the features to be spliced in the second layer. After the features are spliced, they are convoluted and pooled to obtain the output features of the first layer. After the features are spliced, the segmentation labels of the KOA29 structures of the same size as the input MRI are obtained by convolution and pooling, that is, the segmentation labels are distributed in the input matrix in the form of different voxel values. Then, the MRI and structural segmentation labels are combined to count the volume, thickness, average signal intensity and other parameters of different structures to achieve multi-structure key indicator measurement.
[0108] The entire network needs to train the convolutional neural network and the diffusion network. The large language model is a pre-trained model and only requires its encoder. The training is divided into two stages. The first stage is the training of the convolutional neural network, and the second stage is the training of the diffusion network.
[0109] Specifically, set the input MRI to be x, and the corresponding segmentation label to be y. First, in the first stage, input x at the input end of the convolutional neural network. Since only the first part of the network is trained, upsampling is performed directly after the first high-order feature is obtained. Finally, the second high-order feature is the output of the network, which is trained using the cross entropy loss function to obtain the pre-trained first part of the network model. The second stage is to jointly train the convolutional neural network and the diffusion network, that is, the network weights obtained from the first stage of training are used as the initialization weights. First, y is noised, specifically, the 3D Gaussian noise is fused with y in different proportions, thereby obtaining multiple segmentation label sets {y 1 ,y 2 ,y 3 ……,y n}. Each label needs to be trained.
[0110] For the sake of clarity, the segmentation label is y 1 For example, the input of the convolutional neural network is MRI, and the input of the diffusion model is y 1 , the output is y.
[0111] In the second stage of training, the segmentation labels with different proportions of noise are repeatedly used as the input of the diffusion model, x is used as the input of the convolutional neural network, and the doctor's prior knowledge, that is, the text structure information, is used as the input of the large language model. After the encoder of the pre-trained large prediction model, the text information containing prior knowledge will obtain the text feature vector.
[0112] By fusing and aligning the above-mentioned features of different modalities, the diffusion network can learn how to recover the true label from the noise. When training the model, the loss function is the noise prediction loss of the diffusion model (L D ) and segmentation loss (L d ), that is, the total loss L = αL D +(1-α)L d .
[0113] During model inference, the input of the diffusion network is 3D Gaussian noise, the input of the convolutional neural network is the MRI to be inferred, and the input of the large language model is the same key structure text information as in training, that is, the doctor's prior knowledge. The model output is the segmentation label of the MRI and the quantitative indicator measurement value of the key structure.
[0114] Among them, the spatial attention mechanism is expressed as, let the input feature map be F∈R C×H×W , where C represents the number of channels, H and W represent the height and width respectively. Spatial attention map M s ∈R H×W The calculation process is expressed as:
[0115] Ms =σ(Conv2D([AvgPool(F);MaxPool(F)]))
[0116] Among them, AvgPool(F) and MaxPool(F) represent global average pooling and global maximum pooling operations respectively, Conv2D represents a two-dimensional convolution operation, [] represents a concatenation operation, σ is a sigmoid function, and the final spatial attention feature map is, F s =M s ⊙F, represents element-by-element multiplication.
[0117] The channel attention mechanism is expressed as, let the input feature map be F∈R C×H×W , channel attention map M c ∈R C×1×1 The calculation process can be expressed as:
[0118] M c =σ(MLP(AvgPool(F)+MaxPoool(F)))
[0119] Among them, MLP represents a multi-layer perceptron, which consists of a shared fully connected layer. AvgPool(F) and MaxPool(F) represent global average pooling and global maximum pooling operations respectively, σ is the sigmoid function, and the final channel attention feature map is: F c =M c ⊙F.
[0120] The cross attention mechanism is expressed as, assuming that the feature map is represented as F∈R C×H×W , F 1 and F 2 They are respectively derived from the feature map, and the correlation between them is captured by calculating the cross attention of the two. The overall formula is:
[0121]
[0122] Among them, W Q Represents the feature map F 1 The weight matrix for linear transformation of Query, W K Represents the feature map F 2 The weight matrix for the linear transformation of Key, W V Represents the feature map F 2 The weight matrix for linear transformation of Value. All weight matrices are learned through network training. k is the dimension of Key, and softmax represents the softmax function.
[0123] The above embodiment aims at the KOA problem of MRI, and innovatively designs the segmentation of key parts of KOA based on the diffusion network, while realizing 3D fine segmentation including larger targets and smaller substructures, and measuring and quantitatively calculating key indicators in key structures; innovatively introduces Fourier and inverse Fourier transforms, and uses the cross-attention mechanism to align the doctor's prior text information encoded by the convolutional neural network, diffusion network and large language model, so as to improve the segmentation accuracy; innovatively uses the large language model as an encoder of the doctor's prior knowledge, and combines the encoded text feature vector with the image feature vector to assist the segmentation model in the fine and accurate segmentation of key parts and structures.
[0124] It should be noted that the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging provided in the embodiment of the present application may be executed by a device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging, or a control module in the device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging for executing the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging. In the embodiment of the present application, the device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging provided in the embodiment of the present application is taken as an example to illustrate the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging by a device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging.
[0125] like Figure 3 As shown, in a second aspect of the embodiment of the present application, a device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging is provided, and the device for segmenting key parts of knee osteoarthritis includes: a convolutional neural network, a diffusion network and a pre-trained large language model, including:
[0126] An acquisition module 310 is used to acquire a medical magnetic resonance imaging image of a knee joint;
[0127] A first input module 320, used to input the medical nuclear magnetic resonance image into the convolutional neural network to extract the first high-order features of the medical nuclear magnetic resonance image;
[0128] The second input module 330 is used to input the 3D Gaussian noise of the same spatial size as the medical nuclear magnetic image or the 3D segmentation label of the medical nuclear magnetic image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain the second high-order feature;
[0129] The third input module 340 is used to input the position description and substructure relationship description text of the key structure of the knee joint into the pre-trained large language model to obtain a text feature vector;
[0130] A fusion module 350 is used to fuse and align the first high-order feature, the second high-order feature and the text feature vector to obtain a fused feature;
[0131] The fourth input module 360 is used to input the fusion features into the diffusion network to obtain the segmentation label of the knee joint.
[0132] The device for segmenting the key parts of knee osteoarthritis based on medical magnetic resonance imaging in the embodiment of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0133] The device for segmenting the key parts of knee osteoarthritis based on medical magnetic resonance imaging in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0134] The key part segmentation device for knee osteoarthritis based on medical magnetic resonance imaging provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0135] Alternatively, if Figure 4 As shown, an embodiment of the present application further provides an electronic device 400, including a first processor 401, a first memory 402, and a program or instruction stored in the first memory 402 and executable on the first processor 401. When the program or instruction is executed by the first processor 401, each process of the above-mentioned embodiment of the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0136] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0137] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0138] The hardware structure 500 of the electronic device includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a second memory 509, and a second processor 510 and other components.
[0139] Those skilled in the art will appreciate that the hardware structure 500 of the electronic device may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the second processor 510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0140] It should be understood that in the embodiment of the present application, the input unit 504 may include a graphics processor (Graphics Processing Unit, GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts, a touch detection device and a touch controller. Other input devices 5072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The second memory 509 can be used to store software programs and various data, including but not limited to applications and operating systems. The second processor 510 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the second processor 510 .
[0141] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0142] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0143] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0144] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0145] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0147] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging, characterized in that: The invention is applied to a device for segmenting key parts of knee osteoarthritis, the device for segmenting key parts of knee osteoarthritis comprises: a convolutional neural network, a diffusion network and a pre-trained large language model, and the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging comprises: Obtain medical MRI images of the knee joint; Inputting the medical nuclear magnetic resonance image into the convolutional neural network to extract the first high-order features of the medical nuclear magnetic resonance image; Inputting 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or a 3D segmentation label of the medical nuclear magnetic resonance image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain a second high-order feature; Inputting the position description and substructure relationship description text of the key structures of the knee joint into the pre-trained large language model to obtain a text feature vector; Fusing and aligning the first high-order feature, the second high-order feature, and the text feature vector to obtain a fused feature; The fused features are input into the diffusion network to obtain the segmentation label of the knee joint.
2. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging according to claim 1, characterized in that: The convolutional neural network includes multiple layers; The step of inputting the medical nuclear magnetic resonance image into the convolutional neural network and extracting the first high-order feature of the medical nuclear magnetic resonance image comprises: The medical magnetic resonance imaging is input into the convolutional neural network; The spatial attention and the channel attention of the convolutional neural network are used to strengthen the feature attention, and each layer is independently fused to obtain the first high-order feature of the medical magnetic resonance imaging.
3. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging according to claim 1, characterized in that: The convolutional neural network includes a first layer, a second layer, a third layer and a fourth layer; The step of inputting the medical nuclear magnetic resonance image into the convolutional neural network and extracting the first high-order feature of the medical nuclear magnetic resonance image comprises: Input the 3D image of the medical MRI into the first layer, obtain a first feature map through convolution and pooling, obtain a first spliced feature map with attention after spatial attention and channel attention of the first feature map, and perform channel splicing with the feature map obtained by upsampling; The first feature map is downsampled and convolved to obtain a feature map, which is then convolved and pooled to obtain a second feature map. The second feature map is subjected to spatial attention and channel attention to obtain a second spliced feature map with attention, which is channel-spliced with the feature map obtained by upsampling; The second feature map is downsampled and convolved to obtain a third feature map of the third layer, the third feature map is subjected to spatial attention and channel attention to obtain a third spliced feature map with attention, and channel splicing is performed with the feature map obtained by upsampling; The third feature map is down-sampled and convolved to obtain the fourth feature map of the fourth layer, and the fourth feature map is up-sampled to obtain the opposite features. The features of each layer after channel splicing in the opposite direction are convolved and pooled, and then up-sampled to obtain the features to be spliced of the previous layer to obtain the first high-order features.
4. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging according to claim 1, characterized in that: The diffusion network includes a first layer and a second layer; The step of inputting 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or a 3D segmentation label of the medical nuclear magnetic resonance image containing Gaussian noise of a preset proportion and the first high-order feature into the diffusion network to obtain the second high-order feature includes: Inputting 3D Gaussian noise of the same spatial size as the medical MRI image or a 3D segmentation label of the medical MRI image containing a certain proportion of Gaussian noise into the first layer, and performing convolution and pooling on the image to obtain a first encoding feature of the first layer; The first encoding feature and the first high-order feature are subjected to feature compression of spatial attention and channel attention, and are input into the second layer to obtain a second encoding feature.
5. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging according to claim 4, characterized in that: The diffusion network also includes a third layer and a fourth layer; The step of inputting 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or a 3D segmentation label of the medical nuclear magnetic resonance image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain the second high-order feature also includes: The second encoding feature is concatenated with the opposite same-layer feature using spatial attention and channel attention to obtain the second concatenated feature; The second splicing feature is input into the third layer, and a third encoding feature is obtained through convolution and pooling. The third encoding feature is then subjected to channel attention and spatial attention to obtain a corresponding feature with attention and channel splicing, and a fourth splicing feature is obtained. The fourth concatenated feature is downsampled to obtain the fourth encoded feature, and the second high-order feature is finally obtained through convolution pooling.
6. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging according to claim 1, characterized in that: The text describing the position of the key structures of the knee joint and the substructure relationship is a priori knowledge text; The position description of the key structure of the knee joint and the substructure relationship description text are input into the pre-trained large language model to obtain a text feature vector, which is specifically: The position description of the key structure and the substructure relationship description text are used as the input of the large language model, and the text feature vector is obtained after encoding by the large language model.
7. The method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging according to claim 1, characterized in that: The fusing and aligning the first high-order feature, the second high-order feature, and the text feature vector to obtain a fused feature includes: Transforming the first high-order feature and the second high-order feature into a first frequency domain feature and a second frequency domain feature; Aligning the first frequency domain feature and the second frequency domain feature to obtain an aligned feature; Converting the alignment features to obtain time domain features; The time domain feature is fused and aligned with the text feature vector to obtain a fused feature.
8. A device for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging, characterized in that: The knee osteoarthritis key part segmentation device comprises: a convolutional neural network, a diffusion network and a pre-trained large language model, including: An acquisition module, used for acquiring medical magnetic resonance imaging of the knee joint; A first input module, used to input the medical nuclear magnetic resonance image into the convolutional neural network to extract the first high-order feature of the medical nuclear magnetic resonance image; A second input module is used to input 3D Gaussian noise of the same spatial size as the medical nuclear magnetic resonance image or a 3D segmentation label of the medical nuclear magnetic resonance image containing a certain proportion of Gaussian noise and the first high-order feature into the diffusion network to obtain a second high-order feature; A third input module is used to input the position description and substructure relationship description text of the key structure of the knee joint into the pre-trained large language model to obtain a text feature vector; A fusion module, used for fusing and aligning the first high-order feature, the second high-order feature and the text feature vector to obtain a fused feature; The fourth input module is used to input the fusion feature into the diffusion network to obtain the segmentation label of the knee joint.
9. An electronic device, characterized in that: include: A processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for segmenting key parts of knee osteoarthritis based on medical magnetic resonance imaging as described in any one of claims 1 to 7 are implemented.