Generation and discrimination network fused knee joint skeleton segmentation method and device
By fusing generation and discrimination networks, 3D sequences of knee joint MR and CT are preprocessed and multimodal fused. The segmentation results are then optimized using the trained segmentation generation and discrimination networks, which solves the problem of fuzzy boundaries in knee joint bone segmentation and achieves efficient and accurate bone segmentation.
Patent Information
- Application Number
- CN202411408991.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-10
AI Technical Summary
Existing techniques are not ideal for segmenting knee bones, especially for segments with blurred boundaries such as cartilage, which affects the success rate of surgery.
The method of fusing generation and discrimination networks is adopted. By acquiring 3D sequences of knee joint MR and CT, preprocessing and multimodal fusion are performed. The segmentation results are then optimized using trained segmentation generation and discrimination networks, including data augmentation and post-processing operations.
It improves the accuracy and stability of knee joint bone segmentation, reduces manual intervention, increases work efficiency and accuracy, and ensures data integrity and consistency.
Smart Images
Figure CN119295481A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and device for segmenting knee joint bones by integrating a generation network and a discrimination network. Background Art
[0002] With the maturity of artificial intelligence technology, the diagnosis and treatment of knee joints are also developing towards intelligence. For example, knee rehabilitation, knee replacement and knee reconstruction achieved with the help of robots have liberated doctors' hands to varying degrees. The segmentation of bones in the knee joint is the basis of intelligent knee surgery technology, and the quality of the segmentation results directly affects the success of the entire operation. Therefore, it is important to achieve accurate and efficient segmentation of bones in the knee joint.
[0003] However, currently, the segmentation of the bones in the knee joint is mostly performed by using some traditional segmentation methods in the medical field, such as threshold segmentation, gradient segmentation, etc. These methods do not have ideal segmentation effects on fuzzy boundaries (such as cartilage). Summary of the invention
[0004] The purpose of the present invention is to provide a knee joint bone segmentation method and device that integrates generation and discrimination networks, extracts rich features of each bone through multimodal combination, and improves the segmentation accuracy through a segmentation discrimination network.
[0005] The present invention provides a knee joint bone segmentation method integrating a generation network and a discrimination network, comprising:
[0006] Acquire 3D MR and CT sequences of each bone in the knee joint;
[0007] Preprocessing the 3D images of MR and 3D images of CT respectively;
[0008] The preprocessed MR 3D image map and CT 3D image map are sliced respectively, and the 3D image map slices corresponding to MR and CT are multimodally fused in the channel dimension to obtain a fused multimodal image map;
[0009] The fused multimodal images are input into the trained segmentation generation network to obtain the segmentation results of each bone in the knee joint corresponding to each image;
[0010] The trained segmentation generation network output is post-processed to reconstruct the segmentation results of each bone in the knee joint on each image into the original image size. Post-processing includes removing isolated points, edge smoothing and resampling operations on the segmentation results.
[0011] Preferably, the pre-training step of the trained segmentation generation network includes:
[0012] Obtain 3D images of MR and 3D images of CT, and label the corresponding 3D images;
[0013] Resample the 3D image map of MR, the 3D image map of CT and the corresponding label map to obtain the image map and the target label map with the same voxel size;
[0014] Fusing 3D images of MR and 3D images of CT;
[0015] Performing data enhancement on the fused 3D image, wherein the data enhancement includes scaling, rotating, and changing the brightness, contrast, and saturation of the image;
[0016] The data-enhanced multimodal image is input into the segmentation generation network to obtain the segmentation results of each bone in the knee joint;
[0017] The segmentation results of each bone in the knee joint are input into the segmentation discrimination network to realize the training and verification of the network, and the trained segmentation generation network is obtained through parameter update optimization.
[0018] Preferably, the step of inputting the segmentation results of each bone in the knee joint into a segmentation discrimination network to implement network training and verification, and obtaining a trained segmentation generation network through parameter update optimization further comprises:
[0019] The segmentation results of each bone in the knee joint and the corresponding target annotation map are input into the segmentation discriminant network to obtain the probability that the two match the true segmentation results;
[0020] Calculate the loss value between the segmentation result obtained by the segmentation generation network and the annotated segmentation result and the sum of the two matching loss values of the segmentation discriminant network respectively;
[0021] According to the loss obtained by alternating back-propagation generation network and the loss obtained by segmentation discrimination network, the parameter values of the corresponding neural network are updated alternately to obtain the pre-trained segmentation generation network weights.
[0022] Preferably, the step of inputting the segmentation results of each bone in the knee joint into the segmentation discrimination network to implement the training and verification of the network further comprises:
[0023] Performing an abnormal result filtering operation on the segmentation results of each bone in the knee joint, wherein the filtering operation includes filtering out the segmentation results that do not meet the requirements according to the prior knowledge related to the knee joint, wherein the prior knowledge related to the knee joint refers to that the distances between the femur, patella, tibia and cartilage are relatively concentrated, and the cartilage is flat;
[0024] The filtered segmentation results are reorganized and resampled. The reorganization operation includes reorganizing the corresponding segmentation results input from the same 3D image with overlap to obtain the bone segmentation result of the complete 3D image, and reorganizing them with preset overlapping pixels on the x, y and z axes respectively; the resampling operation includes resampling the reorganized segmentation results according to the original spacing to obtain the final segmentation results of each bone in the knee joint.
[0025] Preferably, the trained segmentation generation network is a neural network based on 3D Unet, comprising a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder and a fifth decoder connected in sequence, wherein each encoder and decoder comprises 2 convolutional layers, batch normalization and activation function are performed after each convolutional layer, the number of filters in the corresponding encoders are 32, 64, 128, 256 and 512 respectively, the number of filters in the corresponding decoders are 512, 256, 128, 64 and 32 respectively, the network input is a multimodal 3D image of 2*96*384*384, and the network outputs a probability map of bone segmentation.
[0026] Preferably, the segmentation discrimination network includes 4 convolutional layers, 1 tiling layer and 1 output layer, each convolutional layer is connected to a pooling layer, the first three pooling layers adopt maximum pooling operation, and the last pooling layer adopts average pooling operation, wherein the number of filters corresponding to the 4 convolutional layers are 64, 128, 256 and 512 respectively.
[0027] Preferably, the pretreatment comprises:
[0028] De-noising and resampling of MR 3D images: obtaining the median of all CT image spacings, resampling all CT images to the median spacing, and performing Gaussian filtering on the CT images to remove noise during image data acquisition;
[0029] De-noising and resampling of CT 3D images: obtaining the median of all MR image spacings, resampling all MR images to the median spacing, and performing Gaussian filtering on the MR images to remove noise during image data acquisition;
[0030] The 3D images of MR and CT are cropped and standardized: standardization operations of subtracting the mean and dividing the variance are performed respectively, and overlapping crop operations are performed on the 3D images of MR and CT to obtain several blocks of 96*384*384 size, and crop operations are performed on the x-axis, y-axis and z-axis with preset step sizes.
[0031] The present invention also provides a knee joint bone segmentation device integrating a generation network and a discriminant network, and implements the knee joint bone segmentation method integrating a generation network and a discriminant network as described in the present invention, comprising:
[0032] The acquisition module is used to obtain 3D sequences of MR and CT of each bone in the knee joint;
[0033] A preprocessing module, used for preprocessing the 3D image map of MR and the 3D image map of CT respectively;
[0034] A fusion module is used to slice the preprocessed MR 3D image map and the CT 3D image map, respectively, and perform multimodal fusion of the MR and CT corresponding 3D image map slices in the channel dimension to obtain a fused multimodal image map;
[0035] The segmentation module is used to input the fused multimodal image into the trained segmentation generation network to obtain the segmentation results of each bone in the knee joint corresponding to each image;
[0036] The post-processing module is used to post-process the output results of the trained segmentation generation network, and reorganize the segmentation results of each bone in the knee joint corresponding to each image into the original image size.
[0037] The present invention also provides an electronic device, comprising:
[0038] A memory, the memory being used to store a processing program;
[0039] A processor, when executing the processing program, implements the knee joint bone segmentation method of the fusion generation and discrimination network as described in the present invention.
[0040] The present invention also provides a readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, the knee joint bone segmentation method of the fusion generation and discrimination network as described in the present invention is implemented.
[0041] With respect to the prior art, the present invention has the following beneficial effects:
[0042] The present invention obtains 3D sequences of MR and CT images of each bone in the knee joint and pre-processes the corresponding 3D image map, thereby removing noise and unifying the voxel size, thereby improving the accuracy of the segmentation result. At the same time, the fused multi-modal image map is input into a trained segmentation generation network to obtain a more accurate segmentation result. The segmentation result is post-processed, and the segmentation results of each bone in the knee joint corresponding to each image map are reorganized into the original image map size, which helps to maintain the integrity and consistency of the data.
[0043] The present invention can increase the generalization ability of the model in different scenarios through data enhancement operations, such as scaling, rotating, and changing the brightness, contrast, and saturation of the image. In addition, the results of the segmentation of each bone in the knee joint are input into the segmentation discriminant network to realize the training and verification of the network, and the trained segmentation generation network is obtained through parameter update optimization, which helps to improve the stability and robustness of the model;
[0044] The present invention slices the 3D images of MR and CT, and performs multimodal fusion on the corresponding 3D image slices in the channel dimension to obtain a fused multimodal image, which helps to simplify the data processing process and improve efficiency. At the same time, the results of the trained segmentation generation network output are post-processed, and the segmentation results of each bone in the knee joint corresponding to each image are reorganized into the original image size, which helps to maintain the integrity and consistency of the data.
[0045] In summary, the solution of the present invention can realize the automatic processing of medical images, reducing the need for manual intervention. This is particularly important for the processing of large-scale medical image data, and can significantly improve work efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A flowchart of the knee joint bone segmentation method of the present invention that integrates generation and discrimination networks;
[0047] Figure 2 The segmentation generation network diagram based on 3D Unet provided by the present invention;
[0048] Figure 3 The segmentation and discrimination network diagram provided by the present invention;
[0049] Figure 4 This is a training flow chart of the segmentation generation network and the segmentation discrimination network provided by the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] Embodiment 1
[0052] The present application can be used for many general or special devices, and there are no specific restrictions on the device environment configuration, etc. It can be used for personal computers, servers, handheld portable devices, tablet devices, and mobile devices, etc.
[0053] like Figure 1 As shown, the present invention provides a knee joint bone segmentation method integrating a generation network and a discrimination network, comprising:
[0054] S1: Acquire 3D sequences of MR and CT images of each bone in the knee joint. The 3D sequence is a three-dimensional data set composed of a series of CT or MRI scan images.
[0055] S2: Preprocessing the 3D image of MR and the 3D image of CT respectively, wherein the preprocessing includes: performing denoising and resampling operations on the 3D image of MR: obtaining the median of the spacing of all CT images, and resampling all CT images to the median spacing, and performing Gaussian filtering on the CT images to remove noise in the image data acquisition process;
[0056] Denoise and resample CT 3D images: Get the median of all MR image spacings, resample all MR images to the median spacing, and perform Gaussian filtering on the MR images to remove noise during image data acquisition; spacing refers to the distance between two adjacent voxels in an image. Ensuring consistent spacing is a key step in medical image preprocessing, which is essential for multimodal image fusion, accurate image segmentation, and 3D reconstruction. Adjusting the spacing of images by resampling or interpolation methods can ensure that different images are aligned in space, thereby providing more accurate diagnostic information.
[0057] The 3D images of MR and CT are cropped and standardized: the standardization operation of subtracting the mean and dividing the variance is performed respectively, and the 3D images of MR and CT are cropped with overlap to obtain several blocks of 96*384*384. In the training stage, the 3D MR and 3D CT images are randomly cropped to obtain a block of 96*384*384, and the standardization operation of subtracting the mean and dividing the variance is performed. The crop operation is performed on the x-axis, y-axis and z-axis with preset step sizes such as 192, 192 and 48, respectively. The crop operation is used to extract specific areas from the original image, which is especially important when training deep learning models because it can increase data diversity and reduce overfitting. The intensity difference between images is eliminated by standardization, making the model training more stable. The quality and consistency of image data are improved by denoising, resampling, cropping and standardization of medical images, laying a solid foundation for subsequent image fusion, segmentation and analysis tasks.
[0058] S3: Slice the preprocessed MR 3D image and CT 3D image respectively, and perform multimodal fusion on the channel dimension of the 3D image slices corresponding to MR and CT to obtain the fused multimodal image. The modality fusion refers to concatenating the CT image and the MR image on the channel dimension, that is, splicing the image data of two different modalities, namely the CT image and the MR image, in the depth direction. Specifically, the data of the CT image and the MR image are stacked according to the channel dimension to form a new three-dimensional data set. This operation allows the model to utilize information from both modalities at the same time, thereby improving the accuracy and robustness of the segmentation. When performing modality fusion, it is necessary to ensure that the images of the two modalities have the same spatial resolution (i.e., the spacing is consistent) to avoid the feature differences caused by inconsistent resolution affecting the fusion effect.
[0059] S4: Input the fused multimodal image into the trained segmentation generation network to obtain the segmentation results of each bone in the knee joint corresponding to each image. Specifically, the 3D MR and CT images are fused on the channel to obtain the input of the 2*96*384*384 segmentation generation network to obtain the segmentation results of each bone in the knee joint. The steps include: inputting the fused image into the segmentation generation network, performing a softmax operation on the result output by the segmentation generation network to obtain the segmentation results of each bone in the knee joint. The segmentation results of each bone in the knee joint include the segmentation results of cartilage, patella, tibia and femur, wherein Softmax is a function that maps the output value range of each output node to [0,1] and constrains the sum of the output values of each output node to 1.
[0060] See also Figure 2 As shown, the trained segmentation generation network is a neural network based on 3D Unet, including a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder and a fifth decoder connected in sequence, wherein each encoder and decoder includes 2 convolutional layers, batch normalization and activation function are performed after each convolutional layer, the number of filters in the corresponding encoders are 32, 64, 128, 256 and 512 respectively, and the number of filters in the corresponding decoders are 512, 256, 128, 64 and 32 respectively, the network input is a multimodal 3D image of 2*96*384*384, and the network outputs a probability map of bone segmentation, wherein the downsampling module is the above-mentioned encoder, and the upsampling module is the above-mentioned decoder.
[0061] See also Figure 3 As shown, the segmentation discrimination network includes 4 convolutional layers, 1 tiling layer and 1 output layer. Each convolutional layer is connected to a pooling layer. The first three pooling layers adopt the maximum pooling operation, and the last pooling layer adopts the average pooling operation. The number of filters corresponding to the 4 convolutional layers are 64, 128, 256 and 512 respectively.
[0062] S5: Post-process the output of the trained segmentation generation network, and reorganize the segmentation results of each bone in the knee joint on each image into the size of the original image. This is because the slices of CT and MR are relatively large, and they are generally sliced on the three axes of xyz, and then the slices are sent to the network, so the slices must be reorganized into the original image size in the end. The post-processing of the segmentation results includes operations such as removing isolated points, edge smoothing, and resampling of the segmentation results.
[0063] In one embodiment, in step S4, the pre-training step of the trained segmentation generation network is as follows: Figure 4 As shown, including:
[0064] S41: obtaining a 3D image of MR and a 3D image of CT, and labeling the corresponding 3D image;
[0065] S42: resampling the 3D image map of MR, the 3D image map of CT and the corresponding label map to obtain an image map and a target label map with consistent voxel sizes;
[0066] S43: fusing the 3D image of MR and the 3D image of CT;
[0067] S44: performing data enhancement on the fused 3D image, wherein the data enhancement includes scaling, rotating, and changing the brightness, contrast, and saturation of the image;
[0068] S45: inputting the data-enhanced multimodal image into the segmentation generation network to obtain the segmentation results of each bone in the knee joint;
[0069] S46: Input the segmentation results of each bone in the knee joint into the segmentation discrimination network to realize the training and verification of the network, and obtain the trained segmentation generation network through parameter update optimization. Those skilled in the art can understand that in this embodiment, it is necessary to judge the true or false of the segmentation results of each bone in the knee joint. In the training stage, the graph containing the segmentation results of each bone obtained by the segmentation generation network and the 3D nuclear magnetic imaging graph are input into the segmentation discrimination network to obtain the probability of belonging to the true bone segmentation result, and the manually labeled bone segmentation result graph and the 3D nuclear magnetic imaging graph are input into the segmentation discrimination network to obtain the probability of belonging to the true bone segmentation result.
[0070] Furthermore, in step S46, the segmentation results of each bone in the knee joint are input into the segmentation discrimination network to realize the training and verification of the network, and the trained segmentation generation network is obtained by parameter update optimization, which further includes:
[0071] S4610: Input the segmentation results of each bone in the knee joint and the corresponding target annotation map into the segmentation discrimination network to obtain the probability that the two match the true segmentation results;
[0072] S4611: Calculate the loss value between the segmentation result obtained by the segmentation generation network and the labeled segmentation result and the sum of the two matching loss values of the segmentation discriminant network respectively;
[0073] S4612: According to the loss obtained by the alternating back-propagation generation network and the loss obtained by the segmentation discrimination network, the parameter values of the corresponding neural networks are alternately updated to obtain the pre-trained segmentation generation network weights.
[0074] The above embodiment can effectively evaluate the accuracy of the segmentation result by inputting the segmentation result and the target annotation graph into the segmentation discriminant network to obtain the probability that the two match the real segmentation result. This evaluation mechanism helps to guide the learning process of the generation network, so that it can better capture the characteristics of each bone in the knee joint, thereby improving the accuracy of the segmentation. By calculating the loss value between the segmentation result obtained by the segmentation generation network and the segmentation result after annotation and the sum of the two matching loss values of the segmentation discriminant network, the performance of the model can be comprehensively measured. This comprehensive loss calculation method helps to enhance the robustness of the model so that it can maintain good performance on different data sets. According to the loss obtained by the alternating back-propagation generation network and the loss obtained by the segmentation discriminant network, alternatingly updating the parameter values of the corresponding neural network is an effective optimization strategy. This strategy helps to quickly converge to a better local minimum and improve the stability and performance of the model. The above scheme can realize the automated processing of medical images and reduce the need for manual intervention. This is particularly important for the processing of large-scale medical image data, and can significantly improve work efficiency and accuracy.
[0075] Furthermore, in step S46, inputting the results of the segmentation of each bone in the knee joint into the segmentation discrimination network to implement the training and verification of the network further includes:
[0076] S4620: performing an abnormal result filtering operation on the segmentation results of each bone in the knee joint, wherein the filtering operation includes filtering out the segmentation results that do not meet the requirements according to the prior knowledge related to the knee joint, wherein the prior knowledge related to the knee joint refers to that the distances between the femur, patella, tibia and cartilage are relatively concentrated, and the cartilage is flat. By performing the abnormal result filtering operation on the segmentation results of each bone in the knee joint, the segmentation results that do not meet the requirements can be filtered out according to the prior knowledge related to the knee joint, which helps to remove unreasonable segmentation results, thereby improving the accuracy of segmentation;
[0077] S4621: Performing a reorganization operation and a resampling operation on the filtered segmentation results, wherein the reorganization operation includes reorganizing the corresponding segmentation results input from the same 3D image map with overlap to obtain the bone segmentation result of the complete 3D image map, and reorganizing with preset overlapping pixels such as 192, 192 and 48 on the x, y and z axes respectively; the resampling operation includes resampling the reorganized segmentation results according to the original spacing to obtain the segmentation results of each bone in the final knee joint, and performing a reorganization operation and a resampling operation on the filtered segmentation results, which can further enhance the robustness of the model. The reorganization operation reorganizes the corresponding segmentation results input from the same 3D image map with overlap to obtain the bone segmentation result of the complete 3D image map, which helps to reduce the error caused by incomplete data, and the resampling operation resamples according to the original spacing to obtain the segmentation results of each bone in the final knee joint, which helps to maintain the consistency and integrity of the data. The above scheme optimizes the data processing flow through abnormal result filtering, reorganization and resampling operations. These operations not only improve the efficiency of data processing, but also ensure the accuracy and reliability of data processing.
[0078] Embodiment 2
[0079] Based on the same concept, the present invention also provides a knee joint bone segmentation device integrating a generation network and a discriminant network, and implements the knee joint bone segmentation method integrating a generation network and a discriminant network as described in the first embodiment, including:
[0080] The acquisition module is used to obtain 3D sequences of MR and CT of each bone in the knee joint;
[0081] A preprocessing module, used for preprocessing the 3D image map of MR and the 3D image map of CT respectively;
[0082] A fusion module is used to slice the preprocessed MR 3D image map and the CT 3D image map, respectively, and perform multimodal fusion of the MR and CT corresponding 3D image map slices in the channel dimension to obtain a fused multimodal image map;
[0083] The segmentation module is used to input the fused multimodal image into the trained segmentation generation network to obtain the segmentation results of each bone in the knee joint corresponding to each image;
[0084] The post-processing module is used to post-process the output results of the trained segmentation generation network, and reorganize the segmentation results of each bone in the knee joint corresponding to each image into the original image size.
[0085] The specific principles, contents and implementation methods of the above modules are as described in Example 1 and will not be repeated here.
[0086] Finally, in order to apply the above-mentioned knee joint bone segmentation method of fusion generation and discrimination network to a length quantification system, device or equipment with relevant hardware conditions, the present application also provides a computer-readable storage medium, in which a computer program is loaded, and when the computer program is executed by a computer, it realizes the functions of the corresponding method embodiment as described above.
[0087] Likewise, the present application also protects a computer-readable storage medium loaded with a computer program for implementing the knee joint bone segmentation method of the fusion generation and discrimination network.
[0088] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program. The computer program includes one or more computer programs. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, DDL (Digital DubDriberLine, digital subscriber line)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density DVD (Digital Video DiDD)), or a semiconductor medium (eg, a solid-state drive (SSD)).
[0089] It should be noted that, in the above-mentioned embodiments, the terms used herein are only for describing specific exemplary embodiments, and are not intended to be restrictive. As used herein, the singular forms "one", "an" and "the or described" may be intended to also include plural forms, unless the context clearly indicates otherwise. The terms "include", "comprise" and "have" are inclusive, and therefore specify the existence of stated features, wholes, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, parts and / or their groups. The method steps, processes and operations described herein should not be interpreted as necessarily requiring the method steps, processes and operations to be performed in the specific order discussed or shown, unless it is specifically indicated that the steps are to be performed in the set order. It should also be understood that additional or alternative steps may be adopted.
[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A knee joint bone segmentation method integrating generative and discriminative networks, characterized in that: include: Acquire 3D MR and CT sequences of each bone in the knee joint; Preprocessing the 3D images of MR and 3D images of CT respectively; The preprocessed MR 3D image map and CT 3D image map are sliced respectively, and the 3D image map slices corresponding to MR and CT are multimodally fused in the channel dimension to obtain a fused multimodal image map; The fused multimodal images are input into the trained segmentation generation network to obtain the segmentation results of each bone in the knee joint corresponding to each image; The trained segmentation generation network output is post-processed, and the segmentation results of each bone in the knee joint corresponding to each image are reassembled into the original image size.
2. The knee joint bone segmentation method of the fusion generation and discrimination network as claimed in claim 1, characterized in that: The pre-training step of the trained segmentation generation network includes: Obtain 3D images of MR and 3D images of CT, and label the corresponding 3D images; Resample the 3D image map of MR, the 3D image map of CT and the corresponding label map to obtain the image map and the target label map with the same voxel size; Fusing 3D images of MR and 3D images of CT; Performing data enhancement on the fused 3D image, wherein the data enhancement includes scaling, rotating, and changing the brightness, contrast, and saturation of the image; The data-enhanced multimodal image is input into the segmentation generation network to obtain the segmentation results of each bone in the knee joint; The segmentation results of each bone in the knee joint are input into the segmentation discrimination network to realize the training and verification of the network, and the trained segmentation generation network is obtained through parameter update optimization.
3. The knee joint bone segmentation method of the fusion generation and discrimination network as claimed in claim 2, characterized in that: The step of inputting the segmentation results of each bone in the knee joint into the segmentation discrimination network to realize the training and verification of the network, and obtaining the trained segmentation generation network through parameter updating and optimization further includes: The segmentation results of each bone in the knee joint and the corresponding target annotation map are input into the segmentation discriminant network to obtain the probability that the two match the true segmentation results; Calculate the loss value between the segmentation result obtained by the segmentation generation network and the annotated segmentation result and the sum of the two matching loss values of the segmentation discriminant network respectively; According to the loss obtained by alternating back-propagation generation network and the loss obtained by segmentation discrimination network, the parameter values of the corresponding neural network are updated alternately to obtain the pre-trained segmentation generation network weights.
4. The knee joint bone segmentation method of the fusion generation and discrimination network as claimed in claim 1, characterized in that: The step of inputting the results of the segmentation of each bone in the knee joint into the segmentation discrimination network to implement the training and verification of the network further includes: Performing an abnormal result filtering operation on the segmentation results of each bone in the knee joint, wherein the filtering operation includes filtering out the segmentation results that do not meet the requirements according to the prior knowledge related to the knee joint, wherein the prior knowledge related to the knee joint refers to that the distances between the femur, patella, tibia and cartilage are relatively concentrated, and the cartilage is flat; The filtered segmentation results are reorganized and resampled. The reorganization operation includes reorganizing the corresponding segmentation results input from the same 3D image with overlap to obtain the bone segmentation result of the complete 3D image, and reorganizing them with preset overlapping pixels on the x, y and z axes respectively; the resampling operation includes resampling the reorganized segmentation results according to the original spacing to obtain the final segmentation results of each bone in the knee joint.
5. The knee joint bone segmentation method of the fusion generation and discrimination network as claimed in claim 1, characterized in that: The trained segmentation generation network is a neural network based on 3D Unet, including a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder and a fifth decoder connected in sequence, wherein each encoder and decoder includes 2 convolutional layers, batch normalization and activation function are performed after each convolutional layer, the number of filters in the corresponding encoders are 32, 64, 128, 256 and 512 respectively, the number of filters in the corresponding decoders are 512, 256, 128, 64 and 32 respectively, the network input is a multimodal 3D image of 2*96*384*384, and the network outputs a probability map of bone segmentation.
6. The knee joint bone segmentation method of the fusion generation and discrimination network as claimed in claim 2, characterized in that: The segmentation discrimination network includes 4 convolutional layers, 1 tiling layer and 1 output layer. Each convolutional layer is connected to a pooling layer. The first three pooling layers adopt the maximum pooling operation, and the last pooling layer adopts the average pooling operation. The number of filters corresponding to the 4 convolutional layers are 64, 128, 256 and 512 respectively.
7. The knee joint bone segmentation method of the fusion generation and discrimination network as claimed in claim 1, characterized in that: The pre-processing comprises: De-noising and resampling of MR 3D images: obtaining the median of all CT image spacings, resampling all CT images to the median spacing, and performing Gaussian filtering on the CT images to remove noise during image data acquisition; De-noising and resampling of CT 3D images: obtaining the median of all MR image spacings, resampling all MR images to the median spacing, and performing Gaussian filtering on the MR images to remove noise during image data acquisition; The 3D images of MR and CT are cropped and standardized: standardization operations of subtracting the mean and dividing the variance are performed respectively, and overlapping crop operations are performed on the 3D images of MR and CT to obtain several blocks of 96*384*384 size, and crop operations are performed on the x-axis, y-axis and z-axis with preset step sizes.
8. A knee joint bone segmentation device integrating a generation network and a discrimination network, characterized in that: The method for segmenting the knee joint skeleton by fusion generation and discrimination networks as described in any one of claims 1 to 7 comprises: The acquisition module is used to obtain 3D sequences of MR and CT of each bone in the knee joint; A preprocessing module, used for preprocessing the 3D image map of MR and the 3D image map of CT respectively; A fusion module is used to slice the preprocessed MR 3D image map and the CT 3D image map, respectively, and perform multimodal fusion of the MR and CT corresponding 3D image map slices in the channel dimension to obtain a fused multimodal image map; The segmentation module is used to input the fused multimodal image into the trained segmentation generation network to obtain the segmentation results of each bone in the knee joint corresponding to each image; The post-processing module is used to post-process the output results of the trained segmentation generation network, and reorganize the segmentation results of each bone in the knee joint corresponding to each image into the original image size.
9. An electronic device, characterized in that: include: A memory, the memory being used to store a processing program; A processor, wherein when executing the processing program, the processor implements the knee joint bone segmentation method of the fusion generation and discrimination network as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that: The readable storage medium stores a processing program, and when the processing program is executed by the processor, the knee joint bone segmentation method of the fusion generation and discrimination network as described in any one of claims 1 to 7 is implemented.