Multi-scale-based three-dimensional focus segmentation and fusion method and device
By performing multi-scale resampling and multimodal fusion methods on medical images, the problems of missing large lesions and loss of small lesions in the prior art are solved, and more accurate lesions are realized.
Patent Information
- Application Number
- CN202311554497.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-20
- Publication Date
- 2025-07-08
AI Technical Summary
The existing medical image segmentation technology cannot provide global information, resulting in missed or missed detection of large lesions, and information loss caused by resampling of small lesions as low-resolution images, resulting in missed detection.
A three-dimensional lesion segmentation and fusion method based on multi-scale is adopted. By resampling medical images at different sizes, testing models of multi-modal and multi-pathological diseases are trained, and the detection results of different modalities are fused to improve the accuracy of the detection results.
It improves the accuracy of the detection results of medical imaging segmentation methods, solves the problem of missed and missed detection of large lesions, and effectively detects small lesions, improving the comprehensiveness and accuracy of the detection.
Smart Images
Figure CN120279262A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer multimedia technology, and in particular to a multi-scale three-dimensional lesion segmentation and fusion method and device. Background Art
[0002] Medical images refer to images generated by the interaction of certain media (such as X-rays, electromagnetic fields, ultrasonic waves, etc.) with the human body. They can display the internal tissue and organ structures and densities of the human body in the form of images, providing information for diagnostic doctors to help them make judgments. Currently, medical images such as CT and MR have been widely used in the daily diagnosis and treatment of hospitals. For example, head CT is the gold standard for diagnosing cerebral hemorrhage, and MRI is the gold standard for diagnosing lesions such as carotid artery plaques and femoral head necrosis. Medical image physicians need to use these medical images to make diagnoses of abnormal lesion images of diseases based on the knowledge of the normal anatomical structure of the human body, combined with clinical symptoms, medical tests, etc., providing accurate basis for clinical diagnosis and treatment.
[0003] In recent years, in order to improve the efficiency of medical image processing and analysis, many medical image computer-aided analysis systems based on artificial intelligence and image processing technologies have been proposed. Due to the unique imaging characteristics of medical devices, the unique rules of doctors' film reading, and the limitations of current hardware devices, most of the current lesion detection methods in medical images use the method of slicing 3D data and inputting it into a deep neural network model for detection and recognition. Among them, the 3D detection method uses the detection method of slicing 3D data, which avoids the direct input of the full image and provides a lesion detection method for 3D data.
[0004] However, the existing medical image segmentation technologies have at least the following deficiencies: 1. In the aspect of detecting large lesions and lesions, the slicing method cannot provide a global information, resulting in easy omission or misdetection of this part; 2. For the detection of small lesions, high-definition images are often required. If the image is resampled into a lower-resolution image to adapt to the hardware, information loss will occur, resulting in the omission of small lesions. Summary of the Invention
[0005] This application aims to solve at least one of the technical problems in the related technologies to some extent.
[0006] To this end, the first object of this application is to propose a multi-scale three-dimensional lesion segmentation and fusion method, which solves the technical problems of omission and misdetection in the existing medical image segmentation methods due to the inability to provide global information or resampling into low-resolution images. A multi-modal detection model is trained based on multi-modal and multi-disease data, and the detection results of different modalities are fused to obtain the final detection result, improving the accuracy of the detection result.
[0007] The second object of the present application is to propose a three-dimensional lesion segmentation and fusion device based on multi-scale.
[0008] The third object of the present application is to propose a computer device.
[0009] The fourth object of the present application is to propose a non-transitory computer-readable storage medium.
[0010] To achieve the above object, the first aspect embodiment of the present application proposes a three-dimensional lesion segmentation and fusion method based on multi-scale, including: obtaining a medical image to be detected, performing resampling of different sizes on the medical image to be detected to obtain multiple medical images of different sizes, respectively intercepting the same-size sub-data blocks from the multiple medical images of different sizes; inputting the intercepted multiple sub-data blocks into a trained lesion detection model for prediction to obtain multiple prediction results, and aggregating the multiple prediction results to obtain the prediction result of the medical image to be detected; performing softmax operation on the prediction result of the medical image to be detected according to the lesion category to obtain the result probability of each lesion category, and performing category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected.
[0011] Optionally, in an embodiment of the present application, before inputting the intercepted multiple sub-data blocks into a trained lesion detection model for prediction to obtain multiple prediction results, it includes:
[0012] Obtaining medical images of multiple different lesions, respectively performing resampling of different sizes on the medical images to obtain multiple medical images of different sizes;
[0013] Respectively intercepting the same-size sub-data blocks from the multiple medical images of different sizes and the labels of the medical images, inputting the intercepted sub-data blocks into a neural network for training, and optimizing and updating the neural network according to the loss function to obtain a trained lesion detection model.
[0014] Optionally, in an embodiment of the present application, the loss function includes cross-entropy loss and dice loss;
[0015] The cross-entropy loss formula is expressed as:
[0016]
[0017] Wherein, M is all lesion categories, y is the label image, p is the probability corresponding to each pixel and each category of the prediction image, and c is the current category;
[0018] The dice loss formula is expressed as:
[0019]
[0020] Wherein, I represents the number of intersecting pixels of each category of the predicted image and the label image, U represents the total number of pixels of each category of the predicted image and the label image, and ε represents a constant value to prevent overflow.
[0021] Optionally, in an embodiment of the present application, the prediction result is the predicted sub-data block. Aggregating multiple prediction results to obtain the prediction result of the medical image to be detected includes:
[0022] Set an empty data block, where the size of the empty data block is (c, l, w, h), c is the number of categories, l is the length of the original data, w is the width of the original data, and h is the height of the original data;
[0023] Multiply the predicted sub-data block by the weight and add it to the empty data block according to the coordinates at the time of interception to obtain the final result block, and use the final result block as the prediction result of the medical image to be detected.
[0024] Optionally, in an embodiment of the present application, multiplying the predicted sub-data block by the weight and adding it to the empty data block according to the coordinates at the time of interception to obtain the final result block, and using the final result block as the prediction result of the medical image to be detected includes:
[0025] Use the argmax function for class division to obtain the connected regions of each predicted sub-data block, as well as the size and original probability value of each connected region;
[0026] Perform prediction using the Gaussian mixture model according to the size of the connected region to obtain the probability that the lesion in the predicted sub-data block belongs to a small lesion;
[0027] Calculate the combined weight by combining the original probability value of the connected region of the predicted sub-data block with the probability of belonging to a small lesion;
[0028] Remap the lesions in multiple predicted sub-data blocks back to the original coordinates according to the category and the corresponding combined weight to obtain the prediction result of the medical image to be detected.
[0029] Optionally, in an embodiment of the present application, before performing prediction using the Gaussian mixture model according to the size of the connected region to obtain the probability that the lesion in the predicted sub-data block belongs to a small lesion, it includes:
[0030] By setting a large-region Gaussian model and a small-region Gaussian model, complete the Gaussian mixture model modeling of each category label to obtain the Gaussian mixture models of all lesion categories.
[0031] Optionally, in an embodiment of the present application, the calculation formula of the combined weight is expressed as:
[0032]
[0033] where ω represents the merging weight, and V origin and V resamle are the voxel volumes of the real physical world before and after sampling, respectively.
[0034] To achieve the above object, an embodiment of the second aspect of the present application provides a multi-scale three-dimensional lesion segmentation and fusion device, including:
[0035] An acquisition module, configured to acquire a medical image to be detected, perform resampling on the medical image to be detected with different sizes to obtain multiple medical images to be detected with different sizes, and respectively intercept sub-data blocks of the same size from the multiple medical images to be detected with different sizes;
[0036] A prediction aggregation module, configured to input the intercepted multiple sub-data blocks into a trained lesion detection model for prediction to obtain multiple prediction results, and aggregate the multiple prediction results to obtain a prediction result of the medical image to be detected;
[0037] A classification module, configured to perform a softmax operation on the prediction result of the medical image to be detected according to the lesion category to obtain the result probability of each lesion category, and perform category division according to different lesion category thresholds to obtain a classification result of the medical image to be detected.
[0038] To achieve the above object, an embodiment of the third aspect of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the multi-scale three-dimensional lesion segmentation and fusion method described in the above embodiment is implemented.
[0039] To achieve the above object, an embodiment of the fourth aspect of the present application provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor, a multi-scale three-dimensional lesion segmentation and fusion method can be executed.
[0040] The multi-scale three-dimensional lesion segmentation and fusion method, device, computer device, and non-transitory computer-readable storage medium according to the embodiments of the present application solve the technical problem that existing medical image segmentation methods may cause missed detection and false detection because they cannot provide global information or resample to low-resolution images. By training a multi-modal detection model based on multi-modal and multi-disease data and fusing the detection results of different modalities to obtain the final detection result, the accuracy of the detection result is improved.
[0041] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings
[0042] The above-described and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:
[0043] Figure 1 It is a schematic flowchart of a multi-scale three-dimensional lesion segmentation and fusion method provided in Embodiment 1 of the present application;
[0044] Figure 2 It is a schematic diagram of the lesion detection model structure, training, and testing process of the multi-scale three-dimensional lesion segmentation and fusion method according to the embodiment of the present application;
[0045] Figure 3 It is a schematic structural diagram of a multi-scale three-dimensional lesion segmentation and fusion device provided in Embodiment 2 of the present application. Detailed Embodiments
[0046] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.
[0047] The multi-scale three-dimensional lesion segmentation and fusion method and device according to the embodiments of the present application will be described below with reference to the accompanying drawings.
[0048] Figure 1 It is a schematic flowchart of a multi-scale three-dimensional lesion segmentation and fusion method provided in Embodiment 1 of the present application.
[0049] As Figure 1 shown, the multi-scale three-dimensional lesion segmentation and fusion method includes the following steps:
[0050] Step 101: Obtain the medical image to be detected, perform resampling on the medical image to be detected with different sizes to obtain multiple medical images to be detected with different sizes, and respectively intercept sub-data blocks of the same size from the multiple medical images to be detected with different sizes;
[0051] Step 102: Input the multiple intercepted sub-data blocks into the trained lesion detection model for prediction to obtain multiple prediction results, and aggregate the multiple prediction results to obtain the prediction result of the medical image to be detected;
[0052] Step 103: Perform a softmax operation on the prediction result of the medical image to be detected according to the lesion category to obtain the result probability of each lesion category, and perform category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected.
[0053] The multi-scale based three-dimensional lesion segmentation and fusion method according to the embodiments of the present application obtains a medical image to be detected, resamples the medical image to be detected at different sizes to obtain multiple medical images to be detected at different sizes, and respectively intercepts sub-data blocks of the same size from the multiple medical images to be detected at different sizes; inputs the multiple intercepted sub-data blocks into a trained lesion detection model for prediction to obtain multiple prediction results, and aggregates the multiple prediction results to obtain a prediction result of the medical image to be detected; performs a softmax operation on the prediction result of the medical image to be detected according to the lesion category to obtain the result probability of each lesion category, and performs category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected. Thereby, it can solve the technical problem that existing medical image segmentation methods may cause missed detection and misdetection due to the inability to provide global information or resampling to low-resolution images, train a multi-modal detection model based on multi-modal and multi-disease data, and fuse the detection results of different modalities to obtain the final detection result, improving the accuracy of the detection result.
[0054] The present application proposes a lesion detection and segmentation method based on multi-scale fusion, which solves the problem of how to fuse detection results to obtain a more accurate final diagnosis result in the case of detecting different diseases at different scales in the fully automatic auxiliary diagnosis technology based on medical image processing and analysis.
[0055] The present application proposes a model training and result fusion method based on input of different modality data. This method mainly includes two parts. One is to train a multi-modal model based on multi-modal and multi-disease data, and the other is to test the model based on the training result and fuse the results of different modalities to obtain the final result.
[0056] Further, in the embodiments of the present application, before inputting the multiple intercepted sub-data blocks into a trained lesion detection model for prediction to obtain multiple prediction results, it includes:
[0057] Obtain medical images of multiple different lesions, and respectively resample the medical images at different sizes to obtain multiple medical images at different sizes;
[0058] Respectively intercept sub-data blocks of the same size from the multiple medical images at different sizes and the labels of the medical images, input the intercepted sub-data blocks into a neural network for training, and optimize and update the neural network according to the loss function to obtain a trained lesion detection model.
[0059] In the embodiments of the present application, during model training, resampling operations of different sizes are performed on the data. The resampling is carried out with equal probability, sampling the original 3D data block into new data blocks α of different sizes, intercepting sub-blocks β of the same size from each data block, performing the same operations on the data labels, and inputting them into the model for training to obtain a model that can process data of different sizes.
[0060] Among them, the label of the medical image is an image with the same size as the original medical image, and the lesion part in the medical image will be represented by a non-zero region in this label.
[0061] Furthermore, in the embodiments of the present application, the loss function includes cross-entropy loss and dice loss;
[0062] The cross-entropy loss formula is expressed as:
[0063]
[0064] Among them, M is all lesion categories, y is the label image, p is the probability corresponding to each pixel and each category of the predicted image, and c is the current category;
[0065] The dice loss formula is expressed as:
[0066]
[0067] Among them, I represents the number of intersecting pixels of each category of the predicted image and the label image, U represents the total number of pixels of each category of the predicted image and the label image, ε represents a constant value to prevent overflow, and in the present application, ε = 0.00001 is taken.
[0068] In the embodiments of the present application, the loss function during the training process is defined as cross-entropy loss and dice loss. Among them, after performing epoch rounds of training on all images in the dataset, when the loss function no longer decreases, the model training is ended, the model parameter values are fixed, and a trained lesion detection model is obtained. In the present application, epoch can be set to 50.
[0069] Furthermore, in the embodiments of the present application, the prediction result is the predicted sub-data block. Aggregating multiple prediction results to obtain the prediction result of the medical image to be detected, including:
[0070] Set an empty data block, where the size of the empty data block is (c, l, w, h), c is the number of categories, l is the length of the original data, w is the width of the original data, and h is the height of the original data;
[0071] Multiply the predicted sub-data block by the weight and add it to the empty data block according to the coordinates during interception to obtain the final result block, and use the final result block as the prediction result of the medical image to be detected.
[0072] In the embodiments of the present application, the trained model is used to predict data. The prediction part is mainly divided into a sampling step, a prediction step, and an aggregation step. The sampling step is similar to the sampling in the training process. Different from the sampling in the training process, in prediction, all the sizes set in the training will be resampled and then input into the neural network for prediction. The prediction step mainly performs forward inference through the neural network. In this step, since there are many result files stored in the memory, which are arrays of prediction results with the same size for a large number of dimensions, it will cause a large amount of internal storage to be occupied. Therefore, a real-time prediction aggregation method is adopted for result aggregation.
[0073] In the embodiments of the present application, the aggregation method mainly sets an empty data block with the same size as the final result, and its size is (c, l, w, h), where c is the number of categories, l is the length of the original data, w is the width of the original data, and h is the height of the original data. For each sub-block of the prediction, according to the coordinates when it is extracted, after multiplying by the weight, it is added to the empty data block. After all are added, the final result block is obtained. The softmax operation is performed on the result block according to the category dimension to obtain the result probability of each category, and the category division is performed according to different category thresholds to obtain the classification result.
[0074] Among them, the predicted results are the results under different scaling scales. The prediction results are data blocks with the same size as the input data, containing different values such as 0, 1, 2... 0 represents the background, and 1, 2, 3, etc. represent different pixel categories of the foreground.
[0075] Further, in the embodiments of the present application, after multiplying the predicted sub-data block by the weight, it is added to the empty data block according to the coordinates at the time of interception to obtain the final result block, and the final result block is used as the prediction result of the medical image to be detected, including:
[0076] Using the argmax function for category division to obtain all the connected regions of each predicted sub-data block, as well as the size and original probability value of each connected region;
[0077] Predict according to the size of the connected region using the Gaussian mixture model to obtain the probability that the lesion in the predicted sub-data block belongs to a small lesion;
[0078] Calculate the combined weight by combining the original probability value of the connected region of the predicted sub-data block with the probability of belonging to a small lesion;
[0079] Remap the lesions in multiple predicted sub-data blocks back to the original coordinates according to the category and the corresponding combined weight to obtain the prediction result of the medical image to be detected.
[0080] In the embodiments of the present application, using the argmax function for category division includes that the argmax function takes the channel with the maximum probability as the final result, that is, the category is divided.
[0081] In the embodiments of the present application, each predicted lesion result has a small block, and this small block is a connected region. Exemplarily, from the perspective of map drawing, if the ocean pixel is 0 and the land pixel is 1, then the small island is a connected region and the ocean is the background.
[0082] In the embodiments of the present application, the original probability value is the probability value predicted by the model, and there will be a probability value prediction on all corresponding category channels. Here, the predicted probabilities of all categories will be used for subsequent calculations.
[0083] In the embodiments of the present application, since the Gaussian mixture model is composed of two Gaussian models, one representing the probability of small lesions and the other representing the probability of large lesions, the probability of small lesions ps is calculated, and the probability of large lesions can be obtained by pb = 1 - ps.
[0084] In the embodiments of the present application, the prediction result of the medical image to be detected is an image that only contains lesion regions of different categories, and different categories are presented in different colors. It can be superimposed with the original image for presentation, but here there is only the result and no original image.
[0085] In the embodiments of the present application, when using the model obtained by the above training for model aggregation, it mainly includes two steps: The first step is to statistically calculate the weights when different category data in the training set are merged, establish a Gaussian mixture model, and thus calculate the influence degree of small lesions of this disease type. The calculation method is as follows:
[0086] First, use the Gaussian mixture model to model the currently labeled connected region according to the occupied volume. The modeling is performed using sklearn, and the number of Gaussian models is set to 2, that is, two types: large regions and small regions. The number of iterations in this application is taken as 100. The Gaussian mixture model modeling for different disease types is completed.
[0087] Then, after predicting the lesion model for the sub - block, use argmax for category division, obtain all the connected regions of each sub - block and the size of each connected region, and use the Gaussian mixture model for prediction according to the size to obtain the probability that the size of this lesion belongs to small lesions. Combine the original corresponding probability values of all the connected regions of the sub - block with the probability of small lesions. Calculate its combined weight, and the calculation formula is: In this formula, ω represents the calculated combined weight, V orugun and V resamle respectively represent the volume sizes of the voxels before and after sampling in the real physical world. This formula is mainly designed under the conditions that some disease types are concerned about the impact of large - area lesions, some disease types are more concerned about the detection of small - area lesions, and some lesions need to statistically calculate the importance of different category lesions related to the area according to the annotation.
[0088] In the second step, the lesions in all current sub-blocks are remapped back to the original coordinates according to the category and the predicted weight ω. The expression is B c,i1:i2,j1:j2,k1:k2 = b [bs＝c] * ω, where B c,i1:i2,j1:j2,k1:k2 is the value at the corresponding coordinates of the c-th category of the aggregated block. i1, i2, j1, j2, k1, k2 are the starting and ending coordinates of each dimension of (l, w, h) when extracting the sub-block respectively. b is the predicted sub-block, and bs is the classification result of the predicted block obtained after the argmax operation.
[0089] Furthermore, in the embodiments of the present application, before predicting the probability that the lesions in the predicted sub-data block belong to small lesions by using the Gaussian mixture model according to the size of the connected region, it includes:
[0090] By setting a large-region Gaussian model and a small-region Gaussian model, the Gaussian mixture model modeling of each category label is completed to obtain the Gaussian mixture models of all lesion categories.
[0091] In the embodiments of the present application, the Gaussian model modeling process is as follows: First, traverse all image blocks, use the measure.label method in the skimage library to obtain the connected regions of all labels, record each category and the voxel size of each connected region through an array, and form an array of voxel values of connected regions for each category, denoted as Lt; Second, use the mixture.GaussianMixture in the sklearn library to construct the Gaussian mixture model, gmm = GaussianMixture(n_components = 2, random_state = 42). Here, n_components represents the number of Gaussian models. In this application, 2 is taken, indicating two Gaussian models for larger and smaller ones, and random_state is the random state, which is taken as 42 in this application; Third, complete the Gaussian mixture model modeling of each category label, and c Gaussian mixture models can be obtained.
[0092] Furthermore, in the embodiments of the present application, the calculation formula of the combined weight is expressed as:
[0093]
[0094] where ω represents the combined weight, V origin and V resamle are the volume sizes of the voxels before and after sampling in the real physical world respectively.
[0095] A comparative experiment was conducted using the fusion method of this application. In the experiment, the segmentation results of the whole lung lesions were fused, showing a certain improvement compared to the non-fusion method. The experimental results are shown in Table 1 below.
[0096]
[0097]
[0098] Table 1
[0099] Judging from the experimental results in Table 1, the resampling prediction of different scales used in this application has a certain improvement on the results and can play an actual application value.
[0100] Figure 2 It is a schematic diagram of the lesion detection model structure, training, and testing process of the multi-scale based three-dimensional lesion segmentation and fusion method of the embodiment of this application.
[0101] As Figure 2 shown, during the training of the lesion detection model, resampling operations of different sizes are performed on the data, and the same operations are performed on the data labels, which are then input into the model for training to obtain a model that can process data of different sizes; the trained model is used to predict the data. The prediction part mainly includes a sampling step, a prediction step, and an aggregation step, including resampling the image to be detected and inputting it into the trained model for prediction to obtain different prediction results, and aggregating the prediction results to obtain the final detection result.
[0102] Figure 3 It is a schematic diagram of the structure of a multi-scale based three-dimensional lesion segmentation and fusion device provided in the second embodiment of this application.
[0103] As Figure 3 shown, the multi-scale based three-dimensional lesion segmentation and fusion device includes:
[0104] An acquisition module 10, configured to acquire the medical image to be detected, perform resampling of different sizes on the medical image to be detected, obtain multiple medical images to be detected of different sizes, and respectively intercept sub-data blocks of the same size from the multiple medical images to be detected of different sizes;
[0105] A prediction aggregation module 20, configured to input the multiple intercepted sub-data blocks into the trained lesion detection model for prediction, obtain multiple prediction results, and aggregate the multiple prediction results to obtain the prediction result of the medical image to be detected;
[0106] A classification module 30, configured to perform softmax operation on the prediction result of the medical image to be detected according to the lesion category, obtain the result probability of each lesion category, and perform category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected.
[0107] The three-dimensional lesion segmentation and fusion device based on multi-scale in the embodiment of the present application includes an acquisition module, which is used to acquire the medical image to be detected, perform resampling of different sizes on the medical image to be detected, obtain multiple medical images to be detected of different sizes, and intercept sub-data blocks of the same size from the multiple medical images to be detected of different sizes respectively; a prediction aggregation module, which is used to input the multiple intercepted sub-data blocks into the trained lesion detection model for prediction, obtain multiple prediction results, and aggregate the multiple prediction results to obtain the prediction result of the medical image to be detected; a classification module, which is used to perform softmax operation on the prediction result of the medical image to be detected according to the lesion category, obtain the result probability of each lesion category, and perform category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected. Thus, the technical problem of missed detection and misdetection caused by the existing medical image segmentation method because it cannot provide global information or resample to an image with low resolution can be solved. A multi-modal detection model is trained based on multi-modal and multi-disease data, and the final detection result is obtained by fusing the detection results of different modalities, improving the accuracy of the detection result.
[0108] To implement the above embodiment, the present application also proposes a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the three-dimensional lesion segmentation and fusion method based on multi-scale described in the above embodiment is implemented.
[0109] To implement the above embodiment, the present application also proposes a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the three-dimensional lesion segmentation and fusion method based on multi-scale described in the above embodiment is implemented.
[0110] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without conflict, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0111] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0112] Any process or method description represented in a flowchart or described otherwise herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process. The scope of the preferred embodiments of the present application includes additional implementations, where functions may be executed in a substantially simultaneous manner or in an order opposite to that shown or discussed, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.
[0113] Logic and / or steps represented in a flowchart or described otherwise herein, for example, may be considered a sequenced list of executable instructions for implementing a logical function and may be specifically implemented in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.
[0114] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following techniques known in the art or a combination thereof can be used: discrete logic circuits with logic gate circuits for implementing logic functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0115] Those of ordinary skill in the art can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.
[0116] In addition, in each embodiment of the present application, the functional units can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0117] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.
Claims
1. A multi-scale based three-dimensional lesion segmentation and fusion method, characterized in that Including the following steps: Obtain the medical image to be detected, perform resampling on the medical image to be detected at different sizes to obtain multiple medical images to be detected at different sizes, and respectively intercept sub-data blocks of the same size from the multiple medical images to be detected at different sizes; Input the multiple intercepted sub-data blocks into the trained lesion detection model for prediction to obtain multiple prediction results, and aggregate the multiple prediction results to obtain the prediction result of the medical image to be detected; Perform a softmax operation on the prediction result of the medical image to be detected according to the lesion category to obtain the result probability of each lesion category, and perform category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected.
2. The method according to claim 1, characterized in that, Before inputting the multiple intercepted sub-data blocks into the trained lesion detection model for prediction to obtain multiple prediction results, it includes: Obtain medical images of multiple different lesions, perform resampling on the medical images at different sizes respectively to obtain multiple medical images at different sizes; Respectively intercept sub-data blocks of the same size from the multiple medical images at different sizes and the labels of the medical images, input the intercepted sub-data blocks into a neural network for training, and optimize and update the neural network according to the loss function to obtain a trained lesion detection model.
3. The method according to claim 2, wherein The loss function includes cross-entropy loss and dice loss; The cross-entropy loss formula is expressed as: Where M is all lesion categories, y is the label image, p is the probability corresponding to each pixel and each category of the prediction image, and c is the current category; The dice loss formula is expressed as: Where I represents the number of intersecting pixels of each category of the prediction image and the label image, U represents the total number of pixels of each category of the prediction image and the label image, and ε represents a constant value to prevent overflow.
4. The method according to claim 1, wherein The prediction result is the predicted sub-data block, and the aggregating the multiple prediction results to obtain the prediction result of the medical image to be detected includes: Set an empty data block, where the size of the empty data block is (c, l, w, h), c is the number of categories, l is the length of the original data, w is the width of the original data, and h is the height of the original data; Multiply the predicted sub-data block by the weight and add it to the empty data block according to the coordinates at the time of interception to obtain the final result block, and use the final result block as the prediction result of the medical image to be detected.
5. The method according to claim 4, wherein The multiplying the predicted sub-data block by the weight and adding it to the empty data block according to the coordinates at the time of interception to obtain the final result block, and using the final result block as the prediction result of the medical image to be detected includes: Use the argmax function for category division to obtain the connected regions of each predicted sub-data block, the size of each connected region, and the original probability value; Perform prediction according to the size of the connected region using the Gaussian mixture model to obtain the probability that the lesion in the predicted sub-data block belongs to a small lesion; Calculate the combined weight by combining the original probability value of the connected region of the predicted sub-data block with the probability of belonging to a small lesion; Remap the lesions in multiple predicted sub-data blocks back to the original coordinates according to the category and the corresponding merging weights to obtain the prediction result of the medical image to be detected.
6. The method according to claim 5, wherein Before predicting the probability that the lesions in the predicted sub-data blocks belong to small lesions by using the Gaussian mixture model according to the size of the connected region, it includes: By setting a large-region Gaussian model and a small-region Gaussian model, complete the Gaussian mixture model modeling for each category label to obtain the Gaussian mixture models of all lesion categories.
7. The method according to claim 5, characterized in that, The calculation formula of the merging weight is expressed as: where ω represents the merging weight, and V origin and V resamle are the voxel volumes of the real physical world before and after sampling, respectively.
8. A three-dimensional lesion segmentation and fusion device based on multi-scale, characterized in that, It includes: An acquisition module, configured to acquire a medical image to be detected, perform resampling of the medical image to be detected with different sizes to obtain multiple medical images to be detected with different sizes, and respectively intercept sub-data blocks of the same size from the multiple medical images to be detected with different sizes; A prediction aggregation module, configured to input the multiple intercepted sub-data blocks into a trained lesion detection model for prediction to obtain multiple prediction results, and aggregate the multiple prediction results to obtain the prediction result of the medical image to be detected; A classification module, configured to perform a softmax operation on the prediction result of the medical image to be detected according to the lesion category to obtain the result probability of each lesion category, and perform category division according to different lesion category thresholds to obtain the classification result of the medical image to be detected.
9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method described in any one of claims 1-7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1-7 is implemented.