A 3D liver lesion detection method and device

Abstract

The application provides a 3D liver lesion detection method and device. The method comprises the following steps: acquiring multi-phase CT images to be detected, constructing 3D data including upper and lower layers of each slice layer to be detected as the center; inputting the 3D data into a detection model, extracting 3D context features of the images, and detecting 2D lesions obtained by center cutting; combining 2D lesions of continuous layer slices belonging to the same lesion into a 3D lesion; inputting the 3D lesion cut in the local image of multiple phases output by the detection model into a false positive identification model to obtain a detection result of the 3D lesion. The application can extract 3D context features by constructing 3D data, can fully utilize lesion information of each phase for lesion detection, and thus improves the lesion detection accuracy; and the false positive identification model is set to filter false positives of the 3D lesion, which can further improve the lesion detection accuracy.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging technology, specifically relating to a 3D liver lesion detection method and device. Background Technology

[0002] Liver cancer is one of the most common malignant tumors, characterized by strong heterogeneity, high susceptibility to metastasis and recurrence, and poor prognosis. Early identification and diagnosis of liver cancer, especially small hepatocellular carcinoma (hepatocellular carcinoma with a single nodule having a maximum diameter less than or equal to 3 cm, or multiple nodules with a sum of maximum diameters ≤ 3 cm), is of profound significance for subsequent treatment. The detection and identification of benign and malignant liver lesions is a prerequisite for liver cancer diagnosis and a crucial component in constructing a computer-aided diagnostic system for liver cancer.

[0003] Liver lesions are widely and complexly distributed, including various benign and malignant lesions, with diverse sizes. Some lesions closely resemble surrounding liver tissue in image features, making overall detection extremely difficult. Existing technologies either rely on traditional image processing methods, utilizing prior statistical knowledge for lesion detection; or they optimize only a specific step in 3D lesion detection in CT scans, lacking end-to-end automated detection capabilities. For example, they require manually removing slices from the original data that do not belong to the liver region, and the model only outputs 2D lesion detection boxes or contours for each slice, unable to directly provide 3D detection results; or they utilize multiple neural networks connected in series to construct complex lesion detection processes. Existing semantic segmentation techniques for liver lesion segmentation and detection struggle with small lesion scenarios and are difficult to perform detection based on full-resolution 3D images. Furthermore, existing technologies lack the ability to learn the 3D contextual features of CT data; extracting features from a single slice or three slices (one above and one below) cannot cover the real-world distribution of lesions along the CT axis. Current technologies lack optimization for detecting small hepatocellular carcinoma lesions (≤3cm or even ≤1cm), thus lacking clinical application value. Furthermore, current technologies fail to integrate multi-phase CT features to optimize for the common false positive phenomenon in lesion detection. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, the present invention provides a 3D liver lesion detection method and device.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] In a first aspect, the present invention provides a 3D liver lesion detection method, comprising:

[0007] Acquire multi-phase CT images of the target slice and construct 3D data centered on each target slice layer, including multiple slices above and below it;

[0008] 3D data is input into the detection model to extract multi-size features and obtain the 3D context features of the image. Based on the central cutting and global attention mechanism, the 2D lesion features of the central slice layer are obtained, and then the probability of the 2D lesion being positive is obtained.

[0009] 2D lesions belonging to the same lesion in continuous slices are combined into a 3D lesion. The probability of a 3D lesion being positive is equal to the sum of the probabilities of all 2D lesions being positive.

[0010] The local images of the 3D lesions output by the detection model, segmented at multiple phases, are input into a false positive identification model to filter out false positives in the 3D lesions and obtain the detection results of the 3D lesions.

[0011] Furthermore, the multiple phases include the plain scan phase, the arterial phase, the portal venous phase, and the delayed phase.

[0012] Furthermore, after acquiring multi-phase CT images, the method also includes adjusting the window width and window level of the CT images to highlight the features of the liver region.

[0013] Furthermore, the 3D data dimension constructed with each slice layer to be detected as the center is (2m+1,h,w), where m is the number of layers above and below the slice layer to be detected, m>1, and h and w are the height and width of the image, respectively.

[0014] Furthermore, the training data for the detection model includes positive sample 3D data and negative sample 3D data. The central layer of the positive sample 3D data consists of slices with lesion annotations, while the central layer of the negative sample 3D data consists of slices without lesion annotations.

[0015] Furthermore, the detection model includes a feature extraction module and a detection module, and also includes a Feature Pyramid Network (FPN) located between the feature extraction module and the detection module, which is used to improve the information fusion between deep features and shallow features and improve the detection capability of small lesions.

[0016] Furthermore, the bounding rectangle of the 2D lesion is used as its detection box, and the detection box data is (x,y,h1,w1), where x and y are the horizontal and vertical coordinates of the upper left corner of the detection box, respectively, and h1 and w1 are the height and width of the detection box, respectively.

[0017] Furthermore, the method for determining whether two consecutive 2D lesions belong to the same 3D lesion includes: if the Intersection over Union (IOU) between the detection frames of two consecutive 2D lesions is greater than a set threshold, then the two consecutive lesions belong to the same 3D lesion. The formula for calculating the IOU is:

[0018]

[0019] In the formula, |A| and |B| are the areas of the two detection boxes, respectively, and |A∩B| is the area of ​​the overlapping part of the two detection boxes.

[0020] Secondly, the present invention provides a 3D liver lesion detection device, comprising:

[0021] The 3D data construction module is used to acquire multi-phase CT images to be detected and construct 3D data centered on each slice layer to be detected, including multiple slices above and below it.

[0022] The 2D lesion detection module inputs 3D data into the detection model, performs multi-size feature extraction, obtains the 3D contextual features of the image, obtains the 2D lesion features of the central slice layer based on the central cutting and global attention mechanism, and then obtains the probability that the 2D lesion is positive.

[0023] The 3D lesion synthesis module is used to combine 2D lesions from consecutive slices belonging to the same lesion into a 3D lesion. The probability of a 3D lesion being positive is equal to the fusion of the probabilities of all 2D lesions being positive.

[0024] The false positive filtering module is used to input the local images of the 3D lesions output by the detection model, which are segmented in multiple phases, into a false positive recognition model to filter out false positives in the 3D lesions and obtain the detection results of the 3D lesions.

[0025] Furthermore, the multiple phases include the plain scan phase, the arterial phase, the portal venous phase, and the delayed phase.

[0026] Compared with the prior art, the present invention has the following beneficial effects.

[0027] This invention constructs 3D data centered on each slice to be detected, including multiple slices above and below it. This fully utilizes the 3D contextual information of CT data to effectively extract 3D image features. Through multi-size feature extraction, this invention can effectively detect liver lesions with significant size distribution differences, especially small liver lesions. Based on multi-phase CT images, this invention can fully utilize lesion information from each phase, thereby improving lesion detection accuracy. Furthermore, by setting up a false positive identification model to filter false positives in 3D lesions, this invention can further improve lesion detection accuracy. Attached Figure Description

[0028] Figure 1 This is a flowchart of a 3D liver lesion detection method according to an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of a false positive identification network.

[0030] Figure 3This is a block diagram of a 3D liver lesion detection device according to an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0032] Figure 1 This is a flowchart of a 3D liver lesion detection method according to an embodiment of the present invention, including the following steps:

[0033] Step 101: Obtain multi-phase CT images to be detected and construct 3D data centered on each slice layer to be detected, including multiple slices above and below it.

[0034] Step 102: Input 3D data into the detection model, perform multi-size feature extraction, obtain 3D context features of the image, obtain 2D lesion features of the central slice layer based on the central cutting and global attention mechanism, and then obtain the probability that the 2D lesion is positive.

[0035] Step 103: Combine 2D lesions from consecutive slices belonging to the same lesion into a 3D lesion. The probability of a 3D lesion being positive is equal to the sum of the probabilities of all 2D lesions being positive.

[0036] Step 104: Input the local images of the 3D lesions cut in multiple phases output by the detection model into a false positive identification model to filter the false positives of the 3D lesions and obtain the detection results of the 3D lesions.

[0037] In this embodiment, step 101 is mainly used to construct 3D data. The CT image to be detected is 2D slice data, which needs to be converted into 3D data before being input into the detection model. The method for constructing 3D data is as follows: taking each slice layer to be detected as the center, continuously taking slices of equal number above and below. Two 3D data constructed in this way may contain many identical slices, especially two adjacent 3D data, but at least one slice will be different, so there will be no completely identical 3D data. Inputting such 3D data into the detection model for feature extraction is beneficial for obtaining contextual features. The 3D data construction method described above is for the CT image to be detected input into the trained detection model (for prediction). The training data during the training of the detection model also needs to be constructed into 3D data in the same way. The difference is that 3D data with positive samples (with lesion annotations) at the center layer and 3D data with negative samples (without lesion annotations) at the center layer need to be constructed separately. It is worth noting that this embodiment is for processing multi-phase CT images, which can make full use of the lesion information of each phase and improve the detection accuracy of lesions.

[0038] In this embodiment, step 102 is mainly used for feature extraction and 2D lesion detection. The constructed 3D data is input into the detection model to obtain the 3D contextual features of the image. To effectively detect small lesions, this embodiment performs multi-size feature extraction. Constructing 3D data for feature extraction is for effectively extracting contextual features, but lesion detection still needs to be performed on 2D lesions; therefore, the 3D features need to be converted into 2D features. This embodiment obtains the 2D lesion features of the central slice layer through center segmentation and a global attention mechanism. Then, the detection model outputs the lesion parameters (location and size) and the probability of being positive based on the 2D lesion features.

[0039] In this embodiment, step 103 is mainly used to construct 3D lesions. The method for constructing 3D lesions involves combining 2D lesions from consecutive slices belonging to the same lesion to form a single 3D lesion. Images belonging to the same lesion should have a high degree of similarity, which allows determination of whether two consecutive slices of 2D lesions belong to the same lesion. The probability of a 3D lesion being positive is equal to the fusion of the probabilities of all 2D lesions constituting the 3D lesion being positive, such as the mean, weighted mean, maximum value, median, etc. Furthermore, there may be significant overlap between candidate boxes of different constructed 3D lesions; therefore, the 3DNMS non-maximum suppression algorithm is also required for deduplication.

[0040] In this embodiment, step 104 is mainly used to obtain the detection results of 3D lesions. To minimize the impact of false positives on detection performance, this embodiment sets up a false positive identification model to filter false positives in the 3D lesion results output by the detection model. The structure of the false positive identification model is as follows: Figure 2As shown, the false positive identification model is a binary classifier, which can be implemented by a convolutional neural network (such as 3D ResNet-18). During model training, the positive inputs are the 3D coordinate bounding box features of the lesion in CT, and the negative inputs are random 3D coordinate bounding box features of other non-lesion areas. Specifically, assuming the coordinates of a lesion in a certain phase of CT are (z, y, x), where z represents the number of slice layers spanned, the lesion is cut out from the CT image to obtain a local image of the lesion in that phase. Similarly, local images of the lesion in other phases are also cut out, and the 3D lesion images of all phases are used together as positive samples for the false positive identification model; for images outside the lesion area, a fixed-size 3D bounding box is randomly cut from the images of each phase to obtain negative samples for model training. After the false positive identification model is trained, the local images of the 3D lesions segmented in each phase output by the detection model can be directly used as input to judge false positives of the 3D lesions and filter false positives to obtain the final detection result of the 3D lesions.

[0041] As an optional embodiment, the multiple phases include a plain scan phase, an arterial phase, a portal venous phase, and a delayed phase.

[0042] This embodiment illustrates four specific phases of CT images: the plain scan phase, the arterial phase, the portal venous phase, and the delayed phase. Existing techniques generally only use the plain scan phase for lesion feature extraction and detection, failing to fully utilize lesion information from other phases. This embodiment processes CT images across all four phases simultaneously, effectively improving lesion detection accuracy.

[0043] As an optional embodiment, the method further includes adjusting the window width and window level of the CT images after acquiring multi-phase CT images to highlight the features of the liver region.

[0044] This embodiment presents a technical solution for preprocessing multi-phase CT images. The preprocessing mainly involves adjusting the window width and window level of the CT images to highlight the characteristics of the liver region, which helps the model learn the features of liver lesions.

[0045] As an optional embodiment, the 3D data dimension constructed with each slice layer to be detected as the center is (2m+1,h,w), where m is the number of layers above and below the slice layer to be detected, m>1, and h and w are the height and width of the image, respectively.

[0046] This embodiment provides a specific structure for 3D data. The 3D data has dimensions (2m+1, h, w). The first dimension represents the number of layers. The central layer is the slice to be detected, and there are m consecutive layers above and below the central layer, for a total of 2m+1 layers. m>1. The value of m cannot be too small, otherwise it will be difficult to effectively obtain contextual features, but it cannot be too large either. Generally, m=5 can be used. The other two dimensions, h and w, are the height and width of the image, respectively.

[0047] As an optional embodiment, the training data of the detection model includes positive sample 3D data and negative sample 3D data. The central layer of the positive sample 3D data is a slice with lesion annotation, and the central layer of the negative sample 3D data is a slice without lesion annotation.

[0048] This embodiment describes the types of 3D data used for training the detection model. The construction method of 3D data in the training phase is basically the same as that in the prediction phase, except that positive sample 3D data and negative sample 3D data need to be constructed separately in the training phase. The construction method is as follows: For any phase CT, the slice layer with lesion annotation is used as the positive sample layer, and the other slice layers are used as the negative sample layers. Assume that the total number of slices is N, and the number of positive sample slices is Np. Taking each positive sample slice as the center slice, expand the m consecutive slices above and below it to form a 3D positive sample with dimensions (2*m+1,h,w), where h and w are the height and width of a single slice image, and the label corresponding to each positive sample is the 2D lesion box annotated by the center slice. When generating negative samples, firstly, Nn slices without lesions are selected from N-Np slices using random sampling. Then, Nn negative sample slices are generated in the same way to generate Nn 3D negative samples with dimensions (2*m+1,h,w).

[0049] As an optional embodiment, the detection model includes a feature extraction module and a detection module, and also includes a Feature Pyramid Network (FPN) located between the feature extraction module and the detection module, which is used to improve the information fusion between deep features and shallow features and improve the detection capability of small lesions.

[0050] This embodiment presents a technical solution to improve the detection capability of small-sized lesions. The detection model mainly consists of a feature extraction module and a detection module. In this embodiment, to improve the detection capability of small-sized lesions, a feature pyramid network is added between the feature extraction module and the detection module. The feature pyramid network is a feature extractor designed based on the concept of feature pyramids, aiming to improve detection accuracy, especially for small-sized targets. Its principle is as follows: after a series of convolutions, a feature map is obtained. Then, it is upsampled and gradually restored step by step, enlarging the feature map without losing high-level semantic information. Finally, the enlarged feature map is used to detect small targets, thereby solving the problem of the difficulty in detecting small targets.

[0051] As an optional embodiment, the bounding rectangle of the 2D lesion is used as its detection box, and the detection box data is (x,y,h1,w1), where x and y are the horizontal and vertical coordinates of the upper left corner of the detection box, respectively, and h1 and w1 are the height and width of the detection box, respectively.

[0052] This embodiment defines the detection frame for 2D lesions. Since the shape of the lesion is irregular, this embodiment uses the area occupied by the bounding rectangle of the 2D lesion as the area occupied by the 2D lesion; this bounding rectangle is called the detection frame of the 2D lesion. The 2D lesion constituting the 3D lesion is actually the detection frame of the 2D lesion. The position and size of the 2D lesion are also represented by the data parameters of the detection frame, including the coordinates of the upper left corner of the detection frame and the length and width of the detection frame.

[0053] As an optional embodiment, the method for determining whether two consecutive 2D lesions belong to the same 3D lesion includes: if the overlap ratio (IOU) between the detection frames of two consecutive 2D lesions is greater than a set threshold, then the two consecutive lesions belong to the same 3D lesion, and the formula for calculating the overlap ratio is:

[0054]

[0055] In the formula, |A| and |B| are the areas of the two detection boxes, respectively, and |A∩B| is the area of ​​the overlapping part of the two detection boxes.

[0056] This embodiment provides a technical solution for determining whether two consecutive 2D lesions belong to the same 3D lesion. As mentioned above, the similarity between two consecutive 2D lesion images can be used to determine whether they belong to the same lesion. To simplify the calculation, this embodiment uses detection boxes instead of the corresponding 2D lesion images, and determines whether they belong to the same lesion based on the degree of overlap between the two detection boxes. The calculation formula is as above, where the numerator is twice the area of ​​the overlapping portion of the two detection boxes, and the denominator is the sum of the areas of the two detection boxes. The areas of the two detection boxes and their overlapping area can be easily obtained from their coordinates and dimensions, thus yielding the degree of overlap.

[0057] Figure 3 This is a schematic diagram of the composition of a 3D liver lesion detection device according to an embodiment of the present invention. The device includes:

[0058] The 3D data construction module 11 is used to acquire multi-phase CT images to be detected and construct 3D data centered on each slice layer to be detected, including multiple slices above and below it.

[0059] The 2D lesion detection module 12 inputs 3D data into the detection model, performs multi-size feature extraction, obtains the 3D context features of the image, obtains the 2D lesion features of the central slice layer based on the central cutting and global attention mechanism, and then obtains the probability that the 2D lesion is positive.

[0060] The 3D lesion synthesis module 13 is used to combine 2D lesions from continuous slices belonging to the same lesion into a 3D lesion. The probability of a 3D lesion being positive is equal to the fusion of the probabilities of all 2D lesions being positive.

[0061] The false positive filtering module 14 is used to input the local images of the 3D lesion output by the detection model in multiple phase segments into a false positive recognition model to filter the false positives of the 3D lesion and obtain the detection result of the 3D lesion.

[0062] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the illustrated method embodiments are similar in principle and technical effect, and will not be described again here. The same applies to the subsequent embodiments, which will not be elaborated upon further.

[0063] As an optional embodiment, the multiple phases include a plain scan phase, an arterial phase, a portal venous phase, and a delayed phase.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A 3D liver lesion detection method, characterized in that, Includes the following steps: Acquire multi-phase CT images of the target slice and construct 3D data centered on each target slice layer, including multiple slices above and below it; 3D data is input into the detection model to extract multi-size features and obtain the 3D context features of the image. Based on the central cutting and global attention mechanism, the 2D lesion features of the central slice layer are obtained, and then the probability of the 2D lesion being positive is obtained. 2D lesions belonging to the same lesion in continuous slices are combined into a 3D lesion. The probability of a 3D lesion being positive is equal to the sum of the probabilities of all 2D lesions being positive. The local images of the 3D lesions output by the detection model, segmented at multiple phases, are input into a false positive identification model to filter out false positives in the 3D lesions and obtain the detection results of the 3D lesions. The detection model includes a feature extraction module and a detection module, and also includes a Feature Pyramid Network (FPN) located between the feature extraction module and the detection module. This network is used to improve the information fusion between deep and shallow features and enhance the detection capability of small lesions. The FPN obtains a feature map after a series of convolutions, and then it is upsampled and restored step by step to enlarge the feature map without losing high-level semantic information. Finally, the detection module uses the enlarged feature map to detect small targets.

2. The 3D liver lesion detection method according to claim 1, characterized in that, The multiple phases include the plain scan phase, the arterial phase, the portal venous phase, and the delayed phase.

3. The 3D liver lesion detection method according to claim 1, characterized in that, The method further includes adjusting the window width and window level of the CT images after acquiring multi-phase CT images to highlight the features of the liver region.

4. The 3D liver lesion detection method according to claim 1, characterized in that, The 3D data dimension constructed with each slice layer to be detected as the center is (2m+1, h, w), where m is the number of layers above and below the slice layer to be detected, m>1, and h and w are the height and width of the image, respectively.

5. The 3D liver lesion detection method according to claim 4, characterized in that, The training data for the detection model includes positive sample 3D data and negative sample 3D data. The central layer of the positive sample 3D data is a slice with lesion annotations, while the central layer of the negative sample 3D data is a slice without lesion annotations.

6. The 3D liver lesion detection method according to claim 1, characterized in that, The bounding rectangle of the 2D lesion is used as its detection box, and the detection box data is (x, y, h1, w1), where x and y are the horizontal and vertical coordinates of the upper left corner of the detection box, respectively, and h1 and w1 are the height and width of the detection box, respectively.

7. The 3D liver lesion detection method according to claim 6, characterized in that, The method for determining whether two consecutive 2D lesions belong to the same 3D lesion includes: if the Intersection over Union (IOU) between the detection frames of two consecutive 2D lesions is greater than a set threshold, then the two consecutive lesions belong to the same 3D lesion. The formula for calculating the IOU is: In the formula, , These represent the areas of the two detection boxes. This is the area of ​​the overlapping portion of the two detection frames.

8. A 3D liver lesion detection device, characterized in that, include: The 3D data construction module is used to acquire multi-phase CT images to be detected and construct 3D data centered on each slice layer to be detected, including multiple slices above and below it. The 2D lesion detection module inputs 3D data into the detection model, performs multi-size feature extraction, obtains the 3D contextual features of the image, obtains the 2D lesion features of the central slice layer based on the central cutting and global attention mechanism, and then obtains the probability that the 2D lesion is positive. The 3D lesion synthesis module is used to combine 2D lesions from consecutive slices belonging to the same lesion into a 3D lesion. The probability of a 3D lesion being positive is equal to the fusion of the probabilities of all 2D lesions being positive. The false positive filtering module is used to input the local images of the 3D lesions output by the detection model in multiple phases into a false positive recognition model to filter the false positives of the 3D lesions and obtain the detection results of the 3D lesions. The detection model includes a feature extraction module and a detection module, and also includes a Feature Pyramid Network (FPN) located between the feature extraction module and the detection module. This network is used to improve the information fusion between deep and shallow features and enhance the detection capability of small lesions. The FPN obtains a feature map after a series of convolutions, and then it is upsampled and restored step by step to enlarge the feature map without losing high-level semantic information. Finally, the detection module uses the enlarged feature map to detect small targets.

9. The 3D liver lesion detection device according to claim 8, characterized in that, The multiple phases include the plain scan phase, the arterial phase, the portal venous phase, and the delayed phase.

Images