A pulmonary embolism detection model training method, device, equipment and storage medium
A weakly supervised training method using examination-level and image-level labels addresses the high cost and complexity of pulmonary embolism detection by eliminating the need for lesion-level annotations, enhancing detection accuracy and efficiency.
Patent Information
- Application Number
- CN202110014028.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-01-06
AI Technical Summary
The high cost and complexity of manually annotating lung CT images for pulmonary embolism detection using deep learning models, coupled with the need for large numbers of expert-annotated samples, hinder efficient and accurate detection.
A weakly supervised training method using examination-level and image-level labels, eliminating the need for lesion-level annotations, and employing a multi-instance learning approach to train a pulmonary embolism detection model.
Reduces labeling costs and improves detection accuracy by leveraging examination-level and image-level labels, enabling effective pulmonary embolism detection without requiring labor-intensive lesion-level annotations.
Smart Images

Figure CN114723654B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of machine learning technology, and in particular to a pulmonary embolism detection model training method, device, equipment and storage medium. Background Art
[0002] Pulmonary embolism (PE) is a fatal disease caused by emboli (most commonly blood clots) that block the pulmonary blood vessels. In mild cases, breathing becomes tense and painful, while in severe cases, life is in danger. Currently, CT pulmonary angiography is the main method for evaluating patients with suspected pulmonary embolism, but a CT scan usually contains hundreds of images, and doctors need to carefully examine them to find the emboli blocking the blood vessels and further evaluate the condition.
[0003] With the development of computer artificial intelligence, computer-aided diagnosis systems are increasingly used in clinical diagnosis, which can reduce the workload of doctors and improve diagnostic efficiency and accuracy.
[0004] However, there are few studies on pulmonary embolism detection algorithms based on deep learning. The main reason is the lack of training samples with lesion annotations. The existing detection algorithms use a supervised learning framework, and the training samples need to be manually annotated. For the detection task, the location of the specific lesion needs to be annotated in the CT image. The annotation work is cumbersome, and a CT sequence contains 200-500 images, and it takes a long time to check the annotations layer by layer. In addition, the number of parameters of the deep learning model is often millions. In order to prevent overfitting, a large number of training samples are required, that is, to train an effective deep learning detection model, a large number of manually annotated samples are required. In order to ensure the quality of the annotation, the annotator needs to be an experienced radiologist, and a multi-blind annotation strategy should be adopted to eliminate doctor preference and reduce missed labels. Therefore, lesion-level annotation of lung CT images is extremely difficult and requires high requirements for the annotator, that is, the annotation cost is very high. Summary of the invention
[0005] The present invention provides a pulmonary embolism detection model training method, device, equipment and storage medium to reduce the labeling cost and improve the detection accuracy of the pulmonary embolism detection model.
[0006] In a first aspect, an embodiment of the present invention provides a pulmonary embolism detection model training method, comprising:
[0007] Acquire a lung CT image sequence sample, wherein the lung CT image sequence sample includes a plurality of lung CT image samples obtained from a single CT examination, wherein the lung CT image sequence sample is annotated with an examination-level label, wherein the lung CT image sample is annotated with an image-level label, wherein the examination-level label is used to indicate whether the lung CT image sequence sample has pulmonary embolism, and wherein the image-level label is used to indicate whether the lung CT image sample has pulmonary embolism;
[0008] Inputting the lung CT image sequence samples into the pulmonary embolism detection model for processing, and obtaining a probability value that the candidate region represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism;
[0009] Calculating the loss of the lung CT image sequence samples as a first loss value based on the probability value corresponding to each spatial point in the lung CT image sequence samples and the inspection-level label;
[0010] Calculating the loss of the lung CT image sample as a second loss value based on the probability value corresponding to each spatial point in the lung CT image sample and the image-level label;
[0011] Parameters of the pulmonary embolism detection model are updated based on the first loss value and the second loss value.
[0012] In a second aspect, an embodiment of the present invention further provides a pulmonary embolism detection method, wherein a pulmonary embolism detection model trained by the pulmonary embolism detection model training method provided in the first aspect of the present invention is used, comprising:
[0013] Acquire a lung CT image sequence, wherein the lung CT image sequence includes a plurality of lung CT images obtained from a single CT examination;
[0014] Inputting the lung CT image sequence into the pulmonary embolism detection model for processing to obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence is a pulmonary embolism;
[0015] A prediction box is marked for the candidate region represented by the spatial point with the probability value greater than a preset value in the lung CT image sequence, wherein the prediction box is used to mark the position of the pulmonary embolism on the acquired lung CT image.
[0016] In a third aspect, an embodiment of the present invention further provides a pulmonary embolism detection model training device, comprising:
[0017] A sample acquisition module, used for acquiring lung CT image sequence samples, wherein the lung CT image sequence samples include multiple lung CT image samples obtained from a single CT examination, wherein the lung CT image sequence samples are annotated with an examination-level label, wherein the lung CT image samples are annotated with an image-level label, wherein the examination-level label is used to indicate whether the lung CT image sequence samples have pulmonary embolism, and wherein the image-level label is used to indicate whether the lung CT image sequence samples have pulmonary embolism;
[0018] A first probability value acquisition module is used to input the lung CT image sequence samples into the pulmonary embolism detection model for processing, and obtain a probability value that the candidate area represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism;
[0019] A first loss value calculation module, used to calculate the loss of the lung CT image sequence samples as a first loss value based on the probability value corresponding to each spatial point in the lung CT image sequence samples and the inspection level label;
[0020] A second loss value calculation module, used to calculate the loss of the lung CT image sample as a second loss value based on the probability value corresponding to each spatial point in the lung CT image sample and the image-level label;
[0021] A parameter updating module is used to update the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value.
[0022] In a fourth aspect, an embodiment of the present invention further provides a pulmonary embolism detection device, comprising:
[0023] An image sequence acquisition module, used to acquire a lung CT image sequence, wherein the lung CT image sequence includes a plurality of lung CT images obtained from a single CT examination;
[0024] A second probability value acquisition module is used to input the lung CT image sequence into the pulmonary embolism detection model for processing, and obtain a probability value that the candidate area represented by each spatial point in the lung CT image sequence is a pulmonary embolism;
[0025] A prediction box labeling module is used to label prediction boxes for candidate areas represented by spatial points in the lung CT image sequence whose probability values are greater than a preset value, and the prediction boxes are used to mark the positions of the pulmonary emboli on the acquired lung CT images.
[0026] In a fifth aspect, an embodiment of the present invention further provides a computer device, including:
[0027] one or more processors;
[0028] A storage device for storing one or more programs;
[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the pulmonary embolism detection model training method as described in the first aspect of the present invention, or implement the pulmonary embolism detection method as described in the second aspect of the present invention.
[0030] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pulmonary embolism detection model training method as described in the first aspect of the present invention, or implements the pulmonary embolism detection method as described in the second aspect of the present invention.
[0031] The pulmonary embolism detection model training method provided by the embodiment of the present invention includes: obtaining a pulmonary CT image sequence sample, inputting the pulmonary CT image sequence sample into the pulmonary embolism detection model for processing, obtaining a probability value that the candidate area represented by each spatial point in the pulmonary CT image sequence sample is a pulmonary embolism, calculating the loss of the pulmonary CT image sequence sample as a first loss value based on the probability value corresponding to each spatial point in the pulmonary CT image sequence sample and the inspection-level label, calculating the loss of the pulmonary CT image sample as a second loss value based on the probability value corresponding to each spatial point in the pulmonary CT image sample and the image-level label, and updating the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value. The weakly supervised training method based on multi-instance learning is used to train the pulmonary embolism detection model, and only inspection-level labeled samples and image-level labeled samples are needed to train the pulmonary embolism detection model, and no lesion-level labeled samples are needed, thereby reducing the labeling cost. In addition, based on the characteristics of doctors writing imaging reports, by using the examination-level labels of the entire lung examination (lung CT image sequence samples) and the image-level labels of a single lung CT image sample, the pulmonary embolism detection model is trained with multi-layer labels, which can improve the detection accuracy of the pulmonary embolism detection model. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A flowchart of a pulmonary embolism detection model training method provided in Example 1 of the present invention;
[0033] Figure 2A , Figure 2B is a flow chart of a pulmonary embolism detection model training method provided in Embodiment 2 of the present invention;
[0034] Figure 2C A training flow chart of a pulmonary embolism detection model provided in an embodiment of the present invention;
[0035] Figure 3A A pulmonary embolism detection method provided in Embodiment 3 of the present invention;
[0036] Figure 3B A diagram showing the prediction results of the pulmonary embolism detection model provided in an embodiment of the present invention;
[0037] Figure 3C A diagram showing the prediction results of the modified pulmonary embolism detection model provided in an embodiment of the present invention;
[0038] Figure 4 A schematic diagram of the structure of a pulmonary embolism detection model training device provided in Embodiment 4 of the present invention;
[0039] Figure 5 A schematic diagram of the structure of a pulmonary embolism detection device provided in Embodiment 5 of the present invention;
[0040] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION
[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0042] Embodiment 1
[0043] Figure 1 This is a flow chart of a pulmonary embolism detection model training method provided in the first embodiment of the present invention. This embodiment is applicable to the situation that the lung CT image is difficult to annotate during the pulmonary embolism detection model training process, the annotation takes a long time, and the requirements for the annotator are high. The method can be executed by the pulmonary embolism detection model training device provided in the embodiment of the present invention. The device can be implemented by software and / or hardware, and is usually configured in a computer device, such as Figure 1 As shown, the method specifically comprises the following steps:
[0044] S101. Obtain lung CT image sequence samples.
[0045] Specifically, CT (Computed Tomography) images, that is, electronic computer tomography images, use precisely collimated X-ray beams, gamma rays, ultrasound, etc., together with highly sensitive detectors to perform cross-sectional scans around a certain part of the human body one by one. The detector receives the X-rays that pass through the layer, converts them into visible light, and then converts them into electrical signals through photoelectric conversion. Then, they are converted into digital signals through analog / digital converters, and input into computers for processing to obtain CT images. Lung CT images are tomographic images of the chest and lungs.
[0046] In the embodiment of the present invention, the lung CT image sequence sample includes multiple (usually including hundreds) lung CT image samples obtained from a CT examination. For different regions, multiple continuous scans are required to obtain multiple lung CT image samples, and then the multiple lung CT image samples can be reconstructed to obtain a three-dimensional CT image. The three-dimensional CT image can make the original plane image three-dimensional, increase the density difference between the diseased tissue and the adjacent normal tissue, and can more clearly show the condition of the diseased tissue.
[0047] The lung CT image sequence samples are annotated with examination-level labels, that is, for the lung CT image sequence samples obtained from a lung examination, the lung examination is annotated with an examination-level label, and the examination-level label is used to indicate whether the lung CT image sequence samples (i.e., the lung examination) have pulmonary embolism. The lung CT image samples are annotated with image-level labels, that is, each lung CT image sample is annotated with an image-level label, and the image-level label is used to indicate whether the lung CT image sample has pulmonary embolism.
[0048] During the diagnosis process, the doctor will record in the imaging report (i.e., for this lung examination) whether the patient is diagnosed with pulmonary embolism, as well as the image number of the lung CT image of pulmonary embolism in the lung CT image sequence obtained during this lung examination, that is, in reality, it is relatively easy to obtain inspection-level labeled samples and image-level labeled samples. The embodiment of the present invention only needs to use inspection-level labeled samples and image-level labeled samples to train the pulmonary embolism detection model, and does not require lesion-level labeled samples, thereby reducing the labeling cost.
[0049] S102: Input the lung CT image sequence samples into the pulmonary embolism detection model for processing to obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism.
[0050] The acquired CT image sequence samples are input into the initialized pulmonary embolism detection model for processing, and the probability value of the candidate region represented by each spatial point in the pulmonary CT image sequence samples is pulmonary embolism is obtained. In the embodiment of the present invention, the spatial point can be a pixel point in the pulmonary CT image sample, and each spatial point can represent a candidate region centered on the spatial point, and the size of the candidate region can be pre-set based on the prior knowledge of the usual size of pulmonary embolism.
[0051] Exemplarily, the pulmonary embolism detection model can perform convolution, pooling, and fusion operations on the samples of the lung CT image sequence, extract features from the candidate regions represented by each spatial point, obtain a feature map, and map the feature map to the (0,1) interval via a classifier to obtain the probability value of the candidate region represented by each spatial point being a pulmonary embolism. Among them, the network structure of the pulmonary embolism detection model can use 3D-resnet-18 as the backbone network (backbone), extract features from the samples of the lung CT image sequence, and then use the feature pyramid network (Feature Pyramid Networks) to fuse the feature map output by the high-level network of 3D-resnet-18 with the feature map output by the low-level network to obtain a fused feature map, and then extract features from the candidate region represented by each spatial point in the fused feature map to obtain the final feature map. Then, the feature map is mapped to the (0,1) interval via a classifier to obtain the probability value of the candidate region represented by each spatial point being a pulmonary embolism. 3D-resnet-18 is rewritten from resnet-18 into a 3D form. Resnet-18 is used for object detection in 2D images. By rewriting the number of channels and the dimension of the convolution kernel of resnet-18 into 3 dimensions, 3D-resnet-18 is obtained.
[0052] It should be noted that the network structure of the above-mentioned pulmonary embolism detection model is an exemplary description of the embodiment of the present invention and is not a limitation. In other embodiments of the present invention, the network structure of the pulmonary embolism detection model may also adopt other structures. For example, the backbone network may also adopt 3D-resnet-34, which is not limited in the embodiment of the present invention.
[0053] S103: Calculate the loss of the lung CT image sequence samples based on the probability value corresponding to each spatial point in the lung CT image sequence samples and the inspection-level label as a first loss value.
[0054] In an embodiment of the present invention, the loss of the lung CT image sequence sample is calculated as the first loss value based on the probability value corresponding to each spatial point in the lung CT image sequence sample and the inspection-level label. Exemplarily, in some embodiments of the present invention, a first attention coefficient (or a first weight) can be configured for the probability value corresponding to each spatial point, and the first attention coefficient represents the percentage of the candidate region represented by the spatial point that represents the amount of information about pulmonary embolism. The larger the first attention coefficient, the more the candidate region can represent pulmonary embolism. Then, the probability values configured with the first attention coefficient are added to obtain the probability value of the presence of pulmonary embolism in the lung CT image sequence sample (i.e., this lung examination), and then the cross entropy loss of the prediction result of the lung CT image sequence sample (i.e., this lung examination) and the inspection-level label is calculated based on the probability value of the presence of pulmonary embolism in the lung CT image sequence sample (i.e., this lung examination) as the first loss value.
[0055] S104. Calculate the loss of the lung CT image sample as a second loss value based on the probability value and image-level label corresponding to each spatial point in the lung CT image sample.
[0056] In an embodiment of the present invention, the loss of the lung CT image sample is calculated as the second loss value based on the probability value and image-level label corresponding to each spatial point in the lung CT image sample. Exemplarily, in some embodiments of the present invention, a second attention coefficient (or second weight) can be configured for the probability value corresponding to each spatial point in the lung CT image sample, and the second attention coefficient represents the percentage of the candidate area represented by the spatial point that represents the amount of information about pulmonary embolism. The larger the second attention coefficient, the more the candidate area can represent pulmonary embolism. Then, the probability values configured with the second attention coefficient are added to obtain the probability value of the presence of pulmonary embolism in the lung CT image sample, and then the cross entropy loss of the prediction result of the lung CT image sample and the image-level label is calculated based on the probability value of the presence of pulmonary embolism in the lung CT image sample as the second loss value.
[0057] S105. Update parameters of the pulmonary embolism detection model based on the first loss value and the second loss value.
[0058] In an embodiment of the present invention, the sum of the second loss values corresponding to each lung CT image sample in the lung CT image sequence samples is calculated, and then the sum is added to the first loss value to obtain a total loss value, and the parameters of the pulmonary embolism detection model are updated based on the total loss. Exemplarily, in some embodiments of the present invention, the total loss value is compared with a preset value, and when the total loss value is greater than the preset value, the parameters of the pulmonary embolism detection model are updated, and the step of obtaining the lung CT image sequence samples is returned to train the pulmonary embolism detection model again until the total loss value is less than or equal to the preset value, and it is determined that the embolism detection model training is completed.
[0059] In an embodiment of the present invention, a weakly supervised training method based on multiple-instance learning (MIL) is used to train a pulmonary embolism detection model. Only inspection-level labeled samples and image-level labeled samples are needed to train the pulmonary embolism detection model. No lesion-level labeled samples are needed, which reduces the labeling cost. Multiple-instance learning is different from supervised learning, semi-supervised learning, and unsupervised learning. It is a learning problem with multiple-instance bags as training units, and each bag contains multiple instances. In multiple-instance learning, the training set consists of a group of multiple-instance bags with classification tags, and each bag contains several instances without classification tags. If a multiple-instance bag contains at least one positive example, the bag is marked as a positive multiple-instance bag (positive bag). In an embodiment of the present invention, a lung CT image sequence sample and a lung CT image sample obtained from a lung examination are used as a bag, and the pulmonary embolism contained in the lung CT image sample is used as an example. The lung CT image sequence samples are annotated with examination-level labels, and the lung CT image samples are annotated with image-level labels, but the lung CT image samples lack lesion-level labels that identify the distance and location of pulmonary embolism.
[0060] Furthermore, in an embodiment of the present invention, based on the characteristics of doctors writing imaging reports, the examination-level labels of the entire lung examination (lung CT image sequence samples) and the image-level labels of a single lung CT image sample are used simultaneously, and the pulmonary embolism detection model is trained by using multi-layer labels. The detection accuracy of the pulmonary embolism detection model can be improved.
[0061] The pulmonary embolism detection model training method provided by the embodiment of the present invention includes: obtaining a pulmonary CT image sequence sample, inputting the pulmonary CT image sequence sample into the pulmonary embolism detection model for processing, obtaining a probability value that the candidate area represented by each spatial point in the pulmonary CT image sequence sample is a pulmonary embolism, calculating the loss of the pulmonary CT image sequence sample as a first loss value based on the probability value corresponding to each spatial point in the pulmonary CT image sequence sample and the inspection-level label, calculating the loss of the pulmonary CT image sample as a second loss value based on the probability value corresponding to each spatial point in the pulmonary CT image sample and the image-level label, and updating the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value. The weakly supervised training method based on multi-instance learning is used to train the pulmonary embolism detection model, and only inspection-level labeled samples and image-level labeled samples are needed to train the pulmonary embolism detection model, and no lesion-level labeled samples are needed, thereby reducing the labeling cost. In addition, based on the characteristics of doctors writing imaging reports, by using the examination-level labels of the entire lung examination (lung CT image sequence samples) and the image-level labels of a single lung CT image sample, the pulmonary embolism detection model is trained with multi-layer labels, which can improve the detection accuracy of the pulmonary embolism detection model.
[0062] Embodiment 2
[0063] Figure 2A , Figure 2B is a flow chart of a pulmonary embolism detection model training method provided by the second embodiment of the present invention. This embodiment is refined on the basis of the above-mentioned first embodiment, and describes in detail the network structure and processing process of the pulmonary embolism detection model, as well as the specific process of multi-example learning, such as Figure 2A and Figure 2B As shown, the method includes:
[0064] S201. Obtain lung CT image sequence samples.
[0065] In an embodiment of the present invention, the lung CT image sequence samples include multiple (usually including hundreds of) lung CT image samples obtained from a single CT examination. Multiple consecutive scans are required for different areas to obtain multiple lung CT image samples. Subsequently, the multiple lung CT image samples can be reconstructed to obtain a stereoscopic three-dimensional CT image.
[0066] The lung CT image sequence samples are annotated with examination-level labels, that is, for the lung CT image sequence samples obtained from a lung examination, the lung examination is annotated with an examination-level label, and the examination-level label is used to indicate whether the lung CT image sequence samples (i.e., the lung examination) have pulmonary embolism. The lung CT image samples are annotated with image-level labels, that is, each lung CT image sample is annotated with an image-level label, and the image-level label is used to indicate whether the lung CT image sample has pulmonary embolism.
[0067] S202 , down-sampling the lung CT image sequence samples four times in succession to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map in sequence.
[0068] Figure 2C The training flow chart of the pulmonary embolism detection model provided in the embodiment of the present invention. In the embodiment of the present invention, 3D-resnet-18 is used as the backbone network, and the lung CT image sequence samples are continuously downsampled 4 times (or called feature extraction) to obtain the first feature map C1, the second feature map C2, the third feature map C3 and the fourth feature map C4 in turn.
[0069] Specifically, Figure 2CAs shown, firstly, the samples of the lung CT image sequence are sampled in parallel. For example, a convolution operation can be performed on the samples of the lung CT image sequence to obtain an initial feature map C0. The convolution step size of the convolution operation is 1, and the resolution of the samples of the lung CT image sequence is not changed. Next, the initial feature map C0 is downsampled. For example, a convolution operation can be performed on the initial feature map C0 with a convolution step size of 2 to obtain a first feature map C1 with a resolution of half the initial feature map C0. Next, the first feature map C1 is downsampled. For example, a convolution operation can be performed on the first feature map C1 with a convolution step size of 2 to obtain a second feature map C2 with a resolution of half the first feature map C1. Next, the second feature map C2 is downsampled. For example, a convolution operation can be performed on the second feature map C2 with a convolution step size of 2 to obtain a third feature map C3 with a resolution of half the second feature map C2. Next, the third feature map C3 is downsampled. For example, a convolution operation may be performed on the third feature map C3 with a convolution step of 2 to obtain a fourth feature map C4 having a resolution of half that of the third feature map C3.
[0070] In the above embodiment, the convolution method is used as an example to exemplify the downsampling in the embodiment of the present invention. In other embodiments of the present invention, other methods (for example, pooling, etc.) may also be used, and the embodiment of the present invention is not limited here.
[0071] S203: Fuse the fourth feature map with the first feature map, the second feature map and / or the third feature map to obtain a fused feature map.
[0072] In a neural network, the feature map output by the low-level network has a higher resolution and contains more position and detail information, but because it undergoes fewer convolutions, it has lower semantics and more noise. The feature map output by the high-level network has stronger semantic information, but has a very low resolution and poor perception of details. In an embodiment of the present invention, the fourth feature map is fused with the first feature map, the second feature map and / or the third feature map to obtain a fused feature map, which strengthens the expression of the low-level network and is conducive to improving the detection accuracy of the pulmonary embolism detection model.
[0073] Exemplarily, in one embodiment of the present invention, a feature pyramid network is used to fuse the feature map output by the high-level network in 3D-resnet-18 with the feature map output by the low-level network to obtain a fused feature map. Figure 2C , the fusion process is as follows:
[0074] 1. Extract the fifth feature map from the fourth feature map.
[0075] In an embodiment of the present invention, a convolutional network is used to extract features from the fourth feature map C4 to obtain a fifth feature map P4. Exemplarily, in this embodiment, the size of the convolution kernel of the convolutional network is 1×1, the step size is 1, and the resolution of the extracted fifth feature map P4 is the same as the resolution of the fourth feature map C4. During the downsampling process, the number of channels of the obtained features continues to increase. By setting the number of channels of the convolution kernel of the convolutional network to a preset number of channels, a fifth feature map P4 with a preset number of channels can be obtained.
[0076] 2. Upsample the fifth feature map to obtain the sixth feature map.
[0077] In a specific embodiment of the present invention, the fifth feature map P4 is upsampled by transposed convolution (also referred to as deconvolution, fractional-step convolution) to obtain a sixth feature map having the same resolution as the third feature map C3.
[0078] 3. The sixth feature map is merged with the third feature map to obtain the seventh feature map.
[0079] In the embodiment of the present invention, the sixth feature map is added to the corresponding elements of the third feature map C3 to achieve the fusion of the sixth feature map and the third feature map C3, and obtain the seventh feature map P3. Exemplarily, a 1×1 convolution kernel can be used to perform a convolution operation on the third feature map C3, and the number of channels of the convolution kernel is a preset number of channels to obtain a feature map with a preset number of channels, and the feature map is added to the data of the corresponding channels of the sixth feature map with the preset number of channels to obtain the seventh feature map P3.
[0080] 4. Upsample the seventh feature map to obtain the eighth feature map.
[0081] In a specific embodiment of the present invention, the seventh feature map P3 is upsampled by transposed convolution to obtain an eighth feature map having the same resolution as the second feature map C2.
[0082] 5. Fuse the eighth feature map with the second feature map to obtain a fused feature map.
[0083] In an embodiment of the present invention, the eighth feature map is added to the corresponding elements of the second feature map C2 to achieve fusion of the eighth feature map and the second feature map C2 to obtain a fused feature map P2. Exemplarily, a 1×1 convolution kernel can be used to perform a convolution operation on the second feature map C2, and the number of channels of the convolution kernel is a preset number of channels to obtain a feature map with a preset number of channels, and the feature map is added to the data of the corresponding channels of the eighth feature map with the preset number of channels to obtain the fused feature map P2.
[0084] It should be noted that, in the above embodiment, the fourth feature map C4 output by the high-level network is fused with the second feature map C2 and the third feature map C3 output by the low-level network as an example to illustrate the embodiment of the present invention. In other embodiments of the present invention, the fourth feature map C4 may be fused with the second feature map C2 and the first feature map C1, or the fourth feature map C4 may be fused with the third feature map C3 and the first feature map C1, or the fourth feature map C4 may be fused with the first feature map C1, the second feature map C2, or the third feature map C3. The embodiment of the present invention is not limited here. In addition, the above embodiment uses the transposed convolution method as an example to implement upsampling. In other embodiments of the present invention, other methods (for example, uppooling or linear interpolation) may also be used to implement upsampling, which is not limited here by the embodiment of the present invention.
[0085] S204, extracting features from the candidate region represented by each spatial point in the fused feature map to obtain a target feature map.
[0086] In an embodiment of the present invention, the spatial points in the fused feature map can be the mapping of the pixel points in the lung CT image sample in the fused feature map P2, and each spatial point can represent a candidate area centered on the spatial point, and the size of the candidate area can be pre-set based on the prior knowledge of the usual size of pulmonary embolism. In the prior art, methods such as selective search are usually used to extract candidate areas, which has a large amount of calculation and low efficiency. In an embodiment of the present invention, by using a windowing method on the feature map, the size of the candidate area is pre-set based on prior knowledge, and each spatial point represents a candidate area, the extraction speed of the candidate area is improved and the amount of calculation is reduced.
[0087] In the embodiment of the present invention, a convolutional network can be used to extract features from the candidate region represented by each spatial point in the fusion feature map P2 to obtain the target feature map X f For example, in a specific embodiment of the present invention, four 3D-RESNET blocks are used as detection and classification heads. While keeping the spatial resolution unchanged, the features of the area represented by each point in the fusion feature map P2 are further extracted to generate the target feature map X f .
[0088] S205 , classify the candidate region represented by each spatial point in the target feature map to obtain a probability value that the candidate region represented by each spatial point is pulmonary embolism.
[0089] In the embodiment of the present invention, the target feature map X fThe spatial points in the fusion feature map P2 can be the mapping of the pixel points in the lung CT image sample. Each spatial point can represent a candidate region centered on the spatial point. The size of the candidate region can be pre-set based on the prior knowledge of the usual size of pulmonary embolism.
[0090] In this embodiment of the present invention, the target feature map X f The candidate region represented by each spatial point in the input classifier is classified to obtain the probability value of the candidate region represented by each spatial point being pulmonary embolism. For example, in a specific embodiment of the present invention, a convolution network with a convolution kernel of 1*1*1 is used to classify the target feature map X f The candidate regions represented by the mid-space points are classified to obtain the target feature map X f The probability value of whether each spatial point in the candidate area represents pulmonary embolism is obtained, and the probability matrix X c .
[0091] S206 , selecting the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sequence samples as the first target probability value.
[0092] Since the probability values corresponding to the top K spatial points in terms of probability value are often larger than the probability values of all the remaining spatial points, we focus on calculating the loss values of the candidate areas represented by the spatial points corresponding to the top K probability values, thereby improving the calculation efficiency while ensuring the calculation accuracy.
[0093] In an embodiment of the present invention, a top K algorithm may be used to select the largest top K probability values from the probability values corresponding to each spatial point in the lung CT image sequence sample as the first target probability value. Specifically, the top K algorithm may include global sorting, local sorting, heap, random selection, divide-and-conquer or reduction-and-conquer, etc., which are not limited in the present invention. Exemplarily, in a specific embodiment of the present invention, the top K algorithm is used to select the largest top 5 probability values from the probability values corresponding to each spatial point in the lung CT image sequence sample as the first target probability value X. c-exam , and obtain a probability matrix including 5 first target probability values.
[0094] S207. Configure a first attention coefficient for the first target probability value.
[0095] Among them, the first attention coefficient represents the first target probability value in the target feature map X f The candidate region represented by the corresponding spatial point in the graph represents the percentage of pulmonary embolism information. The larger the first attention coefficient is, the more the candidate region can represent pulmonary embolism.
[0096] Exemplarily, in one embodiment of the present invention, the process of configuring the first attention coefficient is as follows:
[0097] 1. Extract a first feature vector from the features in the candidate area corresponding to the first target probability value in the target feature map.
[0098] In the embodiment of the present invention, the first target probability value is in the target feature map X f The first feature vector is extracted from the features of the corresponding candidate area in . Exemplarily, in a specific embodiment of the present invention, after obtaining 5 first target probability values X c-exam Then, according to the five first target probability values X c-exam Reversely look up the 5 first target probability values in the target feature map X f The corresponding candidate regions are selected, and then the features in these five candidate regions are input into the fully convolutional network for processing, and five first feature vectors F1 are extracted from the features in these five candidate regions.
[0099] 2. Map the first eigenvector to the interval (0,1) to obtain the first attention coefficient C1 corresponding to the first target probability value.
[0100] Specifically, the five first feature vectors F1 are processed by the softmax function layer respectively, and the first feature vector F1 is mapped to the interval (0,1) to obtain five first target probability values X c-exam The corresponding 5 first attention coefficients C1.
[0101] S208. Multiply the first target probability value by the first attention coefficient to obtain a first value.
[0102] The first target probability value is multiplied by the first attention coefficient to obtain a first value. In a specific embodiment of the present invention, the five first target probability values X c-exam Multiply them by the corresponding first attention coefficient C1 to get 5 first values X o-exam .
[0103] S209: Add the first numerical values corresponding to the first target probability values to obtain a probability value of the presence of pulmonary embolism in the lung CT image sequence samples.
[0104] Exemplarily, in the embodiment of the present invention, the five first values X o-exam Add together to get the probability value y of the presence of pulmonary embolism in the lung CT image sequence sample (i.e., this lung examination) exam .
[0105] S210 , calculating a cross entropy loss as a first loss value based on the probability of the presence of pulmonary embolism in the lung CT image sequence samples and the inspection-level labels.
[0106] In the embodiment of the present invention, the cross entropy loss is calculated based on the probability of pulmonary embolism in the lung CT image sequence samples and the inspection-level label as the first loss value. The calculation formula of the first loss is as follows:
[0107] L1=-[y1·log(y exam )+(1-y1)·log(1-y exam )]
[0108] Among them, L1 is the cross entropy loss of the lung CT image sequence sample (i.e., this lung examination), y1 is the inspection-level label of the lung CT image sequence sample, the inspection-level label of the positive sample (i.e., the sample with pulmonary embolism) is 1, and the inspection-level label of the negative sample (i.e., the sample without pulmonary embolism) is 0.
[0109] S211. Select the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sample as the second target probability value.
[0110] As mentioned above, since the probability values corresponding to the top K spatial points in terms of probability value are often larger than the probability values of all the remaining spatial points, the focus is on calculating the loss values of the candidate areas represented by the spatial points corresponding to the top K probability values, thereby improving the calculation efficiency while ensuring the calculation accuracy.
[0111] In an embodiment of the present invention, a top K algorithm may be used to select the largest top K probability values from the probability values corresponding to each spatial point in the lung CT image sample as the second target probability value. Specifically, the top K algorithm may include global sorting, local sorting, heap, random selection, divide-and-conquer or reduction-and-conquer, etc., which are not limited in the present invention. Exemplarily, in a specific embodiment of the present invention, the top K algorithm is used to select the largest top 5 probability values from the probability values corresponding to each spatial point in the lung CT image sample as the second target probability value X. c-image , a probability matrix including five second target probability values is obtained. Accordingly, each lung CT image sample in the lung CT image sequence sample is processed in the same way to obtain multiple probability matrices.
[0112] S212. Configure a second attention coefficient for the second target probability value.
[0113] Among them, the second attention coefficient represents the second target probability value in the target feature map X f The candidate region represented by the corresponding spatial point in represents the percentage of pulmonary embolism information. The larger the second attention coefficient is, the more the candidate region can represent pulmonary embolism.
[0114] Exemplarily, in one embodiment of the present invention, the process of configuring the second attention coefficient is as follows:
[0115] 1. Extract a second feature vector from the features in the candidate area corresponding to the second target probability value in the target feature map.
[0116] In this embodiment of the present invention, the second target probability value is in the target feature map X f The second feature vector is extracted from the features of the corresponding candidate area in . Exemplarily, in a specific embodiment of the present invention, after obtaining 5 second target probability values X c-image Then, according to the five second target probability values X c-image Reverse search for these 5 second target probability values X c-image In the target feature map X f The corresponding candidate regions are selected, and then the features in these five candidate regions are input into the fully convolutional network for processing, and five second feature vectors F2 are extracted from the features in these five candidate regions.
[0117] 2. Map the second eigenvector to the interval (0, 1) to obtain the second attention coefficient corresponding to the second target probability value.
[0118] Specifically, the five second feature vectors F2 are processed by the softmax function layer respectively, and the second feature vector F2 is mapped to the interval (0,1) to obtain five second target probability values X c-image The corresponding 5 second attention coefficients C2.
[0119] S213. Multiply the second target probability value by the second attention coefficient to obtain a second value.
[0120] The second target probability value is multiplied by the second attention coefficient to obtain a second value. c-image Multiply them by the corresponding second attention coefficient C2 to get 5 second values X o-image .
[0121] S214: Add the second numerical values corresponding to the second target probability values to obtain a probability value of the presence of pulmonary embolism in the lung CT image sample.
[0122] Exemplarily, in the embodiment of the present invention, the five second values X o-image Add them together to get the probability value y of the presence of pulmonary embolism in the lung CT image sample image Correspondingly, each lung CT image sample in the lung CT image sequence samples is processed in the same way to obtain the probability value y of the presence of pulmonary embolism in each lung CT image sample image .
[0123] S215. Calculate a cross entropy loss as a second loss value based on the probability of the presence of pulmonary embolism in the lung CT image sample and the image-level label.
[0124] In the embodiment of the present invention, the cross entropy loss is calculated based on the probability of pulmonary embolism in the lung CT image sample and the image-level label as the second loss value. The calculation formula of the second loss is as follows:
[0125]
[0126] Among them, L2 is the cross entropy loss of the i-th lung CT image sample, is the image-level label of the i-th lung CT image sample, the image-level label of the positive sample (i.e., the sample with pulmonary embolism) is 1, and the image-level label of the negative sample (i.e., the sample without pulmonary embolism) is 0. is the probability value of pulmonary embolism in the i-th lung CT image sample predicted by the model.
[0127] S216. Update the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value.
[0128] Specifically, in the embodiment of the present invention, the sum of the second loss values of each lung CT image sample is calculated to obtain the third loss value, and the calculation formula of the third loss value is as follows:
[0129]
[0130] Wherein, N represents the number of lung CT image samples in the lung CT image sequence samples.
[0131] The sum of the first loss value L1 and the third loss value L3 is calculated to obtain a total loss value.
[0132] The total loss value is compared with the preset value. When the total loss value is greater than the preset value, the parameters of the pulmonary embolism detection model are updated, and the step of obtaining the lung CT image sequence samples is returned to train the pulmonary embolism detection model again. When the total loss value is less than or equal to the preset value, it is determined that the embolism detection model training is completed.
[0133] The pulmonary embolism detection model training method provided in the embodiment of the present invention trains the pulmonary embolism detection model based on a weakly supervised training method of multi-example learning. It only needs to use inspection-level labeled samples and image-level labeled samples to train the pulmonary embolism detection model, and no lesion-level labeled samples are required, thereby reducing the labeling cost. In addition, based on the characteristics of doctors writing imaging reports, the inspection-level labels of the entire lung examination (lung CT image sequence samples) and the image-level labels of a single lung CT image sample are used at the same time. By using multi-level labels to train the pulmonary embolism detection model, the detection accuracy of the pulmonary embolism detection model can be improved. Furthermore, the windowing method and the top K strategy are used to improve the extraction speed of the candidate area and reduce the amount of calculation.
[0134] Embodiment 3
[0135] Figure 3A A pulmonary embolism detection method is provided in Embodiment 3 of the present invention. This embodiment can be used for pulmonary embolism detection. The method adopts a pulmonary embolism detection model trained by the pulmonary embolism detection model training method provided in any of the aforementioned embodiments of the present invention. The method can be performed by a pulmonary embolism detection device provided in an embodiment of the present invention. The device can be implemented in software and / or hardware and integrated in a computer device provided in an embodiment of the present invention. Figure 3A As shown, the method specifically comprises the following steps:
[0136] S301. Acquire a lung CT image sequence.
[0137] In an embodiment of the present invention, the lung CT image sequence includes multiple (usually including hundreds of) lung CT images obtained from a single CT examination. For different areas, multiple consecutive scans are required to obtain multiple lung CT images. Subsequently, the multiple lung CT images can be reconstructed to obtain a stereoscopic three-dimensional CT image.
[0138] S302: Input the lung CT image sequence into the pulmonary embolism detection model for processing to obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence is a pulmonary embolism.
[0139] The pulmonary embolism detection model is a pulmonary embolism detection model trained by the pulmonary embolism detection model training method provided by any of the aforementioned embodiments. Specifically, the network structure of the pulmonary embolism detection model and the processing process of the lung CT image sequence can refer to the processing process of the lung CT image sequence samples in the aforementioned embodiment, and the embodiments of the present invention will not be repeated here. The lung CT image sequence is input into the pulmonary embolism detection model for processing, and the probability value of the candidate area represented by each spatial point in the lung CT image sequence being a pulmonary embolism is obtained.
[0140] S303 , marking a prediction box for a candidate region represented by a spatial point having a probability value greater than a preset value in the lung CT image sequence, wherein the prediction box is used to indicate a position of the pulmonary embolism on the acquired lung CT image.
[0141] In the embodiment of the present invention, the probability value of the candidate region represented by each spatial point being pulmonary embolism is compared with a preset value, and when the probability value is greater than the preset value, the candidate region represented by the spatial point is determined to be pulmonary embolism. Then, a prediction box is marked for the candidate region of pulmonary embolism, and the prediction box is used to mark the position of the pulmonary embolism on the acquired lung CT image.
[0142] The pulmonary embolism detection method provided in an embodiment of the present invention adopts a pulmonary embolism detection model trained by the pulmonary embolism detection model training method provided in any of the aforementioned embodiments of the present invention, including: obtaining a lung CT image sequence, the lung CT image sequence including multiple lung CT images obtained from a single CT examination, inputting the lung CT image sequence into the pulmonary embolism detection model for processing, obtaining a probability value that the candidate area represented by each spatial point in the lung CT image sequence is a pulmonary embolism, marking a prediction box for the candidate area represented by the spatial point in the lung CT image sequence whose probability value is greater than a preset value, the prediction box is used to indicate the position of the pulmonary embolism on the acquired lung CT image, thereby improving the detection accuracy of the pulmonary embolism detection model.
[0143] During actual application, the inventors found that although the prediction frame can roughly locate the lesion, it is not accurate and has problems such as center offset or excessive size, resulting in the prediction frame only roughly indicating the lesion location and failing to mark the exact location of the pulmonary embolism. Figure 3B The prediction result diagram of the pulmonary embolism detection model provided in the embodiment of the present invention is as follows: Figure 3B As shown, the center of prediction box 1 is offset to the left, and the prediction box is too large (the small rectangular box in prediction box 1 is the size of the actual pulmonary embolism), and prediction box 2 is also too large (the small rectangular box in prediction box 2 is the size of the actual pulmonary embolism).
[0144] In some embodiments of the present invention, in order to solve the above problem, after marking the prediction box for the candidate area represented by the spatial point whose probability value is greater than the preset value in the lung CT image sequence, the following steps may also be included:
[0145] S304 , extracting a region of interest from the prediction frame of the lung CT image sequence.
[0146] In an embodiment of the present invention, the region of interest (ROI) may be the region where the prediction box is located, and the region where the prediction box is located is captured as the region of interest; the region of interest may also be a region obtained based on a machine vision algorithm, and the region obtained based on the machine vision algorithm is captured from the region where the prediction box is located as the region of interest, which is not limited in the embodiment of the present invention. The capture operation may be based on the coordinate information of the prediction box or the coordinate information of the edge of the region of interest obtained based on the machine vision algorithm. In the field of image processing, the region of interest is an image region selected from an image, which is the focus of subsequent image analysis. Encircling the region for further processing can reduce processing time and increase accuracy.
[0147] S305 , performing double-threshold binarization processing on the region of interest to obtain a binarized image.
[0148] The extracted region of interest is binarized to obtain a binarized image. Exemplarily, in an embodiment of the present invention, the region of interest is binarized with dual thresholds, i.e., threshold values thresh1 and thresh2 are preset, and thresh1 < thresh2, and the grayscales of pixels in the region of interest with grayscale values in the range of thresh1 to thresh2 are all set to 255, and the grayscales of pixels with other grayscale values are all set to 0.
[0149] S306: Perform morphological processing on the binary image to obtain the maximum connected region.
[0150] Specifically, the binary image is subjected to morphological corrosion and dilation processing to segment independent image elements, and adjacent image elements are connected in the image to obtain the maximum connected area, which is regarded as pulmonary embolism.
[0151] The dilation or erosion operation is to convolve the image (or part of the image) with the kernel. The kernel can be of any shape and size, and it has a separately defined reference point, which we call an anchor point. In most cases, the kernel is a small square or disk with a reference point and a solid disk in the middle. We can also regard the kernel as a template or mask. Dilation is the operation of finding the local maximum value. The kernel and the graphic convolution, that is, calculating the maximum value of the pixels in the area covered by the kernel, and assigning this maximum value to the pixel specified by the reference point, which will gradually expand the highlight area in the image. The erosion operation is the opposite of the dilation operation. Erosion is the operation of finding the local minimum value. The kernel and the graphic convolution, that is, calculating the minimum value of the pixels in the area covered by the kernel, and assigning this minimum value to the pixel specified by the reference point, which will gradually shrink the highlight area in the image.
[0152] S307: Label the maximum connected area and generate a corrected prediction box.
[0153] In the embodiment of the present invention, the maximum connected area is re-labeled to generate a revised prediction box. Figure 3C The prediction result diagram of the modified pulmonary embolism detection model provided by the embodiment of the present invention is as follows: Figure 3C As shown, the center offset of the prediction box is smaller and the size is more suitable for the size of the actual pulmonary embolism, that is, the predicted position of the pulmonary embolism is more accurate.
[0154] Embodiment 4
[0155] Figure 4 A schematic diagram of the structure of a pulmonary embolism detection model training device provided in Embodiment 4 of the present invention is shown in FIG. Figure 4 As shown, the pulmonary embolism detection model training device includes:
[0156] The sample acquisition module 401 is used to acquire lung CT image sequence samples, wherein the lung CT image sequence samples include multiple lung CT image samples obtained from a CT examination, wherein the lung CT image sequence samples are annotated with an examination-level label, wherein the lung CT image samples are annotated with an image-level label, wherein the examination-level label is used to indicate whether the lung CT image sequence samples have pulmonary embolism, and wherein the image-level label is used to indicate whether the lung CT image sequence samples have pulmonary embolism;
[0157] A first probability value acquisition module 402 is used to input the lung CT image sequence samples into the pulmonary embolism detection model for processing, and obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism;
[0158] A first loss value calculation module 403, configured to calculate the loss of the lung CT image sequence samples as a first loss value based on the probability value corresponding to each spatial point in the lung CT image sequence samples and the inspection-level label;
[0159] A second loss value calculation module 404, configured to calculate the loss of the lung CT image sample as a second loss value based on the probability value corresponding to each spatial point in the lung CT image sample and the image-level label;
[0160] The parameter updating module 405 is configured to update the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value.
[0161] In some embodiments of the present invention, the first probability value acquisition module 402 includes:
[0162] A downsampling submodule, used for performing downsampling processing on the lung CT image sequence samples for four consecutive times to obtain a first feature map, a second feature map, a third feature map and a fourth feature map in sequence;
[0163] A feature map fusion submodule, used to fuse the fourth feature map with the first feature map, the second feature map and / or the third feature map to obtain a fused feature map;
[0164] A target feature map extraction submodule is used to extract features from the candidate area represented by each spatial point in the fused feature map to obtain a target feature map;
[0165] The first probability value acquisition submodule is used to classify the candidate region represented by each spatial point in the target feature map to obtain a probability value that the candidate region represented by each spatial point is pulmonary embolism.
[0166] In some embodiments of the present invention, the feature map fusion submodule includes:
[0167] a fifth feature map extracting unit, configured to extract a fifth feature map from the fourth feature map;
[0168] a sixth feature map acquisition unit, configured to perform upsampling processing on the fifth feature map to obtain a sixth feature map;
[0169] a seventh feature map acquisition unit, configured to fuse the sixth feature map with the third feature map to obtain a seventh feature map;
[0170] an eighth feature map acquisition unit, configured to upsample the seventh feature map to obtain an eighth feature map;
[0171] A feature map fusion unit is used to fuse the eighth feature map with the second feature map to obtain a fused feature map.
[0172] In some embodiments of the present invention, the first loss value calculation module 403 includes:
[0173] A first target probability value selection submodule is used to select the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sequence sample as the first target probability value;
[0174] A first attention coefficient configuration submodule, configured to configure a first attention coefficient for the first target probability value, wherein the first attention coefficient represents a percentage of pulmonary embolism information represented by a candidate region represented by a spatial point corresponding to the first target probability value in the target feature map;
[0175] A first numerical calculation submodule, used for multiplying the first target probability value by the first attention coefficient to obtain a first numerical value;
[0176] A first calculation submodule is used to add the first numerical values corresponding to the first target probability values to obtain a probability value of the presence of pulmonary embolism in the lung CT image sequence samples;
[0177] The first loss value calculation submodule is used to calculate the cross entropy loss as the first loss value based on the probability of the presence of pulmonary embolism in the lung CT image sequence samples and the inspection-level label.
[0178] In some embodiments of the present invention, the first attention coefficient configuration submodule includes:
[0179] A first feature vector extraction unit, used to extract a first feature vector from features in a candidate area corresponding to the first target probability value in the target feature map;
[0180] The first attention coefficient acquisition unit is used to map the first feature vector to the interval (0, 1) to obtain a first attention coefficient corresponding to the first target probability value.
[0181] In some embodiments of the present invention, the second loss value calculation module 404 includes:
[0182] A second target probability value selection submodule is used to select the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sample as the second target probability value;
[0183] A second attention coefficient configuration submodule, configured to configure a second attention coefficient for the second target probability value, wherein the second attention coefficient represents a percentage of pulmonary embolism information represented by a candidate region represented by a spatial point corresponding to the second target probability value in the target feature map;
[0184] A second numerical calculation submodule, used for multiplying the second target probability value by the second attention coefficient to obtain a second numerical value;
[0185] a second calculation submodule, configured to add second numerical values corresponding to the second target probability values to obtain a probability value of the presence of pulmonary embolism in the pulmonary CT image sample;
[0186] The second loss value calculation submodule is used to calculate the cross entropy loss as the second loss value based on the probability of the presence of pulmonary embolism in the lung CT image sample and the image-level label.
[0187] In some embodiments of the present invention, the second attention coefficient configuration submodule includes:
[0188] A second feature vector extraction unit, used to extract a second feature vector from features in a candidate area corresponding to the second target probability value in the target feature map;
[0189] The second attention coefficient acquisition unit is used to map the second feature vector to the interval (0, 1) to obtain a second attention coefficient corresponding to the second target probability value.
[0190] In some embodiments of the present invention, the parameter updating module 405 includes:
[0191] A third loss value calculation submodule, used to calculate the sum of the second loss values of each lung CT image sample to obtain a third loss value;
[0192] A total loss calculation submodule, used to calculate the sum of the first loss value and the third loss value to obtain a total loss value;
[0193] A parameter updating submodule, used for updating the parameters of the pulmonary embolism detection model when the total loss value is greater than a preset value, and returning to the step of acquiring a lung CT image sequence sample;
[0194] The training completion determination submodule is used to determine that the training of the pulmonary embolism detection model is completed when the total loss value is less than or equal to a preset value.
[0195] The above-mentioned pulmonary embolism detection model training device can execute the pulmonary embolism detection model training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0196] Embodiment 5
[0197] Figure 5 A schematic diagram of the structure of a pulmonary embolism detection device provided in Embodiment 5 of the present invention is shown in FIG. Figure 5 As shown, the pulmonary embolism detection device comprises:
[0198] An image sequence acquisition module 501 is used to acquire a lung CT image sequence, wherein the lung CT image sequence includes a plurality of lung CT images obtained from a single CT examination;
[0199] A second probability value acquisition module 502 is used to input the lung CT image sequence into the pulmonary embolism detection model for processing, and obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence is a pulmonary embolism;
[0200] The prediction box labeling module 503 is used to label the candidate area represented by the spatial point whose probability value is greater than a preset value in the lung CT image sequence with a prediction box, and the prediction box is used to mark the position of the pulmonary embolism on the acquired lung CT image.
[0201] In some embodiments of the present invention, the above-mentioned pulmonary embolism detection device may further include:
[0202] A region interception module, configured to intercept a region of interest from the prediction frame of the lung CT image sequence after marking a prediction frame for the candidate region represented by the spatial point having a probability value greater than a preset value in the lung CT image sequence;
[0203] A binarization processing module, used for performing a double-threshold binarization process on the region of interest to obtain a binarized image;
[0204] A morphological processing module, used for performing morphological processing on the binary image to obtain a maximum connected area;
[0205] The prediction box correction module is used to label the maximum connected area and generate a corrected prediction box.
[0206] The above-mentioned pulmonary embolism detection device can execute the pulmonary embolism detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0207] Embodiment 6
[0208] Embodiment 6 of the present invention provides a computer device, Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 6 of the present invention is shown in FIG. Figure 6 As shown, the computer device includes a processor 601, a memory 602, a communication module 603, an input device 604 and an output device 605; the number of processors 601 in the computer device can be one or more. Figure 6 A processor 601 is taken as an example; the processor 601, memory 602, communication module 603, input device 604 and output device 605 in the computer device can be connected by a bus or other means. Figure 6 The above processor 601, memory 602, communication module 603, input device 604 and output device 605 can be integrated on a control mainboard of a computer device.
[0209] The memory 602 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as the module corresponding to the pulmonary embolism detection model training method or the pulmonary embolism detection method in the embodiment of the present invention. The processor 601 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 602, that is, the pulmonary embolism detection model training method or the pulmonary embolism detection method provided in the above embodiment is implemented.
[0210] The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 602 may further include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0211] The communication module 603 is used to establish a connection with an external device (such as a smart terminal) and realize data interaction with the external device. The input device 604 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device.
[0212] A computer device provided in this embodiment can execute the pulmonary embolism detection model training method or pulmonary embolism detection method provided in any of the above embodiments of the present invention, and has specific corresponding functions and beneficial effects.
[0213] Embodiment 7
[0214] Embodiment 7 of the present invention provides a storage medium containing computer executable instructions, on which a computer program is stored. When the program is executed by a processor, a pulmonary embolism detection model training method or a pulmonary embolism detection method as provided in any of the above embodiments of the present invention is implemented.
[0215] The pulmonary embolism detection model training method includes:
[0216] Acquire a lung CT image sequence sample, wherein the lung CT image sequence sample includes a plurality of lung CT image samples obtained from a single CT examination, wherein the lung CT image sequence sample is annotated with an examination-level label, wherein the lung CT image sample is annotated with an image-level label, wherein the examination-level label is used to indicate whether the lung CT image sequence sample has pulmonary embolism, and wherein the image-level label is used to indicate whether the lung CT image sample has pulmonary embolism;
[0217] Inputting the lung CT image sequence samples into the pulmonary embolism detection model for processing, and obtaining a probability value that the candidate region represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism;
[0218] Calculating the loss of the lung CT image sequence samples as a first loss value based on the probability value corresponding to each spatial point in the lung CT image sequence samples and the inspection-level label;
[0219] Calculating the loss of the lung CT image sample as a second loss value based on the probability value corresponding to each spatial point in the lung CT image sample and the image-level label;
[0220] Parameters of the pulmonary embolism detection model are updated based on the first loss value and the second loss value.
[0221] The pulmonary embolism detection method adopts a pulmonary embolism detection model trained by the pulmonary embolism detection model training method provided in any of the above embodiments of the present invention, including:
[0222] Acquire a lung CT image sequence, wherein the lung CT image sequence includes a plurality of lung CT images obtained from a single CT examination;
[0223] Inputting the lung CT image sequence into the pulmonary embolism detection model for processing to obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence is a pulmonary embolism;
[0224] A prediction box is marked for the candidate region represented by the spatial point with the probability value greater than a preset value in the lung CT image sequence, wherein the prediction box is used to mark the position of the pulmonary embolism on the acquired lung CT image.
[0225] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the pulmonary embolism detection model training method or pulmonary embolism detection method provided in the embodiment of the present invention.
[0226] It should be noted that, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0227] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a robot, a personal computer, a server, or a network device, etc.) to perform the pulmonary embolism detection model training method or pulmonary embolism detection method described in any embodiment of the present invention.
[0228] It is worth noting that in the above-mentioned device, the various modules, sub-modules and units included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0229] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0230] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0231] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A pulmonary embolism detection model training method, It is characterized in that include: Acquire a lung CT image sequence sample, wherein the lung CT image sequence sample includes a plurality of lung CT image samples obtained from a single CT examination, wherein the lung CT image sequence sample is annotated with an examination-level label, wherein the lung CT image sample is annotated with an image-level label, wherein the examination-level label is used to indicate whether the lung CT image sequence sample has pulmonary embolism, and wherein the image-level label is used to indicate whether the lung CT image sample has pulmonary embolism; Inputting the lung CT image sequence samples into the pulmonary embolism detection model for processing, and obtaining a probability value that the candidate region represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism; Selecting the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sequence sample as the first target probability value; Selecting the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sample as the second target probability value; Calculating a loss of the lung CT image sequence sample based on the first target probability value and the inspection-level label as a first loss value; Calculating a loss of the lung CT image sample based on the second target probability value and the image-level label as a second loss value; Parameters of the pulmonary embolism detection model are updated based on the first loss value and the second loss value.
2. The pulmonary embolism detection model training method according to claim 1, It is characterized in that Inputting the lung CT image sequence samples into the pulmonary embolism detection model for processing to obtain a probability value that the candidate region represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism, including: Downsampling the lung CT image sequence samples four times in succession to obtain a first feature map, a second feature map, a third feature map, and a fourth feature map in sequence; Fusing the fourth feature map with the first feature map, the second feature map and / or the third feature map to obtain a fused feature map; Perform feature extraction on the candidate area represented by each spatial point in the fused feature map to obtain a target feature map; The candidate region represented by each spatial point in the target feature map is classified to obtain a probability value that the candidate region represented by each spatial point is pulmonary embolism.
3. The pulmonary embolism detection model training method according to claim 2, It is characterized in that The fourth feature map is fused with the first feature map, the second feature map and / or the third feature map to obtain a fused feature map, comprising: Extracting a fifth feature map from the fourth feature map; Upsampling the fifth feature map to obtain a sixth feature map; Merging the sixth feature map with the third feature map to obtain a seventh feature map; Upsampling the seventh feature map to obtain an eighth feature map; The eighth feature map is fused with the second feature map to obtain a fused feature map.
4. The pulmonary embolism detection model training method according to claim 2 or 3, It is characterized in that Calculating the loss of the lung CT image sequence sample based on the first target probability value and the inspection-level label as a first loss value includes: Configuring a first attention coefficient for the first target probability value, the first attention coefficient representing a percentage of pulmonary embolism information represented by a candidate region represented by a spatial point corresponding to the first target probability value in the target feature map; Multiplying the first target probability value by the first attention coefficient to obtain a first value; Adding the first numerical values corresponding to the first target probability values to obtain a probability value of the presence of pulmonary embolism in the lung CT image sequence sample; A cross entropy loss is calculated as a first loss value based on the probability of the presence of pulmonary embolism in the lung CT image sequence samples and the inspection-level labels.
5. The pulmonary embolism detection model training method according to claim 4, It is characterized in that Configuring a first attention coefficient for the first target probability value includes: Extracting a first feature vector from features within a candidate area corresponding to the first target probability value in the target feature map; Map the first feature vector to the interval (0, 1) to obtain a first attention coefficient corresponding to the first target probability value.
6. The pulmonary embolism detection model training method according to claim 2 or 3, It is characterized in that Calculating the loss of the lung CT image sample based on the second target probability value and the image-level label as a second loss value includes: Configuring a second attention coefficient for the second target probability value, the second attention coefficient representing the percentage of pulmonary embolism information represented by the candidate region represented by the spatial point corresponding to the second target probability value in the target feature map; Multiplying the second target probability value by the second attention coefficient to obtain a second value; Adding the second numerical values corresponding to the second target probability values to obtain a probability value of the presence of pulmonary embolism in the lung CT image sample; A cross entropy loss is calculated as a second loss value based on the probability of the presence of pulmonary embolism in the lung CT image sample and the image-level label.
7. The pulmonary embolism detection model training method according to claim 6, It is characterized in that Configuring a second attention coefficient for the second target probability value includes: Extracting a second feature vector from features within the candidate area corresponding to the second target probability value in the target feature map; The second feature vector is mapped to the interval (0, 1) to obtain a second attention coefficient corresponding to the second target probability value.
8. The pulmonary embolism detection model training method according to claim 1, It is characterized in that Updating the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value includes: Calculate the sum of the second loss values of each lung CT image sample to obtain a third loss value; Calculating the sum of the first loss value and the third loss value to obtain a total loss value; When the total loss value is greater than a preset value, updating the parameters of the pulmonary embolism detection model, and returning to the step of acquiring a lung CT image sequence sample; When the total loss value is less than or equal to a preset value, it is determined that the training of the pulmonary embolism detection model is completed.
9. A pulmonary embolism detection model training device, It is characterized in that include: A sample acquisition module, used for acquiring lung CT image sequence samples, wherein the lung CT image sequence samples include multiple lung CT image samples obtained from a CT examination, wherein the lung CT image sequence samples are annotated with an examination-level label, wherein the lung CT image samples are annotated with an image-level label, wherein the examination-level label is used to indicate whether the lung CT image sequence samples have pulmonary embolism, and wherein the image-level label is used to indicate whether the lung CT image sequence samples have pulmonary embolism; A first probability value acquisition module is used to input the lung CT image sequence samples into the pulmonary embolism detection model for processing, and obtain a probability value that the candidate area represented by each spatial point in the lung CT image sequence samples is a pulmonary embolism; The first loss value calculation module includes: a first target probability value selection submodule, which is used to select the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sequence sample as the first target probability value; The second loss value calculation module includes: a second target probability value selection submodule, which is used to select the largest first K probability values among the probability values corresponding to each spatial point in the lung CT image sample as the second target probability value; The first loss value calculation module is used to calculate the loss of the lung CT image sequence sample as a first loss value based on the first target probability value and the inspection level label; The second loss value calculation module is used to calculate the loss of the lung CT image sample as a second loss value based on the second target probability value and the image-level label; A parameter updating module is used to update the parameters of the pulmonary embolism detection model based on the first loss value and the second loss value.
10. A pulmonary embolism detection device, It is characterized in that include: An image sequence acquisition module, used to acquire a lung CT image sequence, wherein the lung CT image sequence includes a plurality of lung CT images obtained from a single CT examination; a second probability value acquisition module, configured to input the pulmonary CT image sequence into a pulmonary embolism detection model trained by the pulmonary embolism detection model training method according to any one of claims 1 to 8 for processing, and obtain a probability value that the candidate region represented by each spatial point in the pulmonary CT image sequence is a pulmonary embolism; A prediction box labeling module is used to label prediction boxes for candidate areas represented by spatial points in the lung CT image sequence whose probability values are greater than a preset value, and the prediction boxes are used to mark the positions of the pulmonary emboli on the acquired lung CT images.
11. The pulmonary embolism detection device according to claim 10, It is characterized in that Also includes: A region interception module, configured to intercept a region of interest from the prediction frame of the lung CT image sequence after marking a prediction frame for the candidate region represented by the spatial point having a probability value greater than a preset value in the lung CT image sequence; A binarization processing module, used for performing a double-threshold binarization process on the region of interest to obtain a binarized image; A morphological processing module, used for performing morphological processing on the binary image to obtain a maximum connected area; The prediction box correction module is used to label the maximum connected area and generate a corrected prediction box.
12. A computer device, It is characterized in that include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the pulmonary embolism detection model training method as described in any one of claims 1-8.
13. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the pulmonary embolism detection model training method as described in any one of claims 1-8 is implemented.
Citation Information
Patent Citations
Lung focus detection method and device and training method of image detection model
CN111325739A
Multi-example learning recognition method for fuzzy mode in lung CT image
CN111414956A
