Medical image detection system, training method, and medical analysis method
The medical image detection system using neural networks generates multiple composite resolution feature maps by utilizing feature extraction and feature pyramid networks. Combined with region generation and classification modules, it solves the problem of missed nodule detection in chest X-ray examinations and achieves efficient and accurate nodule detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HTC CORP
- Filing Date
- 2023-03-01
- Publication Date
- 2026-07-24
AI Technical Summary
In current medical imaging technology, chest X-ray examinations are prone to problems with nodules being difficult to identify because they are obscured by other organs, leading to missed diagnoses of nodules, especially under heavy workloads where radiologists find it difficult to identify them accurately.
A neural network-based medical image detection system is adopted, which utilizes a feature extraction unit and a feature pyramid network, combined with a dual-head network architecture, to optimize global prediction and local prediction respectively. Multiple composite resolution feature maps are generated through the feature pyramid network, and nodule prediction is performed using a region generation network and a classification module.
It improves the accuracy and efficiency of nodule detection, reduces the possibility of missed nodule diagnosis in chest radiology, and provides rapid and accurate output information on nodule location and probability of presence.
Smart Images
Figure CN116703816B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a medical image detection system and training method, and more particularly to a medical image detection system capable of generating global prediction of the overall view of a medical image and local prediction of a local region in a medical image. Background Technology
[0002] Various medical imaging technologies are widely used in diagnosing patient diseases or conducting health checkups. Examples include X-ray imaging, computed tomography (CT) imaging, and magnetic resonance imaging (MRI). These technologies can provide crucial information in diagnosing cancer, fractures, internal bleeding, and other conditions. Generally, experienced physicians or specialists are required to interpret the images obtained from these medical imaging techniques and, based on their experience, determine whether the images are normal or abnormal.
[0003] Lung cancer is one of the leading causes of cancer-related deaths worldwide. While pulmonary nodules detected in medical imaging are usually benign, they can be a warning sign of potential lung cancer. Chest X-rays are the most widely used method of chest imaging due to their simplicity and low cost. However, a significant problem with chest X-rays is the high rate of nodules missed by radiologists. Because a chest X-ray is a two-dimensional projection of the patient's chest, nodules can be difficult to identify when obscured by other organs (such as ribs) or foreign objects. Furthermore, with increasing workloads, accurately identifying nodules is not easy for chest radiologists, thus requiring auxiliary tools to reduce the chances of omissions during image analysis. Summary of the Invention
[0004] One embodiment of this disclosure discloses a medical image detection system, including a memory and a processor. The memory stores a neural network model. The processor is coupled to the memory and executes the neural network model, which includes a feature extraction unit, a feature pyramid network, a first output terminal, and a second output terminal. The feature extraction unit extracts multiple intermediate tensors from the medical image, each of which has a resolution. The feature pyramid network is linked to the feature extraction unit and generates multiple composite resolution feature maps based on the intermediate tensors. The first output terminal generates a global prediction based on the multiple composite resolution feature maps, indicating whether a target object exists in the medical image. The second output terminal generates multiple local predictions based on the multiple composite resolution feature maps, each of which represents a predicted location of a target object in the medical image. The processor generates output information based on the medical image, the global prediction, and the multiple local predictions.
[0005] In some examples, at least one of the plurality of composite resolution feature maps is generated by the feature pyramid network based on at least two of the plurality of intermediate tensors. The feature extraction unit includes a plurality of convolutional layers to generate the plurality of intermediate tensors with different resolutions. The plurality of intermediate tensors includes a first intermediate tensor with a higher resolution and a second intermediate tensor with a lower resolution. The feature pyramid network is used to upsample the second intermediate tensor and merge it with the first intermediate tensor to generate one of the plurality of composite resolution feature maps.
[0006] In some examples, the last three layers of the multiple convolutional layers in the feature extraction unit are multiple deformable convolutional layers, and the multiple sampling points of the multiple deformable convolutional layers are unevenly distributed in the multiple sampling windows of the multiple deformable convolutional layers.
[0007] In some examples, the second output includes a region generation network, an alignment module, and a classification module. The region generation network generates multiple regions of interest (ROIs) on the multiple composite resolution feature maps. The alignment module extracts multiple proposed regions from the multiple composite resolution feature maps based on the ROI locations. The classification module classifies each of the multiple proposed regions into a target group or a non-target group, thereby generating multiple local predictions based on the multiple proposed regions classified as belonging to the target group.
[0008] In some examples, the first output includes multiple convolutional layers, a linear layer, and an activation layer used to generate the global prediction.
[0009] In some cases, the medical images include X-ray, computed tomography, or magnetic resonance imaging of the chest, abdomen, or head, with the target object being a nodule or tumor.
[0010] Another embodiment of this disclosure discloses a training method suitable for training a neural network model executed by a medical image detection system. The training method includes: providing a medical image and training basis facts related to the medical image; performing a first image augmentation on the medical image to generate a first augmented medical image; performing a second image augmentation on the medical image to generate a second augmented medical image; generating a global prediction from a first output of the neural network model based on the first augmented medical image; generating multiple local predictions from a second output of the neural network model based on the second augmented medical image; calculating a global loss by comparing the global prediction with the training basis facts; calculating a local loss by comparing the multiple local predictions with the training basis facts; and updating the neural network model via backpropagation based on the global loss and the local losses.
[0011] In some examples, the neural network model includes a feature extraction unit, a feature pyramid network, a first output, and a second output. The feature extraction unit is used to extract multiple intermediate tensors from the first and second augmented medical images. The feature pyramid network is linked to the feature extraction unit to generate multiple first composite resolution feature maps and multiple second composite resolution feature maps based on the multiple intermediate tensors.
[0012] In some examples, the first image augmentation and the second image augmentation are each performed in the following manner: at least one augmentation operation is selected from a plurality of candidate augmentation operations; and the selected at least one augmentation operation is applied to the medical image to generate the first augmented medical image or the second augmented medical image.
[0013] In some cases, the training method selects and applies different amplification operations when performing the first image amplification and the second image amplification.
[0014] Another embodiment of this disclosure discloses a medical analysis method applicable to a medical image detection system executing a neural network model. The medical analysis method includes: acquiring a medical image; extracting multiple intermediate tensors from the medical image using a feature extraction unit of the neural network model, each of the multiple intermediate tensors having a resolution; generating multiple composite resolution feature maps based on the multiple intermediate tensors using a feature pyramid network of the neural network model; generating a global prediction based on the multiple composite resolution feature maps using a first output of the neural network model, the global prediction indicating whether a target object exists in the medical image; generating multiple local predictions based on the multiple composite resolution feature maps using a second output of the neural network model, each of the multiple local predictions indicating the predicted location of a target object in the medical image; and generating output information based on the medical image, the global prediction, and the multiple local predictions.
[0015] Medical image detection systems and medical analysis methods can serve as auxiliary tools to reduce the chances of missing nodules during chest radiology examinations. The training method provides a dual-head amplification architecture to perform two different amplification operations on the two output ends separately, thereby training the different amplification operations of the first and second output ends to be optimized independently. Attached Figure Description
[0016] Figure 1 A schematic diagram of a medical image detection system according to some embodiments of the present disclosure is shown;
[0017] Figure 2 A schematic diagram of the architecture of a neural network model according to some embodiments of this disclosure is shown;
[0018] Figure 3 A schematic diagram is shown illustrating global predictions and multiple local predictions generated by a neural network model corresponding to a medical image according to some embodiments of this disclosure;
[0019] Figure 4 Some embodiments according to this disclosure are shown. Figure 1 and Figure 2 A flowchart of the medical analysis methods performed by the medical image detection system and neural network model in the process;
[0020] Figure 5 Some embodiments according to this disclosure are shown. Figure 2 A schematic diagram of the internal architecture of the feature extraction unit in the diagram;
[0021] Figure 6A , Figure 6B as well as Figure 6CThis diagram illustrates the distribution of sampling points in the sampling windows of different standard convolutional layers.
[0022] Figure 6D A schematic diagram showing the distribution of sampling points in the sampling window of a deformable convolutional layer is provided.
[0023] Figure 7 Some embodiments according to this disclosure are shown. Figure 2 A schematic diagram of the internal architecture of the feature pyramid network;
[0024] Figure 8 Some embodiments according to this disclosure are shown. Figure 2 A schematic diagram of the internal architecture of the first output terminal;
[0025] Figure 9 Some embodiments according to this disclosure are shown. Figure 2 A schematic diagram of the internal architecture of the second output terminal;
[0026] Figure 10 A schematic diagram of a medical image detection system in a training process is shown according to some embodiments of this disclosure;
[0027] Figure 11 This diagram illustrates the operations performed during the training phase of a neural network model, including model augmentation and agent training; and
[0028] Figure 12 The following are examples of embodiments suitable for training. Figure 10 A flowchart illustrating a training method for a neural network model executed by a medical image detection system.
[0029] Symbol explanation:
[0030] 100, 300: Medical Image Detection System
[0031] 120, 320: Memory
[0032] 140,340: Processors
[0033] 160, 360: Interface devices
[0034] 180, 380: Neural Network Model
[0035] 182,382: Feature extraction units
[0036] 184,384: Feature Pyramid Network
[0037] 186,386: First output terminal
[0038] 188,388: Second output terminal
[0039] 184a, 184d, 184g: Ascending Sampler
[0040] 184b, 184e, 184h: 1x1 convolutional layers
[0041] 184c, 184f, 184i: Adders
[0042] 186a, 186c: Convolutional layers
[0043] 186b, 186d: Rectifying linear units
[0044] 186e: Pooling layer
[0045] 186f: Linear layer
[0046] 186g: Softmax layer
[0047] 188a: Region Generation Network
[0048] 188b: Alignment Module
[0049] 188c: Classification Module
[0050] 390: Amplification Model
[0051] 392: Training Agent
[0052] 200: Medical Analysis Methods
[0053] S210, S220, S230, S240, S242, S250: Steps
[0054] 400: Training Methods
[0055] S410, S420, S422, S430, S440, S450, S452: Steps
[0056] S460, S462, S470: Steps
[0057] IMGi, IMGt: Medical Imaging
[0058] Ts, T1, T2, T3, T4, T5, T6, T7, T8: Intermediate tensors
[0059] CL1, CL2, CL3, CL4, CL5, CL6: Convolutional layers
[0060] CL7, CL8: Convolutional layers
[0061] SCL1, SCL2, SCL3: Standard convolutional layers
[0062] DCL: Deformable Convolutional Layer
[0063] W1, W2, W3, W4: Sampling windows
[0064] SAMP, dSAMP: Sampling points
[0065] MFM: Composite Resolution Feature Map
[0066] MFM1, MFM2, MFM3, MFM4: Composite resolution feature map
[0067] GP: Global Prediction
[0068] LPs, LP1, LP2, LP3, LP4, LP5: Local Prediction
[0069] INFO: Output Information
[0070] ROI1, ROI2, ROI3: Location of the region of interest
[0071] ROI4, ROI5, ROI6: Location of Region of Interest
[0072] PR: Proposed Area
[0073] TAR: Target Group
[0074] NTG: Non-target group
[0075] TG: Training Basic Facts
[0076] IMGa1: First Amplified Medical Imaging
[0077] IMGa2: Second Amplified Medical Imaging
[0078] MFMG: First Composite Resolution Feature Map
[0079] MFML: Second Composite Resolution Feature Map
[0080] BCHin: Input batch
[0081] BCHout: Output Batch Detailed Implementation
[0082] The following disclosure provides numerous different embodiments or examples for implementing various features of this disclosure. Elements and configurations in the specific examples are used in the following discussion to simplify this disclosure. Any examples discussed are for illustrative purposes only and do not in any way limit the scope or meaning of this disclosure or its examples. Where appropriate, the same reference numerals are used between the drawings and in the corresponding text description to represent the same or similar elements.
[0083] Please see Figure 1The illustration shows a schematic diagram of a medical image detection system 100 according to some embodiments of the present disclosure. In some implementations, the medical image detection system 100 is used to detect or predict the presence of target objects in medical images, and the detection results help to examine, evaluate or diagnose patients based on the content of the medical images.
[0084] like Figure 1 As shown, the medical image detection system 100 includes a memory 120, a processor 140, and an interface device 160. The memory 120 stores computer-executable instructions that can be executed by the processor 140. In some embodiments, the memory 120 may include dynamic memory, static memory, hard disk, or flash memory, etc. The interface device 160 is used to receive input data (e.g., input medical images, instructions, voice commands, or keyboard input, etc.) and / or display output content. In some embodiments, the interface device 160 may include a keyboard, display, touch panel, microphone, network transceiver, speaker, etc. The processor 140 is used to execute a neural network model 180, wherein the data or parameters of the neural network model 180 may be stored in the memory 120. In some embodiments, the processor 140 may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or other equivalent processing circuits.
[0085] Please refer to the following: Figure 2 This document illustrates a schematic diagram of the architecture of a neural network model 180 according to some embodiments of the present disclosure. The neural network model 180, executed by processor 140, is used to detect a medical image IMGi and generate a global prediction GP and multiple local prediction LPs for the medical image IMGi. In some embodiments, the global prediction GP is used to indicate whether a target object exists in the medical image. Each local prediction LP is used to represent the predicted location of a target object in the medical image IMGi.
[0086] In some embodiments, the medical image IMGi can be an X-ray, computed tomography (CT) image, or magnetic resonance imaging (MRI) image of the patient's chest, abdomen, or head. The target object can be a nodule and / or tumor present on the patient and appearing in the medical image IMGi. For simplicity of explanation, the following discussion will assume that the medical image IMGi is an X-ray image of the patient's chest, and the target object will be assumed to be a nodule.
[0087] The above combination (i.e., detecting nodules in chest X-ray images) is an exemplary application of neural network model 180, but this disclosure is not limited thereto.
[0088] In this example, the global prediction GP generated by neural network model 180 pertains to any nodules in the medical image IMGi, while the multiple local predictions LPs generated by neural network model 180 pertain to the predicted locations of individual nodules in the medical image IMGi. Please refer to [further details omitted]. Figure 3 This illustrates a schematic diagram of the global prediction GP and multiple local prediction LPs generated by the neural network model 180 corresponding to a medical image IMGi in some embodiments according to this disclosure. Figure 3 As shown, the global prediction GP generated by the neural network model 180 can be a probability value label, such as 85%, to indicate that the medical image IMGi has a high probability of containing at least one nodule. Figure 2 and Figure 3 As shown, in this embodiment, the neural network model 180 generates multiple local predictions (LPs), including five local predictions LP1, LP2, LP3, LP4, and LP5, which are used to identify five possible locations and regions where nodules exist in the medical image IMGi. In this example, local prediction LP1 includes a local bounding box to show the location and size of a predicted nodule in the medical image IMGi. Local prediction LP2 includes another local bounding box to show the location and size of another predicted nodule in the medical image IMGi. Similarly, local predictions LP3, LP4, and LP5 show three more local bounding boxes.
[0089] This disclosure presents a neural network model 180 that proposes a composite task lung nodule detection algorithm based on a dual-head network (DHN) architecture to simultaneously generate global prediction (GP) and multiple local predictions (LPs). For example... Figure 2As shown, the neural network model 180 includes a feature extraction unit 182, a feature pyramid network (FPN) 184, a first output terminal 186, and a second output terminal 188. Please refer to the following documents: Figure 4 It illustrates some embodiments according to this disclosure. Figure 1 and Figure 2 The flowchart shows the medical analysis method 200 executed by the medical image detection system 100 and the neural network model 180.
[0090] In some embodiments, the feature extraction unit 182, the feature pyramid network 184, the first output terminal 186, and the second output terminal 188 may be software functional modules in the neural network model 180, each of which can be implemented by a software program or computer-executable instructions. Details regarding the software architecture of the feature extraction unit 182, the feature pyramid network 184, the first output terminal 186, and the second output terminal 188 will be further explained in subsequent paragraphs.
[0091] like Figure 1 , Figure 2 as well as Figure 4 As shown, step S210 is performed to provide the medical image IMGi to the feature extraction unit 182 of the neural network model 180. In some embodiments, the medical image IMGi may be an X-ray image taken of the patient's chest area. If the patient has lung cancer, one or more nodules will appear in the medical image IMGi. If the patient's lungs are healthy, the medical image IMGi may not show any nodules.
[0092] like Figure 1 , Figure 2 as well as Figure 4 As shown, in step S220, the feature extraction unit 182 of the neural network model 180 extracts multiple intermediate tensors Ts from the medical image IMGi. Please refer to [link / reference needed]. Figure 5 It illustrates some embodiments according to this disclosure. Figure 2 A schematic diagram of the internal architecture of the feature extraction unit 182 is shown. The feature extraction unit 182 contains several convolutional layers to generate multiple intermediate tensors Ts based on the medical image IMGi. Figure 5In this embodiment, the feature extraction unit 182 includes eight convolutional layers CL1 to CL8 sequentially coupled. Convolutional layer CL1 may contain convolutional kernels on different channels and can be used to perform convolution operations on the medical image IMGi to generate an intermediate tensor T1. The intermediate tensor T1 generated by convolutional layer CL1 is passed to convolutional layer CL2, which can then perform convolution operations on the intermediate tensor T1 to generate an intermediate tensor T2. Similarly, convolutional layers CL3 to CL8 can sequentially perform convolution operations to generate intermediate tensors T3 to T8 respectively.
[0093] In some embodiments, the feature extraction unit 182 may further include pooling layers (not shown) and / or activation layers (not shown) between or before these convolutional layers CL1 to CL8. Because convolutional layers CL1 to CL8 may contain different sampling rates and due to the effects of the pooling layers, intermediate tensors T1 to T8 each have different resolutions. For example, the spatial resolution of intermediate tensor T5 may be 128x128; the spatial resolution of intermediate tensor T6 may be 64x64; the spatial resolution of intermediate tensor T7 may be 32x32; and the spatial resolution of intermediate tensor T8 may be 16x16. The resolutions of these intermediate tensors T5 to T8 are affected by the sampling rates of the convolutional layers CL5 to CL8 and the pooling layers between convolutional layers CL3 to CL8, and this disclosure is not limited to specific numbers.
[0094] In this example, intermediate tensor T5 has a relatively high spatial resolution of 128x128, compared to the spatial resolution of intermediate tensor T6 (64x64). Similarly, intermediate tensor T6 has a relatively high spatial resolution of 64x64, compared to the spatial resolution of intermediate tensor T7 (32x32). On the other hand, intermediate tensor T7 has a relatively high spatial resolution of 32x32, compared to the spatial resolution of intermediate tensor T8 (16x16). Each cell in intermediate tensor T5 tends to represent more concrete and local features, corresponding to a relatively small area of the medical image IMGi. Each cell in intermediate tensor T8 tends to represent more abstract features, corresponding to a relatively large area of the medical image IMGi.
[0095] In some embodiments, a portion of the convolutional layers CL1 to CL8 may be deformable convolutional layers, which differ from standard convolutional layers. Multiple sampling points on a standard convolutional layer are evenly distributed within the sampling window of that standard convolutional layer. Please refer to [further details omitted]. Figure 6A , Figure 6B as well as Figure 6CThis diagram illustrates the distribution of sampling points within the sampling windows W1 to W3 of the standard convolutional layers SCL1 to SCL3. Figure 6A As shown, each cell in the sampling window W1 of the standard convolutional layer SCL1 is selected as a sampling point (SAMP). Figure 6B As shown, the sampling points (SAMP) are spaced one unit apart and are uniformly distributed within the sampling window W2 of the standard convolutional layer SCL2. Figure 6C As shown, the sampling points SAMP are spaced two units apart and are evenly distributed in the sampling window W3 of the standard convolutional layer SCL3.
[0096] Please refer to the following: Figure 6D This diagram illustrates the distribution of sampling points within the sampling window W4 of the deformable convolutional layer (DCL). Figure 6D As shown, the sampling points dSAMP of the deformable convolutional layer DCL are offset by different displacement vectors from their original positions. Therefore, the sampling points dSAMP are unevenly distributed in the sampling window W4 of the deformable convolutional layer DCL.
[0097] If a standard convolutional layer is used (such as...) Figures 6A to 6C The standard convolutional layers SCL1 to SCL3 shown have a fixed receptive field. On the other hand, deformable convolutional layers can provide a dynamic receptive field when detecting regions of interest (e.g., nodules). In practical applications, since the shape and contour of nodules change with the actual situation, using deformable convolution helps in finding the image features of nodules. In some embodiments, the last three layers (i.e., convolutional layers CL6 to CL8) of all convolutional layers CL1 to CL8 in the feature extraction unit 182 are deformable convolutional layers, see [link to documentation]. Figure 6D The deformable convolutional layer DCL is shown in the diagram. The other convolutional layers CL1 to CL5 use standard convolutional layers, see [link to diagram]. Figures 6A to 6C The standard convolutional layers SCL1 to SCL3 are shown in the diagram.
[0098] like Figure 1 , Figure 2 as well as Figure 4 As shown, in step S230, the feature pyramid network 184 of the neural network model 180 generates multiple composite resolution feature maps (MFMs) based on intermediate tensors T5 to T8. The feature pyramid network 184 is coupled to the feature extraction unit 182, and the feature pyramid network 184 and the feature extraction unit 182 are linked. Please refer to [further details omitted]. Figure 7 It illustrates some embodiments according to this disclosure. Figure 2 A schematic diagram of the internal architecture of the feature pyramid network 184.
[0099] In some embodiments, the feature pyramid network 184 receives intermediate tensors T5 to T8 from the feature extraction unit 182 and generates a composite resolution feature map (MFM) accordingly. Figure 7 The embodiment shown has a feature pyramid network 184 comprising a set of upsamplers 184a, 1x1 convolutional layers 184b, and adders 184c between two adjacent intermediate tensors T7 and T8.
[0100] In some embodiments, the intermediate tensor T8 (with a spatial resolution of 16x16) generated by the feature extraction unit 182 can be copied into a composite resolution feature map MFM1. This composite resolution feature map MFM1 (based on the intermediate tensor T8) is upsampled to 32x32 by the upsampler 184a. The intermediate tensor T7 (with a spatial resolution of 32x32) is processed by a 1x1 convolutional layer 184b. The adder 184c is used to merge the intermediate tensor T7 after the 1x1 convolution with the upsampled composite resolution feature map MFM1 to form another composite resolution feature map MFM2 with a spatial resolution of 32x32. In this case, the composite resolution feature map MFM2 carries features from both the intermediate tensor T8 and the intermediate tensor T7. In other words, the composite resolution feature map MFM2 is generated based on two intermediate tensors T7 and T8 with different spatial resolutions (e.g., 32x32 and 16x16).
[0101] Similarly, the feature pyramid network 184 further includes another set of upsamplers 184d, 1x1 convolutional layers 184e, and adders 184f between two adjacent intermediate tensors T6 and T7. In this example, the composite resolution feature map MFM2 is upsampled to 64x64 by upsampler 184d. The intermediate tensor T6 (with a spatial resolution of 64x64) is processed by the 1x1 convolutional layer 184e. Adder 184f is used to merge the intermediate tensor T6 after 1x1 convolution with the upsampled composite resolution feature map MFM2 to form another composite resolution feature map MFM3 with a spatial resolution of 64x64. In this case, the composite resolution feature map MFM3 simultaneously carries features from the three intermediate tensors T6, T7, and T8. In other words, the composite resolution feature map MFM3 is generated based on three intermediate tensors T6, T7 and T8 with different spatial resolutions (e.g. 64x64, 32x32 and 16x16).
[0102] Similarly, the feature pyramid network 184 further includes another set of upsamplers 184g, 1x1 convolutional layers 184h, and adders 184i between two adjacent intermediate tensors T5 and T6 to generate a composite resolution feature map MFM4. In this case, the composite resolution feature map MFM4 simultaneously carries features from four intermediate tensors T5, T6, T7, and T8. In other words, the composite resolution feature map MFM4 is generated based on four intermediate tensors T5, T6, T7, and T8 with different spatial resolutions (e.g., 128x128, 64x64, 32x32, and 16x16).
[0103] In this case, composite resolution feature maps MFM2, MFM3, and MFM4 are generated from at least two intermediate tensors among intermediate tensors T5, T6, T7, and T8 with different resolutions. Multiple composite resolution feature maps (MFMs) carry the detected pattern features from multiple intermediate tensors at different resolutions; therefore, multiple composite resolution feature maps (MFMs) are helpful in finding various nodules with different size characteristics.
[0104] In the above Figure 7 In the illustrated embodiment, the feature pyramid network 184 is exemplified by having three sets of circuits configured between four intermediate tensors T5 to T8 to generate four composite resolution feature maps MFM1 to MFM4. This disclosure is not limited thereto. In some other embodiments, the feature pyramid network 184 may have a different number of layers.
[0105] like Figure 2 as well as Figure 4 As shown, in some embodiments, multiple composite resolution feature maps (MFMs) are simultaneously transmitted to the first output terminal 186 and the second output terminal 188. Step S240 involves the first output terminal 186 generating a global prediction GP based on the composite resolution feature maps (MFMs). The global prediction GP is used to indicate the presence of any nodules in the medical image IMGi. Please refer to [further details omitted]. Figure 8 It illustrates some embodiments according to this disclosure. Figure 2 A schematic diagram of the internal architecture of the first output terminal 186. In some embodiments, the first output terminal 186 may include convolutional layers 186a and 186c, rectified linear units (ReLU) 186b and 186d, a pooling layer 186e, a linear layer 186f, and a softmax layer 186g.
[0106] like Figure 8As shown, multiple composite resolution feature maps (MFMs) are input to the first output terminal 186. The multiple MMFs are passed through convolutional layers 186a, rectified linear units 186b, convolutional layers 186c, and rectified linear units 186d, and then max-pooled by pooling layer 186e to generate a single vector. Linear layers 186f and Softmax layers 186g are applied to this single vector to obtain a probability, representing the probability of a nodule present in the image scan. In some embodiments, the global prediction GP generated by the first output terminal 186 is represented by probability labels, for example... Figure 3 The figure shown is 85%.
[0107] like Figure 2 as well as Figure 4 As shown, in step S242, the second output terminal 188 generates local predictions LPs based on the composite resolution feature map MFM. Please refer to [link / reference needed]. Figure 9 It illustrates some embodiments according to this disclosure. Figure 2 A schematic diagram of the internal architecture of the second output terminal 188. (See diagram below.) Figure 2 as well as Figure 9 As shown, the second output terminal 188 includes a region proposal network (RPN) 188a, an alignment module 188b, and a classification module 188c. The region proposal network 188a generates multiple regions of interest (ROI) locations on the composite resolution feature map (MFM). In some embodiments, the region proposal network 188a can propose multiple ROI locations, such as ROI1, ROI2, ROI3, ROI4, ROI5, and ROI6 on the MFM. The alignment module 188b extracts multiple proposed regions (PRs) from the MFM based on the ROI locations ROI1, ROI2, ROI3, ROI4, ROI5, and ROI6. The classification module 188c classifies these proposed regions (PRs) into either a target group (TAR) or a non-target group (NTG). Multiple local predictions (LPs) for the second output terminal 188 are generated based on the multiple proposed regions (PRs) classified as belonging to the target group (TAR). Multiple local prediction LPs each (see also) Figure 3 Each of LP1 to LP5 in the IMGi corresponds to a predicted location of a nodule in the medical image.
[0108] In some embodiments, the processor 140 can integrate medical image IMGi, global prediction GP, and local prediction LP into a single output information INFO. Figure 3As shown. In this example, the output information INFO may present the probability of a nodule in the medical image IMGi and the potential location of the nodule in the medical image IMGi. In some embodiments, the output information INFO may be displayed on the interface device 160.
[0109] like Figure 3 As shown, in some embodiments, the output information (INFO) is in the form of an output image that combines the probability of the corresponding global prediction (GP) with the potential locations of the corresponding multiple local predictions (LPs). In other embodiments, the output information (INFO) may take other forms, such as text descriptions, coordinate indicators, plots, tables, sound indicators, or a combination of the above.
[0110] Users (such as patients, doctors, radiologists, nurses, or medical specialists) can input medical images (IMGi) into the medical image detection system 100 and subsequently view or receive the output information (INFO) on the interface device 160. This output information (INFO) helps users quickly understand the examination results reflected in the medical image (IMGi). With the rapidly increasing workload of radiologists, the medical image detection system 100 and medical analysis methods 200 can serve as auxiliary tools to reduce the possibility of missed nodules during chest radiology examinations.
[0111] It should be noted that, in order to ensure that the first output terminal 186 and the second output terminal 188 can generate appropriate predictions for medical image IMGi, the neural network model 180 needs to be trained in advance based on training data (such as known chest X-ray images from historical medical records and the corresponding nodule labels). The training process of the neural network model 180 will be discussed in the following paragraphs.
[0112] Please refer to the following: Figure 10 , Figure 10 A schematic diagram of a medical image detection system 300 in a training process according to some embodiments of this disclosure is shown. Figure 10 As shown, the medical image detection system 300 includes a memory 320, a processor 340, and an interface device 360. The processor 340 is used to execute a neural network model 380. Figure 10 The memory 320, processor 340, interface device 360, and neural network model 380 are similar to those in the previous embodiment. Figures 1 to 9 The memory 120, processor 140, interface device 160, and neural network model 180 discussed earlier. It should be noted that in the training process of neural network model 380, processor 340 further executes augmentation model 390 and training agent 392.
[0113] Data augmentation is a technique used to increase the diversity of training data and thereby improve the generality of a model. Data augmentation techniques can expand a limited amount of training data to include a larger number of samples. Some embodiments of this disclosure provide a dual-head augmentation technique that employs independent data augmentation strategies to multiple outputs, thereby fully optimizing the desired objectives of each output.
[0114] Please refer to the following: Figure 11 as well as Figure 12 , Figure 11 A schematic diagram is shown illustrating the operations performed by augmenting model 390 and training agent 392 during the training phase of neural network model 380. Figure 12 A flowchart of a training method 400 according to some embodiments is shown. The training method 400 is suitable for training... Figure 10 The neural network model 380 executed by the medical image detection system 300 in the system.
[0115] like Figure 11 as well as Figure 12 As shown, step S410 is executed to provide a medical image IMGt and training basis facts TG related to the medical image IMGt. In this example, the medical image IMGt and training basis facts TG can be obtained from historical medical records. The medical image IMGt can be a real chest X-ray image, and the corresponding training basis facts TG (e.g., medical diagnosis of the presence of nodules in the medical image IMGt and label boxes about found nodules) can also be obtained from historical medical records. This medical image IMGt is then passed to the augmentation model 390.
[0116] like Figure 11 as well as Figure 12 As shown, in step S420, the amplification model 390 performs a first image amplification on the medical image IMGt to generate a first amplified medical image IMGa1. Furthermore, in step S422, the amplification model 390 performs a second image amplification on the medical image IMGt to generate a second amplified medical image IMGa2.
[0117] In some embodiments, the first image augmentation in step S420 is achieved by selecting at least one augmentation operation and applying the selected at least one augmentation operation to the medical image IMGt to generate a first augmented medical image IMGat. In some embodiments, this augmentation operation is...
[0118] The image amplification operation is selected from multiple candidate amplification operations. For example, candidate amplification operations include random interference insertion, random brightness adjustment, random contrast adjustment, image shearing, image flipping, and image rotation. As an example, image shearing is selected and applied to a medical image IMGt to generate... Figure 11 The first amplified medical image IMGa1 is shown, but this disclosure is not limited thereto. One or more candidate amplification operations may be selected and executed in step S420 to generate the first amplified medical image IMGa1. For example, random contrast adjustment and image flipping may be selected simultaneously to generate the first amplified medical image (not shown in the figure).
[0119] On the other hand, the second image augmentation in step S422 is achieved by selecting at least one augmentation operation from a plurality of candidate augmentation operations and applying the selected at least one augmentation operation to the medical image IMGt to generate a second augmented medical image IMGa2. As an example, random brightness adjustment is selected and applied to the medical image IMGt to generate... Figure 11 The first amplified medical image IMGa2 is shown, but this disclosure is not limited thereto. One or more of the above candidate amplification operations may be selected and executed in step S422 to generate a second amplified medical image IMGa2.
[0120] In some embodiments, the first image amplification in step S420 and the second image amplification in step S422 can be performed according to different amplification operations selected by each.
[0121] In some embodiments, during the selection of the amplification operation, one of several candidate amplification operations is selected according to uniform sampling probability. In other words, when there are six different candidate amplification operations, such as random interference insertion, random brightness adjustment, random contrast adjustment, image cropping, image flipping, and image rotation, each candidate amplification operation has a 1 / 6 chance of being selected, and only one candidate amplification operation is used in a single image amplification.
[0122] In other embodiments, during the selection of amplification operations, each of these candidate amplification operations is selected independently based on a binomial sampling probability. In other words, when there are six different candidate amplification operations, each of the six candidate amplification operations has its own independent sampling probability of being selected, and one or more candidate amplification operations may be used in a single image amplification.
[0123] In some embodiments, in step S430, the first amplified medical image IMGa1 and the second amplified medical image IMGa2 are combined together as an input batch BCHin and fed into the feature extraction unit 382 and the feature pyramid network 384.
[0124] Feature extraction unit 382 is used to extract multiple intermediate tensors Ts from the first amplified medical image IMGa1 and the second amplified medical image IMGa2 in the input batch BCHin. Feature pyramid network 384 is linked to feature extraction unit 382. For example... Figure 12 As shown, in step S440, the feature pyramid network 384 generates an output batch BCHout based on multiple intermediate tensors Ts. The output batch BCHout contains multiple first composite resolution feature maps MFMG (corresponding to the first amplified medical image IMGa1) and second composite resolution feature maps MGML (corresponding to the second amplified medical image IMGa2).
[0125] like Figure 11 as well as Figure 12 As shown, in step S450, the first output terminal 386 generates a global prediction GP based on multiple first composite resolution feature maps (MFMG) in the output batch BCHout. Furthermore, in step S452, the second output terminal 388 generates multiple local predictions (LPs) based on multiple second composite resolution feature maps (MGML) in the output batch BCHout.
[0126] Figure 11 The method of generating global prediction (GP) and multiple local prediction (LPs) using the first output terminal 386 and the second output terminal 388 is similar to... Figure 2 , Figure 8 as well as Figure 9 The methods for generating global prediction (GP) and multiple local predictions (LPs) using the first output terminal 186 and the second output terminal 188, as discussed in previous embodiments, will not be repeated here. The main difference between this embodiment and previous embodiments is that the first output terminal 386 generates the global prediction GP based on multiple first composite resolution feature maps (MFMG) corresponding to the first amplified medical image IMGa1, while the second output terminal 388 generates multiple local predictions (LPs) based on multiple second composite resolution feature maps (MGML) corresponding to the second amplified medical image IMGa2. In other words, the first output terminal 386 and the second output terminal 388 are trained based on different amplification operations.
[0127] like Figure 11 as well as Figure 12 As shown, the global prediction GP and multiple local prediction LPs are sent to the training agent 392. Step S460 is executed, in which the training agent 392 calculates the global loss by comparing the global prediction GP with the training base facts TG.
[0128] The primary objective of the first output terminal 386 is to classify the presence of nodules in the current image scan (i.e., medical image IMGt). In some embodiments, the global loss L global The calculation uses a weighted cross-entropy algorithm. When the training basis fact TG is true (nodes exist), the lower the probability label of the global predicted GP, the lower the global loss L. global The larger the value, the lower the probability label of the global prediction GP; conversely, the higher the probability label of the global prediction GP, the lower the global loss L. global The smaller the value, the better. On the other hand, when the training basis fact TG is negative (no nodes exist), if the probability label of the global prediction GP is higher, the global loss L will be lower. global The larger the value, the lower the probability label of the global prediction GP, and vice versa. global The smaller.
[0129] Furthermore, in step S462, the training agent 392 calculates the local loss by comparing multiple local predictions (LPs) with the training base facts (TG). In some embodiments, the local loss L... local The calculation is performed using the following equation 1:
[0130] L local =l obj +l reg +l cls +l bbox …(Equation 1)
[0131] During the training phase, the region generation network in the second output 388 (see [link]) Figure 2 and Figure 9 The region generation network 188a) in the second output terminal 188 causes object loss during the background and foreground classification process. obj Furthermore, the regression loss l is obtained from the interval distance calculated between the region boxes proposed by the region generation network and the bounding boxes in the training base facts. reg The classification module in the second output terminal 388 (see [link]). Figure 2 and Figure 9 The classification module 188c in the second output terminal 188 generates a classification loss during feature classification. cls The bounding box loss l is calculated based on the margin distance between the updated local predicted bounding boxes and the bounding boxes in the training base facts. bbox In some embodiments, the training agent 392 may calculate the regression loss l according to the Smooth L1 Loss algorithm. reg and bounding box loss l bboxThe training agent 392 can calculate the object loss l using the weighted cross-entropy algorithm. obj And classification loss cls .
[0132] In some embodiments, the training agent 392 further employs a global loss L global With local loss L local After weighting and summing, the composite task loss L is calculated. multi For example, calculate using the following equation 2:
[0133] L multi =λ1*L global +λ2*L local …(Equation 2)
[0134] In Equation 2, λ1 is the weight of the global loss and λ2 is the weight of the local loss.
[0135] like Figure 11 and Figure 12 As shown, step S470 is executed, where the training agent 392 backpropagates the data based on the global loss L. global With local loss L local (or based on the composite task loss L) multi Update neural network model 380.
[0136] In typical image classification or object detection tasks, a single augmentation strategy (e.g., a pre-defined set of random image transformations) is used for each training image. However, training a dual-output architecture with a single augmentation strategy can easily lead to significant optimization of one output while the other performs only moderately. To ensure that the target performance of each output is fully optimized, this disclosure... Figure 11 The system provides a dual-head amplification architecture. Amplification model 390 is used to perform two amplification operations on the two output ends separately. In this case, the different amplification operations used to train the first output end 386 and the second output end 388 can be optimized separately, thereby reducing the global loss L separately. global With local loss L local To minimize (or reduce the loss of the composite task L) multi minimize).
[0137] While specific embodiments of this disclosure have been disclosed with respect to the above embodiments, these embodiments are not intended to limit this disclosure. Various alternatives and modifications can be made by those skilled in the art without departing from the principles and spirit of this disclosure. Therefore, the scope of protection of this disclosure is determined by the appended claims.
[0138] Although this disclosure has described some embodiments in considerable detail, other similar implementations are still possible. Therefore, the spirit and scope of the appended claims should not be limited to the embodiments described above.
[0139] For those skilled in the art, various modifications and variations can be made to the structure of this disclosure without departing from the scope or spirit of this disclosure. Based on the above embodiments, this disclosure covers various possible modifications and variations falling within the scope of the appended claims.
Claims
1. A medical image detection system, characterized in that, This medical image detection system includes: A memory for storing a neural network model; and A processor, coupled to the memory, is used to execute the neural network model, which includes: A feature extraction unit is used to extract multiple intermediate tensors from a medical image, each of the multiple intermediate tensors having a resolution. The feature extraction unit includes multiple convolutional layers to generate the multiple intermediate tensors with different resolutions. The last three layers of the multiple convolutional layers in the feature extraction unit are multiple deformable convolutional layers, and the multiple sampling points of the multiple deformable convolutional layers are non-uniformly distributed in the multiple sampling windows of the multiple deformable convolutional layers. A feature pyramid network, linked to the feature extraction unit, is used to generate multiple composite resolution feature maps based on the multiple intermediate tensors; A first output terminal is used to generate a global prediction based on the plurality of composite resolution feature maps, the global prediction being used to indicate whether a target object exists in the medical image; as well as A second output terminal is used to generate multiple local predictions based on the multiple composite resolution feature maps, each of which represents the predicted location of a target object in the medical image. The processor is used to generate output information based on the medical image, the global prediction, and the multiple local predictions.
2. The medical image detection system of claim 1, wherein at least one of the plurality of composite resolution feature maps is generated by the feature pyramid network based on at least two of the plurality of intermediate tensors, the plurality of intermediate tensors including a first intermediate tensor having a higher resolution and a second intermediate tensor having a lower resolution, the feature pyramid network being used to upsample the second intermediate tensor and merge it with the first intermediate tensor to generate one of the plurality of composite resolution feature maps.
3. The medical image detection system as claimed in claim 1, wherein the second output terminal comprises: A region generation network is used to generate multiple regions of interest locations on the multiple composite resolution feature maps; A matching module is used to extract multiple proposed regions from the multiple composite resolution feature maps based on the locations of the multiple regions of interest. as well as A classification module is used to classify each of the plurality of proposed regions into a target group or a non-target group, thereby generating the plurality of local predictions based on the plurality of proposed regions classified into the target group.
4. The medical image detection system of claim 1, wherein the first output terminal includes a plurality of additional convolutional layers, a linear layer and an activation layer, the activation layer being used to generate the global prediction.
5. The medical image detection system of claim 1, wherein the medical image comprises an X-ray image, a computed tomography image, or an magnetic resonance imaging image relating to a chest, an abdomen, or a head, and the target object comprises a nodule or a tumor.
6. A training method, characterized in that, This training method is applicable to training a neural network model executed by a medical image detection system. The training method includes: Provide a medical image and a set of training facts related to that medical image; The medical image is amplified to produce a first amplified medical image. The medical image is augmented with a second image to produce a second augmented medical image; A global prediction is generated from a first output of the neural network model based on the first augmented medical image. The neural network model includes a feature extraction unit for extracting multiple intermediate tensors from the first augmented medical image and the second augmented medical image. The feature extraction unit includes multiple convolutional layers to generate the multiple intermediate tensors with different resolutions. The last three layers of the multiple convolutional layers in the feature extraction unit are multiple deformable convolutional layers. The multiple sampling points of the multiple deformable convolutional layers are non-uniformly distributed in the multiple sampling windows of the multiple deformable convolutional layers. Based on the second amplified medical image, multiple local predictions are generated from a second output of the neural network model; A global loss is calculated by comparing the global prediction with the training base facts; A local loss is calculated by comparing the plurality of local predictions with the training base facts; and The neural network model is updated using backpropagation based on the global loss and the local loss.
7. The training method of claim 6, wherein the neural network model further comprises: A feature pyramid network, linked to the feature extraction unit, is used to generate multiple first composite resolution feature maps and multiple second composite resolution feature maps based on the multiple intermediate tensors.
8. The training method of claim 6, wherein the first image augmentation and the second image augmentation are each performed in the following manner: At least one amplification operation is selected from multiple candidate amplification operations; and The selected amplification operation is applied to the medical image to generate the first amplified medical image or the second amplified medical image.
9. The training method as described in claim 8, wherein when performing the first image amplification and the second image amplification, the training method selects and applies different amplification operations respectively.
10. A medical analysis method, characterized in that, This medical analysis method is applicable to a medical image detection system that executes a neural network model. The medical analysis method includes: Obtain a medical image; Using a feature extraction unit of the neural network model, multiple intermediate tensors are extracted from the medical image. Each of the multiple intermediate tensors has a resolution. The feature extraction unit includes multiple convolutional layers to generate the multiple intermediate tensors. The last three layers of the multiple convolutional layers in the feature extraction unit are multiple deformable convolutional layers. The multiple sampling points of the multiple deformable convolutional layers are non-uniformly distributed in the multiple sampling windows of the multiple deformable convolutional layers. Using a feature pyramid network of this neural network model, multiple composite resolution feature maps are generated based on the multiple intermediate tensors; Using a first output of the neural network model, a global prediction is generated based on the multiple composite resolution feature maps. This global prediction indicates whether a target object exists in the medical image. Using a second output of the neural network model, multiple local predictions are generated based on the multiple composite resolution feature maps, each of which represents the predicted location of a target object in the medical image; and An output information is generated based on the medical image, the global prediction, and the multiple local predictions.