Nodule detection apparatus and device

Abstract

An embodiment of the present application discloses a nodule detection device and equipment, which comprises: a first acquisition module configured to acquire a three-dimensional image to be detected, the three-dimensional image to be detected displaying a first target object; a flipping module configured to flip the three-dimensional image to be detected along at least one flipping direction to obtain a flipped three-dimensional image, each flipping direction corresponding to at least one flipped three-dimensional image; a first prediction module configured to obtain a first nodule prediction result of the three-dimensional image to be detected by using a nodule prediction model; a second prediction module configured to obtain a second nodule prediction result of the flipped three-dimensional image by using the nodule prediction model; and a detection module configured to obtain a nodule detection result of the first target object according to the first nodule prediction result and the second nodule prediction result. The above scheme can solve the technical problem of the related art that the neural network prediction time length increases and the overall receptive field is limited when the lung three-dimensional image is divided into blocks.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a nodule detection device and equipment. Background Technology

[0002] Lung cancer is one of the leading causes of cancer-related deaths. Early manifestations of lung cancer often include the appearance of nodules in the lungs. Based on the type, size, density, and rate of nodule formation, management can be tailored according to different screening guidelines. Currently, imaging methods for detecting lung nodules are the primary method for lung cancer diagnosis. The main imaging approach involves low-dose computed tomography (LDCT) screening. Imaging methods can reduce lung cancer mortality. However, this method requires a significant workload for physicians in extracting information about the nodules from the images.

[0003] In the detection of lung nodules and the reduction of false positives, with the development of deep learning algorithms, devices deployed with deep neural networks (DNNs), especially convolutional neural networks (CNNs), are increasingly being used by doctors to reduce their workload. For example, there are nodule detection devices deploying DeepLung neural networks built on residual learning and densely connected dual-channel connection modules. Another example is nodule detection devices deploying single-stage, single-scale S4ND neural networks built on YOLO. Yet another example is nodule detection devices deploying 3DFPN neural networks built on fully connected networks and dense anchors. However, when these devices perform lung nodule detection, the neural networks are limited by the graphics processing unit (GPU) memory, requiring the 3D lung image to be divided into blocks before processing by the neural network. At this time, the prediction time of the neural network in the device will increase with the increase of the number of blocks. For example, when the number of blocks is 12, the prediction time of the neural network is 12 times the prediction time of a single prediction, and the overall receptive field of the neural network is also limited to the block area. Summary of the Invention

[0004] One embodiment of this application provides a nodule detection device and apparatus to solve the technical problems in the related art of increasing neural network prediction time and limiting the overall receptive field caused by segmenting three-dimensional lung images into blocks.

[0005] In a first aspect, one embodiment of this application provides a nodule detection device, comprising:

[0006] The first acquisition module is used to acquire a three-dimensional image to be detected, wherein the three-dimensional image to be detected displays a first target object;

[0007] A flipping module is used to flip the three-dimensional image to be detected along at least one flipping direction to obtain a flipped three-dimensional image, wherein each flipping direction corresponds to at least one flipped three-dimensional image;

[0008] The first prediction module is used to obtain the first nodule prediction result of the three-dimensional image to be detected using the nodule prediction model;

[0009] The second prediction module is used to obtain the second nodule prediction result of the flipped three-dimensional image using the nodule prediction model.

[0010] The detection module is used to obtain the nodule detection result of the first target object based on the first nodule prediction result and the second nodule prediction result.

[0011] Secondly, one embodiment of this application also provides a nodule detection device, comprising:

[0012] One or more processors;

[0013] Memory, used to store one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors perform calculations corresponding to the nodule detection device as described in the first aspect.

[0015] The above-described method, which involves acquiring a 3D image of the target object containing the first target object through a first acquisition module, then flipping the 3D image to obtain a flipped 3D image through a flipping module, followed by the first prediction module and the second prediction module using a nodule prediction model to obtain a first nodule prediction result for the 3D image to be detected and a second nodule prediction result for the flipped 3D image, respectively, and finally, the detection module obtaining the nodule detection result of the first target object based on the first and second nodule prediction results, solves the technical problems in related technologies where segmenting 3D lung images leads to increased neural network prediction time and limited overall receptive field. This process eliminates the need to segment the 3D image to be detected and allows for direct nodule detection based on the complete first target object. Furthermore, compared to related technologies that rely on post-processing such as non-maximum suppression (NMS) to eliminate overlap predictions for detection results at different angles and thus fail to achieve an integrated effect, the aforementioned technique obtains nodule prediction results at different angles by flipping the 3D image, and then obtains the final nodule detection result based on the nodule prediction results at different angles. This can easily achieve the integrated effect of the nodule prediction model at different angles. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of a nodule detection device provided in one embodiment of this application;

[0017] Figure 2 A flowchart illustrating the operation of a nodule detection device according to one embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the structure of a nodule prediction model provided in one embodiment of this application;

[0019] Figure 4 This is a schematic diagram of a prediction network structure provided in one embodiment of this application;

[0020] Figure 5 A target feature map provided in one embodiment of this application;

[0021] Figure 6 Another target feature map provided in one embodiment of this application;

[0022] Figure 7 Another target feature map provided in one embodiment of this application;

[0023] Figure 8 This is a schematic diagram of the structure of a nodule prediction device provided in one embodiment of this application. Detailed Implementation

[0024] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0025] One embodiment of this application provides a nodule detection device that can be used to detect nodules formed inside the human body. This nodule detection device can be integrated into a nodule detection equipment. The nodule detection equipment can consist of two or more physical entities, or it can consist of a single physical entity. The nodule detection equipment can be a desktop computer, an interactive smart tablet, a computed tomography (CT) scanner, a low-dose spiral CT scanner, or other similar devices.

[0026] In one embodiment, the nodule detection device is equipped with a convolutional neural network capable of detecting nodules. This convolutional neural network is a single-stage, single-scale nodule detection network. In one embodiment, the nodule detection device is described using the detection of pulmonary nodules as an example. In this case, the convolutional neural network deployed in the nodule detection device is a single-stage, single-scale pulmonary nodule detection network (S2PND network). For example, when S2PND detects pulmonary nodules, it is not necessary to segment the lung image. A complete lung image can be input into the S2PND network, and the nodule detection result can be predicted based on the complete lung image. This solves the technical problems in related technologies where segmenting 3D lung images into blocks increases the neural network prediction time and limits the overall receptive field.

[0027] In one embodiment, Figure 1 This is a schematic diagram of a nodule detection device provided in one embodiment of this application, with reference to... Figure 1 When the nodule detection device predicts nodules based on a complete three-dimensional image of the lungs, it specifically includes a first acquisition module 101, a flipping module 102, a first prediction module 103, a second prediction module 104, and a detection module 105.

[0028] The system comprises: a first acquisition module 101 for acquiring a three-dimensional image to be detected, the three-dimensional image displaying a first target object; a flipping module 102 for flipping the three-dimensional image to be detected along at least one flipping direction to obtain a flipped three-dimensional image, each flipping direction corresponding to at least one flipped three-dimensional image; a first prediction module 103 for obtaining a first nodule prediction result of the three-dimensional image to be detected using a nodule prediction model; a second prediction module 104 for obtaining a second nodule prediction result of the flipped three-dimensional image using a nodule prediction model; and a detection module 105 for obtaining a nodule detection result of the first target object based on the first nodule prediction result and the second nodule prediction result.

[0029] For example, the first acquisition module 101 acquires the three-dimensional image to be detected. The three-dimensional image to be detected refers to the three-dimensional image used for nodule detection. The three-dimensional image to be detected displays a complete first target object. The target object refers to the human or animal part where the nodule is detected; the first target object is the target object for the current nodule detection. In one embodiment, when the nodule detection device detects a lung nodule in a human body, the first target object refers to the lung region of the human body. At this time, the three-dimensional image to be detected displays a complete three-dimensional image of the lung region. The source of the three-dimensional image to be detected is not currently limited. For example, a low-dose spiral CT scan of the human lung region can be used to obtain a three-dimensional scan image of the lung. Then, the first acquisition module 101 acquires the three-dimensional scan image and uses it as the three-dimensional image to be detected. In one embodiment, to reduce useless information (such as information expressed by the background region) in the three-dimensional image to be detected, the first acquisition module 101 can further process the three-dimensional scan image after acquiring it, extracting only the image of the lung region from the three-dimensional scan image and using it as the three-dimensional image to be detected. The processing method can be set according to the actual situation. For example, the first acquisition module 101 performs image segmentation on the three-dimensional scan image to segment out the mask corresponding to the lung region. This mask can reflect the position and shape of the lung region. Then, the lung region is extracted from the three-dimensional scan image according to the mask to obtain the three-dimensional image to be detected. In one embodiment, in order to facilitate the subsequent processing of the S2PND network, the first acquisition module 101 scales the three-dimensional image to be detected to a fixed size to ensure that the size of the three-dimensional image input to the S2PND network is fixed. In one embodiment, the three-dimensional image to be detected is located in a three-dimensional coordinate system, where each coordinate point has a corresponding three-dimensional coordinate.

[0030] For example, after the first acquisition module 101 acquires the 3D image to be detected, the flipping module 102 flips the 3D image to be detected. In one embodiment, the flipped 3D image is denoted as the flipped 3D image. The flipping direction used by the flipping module 102 during flipping can be set according to the actual situation. There can be multiple flipping directions. In this case, each flipping direction can obtain at least one flipped 3D image. Each flipped 3D image can show the first target object from different angles, which is convenient for subsequent nodule detection. In one embodiment, the x-axis, y-axis and z-axis directions in the 3D coordinate system where the 3D image to be detected is located are respectively taken as flipping directions. Then, the flipping module 102 flips the 3D image to be detected along the three flipping directions to obtain flipped 3D images in the three flipping directions. Optionally, the flipping angle of each flipping direction can be set according to the actual situation. For example, a flipping function is preset, and the flipping angle is determined by the flipping function. In this case, it is only necessary to substitute the 3D image to be detected and the flipping angle into the flipping function to obtain the flipped image. Optionally, the flipping angle of each flipping direction can also be set randomly.

[0031] For example, the first prediction module 103 acquires the three-dimensional image to be detected output by the first acquisition module 101 and inputs it into the nodule prediction model. The nodule prediction model is a neural network used to predict nodules inside the first target object. It can be understood that when detecting lung nodules, the nodule prediction model is an S2PND network. The model structure of the nodule prediction model can be set according to actual needs. In one embodiment, the nodule prediction model consists of three parts: a backbone network, a feature pyramid network (FPN) network, and a prediction network. Among them, the backbone network is a 3D convolutional network, which has multiple convolutional layers. Each convolutional layer can be composed of multiple 3D residual convolutional networks. The number and structure of the residual convolutional networks can be set according to actual needs. The backbone network is used to extract image features of the three-dimensional image. When the three-dimensional image to be detected is input into the nodule prediction model, the backbone network extracts the image features of the three-dimensional image to be detected. At this time, multiple convolutional layers in the backbone network can output image features. The FPN network is a highly efficient feature extraction network. In the nodule prediction model, the FPN network can fuse and enhance the image features extracted by each convolutional layer in the backbone network. The structure of the FPN network can be set according to actual needs. The prediction network is used to obtain the final feature map based on the fused and enhanced features output by the FPN network. The final feature map shows the features related to nodules identified by the nodule prediction model. Afterwards, the prediction network can also identify the final feature map and output the final nodule prediction result. The nodule prediction result is used to describe the identified nodules. The content of the nodule prediction result can be set according to the actual situation. In one embodiment, the nodule prediction result includes at least one of the following: nodule classification prediction result, size prediction result, and offset prediction result. The classification prediction result is used to describe the confidence that the feature points in the final feature map are nodules. It can be understood that the higher the confidence, the higher the probability of detecting a nodule. The classification prediction result can be presented as a classification prediction heatmap. This heatmap is a three-dimensional image and is used to reflect the confidence that each feature point is a nodule. Size prediction results are used to describe the size of the nodules. For a 3D image, the size prediction results include size information in three directions (x-axis, y-axis, and z-axis), from which information such as the nodule diameter can be obtained. Size prediction results can be presented as a size regression prediction map, which is a 3D image used to show the nodule size. Offset prediction results can be understood as the positional deviation between the nodule in the 3D image and its position on the final feature map. The position of the nodule in the 3D image can be deduced from the offset prediction results. For a 3D image, the nodule position includes positional coordinates in three directions (x-axis, y-axis, and z-axis). In one embodiment, the nodule position is represented by the center point of the region where the nodule is located. Offset prediction results can be presented as an offset regression prediction map, which is a 3D image used to show the nodule offset.It should be noted that the network type used in the prediction network is related to the nodule prediction result. For example, when the nodule prediction result includes classification prediction results, the prediction network includes a convolutional network for obtaining the final feature map and a classification network for obtaining the nodule classification result. The classification network is used to obtain the confidence score of classifying a sample (i.e., a feature point in the final feature map) as a nodule. When the nodule prediction result includes size prediction results, the prediction network includes a convolutional network for obtaining the final feature map and a regression network for obtaining the size prediction result. The regression network regresses the nodule size using a single-scale anchor box with a fixed size. When the nodule prediction result includes offset prediction results, the prediction network includes a convolutional network for obtaining the final feature map and a regression network for obtaining the offset prediction result. This regression network regresses the nodule position offset using the downsampling ratio used when mapping the 3D image to the final feature map as a reference size. It can be understood that when the nodule prediction result includes multiple components, all components share the same final feature map, that is, they share the same convolutional network to obtain the final feature map.

[0032] In one embodiment, when the nodule prediction model processes the 3D image to be detected, the nodule prediction result output by the nodule prediction model is recorded as the first nodule prediction result. That is, the first prediction module 103 outputs the first nodule prediction result.

[0033] For example, the second prediction module 104 acquires the flipped 3D image output by the flipping module 102 and inputs it into the nodule prediction model. At this time, the nodule prediction model processes the flipped 3D image and obtains the corresponding nodule prediction result. In one embodiment, the nodule prediction result obtained from the flipped 3D image is recorded as the second nodule prediction result. Each flipped 3D image corresponds to one second nodule prediction result. It can be understood that the first nodule prediction result can be considered as a nodule prediction result obtained based on the original angle, and the second nodule prediction result can be a nodule prediction result obtained based on the flipped angle; both reflect the nodule prediction result of the first target object from different angles. In one embodiment, the content included in the second nodule prediction result is the same as that included in the first nodule prediction result. That is, both the first and second nodule prediction results include at least one of the nodule classification prediction result, size prediction result, and offset prediction result. The second nodule prediction result and the first nodule prediction result are generated using the same technical means, which will not be elaborated upon here.

[0034] For example, the detection module 105 acquires the first nodule prediction result output by the first prediction module 103 and the second nodule prediction result output by the second prediction module 104. Then, based on the first and second nodule prediction results, the final detection result is obtained, i.e., the nodule detection result of the first target object. In one embodiment, both the first and second nodule prediction results include at least one of a nodule classification prediction result, a size prediction result, and an offset prediction result. Correspondingly, the nodule detection result includes a classification detection result corresponding to the classification prediction result, a size detection result corresponding to the size prediction result, and an offset detection result corresponding to the offset prediction result. That is, the nodule detection result includes at least one of a classification detection result, a size detection result, and an offset detection result, the specific content of which is the same as the first and second nodule prediction results, and the nodule detection result is the final result obtained by integrating the first and second nodule prediction results.

[0035] The second nodule prediction result is the nodule prediction result of the first target object at the corresponding flip angle. To integrate the first and second nodule prediction results, the second nodule prediction result needs to be processed first. This involves restoring the second nodule prediction result according to the flip angle, ensuring that the second nodule prediction result and the first nodule prediction result have the same baseline (i.e., both are at the original angle). This guarantees the integration of the second and first nodule prediction results based on the original angle. The method for restoring the second nodule prediction result can be set according to the actual situation. For example, if the second nodule prediction result includes classification prediction results or size prediction results, it can be directly restored and flipped according to the flip angle. When the second nodule prediction result includes offset prediction results, restoring and flipping according to the flip angle cannot ensure that the second nodule prediction result and the first nodule prediction result have the same baseline. Therefore, a flip formula is used to process the second nodule prediction result. This flip formula is based on the flip function used during flipping. The flip formula mathematically transforms the offset prediction result, ensuring that the transformed second nodule prediction result has the same baseline as the first nodule prediction result. After processing the prediction results of the second nodule, the prediction results of the first and second nodules can be processed using a weighted average method. The weights corresponding to each nodule prediction result can be set according to the actual situation. After averaging the weights of the nodule prediction results, the nodule detection result of the first target object can be obtained. Optionally, after obtaining the nodule detection results, the detection module 105 can mark the three-dimensional image to be detected based on the nodule detection results, that is, display the confidence level, diameter (or size), and location of the nodules. This marking can be directly reflected in the three-dimensional image to be detected, or it can be independent of the three-dimensional image to be detected. It can be understood that when the first target object contains multiple nodules, all multiple nodules can be reflected through the nodule detection results.

[0036] Optionally, after obtaining the nodule detection results, the nodule detection results can also be output to the false positive reduction algorithm model to further classify and judge the nodule detection results, thereby improving the accuracy of the nodule detection results.

[0037] The above-described method, which involves acquiring a 3D image containing a first target object through a first acquisition module, then flipping the 3D image to obtain a flipped 3D image, followed by using a first prediction module and a second prediction module to obtain a first nodule prediction result for the 3D image to be detected and a second nodule prediction result for the flipped 3D image, respectively, using a nodule prediction model, and finally, using a detection module to obtain the nodule detection result of the first target object based on the first and second nodule prediction results, solves the technical problems in related technologies where segmenting 3D lung images leads to increased neural network prediction time and limited overall receptive field. This process eliminates the need to segment the 3D image to be detected, allowing direct nodule detection based on the complete first target object. Furthermore, compared to related technologies that rely on post-processing such as NMS to eliminate overlap predictions for detection results from different angles, thus failing to achieve integration, this method obtains nodule prediction results from different angles by flipping the 3D image, and then obtains the final nodule detection result based on these results, easily achieving the integration effect of the nodule prediction model at different angles.

[0038] In one embodiment of this application, both the first nodule prediction result and the second nodule prediction result include classification prediction results and / or size prediction results. The detection module 105 includes: a restoration and flipping unit, used to restore and flip the second nodule prediction result along the corresponding flipping direction to obtain a third nodule prediction result; and a first weighted averaging unit, used to perform a weighted average of the first nodule prediction result and the third nodule prediction result to obtain the nodule detection result of the first target object.

[0039] For example, when both the first and second nodule prediction results include classification prediction results and / or size prediction results, the detection module 105 includes a restoration and flipping unit and a first weighted averaging unit. The restoration and flipping unit is used to restore and flip the second nodule prediction result according to the flipping direction corresponding to the second nodule prediction result. Restoration and flipping can be understood as reversing the second nodule prediction result according to the flipping direction. After flipping, the second nodule prediction result has the same baseline as the first nodule prediction result. In one embodiment, the restored and flipped second nodule prediction result is recorded as the third nodule prediction result. The third nodule prediction result includes classification prediction results and / or size prediction results. The first weighted averaging unit is used to perform a weighted average of the first and third nodule prediction results to obtain the nodule detection result. At this time, the nodule detection result includes classification detection results and size prediction results.

[0040] In one embodiment of this application, both the first nodule prediction result and the second nodule prediction result include offset prediction results. The detection module 105 includes: a transformation unit, used to perform a flip transformation on the second nodule prediction result using a flip formula to obtain a fourth nodule prediction result; and a second summing and averaging unit, used to perform a weighted average of the first nodule prediction result and the fourth nodule prediction result to obtain the nodule detection result of the first target object.

[0041] For example, when both the first and second nodule prediction results include offset prediction results, the detection module 105 includes a transformation unit and a second weighted averaging unit. The flipping unit performs a simple mathematical transformation on the second nodule prediction result according to a pre-set flipping formula, so that the transformed second nodule prediction result has the same baseline effect as the first nodule prediction result. In one embodiment, the transformed second nodule prediction result is denoted as the fourth nodule prediction result, which is the offset prediction result. In one embodiment, the flipping formula is:

[0042] R_offset_n=(S n -1) / S n -flip(f_R_offset(flip(image,axis=n)),axis=n),

[0043] Where R_offset_n represents the prediction result of the fourth nodule, f_R_offset represents the prediction function used by the nodule prediction model to generate the offset prediction result, flip represents the flip function, axis represents the flip direction, axis = n, image represents the 3D image to be detected, and S nThis represents the size ratio between the flipped 3D image and the target feature map when the flipping direction is n. The target feature map is the final feature map generated during the nodule prediction model processing. In one embodiment, axis = n indicates that the current flipping direction is n, where n can be the direction corresponding to the x-axis, y-axis, or z-axis in the 3D coordinate system. As shown in the above formula, after the 3D image to be detected is flipped along the flipping direction n using the flipping function `flip`, the corresponding flipped 3D image can be obtained. At this time, the flipped 3D image is represented by `flip(image, axis = n)` in the flipping formula. Then, the flipped 3D image is input to the nodule prediction model, and after processing by the prediction function `f_R_offset` in the nodule prediction model, the second nodule prediction result (currently the offset prediction result) can be obtained. At this time, the second nodule prediction result can be represented in the flipping formula as `f_R_offset(flip(image, axis = n))`. Then, substituting the second nodule prediction result (f_R_offset(flip(image,axis=n)) into the flip formula yields the fourth nodule prediction result R_offset_n. Here, the final feature map obtained from the prediction network of the nodule prediction model is denoted as the target feature map. It can be understood that flipping the 3D image can be done according to the size ratio S. n Mapped onto the target feature map.

[0044] For example, the second weighted averaging unit performs a weighted average of the first nodule prediction result and the fourth nodule prediction result to obtain the nodule detection result. At this time, the nodule detection result includes the offset detection result.

[0045] It is understood that when the first nodule prediction result and the second nodule prediction result include classification prediction result, size prediction result and offset prediction result, the detection module 105 may include a restoration and flipping unit, a transformation unit and a weighted averaging unit. The weighted averaging unit integrates the first weighted averaging unit and the second weighted averaging unit. The restoration and flipping unit is used to restore and flip the classification prediction result and the size prediction result in the second nodule prediction result respectively. The transformation unit is used to perform mathematical transformation on the offset prediction result in the second nodule prediction result. Then, the weighted averaging unit averages and weights the classification prediction result in the first nodule prediction result and the second nodule prediction result to obtain the classification detection result, averages and weights the size prediction result in the first nodule prediction result and the second nodule prediction result to obtain the offset detection result. The classification detection result, the size detection result and the offset detection result constitute the final nodule detection result.

[0046] As mentioned above, for classification prediction results and size prediction results, the detection module can easily integrate the prediction results from different angles by restoring and flipping and weighted averaging. For offset prediction results, the detection module can easily integrate the prediction results from different angles by weighted averaging through simple mathematical transformations, thereby achieving the integration effect of the nodule prediction model. No additional training of the nodule prediction model is required, and the detection accuracy is also improved.

[0047] In one embodiment of this application, the first nodule prediction result includes a classification prediction result, and the nodule prediction device further includes: a first neighborhood suppression module, used to perform neighborhood suppression on the first nodule prediction result by max pooling after obtaining the first nodule prediction result of the three-dimensional image to be detected using the nodule prediction model; the second nodule prediction result includes a nodule classification result, and the nodule prediction device further includes: a second neighborhood suppression module, used to perform neighborhood suppression on the second nodule prediction result by max pooling after obtaining the second nodule prediction result of the flipped three-dimensional image using the nodule prediction model.

[0048] For example, given the sparse distribution of lung nodule lesions in the human body, it is known that only one nodule can exist in any region of the classification prediction result. Therefore, when the first nodule prediction result and the second nodule prediction result include the classification prediction result, neighborhood suppression can be applied to the classification prediction result to satisfy the sparse distribution of lung nodule lesions in the human body. Neighborhood suppression refers to considering only the highest confidence level in the spatial neighborhood of any point in the classification prediction result, suppressing other confidence levels within that spatial neighborhood. In one embodiment, the size of the spatial neighborhood can be set according to actual conditions; for example, the size of the spatial neighborhood is 3*3*3. In one embodiment, neighborhood suppression can be implemented through max pooling, such as through a single-layer 3*3*3 max pooling. Max pooling refers to taking the classification prediction result with the highest confidence level in the spatial neighborhood. The PyTorch code for implementing neighborhood suppression is as follows:

[0049] hmax=nn.functional.max_pool3d(heatmap,kernel=3,stride=1)

[0050] keep=(hmax==heat).float()

[0051] heatmap = heatmap * keep

[0052] In one embodiment, to achieve neighborhood suppression of the classification prediction results, the nodule prediction device further includes a first neighborhood suppression module and a second neighborhood suppression module. After the nodule prediction model outputs the classification prediction result of the three-dimensional image to be detected (i.e., the first nodule prediction result), the first neighborhood suppression module uses max pooling to suppress the neighborhood of the classification prediction result. Similarly, after the nodule prediction model outputs the classification prediction result of the flipped three-dimensional image (i.e., the second nodule prediction result), the second neighborhood suppression module uses max pooling to suppress the neighborhood of the classification prediction result.

[0053] Optionally, to ensure the accuracy of the classification prediction results, a confidence threshold can be set. This confidence threshold can be set according to actual needs. After the first neighborhood suppression module performs neighborhood suppression on the classification prediction results in the first nodule prediction results, it can also compare the confidence of the retained classification prediction results with the confidence threshold to filter out classification prediction results that are lower than or equal to the confidence threshold, so as to retain classification prediction results with high confidence.

[0054] It should be noted that the time consumed by neighborhood suppression is relatively small and can be ignored.

[0055] Compared to related technologies, neural networks using anchors require post-processing with Non-Maximum Suppression (NMS) after outputting the prediction results to eliminate overlapping candidate prediction boxes. Furthermore, NMS requires selecting the prediction box with the highest confidence among overlapping prediction boxes and calculating the Intersection over Union (IOU) of the two prediction boxes, resulting in a time complexity of O(N). 2 ), where N is the number of prediction boxes. When the first neighborhood suppression module and the second neighborhood suppression module perform neighborhood suppression on the classification prediction results through max pooling operation, they can quickly realize neighborhood suppression post-processing.

[0056] In one embodiment of this application, the first acquisition module 101 includes: an acquisition unit for acquiring an original three-dimensional image containing a first target object; a segmentation unit for performing image segmentation on the original three-dimensional image to obtain a target object segmentation mask; an extraction unit for extracting a target object region image from the original three-dimensional image based on the target object segmentation mask; and a scaling unit for scaling the target object region image to a set size to obtain a three-dimensional image of the first target object to be detected.

[0057] For example, the acquisition module 101 includes: an acquisition unit, a segmentation unit, an extraction unit, and a scaling unit. The acquisition unit is used to acquire the original 3D image, which can be considered as a 3D image obtained directly from the first target object. For example, after scanning the target object using low-dose spiral CT, the acquired scanned 3D image is the original 3D image.

[0058] The segmentation unit generates a target object segmentation mask. This mask reflects the position and shape of the first target object in the original 3D image. In one embodiment, when the first target object is the lung region of a human body, the target object segmentation mask can be denoted as `lung_mask`. The segmentation unit obtains the target object segmentation mask by performing image segmentation on the original 3D image. Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and extracting the target of interest (i.e., the target object). The techniques used in image segmentation are not currently limited. For example, a threshold-based method can be used to segment the original 3D image. This threshold-based method involves setting a reasonable threshold, comparing each pixel in the original 3D image with the threshold, and performing image segmentation based on this comparison to obtain the target object segmentation mask describing the first target object. Another example is using a neural network method to segment the original 3D image. This neural network method involves constructing and training a segmentation network, then inputting the original 3D image into the segmentation network to output the target object segmentation mask. It is understood that the target object segmentation mask is also a 3D image.

[0059] Since the target object segmentation mask reveals the region where the first target object is located in the original 3D image, after the segmentation unit obtains the target object segmentation mask, the extraction unit can extract the pixels of the target object region in the original 3D image based on the region described in the target object segmentation mask, thus obtaining a target object region image describing the complete target object. In one embodiment, when the first target object is the human lung region, the target object region image can be denoted as lung_voi. It is understood that the target object region image is also a 3D image.

[0060] To facilitate subsequent processing by the nodule prediction model, in one embodiment, a scaling unit scales the target object region image to a set size. The length, width, and height of this set size can be set according to actual needs. In one embodiment, the scaling unit uses a bilinear interpolation algorithm to scale the target object region image to the set size. In practical applications, other methods can also be used to scale the target object region image. The scaled target object region can then be used as the 3D image to be detected. It is understood that in practical applications, when the size of the target object region image is the set size, it is not necessary to scale the target object region.

[0061] As described above, the acquisition module can obtain a three-dimensional image of the target object containing only the complete first target object and with a fixed size by performing image segmentation, extraction and scaling operations on the original three-dimensional image, which is convenient for subsequent processing by the nodule prediction model.

[0062] The following is an exemplary description of the workflow of the nodule detection device in obtaining the prediction result of the first nodule. Figure 2 A flowchart of a nodule detection device provided in one embodiment of this application is shown below. Figure 2 The nodule detection device's acquisition module first acquires the chest CT sequence (i.e., the original 3D image) obtained from a low-spiral CT scan. Then, the acquisition module performs image segmentation on the lung region within the chest CT sequence to extract the lung region image (i.e., the target object region image) and scales it to a set size (i.e., obtain the 3D image to be detected). The scaling and extraction processes are performed within... Figure 2 The images are merged together. Then, the first prediction module inputs the 3D image to be detected into the S2PND network (i.e., the nodule prediction model) to obtain the first nodule prediction result. Based on the first nodule prediction result, the position and size of the nodules in the 3D image to be detected are determined. The first nodule prediction result includes classification prediction result, size prediction result and offset prediction result.

[0063] The above process eliminates the need to segment the 3D image to be detected, shortens the prediction time of the nodule prediction model, and avoids limiting the overall receptive field of the nodule prediction model.

[0064] It should be noted that the process of obtaining the nodule detection result described above is the application process of the nodule prediction model. The nodule prediction device can also realize the training process of the nodule prediction model. It can be understood that when the nodule prediction device trains the nodule prediction model, it can be integrated into the aforementioned nodule prediction device or into other nodule prediction devices, and after training is completed, the nodule prediction model is deployed in the aforementioned nodule prediction device. In one embodiment of this application, when training the nodule prediction model, the nodule prediction device further includes: a second acquisition module, a third prediction module, and a training module.

[0065] The second acquisition module is used to acquire a label 3D image, which displays a second target object and also marks the nodule marking results of the second target object; the third prediction module is used to obtain the fifth nodule prediction result of the label 3D image using a nodule prediction model; and the training module is used to train the nodule prediction model based on the fifth nodule prediction result and the nodule marking results.

[0066] For example, the second acquisition module is used to acquire the 3D image used during the training of the nodule prediction model. In one embodiment, the 3D image used during training is designated as a label 3D image. In another embodiment, the label 3D image displays a target object of the same type as the 3D image to be detected, and the target object displayed in the label 3D image is designated as the second target object. For example, the label 3D image displays the lung region of the second target object. The label 3D image and the 3D image to be detected have the same size. The nodules of the second target object in the label 3D image are known nodules. When acquiring the label 3D image, the nodule labeling result of the second target object in the label 3D image is also specified. The nodule labeling result is used to describe the relevant information of the nodules in the label 3D image, and the content of the nodule labeling result is related to the nodule prediction result output by the nodule prediction model. For example, when the nodule prediction result includes classification prediction results, the nodule labeling result includes classification labeling results and location labeling results. The location labeling result describes the position of the nodule center point in the label's 3D image, which is the 3D coordinate of the nodule center point. The classification labeling result defines the nodule category corresponding to the nodule center point position as a nodule. In one embodiment, the nodule category includes two categories, 0 and 1, where 1 represents a nodule and 0 represents a non-nodule. In this case, in the label's 3D image, the nodule category corresponding to the position of the nodule center point is 1, and the nodule category corresponding to the position of the non-nodule center point is 0. The location labeling result and classification labeling result can clearly identify the nodule in the 3D image to be detected and its position. As another example, when the nodule prediction result includes size prediction results, the nodule labeling result includes diameter labeling results. The diameter labeling result shows the diameter of the nodule in the label's 3D image, which is the diameter of the nodule in three directions (x-axis, y-axis, and z-axis). Generally, nodules are circular in shape, and the size of the nodule can be inferred from its diameter. For example, when the nodule prediction result includes the offset prediction result, the nodule labeling result includes the classification labeling result and the location labeling result. Also, when the nodule prediction result includes the classification prediction result, the size prediction result, and the offset prediction result, the nodule labeling result includes the classification labeling result, the location labeling result, and the diameter labeling result.

[0067] It should be noted that the source of the 3D label images and the method for determining the nodule labeling results are not limited in this embodiment. It is understood that the number of 3D label images acquired by the second acquisition module during each training process is currently not limited.

[0068] For example, the third prediction module inputs the label 3D image into the nodule prediction model, and the nodule prediction model outputs the nodule prediction result. In one embodiment, the nodule prediction result output during the training process of the nodule prediction model is recorded as the fifth nodule prediction result. It is understood that the accuracy of the fifth nodule prediction result output when the nodule prediction model is built is relatively low, and the accuracy of the fifth nodule prediction result output by the nodule prediction model increases as the training process progresses.

[0069] For example, the training module is used to construct a loss function for the nodule prediction model based on the fifth nodule prediction result and the nodule labeling result, and to update the model parameters of the nodule prediction model through the loss function, thereby achieving the purpose of training the nodule prediction model. The type of loss function can be set according to the actual situation. It can be understood that the more accurate the nodule prediction model, the closer the fifth nodule prediction result and the nodule labeling result are, and the more convergent the loss function. Loss function convergence can be understood as the value of the loss function remaining within a certain range over consecutive training iterations; when the loss function converges, the accuracy of the nodule prediction model reaches a standard. For example, a set of labeled 3D images can be pre-constructed. During each training iteration, the second acquisition module selects labeled 3D images from the set, and the third prediction module inputs the labeled 3D images into the nodule prediction model to obtain the fifth nodule prediction result. Then, the training module constructs a loss function based on the fifth nodule prediction result and the nodule labeling result, and updates the model parameters of the nodule prediction model based on the loss function. At this point, one training iteration is considered complete, and the above process is repeated to start a new training iteration until the loss function converges.

[0070] The above-described method, which involves acquiring labeled 3D images with clear nodule markings through the second acquisition module, followed by the third prediction module inputting these images into the nodule prediction model to obtain the fifth nodule prediction result, and then the training module training the nodule prediction model based on the nodule marking result and the fifth nodule prediction result, solves the technical problems in related technologies where segmenting lung 3D images leads to increased neural network prediction time and limited overall receptive field. This process eliminates the need to segment the 3D image to be detected, allowing direct nodule detection based on the complete second target object, thus omitting the segmentation and merging processes before and after the nodule prediction model and significantly reducing prediction time.

[0071] In one embodiment of this application, the nodule prediction model includes a 3D convolutional network, a feature pyramid network, and a prediction network. The third prediction module includes: an extraction unit for extracting image features of the label 3D image using the 3D convolutional network; an enhancement and fusion unit for fusing and enhancing the image features using the feature pyramid network to obtain fused and enhanced features; and a classification and regression unit for processing the fused and enhanced features using the prediction network to generate the final target feature map and outputting the fifth nodule prediction result based on the target feature map.

[0072] For example, Figure 3 This is a schematic diagram of the structure of a nodule prediction model provided in one embodiment of this application. (Reference) Figure 3 The nodule prediction model includes a 3D convolutional network ( Figure 3 (referred to as backbone) and feature pyramid network ( Figure 3 The network is referred to as the simple feature pyramid net and the prediction network ( Figure 3 (Referring to as Head net). The 3D convolutional network serves as the backbone network of the nodule prediction model, primarily used to extract image features from the input 3D image. In one embodiment, the 3D convolutional network consists of multiple convolutional layers. Figure 3 Taking four convolutional layers as an example, the 3D convolutional network includes convolutional layers C1, C2, C3, and C4. Each convolutional layer consists of multiple residual convolutional networks. For example, convolutional layers C1 to C4 each contain [3, 4, 6, 3] residual convolutional networks. The structure of each residual convolutional network can be set according to the actual situation. For example, each residual convolutional network adopts a resent34-3d structure, that is, it adopts a three-dimensional resnet34, and the number of feature layer channels of convolutional layers C1 to C4 are [32, 64, 64, 64], respectively. For example, after the label 3D image is input into the 3D convolutional network, convolutional layers C2 to C4 all obtain the image features output by the previous convolutional layer and obtain the image features of this layer. Furthermore, the image features output by convolutional layers C2 to C4 are the image features output by the 3D convolutional network.

[0073] The Feature Pyramid Network is used to fuse and enhance image features output from 3D convolutional layers, specifically fusing and enhancing the image features output from convolutional layers C2 to C4. For example, refer to... Figure 3 The feature pyramid network uses bilinear interpolation for upsampling and is combined with the lateral channel (i.e., Figure 3The image features output from the left-hand convolutional layer (C4) are summed (after a 1x1x1 convolution) and then subjected to a 3x3x3 single-layer convolution to obtain the fused enhanced features. At this point, sub-network P4 of the feature pyramid network acquires the image features output from convolutional layer C4, passes these features through a 1x1x1 convolutional layer, and then through a 3x3x3 single-layer convolution to obtain the fused enhanced features output by sub-network P4. Next, sub-network P3 performs bilinear interpolation upsampling on the fused enhanced features output by sub-network P4 to obtain the upsampled result. Simultaneously, sub-network P3 acquires the image features output from convolutional layer C3, passes these features through a 1x1x1 convolutional layer, adds them to the upsampled result, and then passes them through a 3x3x3 single-layer convolution to obtain the fused enhanced features output by sub-network P3. The processing method for sub-network P2 is the same as that for sub-network P3, and will not be elaborated here. The fused enhanced features output by sub-network P2 are the final fused enhanced features output by the feature pyramid network.

[0074] The prediction network outputs the corresponding nodule prediction result based on the fused enhanced features. The prediction network consists of two sub-networks. One sub-network obtains the final feature map based on the fused enhanced features. In one embodiment, the final feature map is denoted as the target feature map, which is the final feature map related to the nodule. The target feature map contains feature points describing the nodule. Optionally, the aspect ratio of the target feature map is the same as or very similar to that of the 3D image of the input nodule prediction model, and the aspect ratio of the target feature map is smaller than that of the 3D image. In this case, for the input label 3D image, the 3D coordinates used to describe the nodule can be downsampled to the target feature map. For example, this sub-network is a convolutional network and can have multiple convolutional layers. In one embodiment, this sub-network consists of four 3*3*3 convolutional layers; the structure of each convolutional layer is not limited in the embodiment. The other sub-network of the prediction network outputs the fifth nodule prediction result based on the target feature map. The specific structure of this sub-network is related to the content included in the fifth nodule prediction result. For example, if the fifth nodule prediction result includes a classification prediction result, then this sub-network consists of a classification network. This classification network determines the confidence level of each feature point in the target feature map as a nodule, thereby obtaining the classification prediction result. When the fifth nodule prediction result includes a size prediction result or an offset prediction result, this sub-network consists of a regression network. This regression network performs size or offset regression on the target feature map, thereby obtaining the size prediction result or the offset prediction result. In one embodiment, Figure 4 This is a schematic diagram of a prediction network structure provided in one embodiment of this application. (Reference) Figure 4 Taking the nodule prediction results, which include nodule classification prediction results, size prediction results, and offset prediction results, as an example, the prediction network consists of four subnetworks, one of which is a... Figure 4 The feature map network in the pyramid is used as a branch network, containing 4 convolutional layers, to obtain the target feature map based on the fused and enhanced features output by the pyramid feature network. Figure 4 for Figure 3 Taking the specific structure of the prediction network shown as an example, when a 3D feature map of size D*H*W*1 is input, a target feature map of size D / 4*H / 4*W / 4*C can be obtained, where D is the length of the 3D feature map, H is the height of the 3D feature map, W is the width of the 3D feature map, and the number of channels in the 3D feature map is 1. C is the number of channels in the target feature map. A subnetwork ( Figure 4 The Classification network in the dataset is a classification network that outputs a classification prediction heatmap (i.e., a classification prediction result) of D / 4*H / 4*W / 4*1 based on the target feature map. This classification prediction result shows one nodule category, i.e., the predicted nodule. A subnetwork ( Figure 4 The Offset Regression network in the model is a regression network used to output a D / 4*H / 4*W / 4*3 offset regression prediction map (i.e., offset prediction result) based on the target feature map. This offset prediction result shows the offset of the nodule in three directions (x-axis, y-axis, and z-axis). A sub-network ( Figure 4 The Size Regression network in the diagram is a regression network used to output a size regression prediction map (i.e., size prediction result) of D / 4*H / 4*W / 4*3 based on the target feature map. This size prediction result shows the diameter length of the nodule in three directions (x-axis, y-axis, and z-axis). It can be understood that the structure and network parameters of the two regression networks are set according to the actual situation. To distinguish between the two regression networks, the regression network for size prediction is denoted as the first regression network, and the regression network for offset prediction is denoted as the second regression network. Figure 4 It can be seen that the classification network and the two regression networks share a single feature map network.

[0075] In training Figure 3 In the nodule prediction model shown, the third prediction module includes an extraction unit, an enhancement fusion unit, and a classification regression unit. The extraction unit is used to extract image features from the label's 3D image using a 3D convolutional network. Figure 3 The middle part represents the features output by convolutional layers C2 to C4. The enhancement fusion unit is used to obtain fused and enhanced features using the feature pyramid network. Figure 3 (The features output by subnetwork P2 are shown in the image). The classification and regression unit is used to obtain the prediction result of the fifth nodule using the prediction network. In one embodiment, the prediction result of the fifth nodule includes the nodule's classification prediction result, size prediction result, and offset prediction result. In this case, the reference... Figure 4The prediction network consists of a branch network, a classification network, a first regression network, and a second regression network. The branch network is used to generate a target feature map based on the fused enhanced features. The classification network is used to output the classification prediction result based on the target feature map. The first regression network is used to output the size prediction result based on the target feature map. The second regression network is used to output the offset prediction result based on the target feature map.

[0076] It is understandable that when training a nodule prediction model, if the device has more GPU memory, the nodule prediction model can be replaced with a more complex network, such as using a residual convolutional network of resent101 or resent121 for each convolutional layer of a 3D convolutional network, or using a feature pyramid network of BIFPN or PAFPN.

[0077] The nodule prediction model described above uses a simple 3D convolutional network and feature pyramid network for feature extraction and fusion enhancement. It also uses a single branch network to obtain the target feature map to achieve both classification and regression. Compared with neural networks in related technologies, the nodule prediction model uses a simpler and lighter 3D convolutional network and feature pyramid network, and designs a more concise and efficient prediction network for classification and regression. It compresses two or more branch networks into one branch network, which greatly reduces the number of model parameters.

[0078] In one embodiment of this application, the training of a nodule prediction model is described using the example of a fifth nodule prediction result including nodule classification prediction result, size prediction result, and offset prediction result, and a nodule labeling result including nodule classification labeling result, location labeling result, and diameter labeling result. The training module includes: a mapping unit, used to map the three-dimensional coordinates recorded in the location labeling result to the target feature map according to a set downsampling ratio, and perform rounding operations to clearly define the nodule mapping feature points in the target feature map; the three-dimensional coordinates are the three-dimensional coordinates of the nodule in the label three-dimensional image, and the target feature map is the final feature map generated during the nodule prediction model processing; a search unit, used to search for positive sample feature points and negative sample feature points in the target feature map based on the nodule mapping feature points; a first loss construction unit, used to construct a classification loss function based on the classification prediction result and classification labeling result corresponding to the positive sample feature points and negative sample feature points; and a first round... The system comprises the following units: a classification unit, which draws a single-size 3D anchor box of a set size on the target feature map and obtains the size labeling result based on the single-size 3D anchor box and the diameter labeling result; a second loss construction unit, which constructs a first regression loss function based on the size labeling result and the size prediction result; a second regression unit, which obtains the offset labeling result based on the nodule mapping feature points, downsampling ratio, and position labeling result; a third loss construction unit, which constructs a second regression loss function based on the offset labeling result and the offset prediction result; and a parameter update unit, which updates the model parameters of the nodule prediction model based on the classification loss function, the first regression function, and the second regression loss function.

[0079] For example, when the prediction result of the fifth node includes the classification prediction result, the size prediction result, and the offset prediction result, the training module includes: a mapping unit, a search unit, a first loss construction unit, a first regression unit, a second loss construction unit, a second regression unit, a third loss construction unit, and a parameter update unit.

[0080] Specifically, the mapping unit can map the 3D coordinates of a nodule displayed in the label's 3D image to the target feature map, thereby determining the nodule's position in the target feature map. It can be understood that, for the label's 3D image, a nodule can be considered a target; in this case, a nodule in the label's 3D image can be identified as {T}. j}, T j Let T represent the j-th node. For node T... j In this regard, the nodule labeling result can be represented as T j =(p z (j) ,p y (j) ,p x (j) ,d z (j) ,d y(j) ,d x (j) ,c (j) ), where (p z (j) ,p y (j) ,p x (j) (d) represents the location marker of the nodule in the label's 3D image, specifically the 3D coordinates of the nodule's center point in the label's 3D image. z (j) ,d y (j) ,d x (j) The value represents the diameter of the nodule in the 3D image of the label, specifically the diameter of the nodule in three directions (x-axis, y-axis, and z-axis) of the 3D image of the label. (j) This refers to the classification labeling result of nodules in the 3D image, specifically whether the 3D coordinates corresponding to the center point of the nodule indicate a nodule type. (This can be understood as c) (j) It can be 0 or 1, where 1 indicates that the corresponding 3D coordinates are nodules, and 0 indicates that the corresponding 3D coordinates are non-nodules. In one embodiment, the mapping unit can map the position marking result (p z (j) ,p y (j) ,p x (j) When mapping to the target feature map, a set downsampling ratio is used, where the downsampling ratio is denoted as S. n , n∈(z,y,x), meaning that there is a corresponding downsampling ratio in the direction corresponding to each coordinate axis. S n It can be set according to the actual situation. For example, when the diameter of the nodule in a certain direction is 2nm-32nm, and the following settings are adopted... Figure 3 When using the nodule prediction model shown, the downsampling ratio in all three directions can be set to 4. For example, using... Figure 3 When using the nodule prediction model, the downsampling ratio in all three directions can be set to 8 when the fused enhancement features are output from subnetwork P3. For example, in... Figure 3Based on the nodule prediction model, a subnetwork P1 is added above subnetwork P2. When subnetwork P1 outputs fused and enhanced features, the downsampling ratio in all three directions can be set to 2. In one embodiment, when mapping the location marking result to the target feature map according to the downsampling ratio, its three-dimensional coordinates in the target feature map may not be integers (i.e., not the three-dimensional coordinates of the feature point). In this case, it is necessary to round the three-dimensional coordinates of the location marking result in the target feature map to ensure that the three-dimensional coordinates are integers (i.e., rounded to a feature point in the target feature map). At this time, the feature point corresponding to the integer three-dimensional coordinates can be considered as the feature point representing the nodule. In one embodiment, this feature point is recorded as the nodule mapping feature point. The three-dimensional coordinates of the nodule mapping feature point on the target feature map can be expressed as (p z_int (j) ,p y_int (j) ,p x_int (j) For example, the implementation method of the rounding operation can be set according to the actual situation, such as rounding down or rounding to the nearest integer. In one embodiment, rounding down is used as an example. In this case, the nodule mapping feature point can be represented as follows: Where, n∈(z,y,x), S n p is the downsampling ratio in the n-direction. n p represents the three-dimensional coordinates of the location marking result in the n-direction. n_int The coordinates of the nodule mapping feature points in the n-direction are given. This is a floor operation.

[0081] After obtaining the nodule mapping feature points, the search unit searches for positive and negative sample feature points in the target feature map based on these points. Positive sample feature points describe nodules, while negative sample feature points describe non-nodules. For example, Figure 5 A target feature map provided in one embodiment of this application, with reference to Figure 5 This is a two-dimensional representation of the local target feature map, showing a two-dimensional local target feature map in a certain direction. (Reference) Figure 5 The target feature map contains 25 feature points (intersections of the dashed lines and intersections of the dashed lines with the boundary). Based on the anchor detection method, K anchor boxes can be predefined on the target feature map by windowing. The size and proportion of different anchor boxes are different. Generally, the number of anchor boxes is 9 or 12. Figure 5Three anchor boxes are shown. Then, the IOU value between the anchor boxes and the ground truth bounding boxes (the bounding boxes containing the actual nodules, which can be obtained from the 3D image of the labels) is calculated, and two thresholds T1 and T2 are set. T1 and T2 can be set according to actual conditions, such as 0.7 and 0.3 respectively. When the IOU of an anchor box is greater than T1, it is considered a positive sample. Figure 5 The anchor frame (L1) is considered a negative sample when the IOU of the anchor frame is less than T2. Figure 5 In the anchor box (L2), the remaining feature points of the target feature map are ignored when calculating the classification loss function. Then, based on the size of the anchor box for positive samples, the offset and size between the anchor box and the actual nodule are regressed (i.e., offset labeling and size labeling results). This method requires constructing anchor boxes of different sizes. For example, Figure 6 This is another target feature map provided in one embodiment of the present application. Figure 6 and Figure 5It is the same two-dimensional local target feature map, containing 25 feature points. Based on the anchor-free detection method, when mapping the 3D coordinates recorded in the location labeling results to the target feature map, the feature point S' closest to the 3D coordinates S' (the coordinates of the actual nodule in the target feature map) in the target feature map is determined as a positive sample feature point, and the remaining samples in the target feature map are determined as negative sample feature points. The offset and size (i.e., offset labeling results and size labeling results) are directly regressed based on the positive sample feature points, and the L1 loss function is calculated. This method lacks a transition between positive and negative sample feature points, leading to training interference. Furthermore, when nodules of different sizes have similar relative regression errors, the difference in the L1 loss function is significant, which is detrimental to model training. In one embodiment, to avoid the above problems, a size-anchor (fixed anchor frame size) detection method is used. In this case, since the nodule mapping feature points are the feature points corresponding to the nodule in the label 3D image in the target feature map, the nodule mapping feature points are directly used as positive sample feature points. Because the nodule has a certain size, other feature points in the neighborhood of the positive sample feature points may also be positive or negative sample feature points. To avoid these feature points affecting the nodule prediction... To mitigate the impact of the measurement results, these feature points are treated as ignored sample feature points. Ignored sample feature points mean that these feature points are ignored in subsequent processing. Then, the remaining sample points in the target feature map are treated as negative sample feature points. Based on this, the search unit includes: a positive sample determination subunit, used to treat nodule mapping feature points as positive sample feature points; and a negative sample determination subunit, used to find negative sample feature points in the target feature map. Negative sample feature points are all feature points in the target feature map other than positive sample feature points and ignored sample feature points. Ignored sample feature points are all feature points within the neighborhood of positive sample feature points. Feature points. For example, the positive sample determination subunit directly uses the nodule-mapped feature points as positive sample feature points. The negative sample determination subunit searches for feature points within the neighborhood of the positive sample feature points in the target feature map as ignored sample feature points, and uses the remaining feature points in the target feature map as negative sample feature points. It can be understood that when multiple nodules exist in the label's 3D image, the positive sample feature points and ignored sample feature points in the target feature map can be identified based on each nodule, and the remaining feature points in the target feature map can then be used as negative sample feature points. It can be understood that the aforementioned neighborhood can be set according to the actual situation; for example, Figure 7 This is yet another target feature map provided in one embodiment of this application. (See reference...) Figure 7 and Figure 5 This is a two-dimensional local target feature map containing 25 feature points, for reference. Figure 7 In the size-anchor detection method, feature point p n_int These are nodule mapping feature points, which are used as positive sample feature points. Feature point p n_intThe ignored sample feature points in the neighborhood are feature points p. n_int The surrounding 8 feature points, Figure 7 In this study, the ignored sample feature points are denoted as feature point h, and the remaining 16 feature points are negative sample feature points. In practical applications, other methods can also be used to determine positive and negative sample feature points, such as determining the ignored sample feature points based on the nodule size or the coverage area when the nodule is mapped to the target feature map, and then determining the positive and negative sample feature points.

[0082] As can be understood, the classification label for a positive sample feature point is a nodule, with a value of 1, while the classification label for a negative sample feature point is a non-nodule, with a value of 0. Positive and negative sample feature points also have corresponding classification prediction results, which are obtained through a nodule prediction model and used to represent the confidence level that the feature point is a nodule.

[0083] In one embodiment, the first loss construction unit can construct a loss function based on the classification prediction results and classification label results corresponding to positive and negative sample feature points. This loss function reflects the difference between the classification label results and the classification prediction results corresponding to positive and negative sample feature points. In one embodiment, this loss function is denoted as the classification loss function. The classification loss function can be set according to the actual situation. In one embodiment, the standard Focal loss function is used as the classification loss function. In this case, the classification loss function can be expressed as:

[0084] FL(p t )=-α t (1-p t ) γ log(p t )

[0085] Where, p t =pt + (1-p)(1-t), α t = αt + (1-α)(1-t), where p is the classification prediction result, i.e., the confidence score of the corresponding feature point in the target feature map as a nodule, p∈[0,1], t is the classification label result, t∈{0,1}, when the corresponding feature point in the target feature map is a positive sample feature point, t is 1, when the corresponding feature point is a negative sample feature point, t is 0, and the sample feature points are ignored and not included in the calculation of the classification loss function. α and γ are parameters, and their specific values ​​can be set according to the actual situation. In one embodiment, α is 0.25 and γ is 2. FL(p t The classification loss function is denoted as . It can be understood that by substituting the classification prediction results and classification label results of each feature point into the above formula, the specific value of the classification loss function can be obtained.

[0086] For example, when the size-anchor detection method regresses size, it pre-constructs a single-size 3D anchor frame of a predetermined size. The anchor frame can be understood as a priori bounding box constructed with the anchor point as the center. A single-size 3D anchor frame refers to a 3D anchor frame with fixed length, width, and height. The size of the single-size 3D anchor frame can be set according to the actual situation. In one embodiment, since the nodule is roughly circular, the single-size 3D anchor frame is a cube. In another embodiment, when the size of the label's 3D image is D*H*W, the size of the single-size 3D anchor frame is AD*AH*AW, and AD=AH=AW. In this case, the size of the single-size 3D anchor frame can be expressed as A. z =A y =A x =A. The value of A can be set according to the actual situation, for example, using... Figure 3 When using the nodule prediction model shown, A can be set to 8. For example, Figure 7 The diagram illustrates a two-dimensional frame (square) of a single-size 3D anchor frame (s) in one direction. The frame has a width of AW and a height of AH. In one embodiment, the first regression unit normalizes the diameter marking result using the single-size 3D anchor frame. In another embodiment, this normalized result is recorded as the size marking result. That is, the size marking result is regressed using the single-size 3D anchor frame. In one embodiment, the formula for the size marking result is: size n =log(d n / A n ), where n∈(z,y,x), size n The dimension marking result in the n-direction, d n For the diameter marking results in the n-direction, A n Let A be the single-dimensional three-dimensional anchor frame in the n-direction. Optionally, when the diameter of the nodule in a certain direction is 2nm to 32nm and A is 8, d / A∈[1 / 4,4].

[0087] In one embodiment, the second loss construction unit can construct a loss function based on the size marking results and the size prediction results. This loss function can reflect the difference between the size marking results and the size prediction results. In one embodiment, this loss function is denoted as the first regression loss function. The first regression loss function can be set according to the actual situation. In one embodiment, the smoothL1 loss function is used as the first regression loss function. In this case, the first regression loss function can be expressed as:

[0088]

[0089] Where, n∈(z,y,x), The dimension marking results in the n-direction. Here, || represents the predicted dimension along the n-axis, and || represents the L1 norm. It can be understood that 0.5 in the above formula is a parameter that can be changed according to the actual situation. L1 (x) is the first regression loss function. It can be understood that after substituting the size prediction result and the size labeling result into the above formula, the specific value of the first regression loss function can be obtained.

[0090] For example, there are some differences between the 3D coordinates of the nodule-mapped feature points in the target feature map and the 3D coordinates recorded in the location marking results mapped to the 3D coordinates in the target feature map. These differences can reflect the offset of the actual nodule in the target feature map. For example, refer to... Figure 7 The three-dimensional coordinates of point S are the three-dimensional coordinates recorded in the location marking results mapped to the three-dimensional coordinates in the target feature map, and their coordinates are the same as those of the nodule mapping feature point p. n_int There are some differences. In one embodiment, when the size-anchor detection method regresses the offset, the second regression unit maps the position marking result to the target feature map using a downsampling ratio and calculates the difference between the position marking result and the single-scale difference between the position marking result and the nodule mapping feature point to obtain the offset marking result. That is, the offset marking result is regressed using a downsampling ratio. It can be understood that since the mapping unit has already determined the downsampling ratio and the three-dimensional coordinates of the position marking result mapped to the target feature map, as well as the three-dimensional coordinates of the nodule mapping feature point, the second regression unit can directly calculate the difference between the two three-dimensional coordinates. In one embodiment, the formula for calculating the offset marking result is: offset n =p n / S n -p n_int Where n∈(z,y,x), offset n The result is labeled with the offset in the n-direction, S n S represents the downsampling ratio in the n-direction, which can be understood as S z =S y =S x =S, meaning the downsampling ratio is the same in all three directions, S. In one embodiment, S = 4, p n p represents the three-dimensional coordinates of the location marking result in the n-direction. n_int The coordinates of the nodule mapping feature points in the n-direction are given.

[0091] In one embodiment, the third loss construction unit can construct a loss function based on the offset labeling result and the offset prediction result. This loss function can reflect the difference between the offset labeling result and the offset prediction result. In one embodiment, this loss function is denoted as the second regression loss function. The second regression loss function can be set according to the actual situation. In one embodiment, the smoothL1 loss function is used as the second regression loss function. At this time, the second regression loss function can be expressed as:

[0092]

[0093] Where, n∈(z,y,x), The results are labeled with the offset in the n-direction. The offset prediction result is in the n-direction, and || is the L1 norm. It can be understood that 0.5 in the above formula is a parameter that can be changed according to the actual situation. (smooth 2) L1 (x) is the second regression loss function. It can be understood that by substituting the offset prediction result and the offset labeling result into the above formula, the specific value of the second regression loss function can be obtained.

[0094] Since the first regression loss function and the second regression loss function differ only in the data type substituted, the first regression loss function and the second regression loss function can be combined and written as:

[0095]

[0096] Where, n∈(z,y,x), O * O' is the true value in the n-direction (i.e., the dimension mark result or offset mark result), O' is the predicted value in the n-direction (i.e., the dimension prediction result or offset prediction result), and || is the L1 norm. It can be understood that 0.5 in the above formula is a parameter, which can be changed according to the actual situation.

[0097] For example, the parameter update unit can construct the total loss function of the nodule prediction model based on the classification loss function, the first regression function, and the second regression loss function. In one embodiment, the total loss function can be expressed as:

[0098]

[0099] Where, N pos N offset and N size Equal, both representing the number of positive sample feature points in the target feature map. FL represents the classification loss function, smooth 1 L1 This represents the first regression function, smooth 2. L1 The second regression function is represented by the parameter update unit. After substituting each loss function into the total loss function, the parameter update unit obtains the value of the total loss function and updates the model parameters of the nodule prediction model based on the value of the total loss function to improve the accuracy of the nodule prediction model. At this point, one training process of the nodule prediction model can be considered complete. Afterward, a new training process can be started to continue training the nodule prediction model until the total loss function converges.

[0100] Understandably, in practical applications, nodule prediction results may include at least one of classification prediction results, size prediction results, and offset prediction results. Correspondingly, the loss function of the nodule prediction model may include at least one of classification loss function, first regression loss function, and second regression loss function.

[0101] As described above, by using nodule-mapped feature points as positive sample feature points and implementing regression of size and offset using a single-size 3D anchor box and downsampling ratio, it is possible to effectively detect sparsely distributed targets (i.e., nodules). Furthermore, the loss function is simpler and has fewer hyperparameters. The training process only requires constructing a proportional size (i.e., a fixed-size anchor box) based on the nodule size range. Moreover, when using the focalloss function as the classification loss function, there is no need to design other loss parameters, which simplifies the training complexity.

[0102] The following example uses a 2080TI graphics card (11GB VRAM) for nodule detection to illustrate the graphics card performance during nodule prediction model training and application. In one embodiment, the size of the 3D image input to the nodule prediction model is 224*224*224, and the batch size is 1. In this case, the graphics card performance is shown in Table 1.

[0103] Input dimensions Training memory usage Predicted video memory usage 224*224*224 8.7G 2.7G

[0104] Table 1

[0105] Referring to Table 1, when training the nodule prediction model, the GPU memory usage was 8.7G, and when applying the nodule training model, the GPU memory usage was 2.7G, both showing good performance.

[0106] In one embodiment, LUNA16 was used as the dataset containing the labeled 3D images, and 10-fold cross-validation was employed as the algorithm for testing the nodule prediction model. In this case, the FROC (free-response ROC) score of the nodule prediction model during training is shown in Table 2 below:

[0107] method Data partitioning FROC score (%) DeepLung 10-fold 84.2 S4ND 10-fold 89.7 3DFPN 10-fold 91.9 S2PND(ours) 10-fold 92.1 S2PND_flips(ours) 10-fold 93.5

[0108] Table 2

[0109] Table 2 shows that the FROC scores of the DeepLung neural network, S4ND neural network, and 3D FPN neural network mentioned in the related technologies are 84.2, 89.7, and 91.9, respectively. The nodule prediction model S2PND achieves a high FROC score of 92.1. Furthermore, after adding flipping during the nodule prediction model application, the FROC score of the S2PND_flips model reaches 93.5, significantly higher than the FROC scores of the related technologies.

[0110] It is worth noting that in the embodiments of the nodule prediction device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0111] Figure 8 This is a schematic diagram of the structure of a nodule prediction device provided in one embodiment of this application. Figure 8 As shown, the nodule prediction device includes a processor 20, a memory 21, an input device 22, and an output device 23; the number of processors 20 in the nodule prediction device can be one or more. Figure 8 Taking a processor 20 as an example, the processor 20, memory 21, input device 22, and output device 23 in the nodule prediction device can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0112] The memory 21, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the algorithm execution of the nodule prediction device in this embodiment (e.g., the first acquisition module, flipping module, first prediction module, second prediction module, and detection module in the nodule prediction device). The processor 20 executes various functional applications and data processing of the nodule prediction device by running the software programs, instructions, and modules stored in the memory 21, that is, it realizes the calculations corresponding to the nodule prediction device described above.

[0113] The memory 21 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the nodule prediction device. Furthermore, the memory 21 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 21 may further include memory remotely located relative to the processor 20, which can be connected to the nodule prediction device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The input device 22 can be used to receive input digital or character information, generate key signal inputs related to user settings and function control of the nodule prediction device, and acquire raw three-dimensional images or three-dimensional images to be detected. The output device 23 may include a display device such as a display screen.

[0115] The aforementioned nodule prediction device includes the aforementioned nodule prediction apparatus and has the corresponding functions and beneficial effects.

[0116] Furthermore, embodiments of this application also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the calculations corresponding to the nodule prediction device provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0118] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A nodule detection device, characterized in that, include: The first acquisition module is used to acquire a three-dimensional image to be detected, wherein the three-dimensional image to be detected displays a complete first target object, the first target object being the target object for the current nodule detection, and the target object referring to the human or animal part of the nodule to be detected; A flipping module is used to flip the three-dimensional image to be detected along at least one flipping direction to obtain a flipped three-dimensional image, wherein each flipping direction corresponds to at least one flipped three-dimensional image; The first prediction module is used to obtain the first nodule prediction result of the three-dimensional image to be detected using the nodule prediction model; The second prediction module is used to obtain the second nodule prediction result of the flipped three-dimensional image using the nodule prediction model. The detection module is used to obtain the nodule detection result of the first target object based on the first nodule prediction result and the second nodule prediction result; The 3D labeled image used in training the nodule prediction model includes classification and location labels of nodules in the second target object. Both the first and second nodule prediction results include nodule classification prediction results. During training, the nodule prediction model maps the 3D coordinates recorded by the location labels of the 3D labeled image to the target feature map according to a set downsampling ratio and performs rounding to identify nodule mapping feature points in the target feature map. Based on the nodule mapping feature points, positive and negative sample feature points are found in the target feature map. A classification loss function is constructed based on the classification prediction results and classification labels corresponding to the positive and negative sample feature points to train the nodule prediction model. The target feature map is the final feature map generated during the nodule prediction model processing. The classification prediction result is the nodule prediction result output by the nodule prediction model based on the 3D labeled image. The positive sample feature points are nodule mapping feature points. The negative sample feature points are other feature points in the target feature map besides the positive sample feature points and ignored sample feature points. The ignored sample feature points are all feature points in the neighborhood of the positive sample feature points.

2. The nodule detection device according to claim 1, characterized in that, The first nodule prediction result and the second nodule prediction result also include at least one of the nodule size prediction result and the offset prediction result.

3. The nodule detection device according to claim 2, characterized in that, Both the first nodule prediction result and the second nodule prediction result include classification prediction results and size prediction results. The detection module includes: The recovery and flipping unit is used to recover and flip the second nodule prediction result along the corresponding flipping direction to obtain the third nodule prediction result; The first weighted averaging unit is used to perform a weighted average of the first nodule prediction result and the third nodule prediction result to obtain the nodule detection result of the first target object. Both the first nodule prediction result and the second nodule prediction result include offset prediction results. The detection module includes: The transformation unit is used to perform a flip transformation on the second nodule prediction result using a flip formula to obtain the fourth nodule prediction result. The second weighted averaging unit is used to perform a weighted average of the first nodule prediction result and the fourth nodule prediction result to obtain the nodule detection result of the first target object.

4. The nodule detection device according to claim 3, characterized in that, The flipping formula is: R_offset_n = (S n -1) / S n - flip(f_R_offset(flip(image, axis=n)), axis=n), Where R_offset_n represents the prediction result of the fourth nodule, f_R_offset represents the prediction function used by the nodule prediction model to generate the offset prediction result, flip represents the flip function, axis represents the flip direction, axis=n, image represents the 3D image to be detected, and S n This represents the size ratio between the flipped 3D image and the target feature map when the flipping direction is n.

5. The nodule detection device according to claim 2, characterized in that, The first nodule prediction result includes classification prediction results. The nodule detection device further includes: The first neighborhood suppression module is used to perform neighborhood suppression on the first nodule prediction result of the three-dimensional image to be detected by using the nodule prediction model and then using max pooling. The second nodule prediction result includes classification prediction results. The nodule detection device further includes: The second neighborhood suppression module is used to perform neighborhood suppression on the second nodule prediction result after obtaining the second nodule prediction result of the flipped 3D image using the nodule prediction model, and then using max pooling.

6. The nodule detection device according to claim 1, characterized in that, The first acquisition module includes: The acquisition unit is used to acquire the original three-dimensional image containing the first target object; A segmentation unit is used to perform image segmentation on the original three-dimensional image to obtain a target object segmentation mask; The extraction unit is used to extract the target object region image from the original three-dimensional image based on the target object segmentation mask; The scaling unit is used to scale the target object region image to a set size to obtain the three-dimensional image of the first target object to be detected.

7. The nodule detection device according to claim 1, characterized in that, Also includes: The second acquisition module is used to acquire a three-dimensional image of the label, the three-dimensional image of the label showing a second target object, and the three-dimensional image of the label also marking the nodule marking results of the second target object; The third prediction module is used to obtain the fifth nodule prediction result of the label three-dimensional image using the nodule prediction model; The training module is used to train the nodule prediction model based on the fifth nodule prediction result and the nodule labeling result.

8. The nodule detection device according to claim 7, characterized in that, The nodule prediction model includes a 3D convolutional network, a feature pyramid network, and a prediction network. The third prediction module includes: The extraction unit is used to extract image features of the label's three-dimensional image using the 3D convolutional network; An enhanced fusion unit is used to fuse and enhance the image features using the feature pyramid network to obtain fused enhanced features; The classification and regression unit is used to process the fused enhanced features using the prediction network to generate the final target feature map, and output the prediction result of the fifth nodule based on the target feature map.

9. The nodule detection device according to claim 8, characterized in that, The fifth nodule prediction results include nodule classification prediction results, size prediction results, and offset prediction results. The prediction network consists of a branch network, a classification network, a first regression network, and a second regression network. The branch network is used to generate a target feature map based on the fusion enhancement features. The classification network is used to output a classification prediction result based on the target feature map. The first regression network is used to output a size prediction result based on the target feature map. The second regression network is used to output an offset prediction result based on the target feature map.

10. The nodule detection device according to claim 7, characterized in that, The fifth nodule prediction result includes nodule classification prediction result, size prediction result, and offset prediction result; the nodule marking result includes nodule classification marking result, location marking result, and diameter marking result. The training module includes: The mapping unit is used to map the three-dimensional coordinates recorded in the location marking result to the target feature map according to a set downsampling ratio, and to perform a rounding operation to clarify the nodule mapping feature points in the target feature map. The three-dimensional coordinates are the three-dimensional coordinates of the nodule in the label three-dimensional image. The search unit is used to search for positive sample feature points and negative sample feature points in the target feature map based on the nodule mapping feature points; The first loss construction unit is used to construct a classification loss function based on the classification prediction results and classification label results corresponding to the positive sample feature points and the negative sample feature points; The first regression unit is used to draw a single-size three-dimensional anchor frame of a set size on the target feature map, and to obtain the size marking result based on the single-size three-dimensional anchor frame and the diameter marking result; The second loss construction unit is used to construct a first regression loss function based on the size labeling result and the size prediction result; The second regression unit is used to obtain the offset marking result based on the nodule mapping feature points, the downsampling ratio, and the position marking result; The third loss construction unit is used to construct a second regression loss function based on the offset labeling result and the offset prediction result; The parameter update unit is used to update the model parameters of the nodule prediction model according to the classification loss function, the first regression loss function, and the second regression loss function.

11. The nodule detection device according to claim 10, characterized in that, The search unit includes: A positive sample determination subunit is used to use the nodule mapping feature points as positive sample feature points; The negative sample determination subunit is used to find negative sample feature points in the target feature map.

12. The nodule detection device according to claim 10, characterized in that, The single-size three-dimensional anchor frame is a cube-shaped anchor frame.

13. The nodule detection device according to claim 10, characterized in that, The classification loss function adopts the Focalloss function, and both the first regression loss function and the second regression loss function adopt the smooth L1 loss function.

14. A nodule detection device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the calculations corresponding to the nodule detection device as described in any one of claims 1-13.

Images