Method, apparatus and storage medium for processing medical image

Abstract

The application discloses a medical image processing method, device and equipment and a storage medium. The method comprises the following steps: inputting a target medical image, performing feature extraction on the target medical image to obtain a feature map; determining at least one candidate box corresponding to a candidate object from the feature map according to the image features of the target medical image, wherein the candidate box represents the area occupied by the image features corresponding to the candidate object in the feature map, different candidate boxes correspond to different positions and sizes, and the position and size of the candidate box match the position and size of the candidate object; classifying the image features in the candidate box to obtain a classification result corresponding to the candidate box; and determining the position of a target object in the target medical image according to the classification result. The candidate boxes with different positions and sizes are determined based on the image features of the target medical image, the number of candidate boxes is reduced, and the detection efficiency is improved. The position and size of the candidate box are determined by fusing the image features, which helps to improve the accuracy of the detection result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for processing medical images. Background Technology

[0002] Organ nodules often appear when human organs develop diseases, thus serving as an important indicator for diagnosing organ lesions. Common organ nodules include lung nodules, breast nodules, and thyroid nodules, among others.

[0003] Medical professionals typically use instruments to image organs, thereby detecting organ nodules based on these images. These images are two-dimensional image sequences obtained through three-dimensional scanning; for example, they can be computed tomography (CT) images or magnetic resonance (MR) images. Taking a CT scanner as an example, organs are scanned using a CT scanner, and organ nodules are detected and classified based on the resulting CT images. In related technologies, when using a convolutional neural network (CNN) to detect organ nodules in CT images, candidate regions of a preset size are typically defined, and then organ nodules are detected within each candidate region to ultimately determine the location of the organ nodules in the CT image.

[0004] In related technologies, to improve recall, a sliding window approach is typically used to generate dense candidate boxes (all of the same size), and each candidate box is then detected. However, in reality, organ nodules are sparsely distributed and vary considerably in size, resulting in low detection efficiency and accuracy of organ nodules in these technologies. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for processing medical images, which can improve the detection efficiency and accuracy of target objects in medical images. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a method for processing medical images, the method comprising:

[0007] Input a target medical image, extract features from the target medical image to obtain a feature map, and use the feature map to characterize the image features of the target medical image;

[0008] Based on the image features of the target medical image, at least one candidate box corresponding to a candidate object is determined from the feature map. The candidate object is an object that is suspected to be the target object. The candidate box represents the area occupied by the image features corresponding to the candidate object in the feature map. Different candidate boxes correspond to different positions and sizes, and the position and size of the candidate box match the position and size of the candidate object.

[0009] The image features in the candidate box are classified to obtain the classification result corresponding to the candidate box. The classification result is used to characterize whether the candidate object corresponding to the candidate box is the target object.

[0010] The location of the target object in the target medical image is determined based on the classification results.

[0011] On the other hand, embodiments of this application provide a medical image processing apparatus, characterized in that the apparatus includes:

[0012] The feature extraction module is used to input a target medical image, extract features from the target medical image, and obtain a feature map, which is used to characterize the image features of the target medical image.

[0013] The candidate box determination module is used to determine at least one candidate box corresponding to a candidate object from the feature map based on the image features of the target medical image. The candidate object is an object that is suspected to be the target object. The candidate box represents the area occupied by the image features corresponding to the candidate object in the feature map. Different candidate boxes correspond to different positions and sizes, and the position and size of the candidate box match the position and size of the candidate object.

[0014] A classification module is used to classify the image features in the candidate box to obtain the classification result corresponding to the candidate box. The classification result is used to characterize whether the candidate object corresponding to the candidate box is the target object.

[0015] The location determination module is used to determine the location of the target object in the target medical image based on the classification result.

[0016] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the medical image processing method as described above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the medical image processing method as described above.

[0018] On the other hand, a computer program product is provided that, when run on a computer, causes the computer to perform the medical image processing method as described above.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] After extracting features from the target medical image to obtain a feature map, candidate boxes corresponding to candidate objects at different positions and sizes are determined from the feature map based on the image features of the target medical image. The image features in the candidate boxes are then classified to obtain classification results indicating whether the candidate box contains the target object. Finally, the position of the target object in the target medical image is determined based on the classification results. Compared with the sliding window method used in related technologies to generate dense and uniformly sized candidate boxes, determining candidate boxes with different positions and sizes based on the image features of the target medical image can reduce the number of candidate boxes, thereby improving detection efficiency. At the same time, the fusion of image features when determining the position and size of the candidate boxes increases the probability that the candidate boxes contain the target object, which helps to improve the accuracy of the detection results. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating the principle of a medical image processing method provided in an exemplary embodiment of this application;

[0023] Figure 2 A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown;

[0024] Figure 3 A flowchart illustrating a medical image processing method provided in an exemplary embodiment of this application is shown;

[0025] Figure 4 A flowchart illustrating a medical image processing method provided in another exemplary embodiment of this application is shown;

[0026] Figure 5 yes Figure 4 The diagram illustrates the principle of the medical image processing method.

[0027] Figure 6 This is a schematic diagram illustrating an exemplary embodiment of the candidate box position prediction process in this application;

[0028] Figure 7 This is a schematic diagram illustrating an exemplary embodiment of the feature adjustment process of this application;

[0029] Figure 8 This is a structural block diagram of a medical image processing apparatus provided in an exemplary embodiment of this application;

[0030] Figure 9 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] In related technologies, when detecting target objects (such as organ nodules, which will be used as an example below) in medical images, dense candidate boxes of uniform size are typically generated using a sliding window method (e.g., generating one candidate box every unit) based on a preset candidate box size (e.g., all 3×3×3). Then, target object detection is performed on the image within each candidate box to determine the candidate box containing the target. However, in reality, on the one hand, because organ nodules are sparsely distributed in medical images, the candidate boxes obtained using the sliding window method contain a large number of invalid candidate boxes, and detecting a large number of invalid candidate boxes will seriously affect the detection efficiency; on the other hand, because the size (shape) of organ nodules varies greatly, candidate boxes generated based on the same size cannot meet the detection of organ nodules of varying sizes, and the detection accuracy is low (e.g., candidate boxes that are too small cannot include the complete organ nodule, affecting the detection accuracy).

[0033] In order to solve the problems existing in the relevant technologies, such as Figure 1As shown in this embodiment, the medical image 101 is first feature-extracted by the feature extraction network 102 to obtain a feature map 103. Then, the feature map 103 is input into the position prediction network 104 and the size prediction network 105, respectively. Combining the prediction results (including predicted position and predicted size) from the two networks, several candidate boxes 106 (corresponding to candidate objects) with different positions and sizes (different shapes and sizes) are determined. Since these candidate boxes 106 are predicted based on the image features of the medical image, they are more targeted than candidate boxes generated using a sliding window method (i.e., a higher probability of containing organ nodules, a higher matching degree with organ nodule size, and a higher probability of containing complete organ nodules), and are fewer in number, thereby improving the efficiency and accuracy of subsequent detection. Furthermore, based on the size prediction results output by the size prediction network 105, the feature adjustment network 107 also performs feature adjustment on the feature map 103 to obtain a feature-adjusted map 108 that conforms to the size prediction results.

[0034] After feature adjustment and candidate box determination are completed, the region of interest (ROI) feature map 109 is extracted from the feature adjustment map 108 using the candidate box 106. The ROI feature map 109 is then input into the fully connected layer (FC) layer 110 for fully connected processing. The classification network 111 outputs the classification result indicating whether there is an organ nodule in the candidate box, and the regression network 112 outputs the actual position of the candidate box 106 in the medical image 101. Finally, the detection result 113 is obtained and presented in the medical image 101.

[0035] The method provided in this application determines candidate boxes with different positions and sizes based on the image features of the target medical image, which can reduce the number of candidate boxes and thus improve detection efficiency. At the same time, the image features are fused when determining the position and size of the candidate boxes, which increases the probability that the candidate boxes contain organ nodules and helps to improve the accuracy of the detection results.

[0036] In one possible application scenario, the medical image processing method provided in this application embodiment can be applied to the preprocessing of organ nodule classification. This method can be implemented as an organ nodule localization program and combined with an organ nodule classification program as a preprocessing program for the organ nodule classification program.

[0037] In this scenario, the organ nodule localization program takes a medical image as input, extracts features from the medical image to obtain a feature map, and determines several candidate boxes of different sizes based on the image features contained in the feature map. Then, it uses the candidate boxes to extract the ROI region feature map from the feature map, performs classification and regression on the ROI region feature map, and marks the location of the organ nodule on the medical image based on the classification and regression results.

[0038] To further classify the detected organ nodules (such as true / false nodule classification, benign / malignant nodule classification, and nodule nature classification), the organ nodule localization program takes the nodule information output by the organ nodule localization program (including the position and size of the corresponding localization box of the organ nodule) and the medical image as input, classifies the identified organ nodules, and finally outputs and displays the classification results of the organ nodules.

[0039] Of course, after the organ nodules are located using the organ nodule localization program, medical staff can also manually classify the organ nodules based on the medical images marked with the organ nodules.

[0040] Of course, in addition to the above-mentioned scenarios, the method provided in this application embodiment can also be applied to other scenarios that require the classification of organ nodules. This application embodiment does not limit the specific application scenarios.

[0041] The medical image processing method provided in this application can be applied to computer devices such as terminals or servers. In one possible implementation, the medical image processing method provided in this application can be implemented as an application or part of an application and installed on a terminal used by a doctor, enabling the terminal to automatically locate and label organ nodules based on medical images; or, it can be applied to the backend server of an application, so that the server provides organ nodule location and labeling services to the application in the terminal.

[0042] Please refer to Figure 2 The diagram illustrates an implementation environment provided by an exemplary embodiment of this application. This implementation environment includes a terminal 210 and a server 220, wherein the terminal 210 and the server 220 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0043] The terminal 210 is equipped with an application that requires organ nodule recognition. This application may be a clinical diagnostic assistance application, a medical image processing application, a medical learning assistance application, or other artificial intelligence (AI) applications applied in the medical field. This application embodiment does not limit this.

[0044] Optionally, the terminal 210 can be a mobile terminal such as a tablet computer or a laptop computer, or a terminal such as a desktop computer or a projector computer. This application embodiment does not limit this.

[0045] Server 220 can be implemented as a single server or as a server cluster consisting of a group of servers; it can be a physical server or a cloud server. In one possible implementation, server 220 is the backend server for the application in terminal 210.

[0046] like Figure 2 As shown in this embodiment, the server 220 includes a feature extraction module 2201, a candidate box determination module 2202, a classification module 2203, and a processing module 2204. After the terminal 210 sends a medical image to the server 220 via an application, the server 220 extracts features from the medical image using the feature extraction module 2201 to obtain a feature map. The feature map is then input into the candidate box determination module 2202, which determines at least one candidate box based on the image features. The classification module 2203 then classifies the image features within each candidate box to obtain a classification result. Finally, the processing module 2204 processes the medical image according to the classification result to obtain a processed medical image. After the processed medical image is sent from the server 220 to the terminal 210, the terminal 210 displays the processed medical image. Figure 2 In the middle, terminal 210 displays a medical image with the location of organ nodules marked by positioning boxes.

[0047] In other possible implementations, the feature extraction module 2201, candidate box determination module 2202, classification module 2203 and processing module 2204 described above can also be implemented as part or all of the application. Accordingly, the terminal 210 can perform organ nodule localization and identification locally without the aid of the server 220. This embodiment does not limit this.

[0048] For ease of explanation, the following embodiments are illustrated using a medical image processing method executed by a computer device as an example.

[0049] Please refer to Figure 3This document illustrates a flowchart of a medical image processing method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example, and the method includes the following steps.

[0050] Step 301: Input the target medical image, extract features from the target medical image to obtain a feature map, which is used to characterize the image features of the target medical image.

[0051] In one possible implementation, the target medical image can be a two-dimensional image sequence obtained from a three-dimensional scan of an organ; that is, the target medical image can be a three-dimensional medical image, such as a CT image sequence obtained using a CT scanner or an MRI image sequence obtained using an MRI scanner. In other possible implementations, the target medical image can also be a single two-dimensional medical image, which is not limited in this embodiment.

[0052] Optionally, the computer device inputs the target medical image into a feature extraction network, which extracts image features to obtain a feature map containing the image features. When the target medical image is a three-dimensional medical image, the feature extraction network can be a three-dimensional convolutional neural network, and the resulting feature map is a three-dimensional feature map. This embodiment does not impose any limitations on this.

[0053] Step 302: Based on the image features of the target medical image, determine at least one candidate box corresponding to a candidate object from the feature map. The candidate object is an object that is suspected to be the target object. The candidate box represents the area occupied by the image features corresponding to the candidate object in the feature map. Different candidate boxes correspond to different positions and sizes, and the position and size of the candidate box match the position and size of the candidate object.

[0054] Unlike related technologies that do not consider the image features of medical images and directly generate dense candidate boxes using the sliding window method according to preset sizes, in order to reduce the number of generated candidate boxes and improve the matching degree between the candidate box size and the actual size of the target object in the target medical image, in this embodiment, the computer device determines the position and size of at least one candidate object and its corresponding candidate box in the feature map according to the image features of the target medical image, so as to perform target object detection based on the determined candidate boxes.

[0055] The target object can be a lesion such as an organ nodule or a tumor. Correspondingly, the candidate object can be a suspected organ nodule or a suspected tumor or other suspected lesion. This application embodiment only uses an organ nodule as an example for illustration, but does not constitute a limitation.

[0056] Because the distribution of target objects in the same medical image is sparse and there are large differences in the size of individual target objects, the positions and sizes of different candidate boxes determined based on image features are different.

[0057] In one possible implementation, when the target medical image is a three-dimensional medical image, the candidate box is a three-dimensional candidate box, wherein the width of the three-dimensional candidate box is the number of two-dimensional medical images contained in the candidate box.

[0058] Optionally, a candidate box prediction network can be pre-trained on a computer device. This network takes a feature map as input and outputs the position and size of the candidate box within the feature map. The implementation method for the candidate box determination process will be detailed in the following embodiments.

[0059] Step 303: Classify the image features in the candidate boxes to obtain the classification results corresponding to the candidate boxes. The classification results are used to characterize whether the candidate object corresponding to the candidate box is the target object.

[0060] In one possible implementation, the computer device extracts a Region of Interest (ROI) feature map from the feature map based on the determined position and size of the candidate bounding box, and further classifies the ROI feature map to obtain the classification result corresponding to the candidate bounding box. Wherein, when the candidate bounding box is a three-dimensional candidate bounding box, the extracted ROI feature map is also a three-dimensional feature map.

[0061] Optionally, the computer device processes the ROI region feature map through a fully connected layer and then inputs it into a pre-trained classification network to obtain the classification result output by the classification network. This classification network is a binary classification network, and its output classification result is used to indicate whether the candidate box contains the target object.

[0062] Step 304: Determine the location of the target object in the target medical image based on the classification results.

[0063] Optionally, when the classification result corresponding to the candidate box contains the target object, the computer device determines the image position of the candidate box in the target medical image based on the position of the candidate box in the feature map, and then processes the target medical image according to the image position to obtain a target medical image with the target object location box labeled. The labeling of the target object position can be done using images or coordinates; this embodiment does not limit this method.

[0064] In summary, in this embodiment, after extracting features from the target medical image to obtain a feature map, candidate boxes corresponding to candidate objects of different positions and sizes are determined from the feature map based on the image features of the target medical image. The image features within the candidate boxes are then classified to obtain classification results indicating whether the candidate boxes contain the target object. Finally, the position of the target object in the target medical image is determined based on the classification results. Compared to related technologies that use a sliding window method to generate dense and uniformly sized candidate boxes, determining candidate boxes with different positions and sizes based on the image features of the target medical image reduces the number of candidate boxes, thereby improving detection efficiency. Furthermore, the fusion of image features when determining the position and size of the candidate boxes increases the probability that the candidate boxes contain the target object, contributing to improved accuracy of the detection results.

[0065] In related technologies, since the candidate boxes are of uniform size, when extracting image features of the ROI region based on the generated dense candidate boxes, the extraction is directly performed from the feature map of the medical image, meaning different candidate boxes share unified image features. However, in this embodiment, the candidate boxes determined based on image features have different sizes (shape and dimensions), so sharing image features among different candidate boxes may lead to lower accuracy in the classification results. To improve the accuracy of the classification results, in this embodiment, after the computer device predicts the size of the candidate box, it further adjusts the feature map based on this size, so that the size features of the candidate box are incorporated into the feature map after feature adjustment. In subsequent processes, the computer device extracts features from the candidate boxes based on the feature map after feature adjustment. The following is an exemplary embodiment for illustration.

[0066] Please refer to Figure 4 This illustration shows a flowchart of a medical image processing method provided in another exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example for illustration, and the method includes the following steps.

[0067] Step 401: Input the target medical image, extract features from the target medical image to obtain a feature map, which is used to characterize the image features of the target medical image.

[0068] The implementation method of this step can refer to step 301 above, and will not be repeated here in this embodiment.

[0069] Indicative, such as Figure 5As shown, taking a three-dimensional lung CT image as an example, the computer device inputs the target medical image 501 into the feature extraction network 502 to obtain the feature map 503 output by the feature extraction network 502. The feature extraction network 502 consists of convolutional layers, batch normalization (BN) + rectified linear unit (ReLU) layers, and pooling layers, and the feature map 503 output by the feature extraction network 502 is a three-dimensional feature map.

[0070] After obtaining the feature map, the computer device performs parallel prediction of candidate box positions and sizes based on the feature map, thereby determining at least one candidate box based on the candidate box positions and sizes. The process of predicting candidate box positions is as described in step 402, and the process of predicting candidate box sizes is as described in step 403.

[0071] Step 402: Based on the image features of the target medical image, determine at least one predicted location from the feature map, wherein the probability of using the predicted location as the center point of the candidate box is higher than the probability of using other locations on the feature map as the center point of the candidate box.

[0072] Because organ nodules are sparsely distributed in medical images, the distribution of the final candidate boxes is also sparse. Accordingly, the purpose of candidate box position prediction is to predict which positions in the feature map should be the center points of the candidate boxes to generate candidate boxes, and which positions should not be the center points of the candidate boxes to reduce the number of generated candidate boxes.

[0073] In one possible implementation, the computer device uses a machine learning algorithm to determine the prediction probability corresponding to each feature position in the feature map. This prediction probability is the probability that the feature position is the center point of the candidate box. Then, based on the prediction probability, at least one prediction position is determined. The probability that the prediction position is the center point of the candidate box is higher than the probability that other positions on the feature map are the center points of the candidate boxes.

[0074] Optionally, this step may include the following sub-steps.

[0075] 1. Input the feature map into the location prediction network to obtain the probability prediction map output by the location prediction network. The probability prediction map is used to represent the probability that each feature position on the feature map is the center point of the candidate box. The size of the probability prediction map is the same as the size of the feature map.

[0076] In one possible implementation, the computer device stores a pre-trained location prediction network that takes a feature map as input and outputs a probability prediction map with the same size as the feature map. The probability prediction map contains the probabilities of the corresponding feature locations in the feature map as the center points of candidate boxes.

[0077] Indicative, such as Figure 5 As shown, the location prediction network consists of a 1×1×1 convolution 504 and an activation function. After the computer device performs convolution processing on the feature map 503 using convolution 504, it further activates the convolution result using activation function 505. Figure 6 As shown, after convolution and activation processing, the computer device obtains a probability prediction map 601 with the same size as the feature map 503. In the probability prediction map 601, the darker the color, the higher the probability of the corresponding position.

[0078] Regarding the training method of the location prediction network, in one possible implementation, the location prediction network is trained based on a sample feature map, wherein the sample feature map is marked with positive sample regions and negative sample regions, the positive sample regions are the central regions of the candidate boxes, and the negative sample regions are the regions outside the positive sample regions.

[0079] Optionally, before training the location prediction network, the region of a predetermined size surrounding the center point of the candidate bounding box (containing the target object) in the sample feature map is first manually labeled as a positive sample region (with a probability of 1). Based on the positive sample regions, negative sample regions (with a probability of 0) are then determined, and these regions are labeled. Next, the computer inputs the sample feature map into the location prediction network to be trained, obtaining the output sample probability prediction map (containing the probability of each feature location being the center point of the candidate bounding box). This positive and negative sample regions serve as supervision for training the location prediction network until the loss of the network meets the convergence condition, thus completing the network training.

[0080] Second, threshold filtering is performed on the probability prediction map to determine the predicted position in the feature map. The probability of the predicted position being the center point of the candidate box is higher than the probability threshold.

[0081] Furthermore, the computer device performs threshold filtering on the probability prediction map based on a probability threshold, filtering out points in the probability prediction map whose probabilities are lower than the probability threshold. Based on the points retained in the filtered probability prediction map, the predicted positions in the feature map are determined. Since the size of the probability prediction map is the same as the size of the feature map, the computer device determines the predicted position of the candidate box center point by mapping the retained points in the probability prediction map to the positions in the feature map.

[0082] Indicative, such as Figure 6 As shown, the computer device performs threshold filtering on the probability prediction map 601 based on a probability threshold of 0.8, thereby obtaining the location prediction result 602. The white area in the location prediction result 602 is the predicted position of the candidate box center in the feature map.

[0083] By determining the predicted location of the candidate box using the above method, while ensuring a high recall rate, the number of subsequent candidate boxes generated can be reduced, which helps to reduce the computational load during subsequent detection and thus improve detection efficiency.

[0084] Step 403: Based on the image features of the target medical image, determine the predicted size corresponding to at least one feature location in the feature map. The predicted size refers to the size of the candidate box constructed with the feature location as the center point of the candidate box.

[0085] Because organ nodules in medical images vary greatly in size, the sizes of the final candidate boxes also vary considerably. Accordingly, the purpose of candidate box size prediction is to predict the optimal size of the candidate box when each feature position in the feature map is used as the center point of the candidate box.

[0086] In one possible implementation, the computer device inputs the feature map into a size prediction network to obtain the predicted size corresponding to each feature location output by the size prediction network. When the target medical image is a three-dimensional medical image, the predicted size is a three-dimensional size (including length, width, and height).

[0087] In one possible implementation, the candidate box of optimal size is defined as the candidate box that generates the maximum intersection over union (IoU) with the neighboring gold standard locator boxes, wherein the neighboring gold standard locator boxes are locator boxes with pre-calibrated positions and sizes (including organ nodules), and the center point of the neighboring gold standard locator boxes is adjacent to the center point of the candidate box of optimal size.

[0088] Since the optimal size of the candidate boxes cannot be directly calculated during the training of the size prediction network, and calculating the IoU is a differentiable operation, the computer device obtains regression candidate boxes through regression based on the sample feature maps during the training process of the size prediction network. Then, it determines the neighboring labeled candidate boxes corresponding to the regression candidate boxes, calculates the IoU between the regression candidate boxes and the neighboring labeled candidate boxes, and then uses the IoU as supervision (maximizing the IoU) to train the size prediction network and learn the optimal size of the candidate boxes.

[0089] Indicative, such as Figure 5 As shown, the size prediction network includes a 1×1×1 convolution 505. After the computer device performs convolution processing on the feature map 503, it obtains the predicted size corresponding to each feature position on the feature map.

[0090] In other possible implementations, when predicting the predicted size corresponding to at least one feature location, in order to reduce the amount of computation, the computer device can select at least one feature location based on the location prediction result and determine the predicted size corresponding to the selected feature location.

[0091] Step 404: Determine at least one candidate box corresponding to a candidate object based on the predicted position and predicted size, wherein the candidate box is centered on the predicted position and uses the predicted size corresponding to the predicted position.

[0092] Once the predicted position and size of the candidate box are determined, the computer device constructs the candidate box with the predicted position as the center and the predicted size of the candidate box corresponding to that predicted position.

[0093] Indicative, such as Figure 5 As shown, the computer device determines at least one candidate box 506 based on the output of the shape prediction branch and the position prediction branch.

[0094] Compared to generating dense candidate boxes of uniform size, the number of candidate boxes determined by the method provided in this application embodiment is significantly reduced, and the size of the candidate boxes matches the size of the actual organ nodules more closely, which helps to improve the accuracy of subsequent classification based on candidate boxes.

[0095] Step 405: Based on the predicted size, perform feature adjustment on the feature map to obtain a feature-adjusted map, wherein the image features at the feature locations in the feature-adjusted map match the predicted size corresponding to the feature locations.

[0096] Through the above steps, the computer device can generate sparse candidate boxes of varying sizes. Different sized candidate boxes correspond to different receptive fields; smaller candidate boxes correspond to smaller receptive fields, vertical candidate boxes correspond to vertical receptive fields, and horizontal candidate boxes correspond to horizontal receptive fields. Therefore, the computer device does not directly extract the ROI region from the feature map using the candidate boxes, but rather needs to adjust the feature map based on the size of the candidate boxes.

[0097] In this embodiment, the computer device adopts the idea of ​​deformable convolution and adjusts the feature map at each position individually according to the predicted size of the candidate box, thereby incorporating the size features of the candidate box into the feature map. This makes the image features contained in the adjusted feature map adaptable to the size of the candidate box at each position, so that the effective range of the adjusted feature map is closer to the size of the candidate box.

[0098] In one possible implementation, this step may include the following sub-steps.

[0099] 1. Determine the feature bias of the feature map based on the predicted size. The feature bias is used to indicate the bias of image features at each feature location in the feature map.

[0100] In related technologies, the feature extraction process at the candidate box position (i, j, k) can be expressed as the following formula:

[0101]

[0102] Where bin(i, j, k) is a feature cube constructed with (i, j, k) as its center, with a fixed side length, and N bin Let x(p) be the number of features contained in the feature cube (for example, if the side length of the feature cube is 3, the number of features is 9), and let x(p) represent the feature extraction process at feature point p.

[0103] Clearly, in the relevant technologies, feature extraction is independent of the size of the candidate boxes (both are fixed sizes), but the position of the goldfish candidate boxes is relevant.

[0104] In this embodiment of the application, in order to incorporate the size of the candidate box into the feature map, such as... Figure 5 and 7 As shown, after predicting the candidate box size, the computer device further performs convolution processing on the size prediction result through a 1×1×1 convolution 507 to obtain the feature bias corresponding to the feature map. The feature bias indicates the (direction) bias of the image features at each feature position in the feature map. When adjusting the features later, the image features at the feature position are fused to the direction indicated by the feature bias according to the feature bias.

[0105] 2. The feature adjustment map is calculated based on the feature map and feature bias.

[0106] Optionally, the computer device feature biasing performs a biasing operation on the features in the feature map to obtain a feature adjustment map, wherein the feature adjustment map has the same size as the feature map.

[0107] In the feature adjustment method provided in this application embodiment, the feature map is adjusted according to the size of the candidate box to form a feature adjustment map. The process of generating the feature adjustment map can be expressed as the following formula.

[0108]

[0109] Here, offset(i, j, k) is the feature offset corresponding to the feature cube bin(i, j, k). In an illustrative example, if the side length of the feature cube is 3, the size of bin(i, j, k) is 3×3×3, and the size of offset(i, j, k) is 3×3×3×3.

[0110] like Figure 5 As shown, the computer device uses feature bias 508 to adjust feature map 503 to obtain feature adjustment map 509.

[0111] Indicative, such as Figure 7 As shown, for feature position 503 in the middle of feature map 503 a After feature offset processing, the feature position is moved to feature position 503b on the periphery of feature position 503a, so that the feature receptive field matches the size of the candidate box.

[0112] Step 406: Extract image features from the feature adjustment map based on the predicted position and predicted size of the candidate box.

[0113] After determining the candidate bounding boxes and adjusting their features through the above steps, the computer device extracts the image features corresponding to each candidate bounding box (i.e., the ROI region) from the feature-adjusted map based on the predicted position and size of each candidate bounding box. (Illustrative example follows.) Figure 5 As shown, the computer device extracts the ROI region feature map 510 from the feature adjustment map 509.

[0114] In one possible implementation, to improve the subsequent classification rate and accuracy, after extracting image features based on candidate boxes, the computer device also needs to perform pooling processing on the extracted image features. This pooling processing can employ max pooling.

[0115] Step 407: Input the extracted image features into the classification network to obtain the classification result output by the classification network.

[0116] For the extracted image features, the computer device inputs the image features into a pre-trained classification network to obtain the classification result output by the classification network. The classification network can be a binary classifier, such as a softmax classifier.

[0117] Indicative, such as Figure 5 As shown, the computer device first performs fully connected processing on the extracted image features through a fully connected network (including fully connected layers and BN+ReLU layers), and then outputs the classification result 512 using the fully connected layer of the classification branch and the activation function 511.

[0118] Optionally, since the predicted position and predicted size determined in the above steps are relative to the feature map, the computer device can also perform candidate box regression through the regression branch while performing classification to determine the position of the candidate box in the feature map mapped in the target medical image. The regression process of the candidate box will not be described in detail in this embodiment.

[0119] Indicative, such as Figure 5As shown, the computer device uses the fully connected layer of the regression branch to regress the output of the fully connected network, thereby obtaining the candidate box regression result 513.

[0120] Step 408: Determine the location of the target object in the target medical image based on the classification results.

[0121] The implementation method of this step can refer to step 304 above, and will not be repeated here in this embodiment.

[0122] In this embodiment, the computer device uses a location prediction network to predict the center point position of the candidate box based on the feature map, and uses a size prediction network to predict the optimal size of the candidate box at each feature position. Thus, at least one candidate box is determined based on the predicted position and predicted size. While reducing the number of candidate boxes, the matching degree between the candidate box and the target object size is improved, thereby improving the accuracy of the subsequent classification results.

[0123] In addition, the computer device determines the feature bias corresponding to the feature map based on the size prediction results of the size prediction network. The feature map is then processed using the feature bias to incorporate the size features of the candidate boxes into the feature map, resulting in a feature adjustment map. Subsequently, the ROI region is extracted based on the feature adjustment map, which helps to improve the accuracy of the subsequent classification results.

[0124] Figure 8 This is a structural block diagram of a medical image processing apparatus provided in an exemplary embodiment of this application. This apparatus can be installed in the computer device described in the above embodiments, such as... Figure 8 As shown, the device includes:

[0125] The feature extraction module 801 is used to input a target medical image, extract features from the target medical image, and obtain a feature map, wherein the feature map is used to characterize the image features of the target medical image.

[0126] The candidate box determination module 802 is used to determine at least one candidate box corresponding to a candidate object from the feature map based on the image features of the target medical image. The candidate object is an object that is suspected to be the target object. The candidate box represents the area occupied by the image features corresponding to the candidate object in the feature map. Different candidate boxes correspond to different positions and sizes, and the position and size of the candidate box match the position and size of the candidate object.

[0127] The classification module 803 is used to classify the image features in the candidate box to obtain the classification result corresponding to the candidate box. The classification result is used to characterize whether the candidate object corresponding to the candidate box is the target object.

[0128] The location determination module 804 is used to determine the location of the target object in the target medical image based on the classification result.

[0129] Optionally, the candidate box determination module 802 includes:

[0130] A location determination unit is used to determine at least one predicted location from the feature map based on the image features of the target medical image, wherein the probability of using the predicted location as the center point of a candidate box is higher than the probability of using other locations on the feature map as the center point of a candidate box.

[0131] The size determination unit is used to determine the predicted size corresponding to at least one feature position in the feature map based on the image features of the target medical image. The predicted size refers to the size of the candidate box constructed with the feature position as the center point of the candidate box.

[0132] A candidate box determination unit is configured to determine at least one candidate box corresponding to the candidate object based on the predicted position and the predicted size, wherein the candidate box is centered on the predicted position and uses the predicted size corresponding to the predicted position.

[0133] Optionally, the position determination unit is used for:

[0134] The feature map is input into the location prediction network to obtain the probability prediction map output by the location prediction network. The probability prediction map is used to characterize the probability that each feature position on the feature map is the center point of the candidate box. The size of the probability prediction map is the same as the size of the feature map.

[0135] The probability prediction map is subjected to threshold filtering to determine the predicted position in the feature map, wherein the probability of the predicted position being the center point of the candidate box is higher than the probability threshold.

[0136] Optionally, the location prediction network is trained based on a sample feature map, which is marked with positive sample regions and negative sample regions. The positive sample regions are the central regions of the calibration candidate boxes, and the negative sample regions are the regions outside the positive sample regions. The calibration candidate boxes contain sample target objects.

[0137] Optionally, the size prediction unit is used for:

[0138] The feature map is input into the size prediction network to obtain the predicted size corresponding to each feature position output by the size prediction network.

[0139] Optionally, the size prediction network is trained under supervision using the intersection-over-union ratio (IoU) between the regression candidate boxes and the calibration candidate boxes, wherein the regression candidate boxes are candidate boxes obtained during the training of the size prediction network.

[0140] Optionally, the device further includes:

[0141] The feature adjustment module is used to adjust the feature map according to the predicted size to obtain a feature adjustment map, wherein the image features at the feature positions in the feature adjustment map are matched with the predicted size corresponding to the feature positions;

[0142] The classification module 803 includes:

[0143] The extraction unit is used to extract image features from the feature adjustment map based on the predicted position and the predicted size corresponding to the candidate box;

[0144] A classification unit is used to input the extracted image features into a classification network to obtain the classification result output by the classification network.

[0145] Optionally, the feature adjustment module includes:

[0146] The bias determination unit is used to determine the feature bias of the feature map based on the prediction size, wherein the feature bias is used to indicate the bias of image features at each feature position in the feature map;

[0147] An adjustment module is used to calculate the feature adjustment map based on the feature map and the feature bias.

[0148] Optionally, the target medical image is a CT image, and the target object is an organ nodule.

[0149] In summary, in this embodiment, after extracting features from the target medical image to obtain a feature map, candidate boxes corresponding to candidate objects of different positions and sizes are determined from the feature map based on the image features of the target medical image. The image features within the candidate boxes are then classified to obtain classification results indicating whether the candidate boxes contain the target object. Finally, the position of the target object in the target medical image is determined based on the classification results. Compared to related technologies that use a sliding window method to generate dense and uniformly sized candidate boxes, determining candidate boxes with different positions and sizes based on the image features of the target medical image reduces the number of candidate boxes, thereby improving detection efficiency. Furthermore, the fusion of image features when determining the position and size of the candidate boxes increases the probability that the candidate boxes contain the target object, contributing to improved accuracy of the detection results.

[0150] In this embodiment, the computer device uses a location prediction network to predict the center point position of the candidate box based on the feature map, and uses a size prediction network to predict the optimal size of the candidate box at each feature position. Thus, at least one candidate box is determined based on the predicted position and predicted size. While reducing the number of candidate boxes, the matching degree between the candidate box and the target object size is improved, thereby improving the accuracy of the subsequent classification results.

[0151] In addition, the computer device determines the feature bias corresponding to the feature map based on the size prediction results of the size prediction network. The feature map is then processed using the feature bias to incorporate the size features of the candidate boxes into the feature map, resulting in a feature adjustment map. Subsequently, the ROI region is extracted based on the feature adjustment map, which helps to improve the accuracy of the subsequent classification results.

[0152] It should be noted that the medical image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the medical image processing apparatus and the medical image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0153] Please refer to Figure 9 This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 900 includes a Central Processing Unit (CPU) 901, a system memory 904 including a random access memory 902 and a read-only memory 903, and a system bus 905 connecting the system memory 904 and the CPU 901. The computer device 900 also includes a basic input / output system (I / O system) 906 that facilitates the transfer of information between various devices within the computer, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.

[0154] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include the input / output controller 910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0155] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include computer-readable media (not shown) such as a hard disk or drive.

[0156] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0157] The memory stores one or more programs, which are configured to be executed by one or more central processing units 901. The one or more programs contain instructions for implementing the methods described above, and the central processing unit 901 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0158] According to various embodiments of this application, the computer device 900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0159] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0160] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the medical image processing method described in any of the above embodiments.

[0161] This application also provides a computer program product that, when run on a computer, causes the computer to execute the medical image processing method provided in the above-described method embodiments.

[0162] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a terminal. The computer-readable storage medium stores at least one instruction, at least one program segment, a code set, or an instruction set. The at least one instruction, the at least one program segment, the code set, or the instruction set is loaded and executed by the processor to implement the medical image processing method described in any of the above method embodiments.

[0163] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments described above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0164] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0165] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A processing method of a medical image, characterized by, The method comprises: feature extraction on a target medical image by a three-dimensional convolutional neural network to obtain a three-dimensional feature map, the three-dimensional feature map being used to represent image features of the target medical image, the target medical image being a CT image sequence; determination of at least one predicted position from the three-dimensional feature map according to the image features of the target medical image, the probability of the predicted position being higher than the probability of other positions on the three-dimensional feature map as a center point of a candidate box, the candidate box representing an area occupied by image features of a candidate object in the three-dimensional feature map, the candidate box being a three-dimensional candidate box, and the candidate object being an object suspected to be a target object; input of the three-dimensional feature map into a size prediction network according to the predicted position to obtain a predicted size corresponding to each predicted position output by the size prediction network, the predicted size being a size of a candidate box constructed with the predicted position as a center point of a three-dimensional candidate box, the size prediction network being supervised by an intersection over union between a regression candidate box and a neighboring labeled candidate box, and being trained by maximizing the intersection over union, the regression candidate box being a candidate box obtained by regression during training of the size prediction network; determination of the candidate box corresponding to at least one candidate object according to the predicted position and the predicted size, the candidate box having the predicted position as a center and adopting the predicted size corresponding to the predicted position; determination of a feature bias of the three-dimensional feature map according to the predicted size, the feature bias being used to indicate a bias condition of image features at each predicted position in the three-dimensional feature map; calculation of a feature adjustment map from the three-dimensional feature map and the feature bias, image features at the predicted position in the feature adjustment map matching the predicted size corresponding to the predicted position; classification of image features in the candidate box to obtain a classification result corresponding to the candidate box, the classification result being used to represent whether the candidate object corresponding to the candidate box is the target object; determination of a position of the target object in the target medical image according to the classification result.

2. The method of claim 1, wherein, The determination of at least one predicted position from the three-dimensional feature map according to the image features of the target medical image comprises: input of the three-dimensional feature map into a position prediction network to obtain a probability prediction map output by the position prediction network, the probability prediction map being used to represent probabilities of each feature position on the three-dimensional feature map as a center point of a candidate box, the size of the probability prediction map being the same as the size of the three-dimensional feature map; threshold filtering on the probability prediction map to determine the predicted position in the three-dimensional feature map, the probability of the predicted position as a center point of a candidate box being higher than a probability threshold.

3. The method of claim 2, wherein, The position prediction network is trained according to a sample feature map, the sample feature map having a positive sample region and a negative sample region labeled therein, the positive sample region being a center region of a labeled candidate box, the negative sample region being a region other than the positive sample region, and the labeled candidate box containing a sample target object.

4. The method according to any one of claims 1 to 3, characterized in that, The classifying the image features in the candidate box to obtain a classification result corresponding to the candidate box comprises: extracting image features from the feature adjustment map according to the predicted position and the predicted size corresponding to the candidate box; inputting the extracted image features into a classification network to obtain the classification result output by the classification network.

5. A medical image processing apparatus characterized by comprising: The device comprises: a feature extraction module configured to extract features of a target medical image by a three-dimensional convolutional neural network to obtain a three-dimensional feature map, the three-dimensional feature map being configured to represent image features of the target medical image, the target medical image being a CT image sequence; a candidate box determination module configured to determine at least one predicted position from the three-dimensional feature map according to image features of the target medical image, the probability of taking the predicted position as a center point of a candidate box being higher than the probability of taking other positions on the three-dimensional feature map as the center point of the candidate box, the candidate box representing a region occupied by image features of a candidate object in the three-dimensional feature map, the candidate object being an object suspected to be a target object, and the candidate box being a three-dimensional candidate box; inputting the three-dimensional feature map into a size prediction network according to the predicted position to obtain a predicted size corresponding to each predicted position output by the size prediction network, the predicted size being a size of a candidate box constructed by taking the predicted position as a center point of a three-dimensional candidate box, the size prediction network being supervised by an intersection over union between a regression candidate box and a neighboring labeled candidate box, and being trained by maximizing the intersection over union; determining the candidate box corresponding to at least one candidate object according to the predicted position and the predicted size, the candidate box taking the predicted position as the center and adopting the predicted size corresponding to the predicted position; a feature adjustment module configured to determine a feature bias of the three-dimensional feature map according to the predicted size, the feature bias being configured to indicate a bias condition of image features at each predicted position in the three-dimensional feature map; calculating a feature adjustment map according to the three-dimensional feature map and the feature bias, image features at the predicted position in the feature adjustment map matching the predicted size corresponding to the predicted position; a classification module configured to classify image features in the candidate box to obtain a classification result corresponding to the candidate box, the classification result being configured to represent whether the candidate object corresponding to the candidate box is the target object; a position determination module configured to determine a position of the target object in the target medical image according to the classification result.

6. A computer device, comprising: The computer device comprises a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the medical image processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the medical image processing method according to any one of claims 1 to 4.

Images