Image processing method, image processing model training method, and related device

Abstract

The application discloses an image processing method, a training method of an image processing model and related devices. The image processing method comprises the following steps: acquiring a medical image, processing the medical image, and obtaining a first prediction map of each first object in the medical image. The first prediction map of each first object contains a first prediction probability value of a pixel point in the medical image belonging to the corresponding first object. Then, based on the first prediction map of each first object in the medical image, a second prediction map of a second object in the medical image is generated. The second prediction map of the second object contains a second prediction probability value of a pixel point in the medical image belonging to the second object. The above scheme can improve the processing efficiency of the medical image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to an image processing method, a training method of an image processing model and related devices. BACKGROUND

[0002] Nowadays, image processing is widely used in the medical field, and different organs or lesion regions in a medical image can be divided out through image processing, and then the divided image can be analyzed. Therefore, efficient processing of medical images is of great significance for subsequent work.

[0003] In a medical image, some organs or lesions usually have a basic unit (referred to as a first object in the present application), and adjacent first objects are separated by a second object. At present, an additional branch network is usually added to further determine the second object on the basis of a processing module for determining the first object from the medical image, which is low in efficiency. Therefore, how to improve the processing efficiency of medical images has become a problem to be solved. SUMMARY

[0004] The present application provides an image processing method, a training method of an image processing model and related devices.

[0005] The first aspect of the present application provides an image processing method, comprising: obtaining a medical image and processing the medical image to obtain a first prediction map of each first object in the medical image, the first prediction map of each first object containing a first prediction probability value of a pixel point in the medical image belonging to the corresponding first object; and generating a second prediction map of each second object in the medical image based on the first prediction map of each first object in the medical image, the second prediction map of the second object containing a second prediction probability value of a pixel point in the medical image belonging to the second object, and adjacent first objects being separated by the second object.

[0006] Therefore, by obtaining a medical image and processing the medical image to obtain a first prediction map of each first object in the medical image, and generating a second prediction map of each second object in the medical image based on the first prediction map of each first object in the medical image, it is not necessary to add an additional branch network, and the second object processing can be realized directly based on the first prediction map of the first object in an end-to-end manner, which helps to improve the processing efficiency of medical images.

[0007] Among them, generating a second prediction map of each second object in the medical image based on the first prediction map of each first object in the medical image comprises: generating a third prediction map of the corresponding second object based on the first prediction maps of the two adjacent first objects on both sides of the second object, and obtaining the second prediction map of the second object based on the normalization of the third prediction map of the second object.

[0008] Therefore, by generating the third prediction map corresponding to the second object based on the first prediction maps of the first objects adjacent to the second object on two sides, and performing normalization based on the third prediction map of the second object, the second prediction map of the second object is obtained. On the one hand, since the third prediction map is generated based on the first prediction maps of the first objects adjacent to the second object on two sides, generating the third prediction map corresponding to the second object based on the first prediction maps of the first objects adjacent to the second object on two sides helps to improve the accuracy of the third prediction map. On the other hand, performing normalization based on the third prediction map of the second object to obtain the second prediction map of the second object so that the second prediction map of the second object contains a fixed sum of second prediction probability values of the pixel points belonging to the corresponding second object in the medical image, thereby improving the processing rate of the image processing method.

[0009] The third prediction map corresponding to the second object based on the first prediction maps of the first objects adjacent to the second object on two sides includes: respectively dilating the first prediction maps of the first objects adjacent to the second object on two sides, and performing intersection operation on the first prediction maps of the first objects adjacent to the second object on two sides after dilation to obtain the third prediction map corresponding to the second object.

[0010] Therefore, by respectively dilating the first prediction maps of the first objects adjacent to the second object on two sides, and then performing intersection operation on the first prediction maps of the first objects adjacent to the second object on two sides after dilation to obtain the third prediction map corresponding to the second object, the objects in the first prediction maps after dilation can be expanded, and on this basis, performing intersection operation on the first prediction maps of the first objects adjacent to the second object on two sides after dilation helps to improve the accuracy of the third prediction map corresponding to the second object.

[0011] The method further includes: after generating the third prediction map corresponding to the second object based on the first prediction maps of the first objects adjacent to the second object on two sides, and before performing normalization based on the third prediction map of the second object to obtain the second prediction map of the second object, performing inverse operation on the third prediction map of the second object, and performing pixel-by-pixel multiplication on the third prediction maps of all the second objects after inverse operation to obtain a background prediction map of the medical image; and performing normalization based on the third prediction map of the second object to obtain the second prediction map of the second object includes: performing normalization based on the third prediction map of the second object and the background prediction map to obtain at least the second prediction map of the second object.

[0012] Therefore, by inversing the third prediction map of the second object, pixel-by-pixel multiplication is performed on the third prediction map of each second object after being inverted, to obtain a background prediction map of the medical image, so that the background image can be processed as much as possible, and then normalization is performed based on the third prediction map of the second object and the background prediction map, to obtain at least the second prediction map of the second object, so that the second prediction map of the second object contains a fixed sum of second prediction probability values of pixel points belonging to the corresponding second object in the medical image, thereby improving the processing rate of the image processing method.

[0013] In the medical image, the lung is contained, the first object is a lung lobe, and the second object is a lung fissure.

[0014] In the medical image, the lung is contained, the first object is a lung lobe, and the second object is a lung fissure.

[0015] Therefore, by dividing the medical image to obtain the first sub-image containing the left lung and the second sub-image containing the right lung, and processing the first sub-image to obtain the first prediction map of each lung lobe in the left lung and processing the second sub-image to obtain the second prediction map of each lung lobe in the right lung, and then generating the second prediction map of the lung fissure in the left lung based on the first prediction map of each lung lobe in the left lung and generating the second prediction map of the lung fissure in the right lung based on the first prediction map of each lung lobe in the right lung, on the one hand, by dividing the medical image to obtain the first sub-image containing the left lung and the second sub-image containing the right lung, and processing each sub-image to obtain the corresponding prediction map, the calculation power can be saved, and then generating the second prediction map of the lung fissure in the left lung based on the first prediction map of each lung lobe in the left lung and generating the second prediction map of the lung fissure in the right lung based on the first prediction map of each lung lobe in the right lung, thereby improving the processing efficiency of the medical image while improving the accuracy of generating the second prediction map.

[0016] The second aspect of the present application provides a training method of an image processing model, comprising: obtaining a sample medical image, the sample medical image being labeled with a first mask diagram of a first object and a second mask diagram of a second object, and processing the sample medical image based on the image processing model to obtain a first probability diagram of each first object in the sample medical image, the first probability diagram of each first object containing a first sample probability value of a sample pixel point in the sample medical image belonging to the corresponding first object; then generating a second probability diagram of the second object in the sample medical image based on the first probability diagram of each first object in the sample medical image, the second probability diagram of the second object containing a second sample probability value of a sample pixel point in the sample medical image belonging to the second object, and adjacent first objects being separated by the second object; obtaining a first loss based on the first mask diagram of the first object and the first probability diagram, and obtaining a second loss based on the second mask diagram of the second object and the second probability diagram; and adjusting network parameters of the image processing model based on at least the first loss and the second loss.

[0017] Therefore, by obtaining a sample medical image and processing the sample medical image based on the image processing model to obtain a first probability diagram of each first object in the sample medical image, then generating a second probability diagram of the second object in the sample medical image based on the first probability diagram of each first object in the sample medical image, and on this basis, obtaining a first loss based on the first mask diagram of the first object and the first probability diagram, and obtaining a second loss based on the second mask diagram of the second object and the second probability diagram, and then adjusting the network parameters of the image processing model based on at least the first loss and the second loss, since the sample medical image is labeled with the first mask diagram of the first object and the second mask diagram of the second object, the first loss is obtained based on the first mask diagram of the first object and the first probability diagram, which helps to improve the accuracy of the first loss, and further, since the second probability diagram of the second object is generated based on the first probability diagram of the first object, which helps to improve the accuracy of the second probability diagram of the second object, and then the second loss is obtained based on the second mask diagram of the second object and the second probability diagram, which improves the accuracy of the second loss while improving the training speed of the image processing model, in addition, the network parameters of the image processing model can be adjusted based on the first loss and the second loss, and then the image processing model can be trained in two dimensions of the first object and the second object, so that the result of the medical image processing of the trained image processing model is more accurate, and the trained image processing model has better robustness.

[0018] The first loss is obtained based on the first mask image and the first probability image of the first object, including: obtaining a first sample probability value of a corresponding sample pixel point predicted to belong to an actual first object based on a first probability image of the sample pixel point actually belonging to the first object; obtaining a difference value between a preset value and the first sample probability value of the sample pixel point predicted to belong to the actual first object, and obtaining a first weight, and obtaining a second weight negatively correlated with the first weight based on the first weight; weighting the difference value and the preset value by using the first weight and the second weight respectively to obtain a loss weight of the sample pixel point; and obtaining the first loss based on the loss weight of the sample pixel point and the first sample probability value predicted to belong to the actual first object.

[0019] Therefore, by obtaining a first sample probability value of a corresponding sample pixel point predicted to belong to an actual first object based on a first probability image of the sample pixel point actually belonging to the first object, obtaining a difference value between a preset value and the first sample probability value of the sample pixel point predicted to belong to the actual first object, obtaining a first weight, and obtaining a second weight negatively correlated with the first weight based on the first weight, weighting the difference value and the preset value by using the first weight and the second weight respectively to obtain a loss weight of the sample pixel point, that is, the attention degree to the first object is the same at the beginning of model training, and after multiple iterations, the prediction of the trained model to the first object can be more accurate, and at this time, more attention is paid to the area close to the second object, thereby improving the training effect of the model. In addition, the first loss is obtained based on the loss weight of the sample pixel point and the first sample probability value predicted to belong to the actual first object, thereby improving the accuracy of the first loss in the model training process as much as possible, and further improving the robustness of the trained image processing model.

[0020] The first weight is obtained based on the training round, and the training round is positively correlated with the first weight.

[0021] Therefore, by obtaining the current training round of the image processing model, and then determining the first weight based on the training round, the attention degree in the training process can be transferred from the first object to the place close to the second object in the training process of the image processing model, thereby improving the training effect of the image processing model as much as possible, and the robustness of the trained image processing model is stronger.

[0022] The second loss is obtained based on the second mask image and the second probability image of the second object, including: obtaining a first displacement field of the second mask image and the second probability image of the registered same second object; and obtaining the second loss based on the first displacement field.

[0023] Therefore, by obtaining the first displacement field of the second mask map and the second probability map of the same second object, and then reflecting the distance between the second mask map and the second probability map of the same second object by registering the first displacement field of the second mask map and the second probability map of the same second object, and obtaining the second loss based on the first displacement field, the accuracy of the second loss can be improved.

[0024] In the method, the registration displacement field is predicted by the image registration model based on the second mask map and the second probability map, and before the second loss is obtained based on the first displacement field, the method further includes: registering the second mask map of the second object and itself based on the image registration model to obtain a second displacement field; and obtaining the second loss based on the first displacement field includes: obtaining the second loss based on a difference between the first displacement field and the second displacement field.

[0025] Therefore, by obtaining the first displacement field of the second mask map and the second probability map of the same second object, and then reflecting the distance between the second mask map and the second probability map of the same second object by registering the first displacement field of the second mask map and the second probability map of the same second object, and obtaining the second loss based on the first displacement field, the accuracy of the second loss can be improved.

[0026] The third aspect of the present application provides an image processing device, comprising: an acquisition module, a processing module and a generation module, the acquisition module is used for acquiring a medical image; the processing module is used for processing the medical image to obtain a first prediction map of each first object in the medical image; wherein the first prediction map of the first object contains a first prediction probability value of a pixel point in the medical image belonging to the corresponding first object; the generation module is used for generating a second prediction map of a second object in the medical image based on the first prediction map of each first object in the medical image; wherein the second prediction map of the second object contains a second prediction probability value of a pixel point in the medical image belonging to the second object, and adjacent first objects are separated by the second object.

[0027] A fourth aspect of this application provides a training apparatus for an image processing model, comprising: an acquisition module, a processing module, a generation module, a measurement module, and an adjustment module. The acquisition module is used to acquire sample medical images; wherein the sample medical images are annotated with a first mask image of a first object and a second mask image of a second object. The processing module is used to process the sample medical images based on the image processing model to obtain a first probability map of each first object in the sample medical images; wherein the first probability map of each first object contains a first sample probability value of a sample pixel in the sample medical images belonging to the corresponding first object. The generation module is used to generate a second probability map of a second object in the sample medical images based on the first probability maps of each first object in the sample medical images, wherein adjacent first objects are separated by the second object; wherein the second probability map of the second object contains a second sample probability value of a sample pixel in the sample medical images belonging to the second object. The measurement module is used to obtain a first loss based on the first mask image and the first probability map of the first object, and to obtain a second loss based on the second mask image and the second probability map of the second object. The adjustment module is used to adjust the network parameters of the image processing model based at least on the first loss and the second loss.

[0028] The fifth aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the image processing method of the first aspect above, or to implement the image processing model training method of the second aspect above.

[0029] The sixth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the image processing method of the first aspect above, or the training method of the image processing model of the second aspect above.

[0030] The above scheme obtains a first prediction map of each first object in the medical image by acquiring and processing the medical image, and then generates a second prediction map of the second object in the medical image based on the first prediction map of each first object. Therefore, there is no need to add an additional branch network. It can directly realize end-to-end processing of the second object based on the first prediction map of the first object, which helps to improve the processing efficiency of medical images.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0033] Figure 1This is a schematic flowchart of an embodiment of the image processing method of this application;

[0034] Figure 2 This is a flowchart illustrating an embodiment of the training method for the image processing model of this application;

[0035] Figure 3 This is a schematic diagram of an embodiment of the second object generation module;

[0036] Figure 4 This is a schematic diagram of the framework of an embodiment of the image processing model training method of this application;

[0037] Figure 5 This is a schematic diagram of the framework of an embodiment of the image processing apparatus of this application;

[0038] Figure 6 This is a schematic diagram of the framework of an embodiment of the training device for the image processing model of this application;

[0039] Figure 7 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0040] Figure 8 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0041] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0042] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0043] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. "Several" means at least one. The terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0044] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the image processing method of this application.

[0045] Specifically, this may include the following steps:

[0046] Step S11: Acquire medical images.

[0047] In one implementation scenario, medical images can be obtained by scanning the area or organ to be treated. Specifically, medical images can be obtained through scanning imaging techniques such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging). For example, a medical image can be obtained by scanning the lungs. Of course, medical images can also be selected from existing medical image databases. The method of acquiring medical images can be determined according to the actual situation and is not specifically limited here.

[0048] Step S12: Process the medical image to obtain the first prediction image of each first object in the medical image.

[0049] In this embodiment of the disclosure, the first predicted image of the first object contains a first predicted probability value of a pixel in the medical image belonging to the corresponding first object. It is understood that the medical image can be obtained by scanning an organ or part, and the first object can be a basic unit within the aforementioned organ or part, with adjacent first objects separated by second objects. For example, the medical image may contain lungs, and the human lung consists of five lobes (i.e., the aforementioned basic unit or first object), which are separated from each other by interlobar fissures (i.e., second objects). Each lobe has an independent bronchial and vascular system, giving them physiologically independent functions; therefore, the pathological characteristics and causes of disease in each lobe are also different. Therefore, by processing the medical image, specifically, prediction can be performed to obtain the first predicted image of each lobe in the medical image; or, the medical image can be processed to separate the first objects in the medical image, thereby obtaining the first predicted image of each first object in the medical image.

[0050] In one implementation scenario, medical images can be processed using an image processing model. The specific training process of this model can be found in the subsequent implementation examples of its training phase, leading to the first predicted images of each first object in the medical image. Alternatively, medical images can be processed using an image processing algorithm, which may include, but is not limited to, algorithms such as U-Net and Mask R-CNN, etc. No specific limitations are specified here.

[0051] In one implementation scenario, medical images can be processed to obtain first predicted images of each first object in the medical image. Unlike the aforementioned implementation methods, to improve processing speed, the medical image can be first segmented to obtain corresponding sub-images, and then the corresponding sub-images can be processed to obtain predicted images. For example, the lungs can be segmented to obtain a first sub-image containing the left lung and a second sub-image containing the right lung. Specifically, a segmentation model can be used to segment the medical image to obtain a first sub-image containing the left lung and a second sub-image containing the right lung. The segmentation model can be, but is not limited to, U-Net, Mask R-CNN, etc., and is not specifically limited here. After obtaining the first sub-image containing the left lung and the second sub-image containing the right lung, the first sub-image can be processed to obtain first predicted images of each lobe in the left lung, and the second sub-image can be processed to obtain second predicted images of each lobe in the right lung. The processing method can refer to the processing method in the aforementioned disclosed embodiments, and is not specifically limited here. Alternatively, the kidneys can be scanned to obtain medical images, which can then be divided into a first sub-image containing the left kidney and a second sub-image containing the right kidney. These sub-images can then be further processed to obtain a first predicted image of the first object. It is understood that the shown area is only one possible area to be processed in practical applications, and does not limit the specific area processed in actual applications. The specific area can be determined based on the actual situation and is not limited here.

[0052] Step S13: Based on the first prediction map of each first object in the medical image, generate the second prediction map of the second object in the medical image.

[0053] In this embodiment of the disclosure, the second prediction map of the second object contains a second predicted probability value of a pixel in the medical image belonging to the second object. It is understood that the second object can be the boundary between adjacent first objects. For example, the first object is a lung lobe, and the second object is a lung fissure. Since the left lung of a human is divided into the upper left lobe and the lower left lobe by a large oblique fissure, and the right lung is divided into the upper right lobe, the middle right lobe, and the lower right lobe by a small horizontal fissure and a large oblique fissure, a second prediction map of the lung fissure in the medical image can be generated based on the first prediction map of each lung lobe in the medical image.

[0054] In one implementation scenario, to generate a second predicted image of a second object in a medical image, the second predicted image of the second object in the medical image can be directly determined based on the first predicted images of the first objects adjacent to the second object. Unlike the aforementioned implementation, to further improve the accuracy of the second predicted image of the second object, a third predicted image of the corresponding second object can be generated first based on the first predicted images of the first objects adjacent to the second object. This third predicted image of the second object contains a third predicted probability value of pixels in the medical image belonging to the corresponding second object; that is, the third predicted probability value of the second object represents the unnormalized second predicted image. After generating the third predicted image of the corresponding second object, normalization is then performed based on the third predicted image of the second object to obtain the second predicted image of the second object. The above method generates a third prediction image of the second object based on the first prediction image of the first objects adjacent to the second object, and then normalizes the third prediction image of the second object to obtain a second prediction image of the second object. On the one hand, since the third prediction image is generated based on the first prediction images of the first objects adjacent to the second object, generating the third prediction image of the second object based on the first prediction images of the first objects adjacent to the second object helps to improve the accuracy of the third prediction image. On the other hand, normalizing the third prediction image of the second object to obtain the second prediction image of the second object ensures that the sum of the second prediction probability values ​​of pixels in the medical image belonging to the corresponding second object is fixed, thereby improving the processing speed of the image processing method.

[0055] In a specific implementation scenario, a third prediction image of the second object is derived based on the first prediction images of the first objects adjacent to the second object. As one possible implementation, the third prediction image of the second object can be obtained by directly intersecting the first prediction images of the first objects adjacent to the second object. Unlike the aforementioned implementation, the first prediction images of the first objects adjacent to the second object can be dilated firstly. It is understood that the overall brightness of the dilated first prediction image will increase, the size of brighter objects in the first prediction image will increase, while the size of darker objects will decrease or even disappear. The third prediction image of the second object is then obtained by intersecting the dilated first prediction images of the first objects adjacent to the second object. The method for intersecting the dilated first prediction images of the first objects adjacent to the second object can be determined according to the actual situation, such as using a for loop, iterator, map, etc., and is not specifically limited here. The above method, by dilating the first prediction maps of the first objects on both sides of the second object respectively, and then finding the intersection of the first prediction maps of the first objects on both sides of the second object after dilation, obtains the third prediction map of the corresponding second object. This can expand the objects in the first prediction map after dilation. On this basis, finding the intersection of the first prediction maps of the first objects on both sides of the second object after dilation helps to improve the accuracy of obtaining the third prediction map of the corresponding second object.

[0056] In one implementation scenario, a third prediction image of the second object can be obtained based on the first prediction image of the first object adjacent to the second object. Specifically, after obtaining the third prediction image of the second object, it can be inverted by subtracting 1 from it. Furthermore, the inverted third prediction images of each second object are multiplied pixel-by-pixel to obtain the background prediction image of the medical image. This process can extract as much background image as possible. Then, normalization is performed based on the third prediction image of the second object and the background prediction image to obtain at least the second prediction image of the second object. This method, by inverting the third prediction image of the second object and multiplying the inverted third prediction images of each second object pixel-by-pixel to obtain the background prediction image of the medical image, can extract as much background image as possible. Normalization is then performed based on the third prediction image of the second object and the background prediction image to obtain at least the second prediction image of the second object. This ensures that the sum of the second prediction probability values ​​of pixels in the medical image belonging to the corresponding second object is fixed, thereby improving the processing speed of the image processing method.

[0057] In one implementation scenario, the first object can be a lung lobe, and the second object can be a lung fissure. To obtain a second predicted image of the lung fissure, the medical image can be divided into a first sub-image containing the left lung and a second sub-image containing the right lung. The first sub-image is processed to obtain a first predicted image of each lung lobe in the left lung, and the second sub-image is processed to obtain a second predicted image of each lung lobe in the right lung. Based on the first predicted images of each lung lobe in the left lung, a second predicted image of the lung fissure in the left lung is generated, and based on the first predicted images of each lung lobe in the right lung, a second predicted image of the lung fissure in the right lung is generated. Specifically, a third predicted image of the corresponding lung fissure can be generated based on the first predicted images of the lung lobes adjacent to the lung fissure, and normalized based on the third predicted image of the lung fissure to obtain a second predicted image of the lung fissure in the left lung. The method for obtaining the second predicted image of the lung fissure in the right lung can refer to the method for obtaining the second predicted image of the lung fissure in the left lung, and will not be repeated here. The above method divides the medical image into a first sub-image containing the left lung and a second sub-image containing the right lung. The first sub-image is processed to obtain a first predicted image of each lobe in the left lung, and the second sub-image is processed to obtain a second predicted image of each lobe in the right lung. Based on the first predicted images of each lobe in the left lung, a second predicted image of the lung fissure in the left lung is generated, and the same applies to the first predicted images of each lobe in the right lung. This method saves computational resources by dividing the medical image into the first and second sub-images containing the left and right lungs and processing each sub-image to obtain the corresponding predicted image. Furthermore, it improves both the processing efficiency of the medical image and the accuracy of the generated second predicted images.

[0058] In one implementation scenario, the image processing method can be determined based on the actual application scenario. That is, the specific content of the predicted image based on the medical image can be determined first, and then the medical image can be processed. As one possible implementation, to obtain each first object in the medical image, the medical image can be processed to obtain a first predicted image of each first object. The first predicted image of the first object contains a first predicted probability value of a pixel in the medical image belonging to the corresponding first object, thus completing the processing of the medical image. As another possible implementation, to obtain a second object in the medical image, the medical image can be processed first to obtain a first predicted image of each first object. Then, based on the first predicted image of each first object, a second predicted image of the second object can be generated. The second predicted image of the second object contains a second predicted probability value of a pixel in the medical image belonging to the second object, thus completing the processing of the medical image. For example, if the first object can be a lung lobe and the second object can be a lung fissure, and only the first predicted image of the lung lobe is needed, the medical image can be processed to obtain the first predicted image of each lung lobe in the medical image. The first predicted image of the lung lobe contains the first predicted probability value of a pixel in the medical image belonging to the corresponding lung lobe, and the processing of the medical image can be completed. If the second predicted image of the lung fissure is needed, after obtaining the first predicted image of each lung lobe in the medical image, the second predicted image of the lung fissure in the medical image can be generated based on the first predicted image of each lung lobe in the medical image. The second predicted image of the lung fissure contains the second predicted probability value of a pixel in the medical image belonging to the lung fissure, and the processing of the medical image can be completed.

[0059] In one implementation scenario, a medical image can be acquired first, and then processed based on an image processing model to obtain a first probability map of each first object in the medical image. The first prediction map of each first object contains a first predicted probability value for a pixel in the medical image belonging to the corresponding first object. Based on the first probability maps of each first object in the medical image, a second probability map of a second object in the medical image is generated. The second prediction map of the second object contains a second predicted probability value for a pixel in the medical image belonging to the second object. As another possible implementation, a medical image can be acquired first, and then processed to obtain a first prediction map of each first object in the medical image, wherein the first prediction map of the first object contains a first predicted probability value for a pixel in the medical image belonging to the corresponding first object. The image processing model is trained on sample medical images labeled with a first mask map of the first object and a second mask map of the second object. The training process of the image processing model can be found in the subsequent training phase embodiments.

[0060] The above scheme obtains a first prediction map of each first object in the medical image by acquiring and processing the medical image, and then generates a second prediction map of the second object in the medical image based on the first prediction map of each first object. Therefore, there is no need to add an additional branch network. It can directly realize end-to-end processing of the second object based on the first prediction map of the first object, which helps to improve the processing efficiency of medical images.

[0061] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the image processing model training method of this application. Specifically, it may include the following steps:

[0062] Step S21: Obtain medical images of the sample.

[0063] In this embodiment, the sample medical image is annotated with a first mask image of a first object and a second mask image of a second object. Furthermore, adjacent first objects are separated by second objects, as detailed in the foregoing embodiments, and will not be repeated here. For example, when the sample medical image includes a lung, it is annotated with a first mask image of a lung lobe and a second mask image of a lung fissure. Moreover, the method for acquiring the sample medical image can refer to the method for acquiring medical images in the foregoing embodiments, and will not be repeated here.

[0064] In one implementation scenario, a second mask image corresponding to the second object can be generated based on the first mask image of the first object. Specifically, the first mask images of the first objects adjacent to the second object can be dilated, and the intersection of the dilated probability images of the first objects adjacent to the second object can be obtained to obtain the second mask image corresponding to the second object.

[0065] Step S22: Process the sample medical image based on the image processing model to obtain the first probability map of each first object in the sample medical image.

[0066] In this embodiment, the first probability map of the first object contains the first sample probability value of a sample pixel in the sample medical image belonging to the corresponding first object. Furthermore, as a possible implementation, the sample medical image can be directly processed based on an image processing model to obtain the first probability map of each first object in the sample medical image. Different from the aforementioned implementation, to improve the speed of model training, the sample medical image can be divided first before training to improve training efficiency. For example, when the sample medical image contains lungs, the sample medical image can be divided first to obtain a first sub-image containing the left lung and a second sub-image containing the right lung. Then, the first sub-image can be processed to obtain the first probability map of each lobe in the left lung, and the second sub-image can be processed to obtain the second probability map of each fissure in the right lung. The method of processing the sample medical image using an image processing model can be determined according to the actual situation and is not specifically limited here.

[0067] Step S23: Based on the first probability map of each first object in the sample medical image, generate the second probability map of the second object in the sample medical image.

[0068] It should be noted that this application does not involve additional network branches, but adopts an end-to-end method to obtain the prediction of the second object. It is understood that if additional network branches are added to predict the second object, the supervision signal from the second object can only be transmitted to a part of the network. By adopting an end-to-end method to obtain the prediction of the second object, this situation is avoided.

[0069] In this embodiment, the second probability map of the second object contains the second sample probability value of a sample pixel in the sample medical image belonging to the second object. Furthermore, as a possible implementation, the sample medical image can be directly processed based on an image processing model to obtain the first probability map of each first object in the sample medical image, and then the second probability map of the second object in the sample medical image can be generated based on the first probability map of each first object in the sample medical image. Different from the aforementioned implementation, to improve the speed of model training, the sample medical image can be divided first before training to improve the efficiency of model training. For example, when the sample medical image contains lungs, the sample medical image can be divided first to obtain a first sub-image containing the left lung and a second sub-image containing the right lung. The first sub-image can then be processed to obtain the first probability map of each lobe in the left lung, and the second sub-image can be processed to obtain the second probability map of each lobe in the right lung. Then, based on the first probability map of each lobe in the left lung, a second probability map of the lung fissure in the left lung can be generated, and based on the first probability map of each lobe in the right lung, a second probability map of the lung fissure in the right lung can be generated. The method of processing the medical images of the samples can be determined according to the actual situation, and no specific restrictions are made here.

[0070] In one implementation scenario, a second probability map of a second object in a sample medical image is generated based on the first probability maps of each first object in the sample medical image. This can be achieved by first generating a third probability map of the corresponding second object based on the first probability maps of the first objects adjacent to the second object in the sample medical image. Specifically, the first probability maps of the first objects adjacent to the second object can be dilated, and the intersection of these dilated first probability maps can be obtained to get the third probability map of the corresponding second object. The third probability map of the second object is then inverted, and the inverted third probability maps of each second object are multiplied pixel-by-pixel to obtain the background probability map of the sample medical image. Based on this, normalization is performed using the third probability map of the second object and the background probability map to obtain the second probability map of the second object in the sample medical image. For example, the specific expression can be represented as follows:

[0071]

[0072]

[0073]

[0074] In this model, the image processing model f(·) predicts the first object probability map as Y = softmax(f(X)). Furthermore, a second object generation module (FGM, fissure generation module) can be added after the last softmax layer. The FGM uses a gradient traceability method to obtain the prediction of the second object. Its core processing flow is as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of an embodiment of the second object generation module. The predicted probability map of the first object is set as Y. Then Y... c Let C be the value of the c-th channel of the probability graph. fiss The number of foreground classes representing the second object, i(c) and ii(c) represent the two first object classes adjacent to the second object class c, and mp(·) represents the maxpooling operation, used to simulate morphological dilation, while pixel-wise multiplication is used to simulate intersection. The third probability map representing the second object is obtained by dilating the first probability maps of the first objects on both sides of the second object, and then intersecting the dilated first probability maps of the first objects on both sides of the second object to obtain the corresponding third probability map of the second object. Next, the third probability map of the second object is inverted, and the inverted third probability maps of each second object are multiplied pixel-by-pixel to obtain the background probability map of the sample medical image. Based on this, normalization is performed using the third probability map of the second object and the background probability map to obtain the second probability map Z of the second object in the sample medical image. cFurthermore, since all operations of the second object generation module are derivable, the supervision signal for the second object can be transmitted back to the entire network of the image processing model.

[0075] Step S24: Based on the first mask map and the first probability map of the first object, obtain the first loss, and based on the second mask map and the second probability map of the second object, obtain the second loss.

[0076] Specifically, a first loss can be obtained based on the first mask map and the first probability map of the first object, and a second loss can be obtained based on the second mask map and the second probability map of the second object. Furthermore, a third loss can be obtained based on the first mask map and the first probability map of the first object. Specifically, the third loss can be calculated based on the overlap between the first mask map and the first probability map of the corresponding first object. The loss functions for the first, second, and third losses can be determined according to the actual situation. For example, the first loss can be obtained using the cross-entropy loss function, the second loss can be obtained using the registration loss function, and the third loss can be obtained using the Dice loss function. No specific restrictions are placed on the choice of loss function here. Additionally, the second loss for predicting the second object is measured using the registration loss; details can be found in the following description, which will not be elaborated here. It should be noted that compared to directly applying the Dice loss function to the second object, the registration loss used in this embodiment can, on the one hand, minimize loss oscillations caused by the thinness of the second object, thus improving training stability; on the other hand, it can enhance the constraint on the shape and structure of the second object, thereby improving the model's accuracy.

[0077] In one implementation scenario, to obtain the first loss, the sample medical images can be segmented before training to improve the efficiency of model training. For example, when the sample medical images contain lungs, the sample medical images are segmented to obtain a first sub-image containing the left lung and a second sub-image containing the right lung. The first sub-image is then processed to obtain a first probability map of each lobe in the left lung, and the second sub-image is processed to obtain a second probability map of each lobe in the right lung. Based on the first probability map of each lobe in the left lung, a second probability map of the lung fissure in the left lung is generated, and based on the first probability map of each lobe in the right lung, a second probability map of the lung fissure in the right lung is generated. Furthermore, based on the second probability map of the lung fissure in the left lung and the first mask map, and based on the second probability map of the lung fissure in the right lung and the first mask map, the first loss is obtained.

[0078] In another implementation scenario, since the first mask icon of the first object is marked with the actual first object to which the pixels in the sample medical image belong, the first sample probability value of the corresponding sample pixel predicted to belong to the actual first object can be obtained based on the first probability map of the first object to which the sample pixel actually belongs. The difference between the preset value and the first sample probability value of the sample pixel predicted to belong to the actual first object can be obtained, and the first weight can be obtained. Based on the first weight, the second weight negatively correlated with the first weight can be obtained. Then, the difference and the preset value are weighted by the first weight and the second weight respectively to obtain the loss weight of the sample pixel. On this basis, the first loss is obtained based on the loss weight of the sample pixel and the first sample probability value predicted to belong to the actual first object. The above method obtains the first sample probability value of the corresponding sample pixel belonging to the first object based on the first probability map of the first object to which the sample pixel actually belongs. It also obtains the first weight by the difference between the preset value and the first sample probability value of the sample pixel belonging to the first object. Based on the first weight, it obtains the second weight that is negatively correlated with the first weight. The first weight and the preset value are weighted by the difference and the second weight respectively to obtain the loss weight of the sample pixel. That is to say, in the early stage of model training, the attention to the first object is the same. However, after multiple iterations, the trained model can make the prediction of the first object more accurate. At this time, more attention is paid to the area close to the second object, which can improve the training effect of the model. In addition, the first loss is obtained based on the loss weight of the sample pixel and the first sample probability value of the sample pixel belonging to the first object, thereby improving the accuracy of the first loss during the model training process and further improving the robustness of the trained image processing model.

[0079] In a specific implementation scenario, the first weight can be a fixed value, such as 0.3, 0.4, 0.5, etc., without specific limitations. Unlike the aforementioned implementation methods, we can first obtain the current training epoch of the image processing model, and then determine the first weight based on the training epoch. The training epoch and the first weight are positively correlated. It is understandable that, since the training epoch can be positively correlated with the first weight during model training, the focus during training can be shifted from the first object to a point closer to the second object, thereby maximizing the model's training performance. For example, the first weight can be increased by 0.01 per epoch based on the image processing model's training epoch; or, if the image processing model's training performance is good, the first weight can be increased by 0.05, 0.06, etc., per epoch based on the image processing model's training epoch. Furthermore, it should be noted that, since the model has not yet converged in the initial stage of training, classification results with large prediction errors will be randomly distributed. Therefore, α can be slowly increased from 0 to 1 during the image processing model's training process. The above method obtains the current training epoch of the image processing model, and then determines the first weight based on the training epoch. In the training process of the image processing model, the focus of training can be shifted from the first object to a place closer to the second object, thereby making the training effect of the image processing model better and the robustness of the trained image processing model stronger.

[0080] For example, the expression for the loss weights can be represented as follows:

[0081]

[0082] Where X∈D is a medical image in the entire training sample set D, x i ∈X, x i A pixel representing the first mask of the first object, g(x) i The first probability map represents the first object to which the sample pixel actually belongs. The first sample probability value represents the predicted value of a sample pixel belonging to the actual first object, and α represents the first weight, α∈[0,1]. The value of α can be referred to the aforementioned disclosed implementation method, and will not be repeated here. The preset value is 1, but the preset value can also be set to 2, 3, etc., without specific limitation. Then, based on the first probability map of the actual first object to which the sample pixel belongs, the first sample probability value of the corresponding sample pixel predicted to belong to the actual first object can be obtained, and the difference between the preset value and the first sample probability value of the sample pixel predicted to belong to the actual first object can be obtained to obtain the first weight. Based on the first weight, the second weight negatively correlated with the first weight can be obtained. Then, the difference and the preset value can be weighted by the first weight and the second weight respectively to obtain the loss weight of the sample pixel. Based on the loss weight of the sample pixel and the first sample probability value predicted to belong to the actual first object, the first loss can be obtained. The expression of the first loss can be expressed as follows:

[0083]

[0084] in, The probability value w represents the first sample value predicted to belong to the actual first object. i Characterizing the loss weights, L ace Representing the first loss, it can be understood that the above formula shows that during the training process, the image processing model will give greater weight to pixels that are prone to errors. In the first object processing task, the pixels with larger errors are those pixels surrounding the second object. Therefore, the image processing model can focus its attention on the region around the second object.

[0085] In one implementation scenario, to obtain the second loss, the first displacement field of the second mask image and the second probability map registered with the same second object can be acquired, and then the second loss can be obtained based on the first displacement field. This method, by acquiring the first displacement field of the second mask image and the second probability map registered with the same second object, and then using the first displacement field to reflect the distance between the second mask image and the second probability map registered with the same second object, and then obtaining the second loss based on the first displacement field, helps to improve the accuracy of the second loss.

[0086] In another implementation scenario, unlike the aforementioned implementation, the registration displacement field is predicted by the image registration model using the second mask image and the second probability map. First, the first displacement field of the second mask image and the second probability map, which are registered to the same second object, can be obtained. Then, based on the image registration model, the second mask image of the second object and itself are registered to obtain the second displacement field. Subsequently, the second displacement field registered by the image registration model using the second mask image of the second object and itself can be obtained. Finally, based on the difference between the first and second displacement fields, a second loss is obtained. As one possible implementation, the difference between the first and second displacement fields can be directly determined to obtain the second loss. Unlike the aforementioned implementation, after determining the difference between the first and second displacement fields, the norm of the displacement field difference can be obtained, and the second loss can be obtained based on the norm of the displacement field difference. The above method obtains the first displacement field of the second mask image and the second probability map of the same second object, and registers the second mask image of the second object with itself based on the image registration model to obtain the second displacement field. Then, the second loss is obtained based on the difference between the first displacement field and the second displacement field. By obtaining the second displacement field, the error of the model itself can be obtained. Then, the second loss is determined by the difference between the first displacement field and the second displacement field, which can effectively offset the error of the model itself, thereby making the second loss more accurate and further improving the training effect of the model.

[0087] For example, the specific expression of the second loss can be represented as follows:

[0088]

[0089]

[0090] Among them, L reg Characterizing the second loss, Z represents the first displacement field, Z represents the second probability map of the second object, and G represents the second displacement field. f (X) represents the second mask image. Characterizing the second displacement field, Indicates from G f (X) to G f The displacement field of (X) should ideally be all zeros, but the actual registration model itself has certain errors. Here, we introduce... This is to compensate for this error. The norm characterizes the difference between the first displacement field and the second displacement field. Specifically, we can first obtain the first displacement field of the second mask image and the second probability map of the same second object, and then register the second mask image of the second object with itself based on the image registration model to obtain the second displacement field. Finally, based on the difference between the first and second displacement fields, we obtain the second loss.

[0091] Step S25: Adjust the network parameters of the image processing model based at least on the first loss and the second loss.

[0092] In one implementation scenario, the network parameters of the image processing model can be adjusted based on the first loss and the second loss. As one possible implementation, the first and second losses can be weighted and summed to obtain the total loss, and the network parameters of the image processing model can be adjusted based on this total loss. A specific expression can be given as follows:

[0093] L=λ1L ace +λ2L dc +λ3L reg

[0094] Where L represents the total loss, L ace Characterizing the first loss, L reg Characterizing the second loss, L dc The third loss is represented by λ1, λ2, and λ3, which are hyperparameters controlling the weights of each loss. These hyperparameters can be determined based on the actual situation; for example, λ1 and λ2 can be fixed at 1, and λ3 can be linearly increased from 0 to 1 during training. Unlike the aforementioned implementation, the total loss can be obtained by directly adding the first, second, and third losses, and the network parameters of the image processing model can be adjusted based on this total loss.

[0095] Furthermore, after training the image processing model, it can be validated. This can be done using validation medical images, specifically a first mask image labeled with the first object and a second mask image labeled with the second object. Alternatively, test medical images can be used for model testing. Or, public datasets can be used for both training and testing; for example, 40 sample medical images from a public dataset can be used for training, and 10 sample medical images from the same dataset can be used for testing. In addition, during the preprocessing stage, the image volume can be uniformly resampled to 1mm*1mm*1mm, and the HU values ​​within the window [-1000, 400] can be normalized to [0, 1]. It is noteworthy that during the training of the image processing model, the complete image volume is split into the left and right lungs to train their respective processing models.

[0096] It should be noted that the trained image processing model can be used to process medical images. For details, please refer to the aforementioned application stage implementation examples, which will not be repeated here.

[0097] In one implementation scenario, the trained image processing model can be integrated into surgical planning software, providing a user-friendly interface for doctors. When using the planning software, doctors only need to import the patient's scanned medical images, and the image processing model can automatically generate the processing results for the first object. Based on the results, the diagnostic focus can be placed on a specific first object, or it can assist in subsequent surgical planning, such as determining the severity of lesions in each lung lobe and assisting in lobectomy, thereby providing convenience for users.

[0098] The above scheme acquires sample medical images and processes them using an image processing model to obtain a first probability map of each first object in the sample medical image. Then, based on the first probability maps of the first objects, a second probability map of the second objects in the sample medical image is generated. On this basis, a first loss is obtained based on the first mask map and the first probability map of the first objects, and a second loss is obtained based on the second mask map and the second probability map of the second objects. Finally, based on at least the first and second losses, the network parameters of the image processing model are adjusted. Since the sample medical images are labeled with the first mask map of the first objects and the second mask map of the second objects, the network parameters are adjusted based on the first mask map and the first probability map of the first objects. The graph shows the first loss, which helps improve its accuracy. Furthermore, since the second probability map of the second object is generated based on the first probability map of the first object, it helps improve the accuracy of the second probability map of the second object. Based on the second mask map and the second probability map of the second object, the second loss is obtained. While improving the accuracy of the second loss, the training speed of the image processing model can also be improved. In addition, the network parameters of the image processing model can be adjusted based on the first and second losses. Thus, the image processing model can be trained through two dimensions: the first object and the second object. This makes the trained image processing model more accurate in processing medical images and gives it better robustness.

[0099] Please see Figure 4 , Figure 4 This is a schematic diagram of the framework of an embodiment of the image processing model training method of this application, as shown below. Figure 4As shown, the sample medical image is a lung image. The first object can be a lung lobe, and the second object can be a lung fissure. Before processing the sample medical image, it can be divided into a first sub-image containing the left lung and a second sub-image containing the right lung, and then processed using an image processing model. Alternatively, the sample medical image can be processed directly using an image processing model. Then, based on the image processing model, the sample medical image is processed to obtain the first probability map of each lung lobe in the sample medical image. Finally, the lung fissure generation module processes the first probability map of each lung lobe in the sample medical image to generate the second probability map of the lung fissure in the sample medical image. Based on this, a third loss is obtained using the first mask map and the first probability map of the lung lobe. Based on the first probability map of the lung lobe to which the sample pixel actually belongs, a first sample probability value is obtained, predicting that the corresponding sample pixel belongs to its actual lung lobe. The difference between a preset value and the first sample probability value is obtained, along with a first weight. A second weight, negatively correlated with the first weight, is then obtained. The first and second weights are used to weight the difference and the preset value respectively, yielding the loss weight for the sample pixel. The first loss is obtained based on the loss weight and the first sample probability value predicting that the sample pixel belongs to its actual lung lobe. A first displacement field is obtained from the second mask map and the second probability map registered with the same lung fissure. The second displacement field is obtained by registering the second mask map of the lung fissure with itself using the image registration model. The second displacement field is then obtained based on the difference between the first and second displacement fields. Based on this, the network parameters of the image processing model are adjusted using the first, second, and third losses.

[0100] The above scheme acquires sample medical images and processes them using an image processing model to obtain a first probability map of each first object in the sample medical image. Then, based on the first probability maps of the first objects, a second probability map of the second objects in the sample medical image is generated. On this basis, a first loss is obtained based on the first mask map and the first probability map of the first objects, and a second loss is obtained based on the second mask map and the second probability map of the second objects. Finally, based on at least the first and second losses, the network parameters of the image processing model are adjusted. Since the sample medical images are labeled with the first mask map of the first objects and the second mask map of the second objects, the network parameters are adjusted based on the first mask map and the first probability map of the first objects. The graph shows the first loss, which helps improve its accuracy. Furthermore, since the second probability map of the second object is generated based on the first probability map of the first object, it helps improve the accuracy of the second probability map of the second object. Based on the second mask map and the second probability map of the second object, the second loss is obtained. While improving the accuracy of the second loss, the training speed of the image processing model can also be improved. In addition, the network parameters of the image processing model can be adjusted based on the first and second losses. Thus, the image processing model can be trained through two dimensions: the first object and the second object. This makes the trained image processing model more accurate in processing medical images and gives it better robustness.

[0101] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0102] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the image processing apparatus of this application. The image processing apparatus 40 includes: an acquisition module 41, a processing module 42, and a generation module 43. The acquisition module 41 is used to acquire a medical image; the processing module 42 is used to process the medical image to obtain a first prediction image of each first object in the medical image; wherein the first prediction image of each first object contains a first prediction probability value of a pixel in the medical image belonging to the corresponding first object, and adjacent first objects are separated by second objects; the generation module 43 is used to generate a second prediction image of a second object in the medical image based on the first prediction images of each first object in the medical image; wherein the second prediction image of the second object contains a second prediction probability value of a pixel in the medical image belonging to the second object.

[0103] In the above scheme, the image processing device 40 can execute the steps in the above image processing embodiments, so there is no need to add additional branch networks. It can directly realize end-to-end processing of the second object based on the first prediction map of the first object, which helps to improve the processing efficiency of medical images.

[0104] In some disclosed embodiments, the generation module 43 includes a generation submodule and a normalization submodule. The generation submodule generates a third prediction map of the second object based on a first prediction map of the first objects adjacent to the second object; the normalization submodule normalizes the third prediction map of the second object to obtain a second prediction map of the second object.

[0105] Therefore, by generating a third prediction image of the second object based on the first prediction image of the first object adjacent to both sides of the second object, and then normalizing the third prediction image of the second object to obtain a second prediction image of the second object, the accuracy of the third prediction image is improved. On the other hand, normalizing the third prediction image of the second object to obtain a second prediction image of the second object ensures that the sum of the second prediction probability values ​​of pixels in the medical image belonging to the second object is fixed, thereby improving the processing speed of the image processing method.

[0106] In some disclosed embodiments, the generation submodule includes an image dilation unit and an image processing unit. The image dilation unit is used to dilate the first prediction images of the first objects on both sides adjacent to the second object, respectively. The image processing unit is used to find the intersection of the first prediction images of the first objects on both sides adjacent to the second object after dilation, so as to obtain the third prediction image of the corresponding second object.

[0107] Therefore, by dilating the first prediction maps of the first objects on both sides of the second object respectively, and then finding the intersection of the first prediction maps of the first objects on both sides of the second object after dilation, a third prediction map of the corresponding second object is obtained. This can further expand the objects in the first prediction map after dilation. On this basis, finding the intersection of the first prediction maps of the first objects on both sides of the second object after dilation helps to improve the accuracy of obtaining the third prediction map of the corresponding second object.

[0108] In some disclosed embodiments, the generation module 43 includes an inversion submodule and a calculation submodule. The inversion submodule inverts the third prediction image of the second object; the calculation submodule multiplies the inverted third prediction images of each second object pixel-by-pixel to obtain a background prediction image of the medical image. The normalization submodule includes a normalization unit, which normalizes the image based on the third prediction image of the second object and the background prediction image to obtain at least a second prediction image of the second object.

[0109] Therefore, by inverting the third prediction image of the second object and multiplying the inverted third prediction images of each second object pixel by pixel, a background prediction image of the medical image is obtained. This allows the background image to be processed as much as possible. Then, based on the third prediction image of the second object and the background prediction image, normalization is performed to obtain at least a second prediction image of the second object. This ensures that the second prediction image of the second object contains a fixed sum of the second prediction probability values ​​of pixels in the medical image belonging to the corresponding second object, thereby improving the processing speed of the image processing method.

[0110] In some disclosed embodiments, the medical image contains a lung, with the first object being a lung lobe and the second object being a lung fissure.

[0111] In some disclosed embodiments, the image processing apparatus 40 includes a segmentation module for segmenting a medical image to obtain a first sub-image containing the left lung and a second sub-image containing the right lung; the processing module 42 includes a processing sub-module for processing the first sub-image to obtain a first predicted image of each lobe in the left lung, and processing the second sub-image to obtain a second predicted image of each lobe in the right lung; the generation module 43 includes a generation sub-module for generating a second predicted image of the lung fissure in the left lung based on the first predicted image of each lobe in the left lung, and generating a second predicted image of the lung fissure in the right lung based on the first predicted image of each lobe in the right lung.

[0112] Therefore, by dividing the medical image into a first sub-image containing the left lung and a second sub-image containing the right lung, and processing the first sub-image to obtain the first predicted image of each lobe in the left lung, and processing the second sub-image to obtain the second predicted image of each lobe in the right lung, a second predicted image of the lung fissure in the left lung is generated based on the first predicted image of each lobe in the left lung, and a second predicted image of the lung fissure in the right lung is generated based on the first predicted image of each lobe in the right lung. On the one hand, dividing the medical image into a first sub-image containing the left lung and a second sub-image containing the right lung, and processing each sub-image to obtain the corresponding predicted image, helps to save computing power. On the other hand, generating the second predicted image of the lung fissure in the left lung based on the first predicted image of each lobe in the left lung, and generating the second predicted image of the lung fissure in the right lung based on the first predicted image of each lobe in the right lung, improves the processing efficiency of the medical image while improving the accuracy of generating the second predicted image.

[0113] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of an embodiment of the training device for the image processing model of this application. The training device 50 for the image processing model includes: an acquisition module 51, a processing module 52, a generation module 53, a measurement module 54, and an adjustment module 55. The system includes the following modules: an acquisition module 51 acquires a sample medical image, which is labeled with a first mask of a first object and a second mask of a second object; a processing module 52 processes the sample medical image based on an image processing model to obtain a first probability map of each first object in the sample medical image, wherein the first probability map of the first object contains a first sample probability value of a sample pixel in the sample medical image belonging to the corresponding first object; a generation module 53 generates a second probability map of a second object in the sample medical image based on the first probability map of each first object in the sample medical image, wherein adjacent first objects are separated by second objects; wherein the second probability map of the second object contains a second sample probability value of a sample pixel in the sample medical image belonging to the corresponding second object; a measurement module 54 obtains a first loss based on the first mask and the first probability map of the first object, and obtains a second loss based on the second mask and the second probability map of the second object; and an adjustment module 55 adjusts the network parameters of the image processing model based on at least the first loss and the second loss.

[0114] In the above scheme, the training device 50 of the image processing model can execute the steps in the training embodiment of the image processing model described above. It acquires sample medical images and processes them based on the image processing model to obtain a first probability map of each first object in the sample medical image. Then, based on the first probability maps of each first object in the sample medical image, it generates a second probability map of a second object in the sample medical image. On this basis, based on the first mask map and the first probability map of the first object, it obtains a first loss, and based on the second mask map and the second probability map of the second object, it obtains a second loss. Finally, based on at least the first loss and the second loss, it adjusts the network parameters of the image processing model. Since the sample medical image is labeled with the first mask map of the first object and the second mask map of the second object... Based on the first mask map and the first probability map of the first object, a first loss is obtained, which helps to improve the accuracy of the first loss. Furthermore, since the second probability map of the second object is generated based on the first probability map of the first object, it helps to improve the accuracy of the second probability map of the second object. Then, based on the second mask map and the second probability map of the second object, a second loss is obtained. While improving the accuracy of the second loss, the training speed of the image processing model can be improved. In addition, the network parameters of the image processing model can be adjusted based on the first loss and the second loss. Thus, the image processing model can be trained through two dimensions: the first object and the second object. This makes the trained image processing model more accurate in processing medical images and gives it better robustness.

[0115] In some disclosed embodiments, the measurement module 54 includes a calculation submodule, an acquisition submodule, a weighting submodule, and a determination submodule. The calculation submodule is used to obtain a first sample probability value predicting that a corresponding sample pixel belongs to the first object based on a first probability map of the first object to which the sample pixel actually belongs. The acquisition submodule is used to obtain the difference between a preset value and the first sample probability value predicting that the sample pixel belongs to the first object, and to obtain a first weight, and based on the first weight, to obtain a second weight negatively correlated with the first weight. The weighting submodule is used to weight the difference and the preset value using the first weight and the second weight respectively to obtain the loss weight of the sample pixel. The determination submodule is used to obtain a first loss based on the loss weight of the sample pixel and the first sample probability value predicting that it belongs to the first object.

[0116] Therefore, by obtaining the first probability map of the first object to which the sample pixel actually belongs, the first sample probability value of the corresponding sample pixel is obtained. The difference between the preset value and the first sample probability value of the sample pixel belonging to the first object is obtained to obtain the first weight. Based on the first weight, a second weight negatively correlated with the first weight is obtained. The difference and the preset value are weighted by the first weight and the second weight respectively to obtain the loss weight of the sample pixel. That is to say, in the early stage of model training, the attention to the first object is the same. However, after multiple iterations, the trained model can make the prediction of the first object more accurate. At this time, more attention is paid to the area close to the second object, which can improve the training effect of the model. In addition, the first loss is obtained based on the loss weight of the sample pixel and the first sample probability value of the sample pixel belonging to the first object, thereby improving the accuracy of the first loss during the model training process and further improving the robustness of the trained image processing model.

[0117] In some disclosed embodiments, the acquisition submodule includes an acquisition unit and a determination unit. The acquisition unit is used to acquire the current training epoch of the image processing model; the determination unit is used to determine a first weight based on the training epoch; wherein the training epoch is positively correlated with the first weight.

[0118] Therefore, by obtaining the current training epoch of the image processing model and determining the first weight based on the training epoch, the focus during the training process can be shifted from the first object to a place closer to the second object, thereby maximizing the training effect of the image processing model and making the trained image processing model more robust.

[0119] In some disclosed embodiments, the measurement module 54 includes a displacement field acquisition submodule and a loss determination submodule. The displacement field acquisition submodule is used to acquire the first displacement field of the second mask map and the second probability map registered with the same second object; the loss determination submodule is used to obtain the second loss based on the first displacement field.

[0120] Therefore, by obtaining the first displacement field of the second mask map and the second probability map registered with the same second object, and then reflecting the distance between the second mask map and the second probability map registered with the same second object through the first displacement field, and then obtaining the second loss based on the first displacement field, it helps to improve the accuracy of the second loss.

[0121] In some disclosed embodiments, the measurement module 54 includes a registration submodule, which is used to register the second mask image of the second object and itself based on the image registration model to obtain a second displacement field; the loss determination submodule includes a calculation unit, which is used to obtain a second loss based on the difference between the first displacement field and the second displacement field.

[0122] Therefore, by obtaining the first displacement field of the second mask image and the second probability map of the same second object, and registering the second mask image of the second object with itself based on the image registration model, the second displacement field is obtained. Then, based on the difference between the first displacement field and the second displacement field, the second loss is obtained. By obtaining the second displacement field, the error of the model itself can be obtained. Then, by using the difference between the first displacement field and the second displacement field, the second loss can be determined, which can effectively offset the error of the model itself, thereby making the second loss more accurate and further improving the training effect of the model.

[0123] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other. The processor 62 is used to execute program instructions stored in the memory 61 to implement the steps of any of the above-described image processing method embodiments, or the steps of any of the above-described image processing model training embodiments. In a specific implementation scenario, the electronic device 60 may include, but is not limited to, a microcomputer, a server, etc. In addition, the electronic device 60 may also include mobile devices such as smartphones, tablets, and smart glasses, which are not limited here.

[0124] Specifically, processor 62 controls itself and memory 61 to implement the steps of any of the above-described image processing method embodiments, or the steps of any of the above-described image processing model training embodiments. Processor 62 may also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 62 may be implemented using integrated circuit chips.

[0125] The above solution allows the electronic device 60 to implement the steps in the above image processing embodiments or the steps in the above image processing model training embodiments. Therefore, there is no need to add additional branch networks. It can directly achieve end-to-end processing of the second object based on the first prediction map of the first object, which helps to improve the processing efficiency of medical images.

[0126] Please see Figure 8 , Figure 8 This is a schematic diagram of a framework of an embodiment of a computer-readable storage medium according to this application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor. The program instructions 71 are used to implement the steps of any of the above-described image processing method embodiments, or the steps of any of the above-described image processing model training embodiments.

[0127] The above-described solution allows the computer-readable storage medium 70 to implement the steps in the above-described image processing embodiments or the steps in the above-described image processing model training embodiments. Therefore, there is no need to add additional branch networks, and end-to-end processing of the second object can be achieved directly based on the first prediction map of the first object, which helps to improve the processing efficiency of medical images.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a medical image; processing the medical image to obtain a first prediction map of each first object in the medical image; wherein the first prediction map of each first object contains a first prediction probability value of a pixel in the medical image belonging to the corresponding first object; generating a second prediction map of a second object in the medical image based on the first prediction map of each first object in the medical image; wherein the second prediction map of the second object contains a second prediction probability value of a pixel in the medical image belonging to the second object, and adjacent first objects are separated by the second object.

2. The method of claim 1, wherein, The generating of the second prediction map of the second object in the medical image based on the first prediction map of each first object in the medical image comprises: generating a third prediction map corresponding to the second object based on the first prediction map of the first object on both sides of the second object; normalizing the third prediction map of the second object to obtain the second prediction map of the second object.

3. The method of claim 2, wherein, The generating of the third prediction map corresponding to the second object based on the first prediction map of the first object on both sides of the second object comprises: respectively dilating the first prediction map of the first object on both sides of the second object; performing intersection on the first prediction map of the first object on both sides of the second object after dilation to obtain the third prediction map corresponding to the second object.

4. The method of claim 2, wherein, After the generating of the third prediction map corresponding to the second object based on the first prediction map of the first object on both sides of the second object, and before the normalizing of the third prediction map of the second object to obtain the second prediction map of the second object, the method further comprises: inverting the third prediction map of the second object; performing pixel-by-pixel multiplication on the inverted third prediction map of each second object to obtain a background prediction map of the medical image; The normalizing of the third prediction map of the second object to obtain the second prediction map of the second object comprises: normalizing the third prediction map of the second object and the background prediction map to obtain at least the second prediction map of the second object.

5. The method according to any one of claims 1 to 4, characterized in that, The medical image contains lungs, the first object is a lung lobe, and the second object is a lung fissure.

6. The method of claim 5, wherein, After the obtaining of the medical image, and before the processing of the medical image to obtain the first prediction map of each first object in the medical image, the method further comprises: dividing the medical image to obtain a first sub-image containing a left lung and a second sub-image containing a right lung; The processing of the medical image to obtain the first prediction map of each first object in the medical image comprises: processing the first sub-image to obtain the first prediction map of each lung lobe in the left lung, and processing the second sub-image to obtain the second prediction map of each lung lobe in the right lung; The generating of the second prediction map of the second object in the medical image based on the first prediction map of each first object in the medical image comprises: generate a second prediction map of lung fissures in the left lung based on the first prediction map of the individual lung lobes in the left lung and generate a second prediction map of lung fissures in the right lung based on the first prediction map of the individual lung lobes in the right lung.

7. A training method for an image processing model, characterized in that, Comprise: Obtain a sample medical image; wherein the sample medical image is labeled with a first mask map of a first object and a second mask map of a second object; Process the sample medical image based on an image processing model to obtain a first probability map of each first object in the sample medical image; wherein the first probability map of each first object contains a first sample probability value of a sample pixel point in the sample medical image belonging to the corresponding first object; Based on the first probability map of each first object in the sample medical image, generate a second probability map of the second object in the sample medical image; wherein the second probability map of the second object contains a second sample probability value of a sample pixel point in the sample medical image belonging to the second object, and adjacent first objects are separated by the second object; Based on the first mask map and the first probability map of the first object, obtain a first loss, and based on the second mask map and the second probability map of the second object, obtain a second loss; At least based on the first loss and the second loss, adjust the network parameters of the image processing model.

8. The method of claim 7, wherein, The first mask map of the first object labels the actual first object to which the pixel point in the sample medical image belongs, and the first loss is obtained based on the first mask map and the first probability map of the first object, comprising: Based on the first probability map of the actual first object to which the sample pixel point belongs, obtain the first sample probability value of the sample pixel point predicted to belong to the actual first object; Obtain the difference between the first sample probability value of the sample pixel point predicted to belong to the actual first object and a preset value, obtain a first weight, and based on the first weight, obtain a second weight negatively correlated with the first weight; Use the first weight and the second weight to weight the difference and the preset value respectively to obtain the loss weight of the sample pixel point; Based on the loss weight of the sample pixel point and the first sample probability value predicted to belong to the actual first object, obtain the first loss.

9. The method of claim 8, wherein, The first weight is obtained, comprising: Obtain the current training round of the image processing model; Determine the first weight based on the training round; wherein the training round is positively correlated with the first weight.

10. The method of claim 7, wherein, The second loss is obtained based on the second mask map and the second probability map of the second object, comprising: Obtain a first displacement field that registers the second mask map and the second probability map of the same second object; Obtain the second loss based on the first displacement field.

11. The method of claim 10, wherein, The registration displacement field is predicted by an image registration model based on the second mask map and the second probability map, and before the second loss is obtained based on the first displacement field, the method further comprises: Register the second mask map of the second object with itself based on the image registration model to obtain a second displacement field; The second loss is obtained based on the first displacement field, comprising: The second loss is obtained based on a difference between the first displacement field and the second displacement field.

12. An image processing apparatus characterized by comprising: The method comprises the steps of: An acquisition module is configured to acquire a medical image. A processing module is configured to process the medical image to obtain a first prediction map of each first object in the medical image, wherein the first prediction map of each first object contains a first prediction probability value of a pixel in the medical image belonging to a corresponding first object. A generation module is configured to generate a second prediction map of a second object in the medical image based on the first prediction map of each first object in the medical image, wherein the second prediction map of the second object contains a second prediction probability value of a pixel in the medical image belonging to the second object, and adjacent first objects are separated by the second object.

13. A training device for an image processing model, characterized in that, The method comprises the steps of: An acquisition module is configured to acquire a sample medical image, wherein the sample medical image is labeled with a first mask map of a first object and a second mask map of a second object. A processing module is configured to process the sample medical image based on an image processing model to obtain a first probability map of each first object in the sample medical image, wherein the first probability map of each first object contains a first sample probability value of a sample pixel in the sample medical image belonging to a corresponding first object. A generation module is configured to generate a second probability map of a second object in the sample medical image based on the first probability map of each first object in the sample medical image, wherein the second probability map of the second object contains a second sample probability value of a sample pixel in the sample medical image belonging to the second object, and adjacent first objects are separated by the second object. A measurement module is configured to obtain a first loss based on the first mask map and the first probability map of the first object, and obtain a second loss based on the second mask map and the second probability map of the second object. An adjustment module is configured to adjust network parameters of the image processing model based on at least the first loss and the second loss.

14. An electronic device, comprising: The program instructions are executed by the processor to implement the image processing method of any one of claims 1 to 6, or the training method of the image processing model of any one of claims 7 to 11.

15. A computer readable storage medium having stored thereon program instructions, wherein, The program instructions are executed by the processor to implement the image processing method of any one of claims 1 to 6, or the training method of the image processing model of any one of claims 7 to 11.

Images