Image processing method and computer-aided diagnosis method for cancer

By fusing object and lesion feature information into the plain CT image processing model, the accuracy of tumor screening is improved, the problem of low detection rate of plain CT is solved, the convenience of examination is expanded, and the risks of enhanced CT are reduced.

CN119048419AActive Publication Date: 2024-11-29ALIBABA (CHINA) CO LTD

Patent Information

Application Number
CN202410927473.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-29
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

In existing technologies, plain CT scans have low accuracy in tumor screening and are difficult to effectively identify lesions. Furthermore, enhanced CT scans are limited by equipment and technology, making them unavailable in all regions, and there is a risk of contrast agent allergy.

Method used

Image processing methods are employed to receive multiple plain CT images, generate object and lesion feature information using an image processing model, fuse feature information, and generate object classification, lesion classification, and segmentation information to improve detection accuracy.

Benefits of technology

It improves the detection accuracy of plain CT in tumor screening, reduces the dependence on contrast agents, expands the convenience of examination, and is suitable for the identification of abnormal objects in various target detection areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048419A_ABST
    Figure CN119048419A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image processing method and a cancer computer-aided diagnosis method, and the image processing method comprises the steps: receiving an image processing task which carries a plurality of target images corresponding to a target detection region and is used for detecting whether an abnormal object exists in the target detection region or not; the multiple target images are input into an image processing model, a detection result corresponding to the target detection area is obtained, the image processing model generates object feature information and focus feature information based on all the target images, object fusion feature information is generated according to the object feature information and the focus feature information, and the object fusion feature information is sent to the target detection area; object classification information is determined according to the object fusion feature information, focus classification information and focus segmentation information are determined according to the focus feature information, and a detection result is generated according to the object classification information, the focus classification information and the focus segmentation information. Through the method, the accuracy of image detection is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular to an image processing method. Background Art

[0002] As people's living standards improve, more and more people begin to pay attention to their health problems. Cancer is an important factor affecting people's physical health. Early diagnosis has a positive effect on cancer treatment. With the continuous development of computer technology, various learning models have gradually been applied to the prediction of various application scenarios. Image processing models have also achieved remarkable success in medical image computer-aided diagnosis (CAD) tasks. Pathological analysis and classification from medical images is an important topic in computer-aided diagnosis.

[0003] At present, the screening technology for tumors is mainly based on enhanced CT. Although enhanced CT has high accuracy, it will expose patients to contrast agents, which may cause allergic reactions or organ failure. In addition, enhanced CT examinations are limited by technology and equipment and cannot be used 24 hours a day in all regions, which affects its convenience. Due to the low resolution of plain scan CT, lesions on plain scan CT are visually difficult to identify, and the detection rate is slightly lower than that of enhanced CT. It is also difficult to mark lesions on plain scan CT. Therefore, how to use plain scan CT for tumor screening and improve the accuracy of tumor screening using plain scan CT. Summary of the invention

[0004] In view of this, an embodiment of this specification provides an image processing method. One or more embodiments of this specification also relate to a CT image processing method, a computer-aided diagnosis method for cancer, a computer-aided diagnosis method for liver cancer, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an image processing method, including:

[0006] receiving an image processing task, wherein the image processing task carries a plurality of target images corresponding to a target detection area, and the image processing task is used to detect whether there is an abnormal object in the target detection area;

[0007] The multiple target images are input into an image processing model to obtain detection results corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each target image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates detection results according to the object classification information, the lesion classification information and the lesion segmentation information.

[0008] According to a second aspect of the embodiments of this specification, a CT image processing method is provided, comprising:

[0009] receiving a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether there is an abnormal object in the target detection area;

[0010] The multiple plain scan CT images are input into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0011] According to a third aspect of the embodiments of this specification, a computer-aided diagnosis method for cancer is provided, comprising:

[0012] receiving a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether a tumor exists in the target detection area;

[0013] The multiple plain scan CT images are input into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-aided diagnosis method for liver cancer is provided, comprising:

[0015] receiving a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a liver region, and the CT image processing task is used to detect whether a tumor exists in the liver region;

[0016] The multiple plain scan CT images are input into an image processing model to obtain a detection result corresponding to the liver area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0017] According to a fifth aspect of the embodiments of this specification, a computer-aided diagnosis system for cancer is provided, including a client and a server, wherein:

[0018] The client is used to send a CT image processing task to the server, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether a tumor exists in the target detection area;

[0019] The server is used to input the multiple plain scan CT images into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0020] According to a sixth aspect of an embodiment of this specification, a computing device is provided, including:

[0021] Memory and processor;

[0022] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0023] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the steps of the above method are implemented when the computer program / instruction is executed by a processor.

[0024] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0025] An embodiment of the present specification provides an image processing method, comprising receiving an image processing task, wherein the image processing task carries multiple target images corresponding to a target detection area, and the image processing task is used to detect whether there is an abnormal object in the target detection area; inputting the multiple target images into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each target image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0026] Through the image processing method provided in the embodiments of this specification, multiple target images are input into the image processing model. In the image processing model, object feature information and lesion feature information are generated based on each target image. At the same time, the object feature information and the lesion feature information are fused to generate object fusion feature information. When the model predicts the final object classification information, in addition to using the object feature information, it also refers to the lesion feature information, fuses the global information and the local information, and improves the detection capability of small lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of an image processing method provided by an embodiment of this specification;

[0028] Figure 2 is a schematic diagram of a model structure of an image processing model provided by an embodiment of this specification;

[0029] Figure 3 is a data processing diagram of an object branch network provided by an embodiment of this specification;

[0030] Figure 4 It is a data processing schematic diagram of a lesion branch network provided by an embodiment of this specification;

[0031] Figure 5 is a data processing schematic diagram of a fusion network provided by an embodiment of this specification;

[0032] Figure 6 is a flowchart of an image processing model training method provided by an embodiment of this specification;

[0033] Figure 7 is a processing flow chart of a CT image processing method provided by an embodiment of this specification;

[0034] Figure 8 is a flowchart of a computer-aided diagnosis method for cancer provided by one embodiment of this specification;

[0035] Fig. 9 It is a flowchart of a computer-aided diagnosis method for liver cancer provided by one embodiment of this specification;

[0036] Fig.10 is an architecture diagram of a computer-aided diagnosis system for cancer provided by one embodiment of this specification;

[0037] Fig.11 is a structural schematic diagram of an image processing device provided by an embodiment of this specification;

[0038] Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0039] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0040] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0043] First, the terms involved in one or more embodiments of this specification are explained.

[0044] CT (Computed Tomography): Computer tomography uses a precisely collimated X-ray beam and a highly sensitive detector to perform cross-sectional scans around a certain part of the human body one by one. It has the characteristics of fast scanning time and clear images, and can be used to detect a variety of diseases.

[0045] Plain scan CT: Also known as a normal scan, it refers to a scan without the administration of contrast agent into the vein.

[0046] Enhanced CT: refers to the method of scanning after intravascular injection of contrast agent. The purpose is to increase the density difference between the diseased tissue and the normal tissue to show the lesions that are not displayed or not displayed clearly on the plain CT. The presence or absence of enhancement and the type of enhancement help to characterize the lesions.

[0047] CAD (computer aided diagnosis): Computer-aided diagnosis refers to the use of imaging, medical image processing technology and other possible physiological and biochemical means, combined with computer analysis and calculation, to assist in the discovery of lesions and improve the accuracy of diagnosis.

[0048] As people's living standards improve, more and more people pay attention to their own health. Tumors are one of the major factors affecting human health. Identifying tumors in medical images requires professional doctors to identify them based on their experience. Due to the limitations of doctors' experience, image recognition and analysis with the help of medical images has become an important topic.

[0049] Current cancer screening technology is mainly based on enhanced CT. Although enhanced CT has high accuracy, it exposes patients to contrast agents, which may cause allergic reactions or organ failure. In addition, enhanced CT examinations are limited by technology and equipment and cannot be used 24 hours a day in all areas, which affects its convenience. Some screening technologies are based on ultrasound diagnosis, but the sensitivity is not high and it is easy to miss a diagnosis. At present, some technologies are based on plain scan CT, but plain scan CT has a low resolution and lesions on organs are difficult to identify visually, so the detection rate is slightly lower than that of enhanced CT. Therefore, how to better detect tumors has become an urgent problem for technicians to solve.

[0050] Based on this, in this specification, an image processing method is provided. One or more embodiments of this specification also relate to a CT image processing method, a computer-aided diagnosis method for cancer, a computer-aided diagnosis method for liver cancer, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0051] See also Figure 1 , Figure 1 A flowchart of an image processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0052] Step 102: Receive an image processing task, wherein the image processing task carries a plurality of target images corresponding to a target detection area, and the image processing task is used to detect whether there is an abnormal object in the target detection area.

[0053] In actual applications, the image processing task sent by the user can be received through the server or the client.

[0054] Specifically, the image processing task can be understood as a task for detecting whether there is an abnormal object in the target detection area. The image processing task carries multiple target images corresponding to the target detection area. Furthermore, the target detection area can be understood as a partition for detecting whether there is an abnormal object, and the abnormal object can be understood as a foreign body in the target detection area. For example, the abnormal object can be a tumor in the human body.

[0055] The target detection area can be any organ in the human body, such as the liver, lungs, stomach, esophagus, etc. By predicting whether there are abnormal objects in the target detection area, the state of the object to be detected can be further judged based on the prediction results, thereby providing assistance for the positioning and precise treatment of abnormal objects.

[0056] The object to be detected can be understood as the object described in the target detection area. For example, if the target image is a CT image of Zhang San's stomach, then the target detection area is the stomach, the CT image is the target image, and the object to be detected is Zhang San. The image processing task is used to detect whether there is a tumor in the stomach area. In practical applications, the object to be detected can be a person or other living organisms, and this is not limited in one or more specific embodiments provided in this specification.

[0057] It should be noted that in one or more embodiments of this specification, the image processing task can be applied to the recognition of various types of medical images, and determine whether there are abnormal objects in the target detection area in the medical image based on the image features. For example, in the application scenario of gastric cancer detection, it is possible to predict whether there is a tumor in the stomach area based on the medical image of the stomach area, thereby helping doctors to accurately locate the abnormal part; in the application scenario of liver cancer detection, it is possible to predict whether there is a tumor in the liver based on the medical image of the liver area, thereby helping doctors to accurately locate the abnormal part, which is convenient for subsequent treatment.

[0058] For example, in the scenario of liver cancer detection, the target image obtained is an image of the liver. Specifically, the multiple target images obtained are plain scan CT images of the liver, and the multiple target images can form a 3D image of the liver area. The abnormal object can be understood as a tumor in the liver. Multiple plain scan CT images corresponding to the liver area are obtained, and image detection processing is performed on the multiple plain scan CT images to detect whether there is a tumor in the liver.

[0059] In practical applications, the abnormal object may be a certain type of cell, a certain type of tissue structure, etc., for example, a malignant tumor, a benign tumor, a hyperplastic tissue, etc. This is not limited in one or more embodiments provided in this specification.

[0060] By receiving the image processing task, multiple target images corresponding to the target detection area carried in the image processing task can be used as input to detect whether there is an abnormal object in the target detection area.

[0061] In a specific implementation manner provided in this specification, before receiving the image processing task, the method further includes:

[0062] receiving an image segmentation task, wherein the image segmentation task carries a plurality of initial images corresponding to a target detection area, and the image segmentation task is used to extract a target image corresponding to the target detection area;

[0063] Each initial image is input into a pre-trained image segmentation model to obtain a target image corresponding to each initial image output by the image segmentation model.

[0064] In the embodiments provided in the embodiments of this specification, the target image can be understood as a close-up image corresponding to the target detection area. In actual applications, the received multiple images may include other areas in addition to the target detection area, and the other areas may affect the target detection area. Therefore, in the method provided in the embodiments of this specification, the image is also segmented.

[0065] Specifically, firstly, an image segmentation task for a plurality of initial images is obtained, wherein the initial images include both the target detection area and the area that affects the target detection area. The image segmentation task is used to extract the target image corresponding to the target detection area from each initial image.

[0066] The multiple initial images are input to a pre-trained image segmentation model for processing. The image segmentation model is trained to identify the target detection area in the initial image, and cut out the target detection area from the initial image to generate a target image corresponding to the target detection area.

[0067] Taking a plain scan CT image as an example, in the method provided in this specification, an initial plain scan CT image is first obtained. The initial plain scan CT image is a plain scan CT image that meets the image quality requirements. The source of the initial plain scan CT image can be multiple CT scanners or the same CT scanner, which is not limited in this specification. After obtaining the initial plain scan CT image, the format of each initial plain scan CT image should be unified. The image segmentation model can be 3DUNet.

[0068] 3DUNet is a deep learning architecture for image segmentation in three-dimensional space. It is an extended version of the U-Net model, which was originally designed for semantic segmentation of two-dimensional biomedical images and is widely recognized for its excellent performance and high accuracy in segmenting small objects. 3DUNet applies this idea to three-dimensional datasets, such as medical images (such as CT, MRI scans, etc.), which is very useful in many medical fields because the data in these fields usually have rich three-dimensional structural information.

[0069] 3DUNet includes an encoder-decoder structure for capturing global context information, restoring lost spatial details and generating accurate pixel-level segmentation labels. The 3D convolution kernel in the 3DUNet structure replaces the 2D convolution kernel in the network, and can simultaneously process the features of the input data in three dimensions (length, width, and height). At the same time, 3DUNet can also effectively fuse three-dimensional features at different levels, which helps to extract complex shape and structural information. This framework has a good effect in medical image segmentation methods. In the method provided in this specification, the initial CT image is cut using the preprocessing strategy of the 3DUNet structure to obtain the target image corresponding to the target detection area for subsequent processing.

[0070] In the method provided in the embodiment of this specification, the detection of the liver area is used as an example for explanation. The initial plain scan CT image is resampled to a size of 0.7*0.7*5mm in the x, y, and z directions, and the 3DUNet segmentation model is used to segment the liver and several surrounding organs on the initial plain scan CT image. The surrounding organs include the gallbladder, intrahepatic blood vessels, spleen, stomach, pancreas, right kidney, inferior vena cava, etc.

[0071] Then, according to the masks of the liver and spleen, the corresponding marking boxes are determined, and then the pixel distance of the preset range is expanded outward on the basis of the marking box to obtain the target image. The purpose of obtaining the liver and spleen regions is to use the spleen as a reference organ of the liver, so that when the subsequent model processes the image, it can not only learn the characteristics of the target detection area (liver), but also learn the characteristics of the reference detection area (spleen), so as to learn that the characteristics of the liver need to be considered, but the characteristics of the spleen do not need to be considered.

[0072] Step 104: Input the multiple target images into an image processing model to obtain detection results corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each target image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates detection results according to the object classification information, the lesion classification information and the lesion segmentation information.

[0073] After receiving the image processing task, multiple target images carried by the image processing task can be obtained from the image processing task, and the multiple target images can be input into the image processing model to obtain the detection results corresponding to the target detection area output by the image processing model. The detection results specifically include object classification information of the object to be detected, lesion segmentation information and lesion classification information on the target detection area.

[0074] The object classification information of the object to be detected can be understood as classification information at the object level, which includes whether there are abnormal objects, the types of abnormal objects, and the probability of abnormal objects corresponding to each category.

[0075] The lesion segmentation information can be understood as the mask information for abnormal objects on the image to be detected. In the method provided in the embodiment of this specification, the lesion segmentation information includes global mask information and local mask information. The global mask information can be understood as the mask information of all abnormal objects in the target detection area, and the local mask information can be understood as the mask information for each type of abnormal object. For example, if there are 3 abnormal objects in a certain target detection area, the lesion segmentation information includes 4 parts, namely, all the mask information of the three abnormal objects, and the mask information corresponding to each of the three abnormal objects.

[0076] The lesion classification information can be understood as determining the type of lesion in only one step after the image processing model is identified, if there is lesion segmentation information. Furthermore, the lesion classification information includes lesion category information and organ category information. In the detection of liver tumors provided in the embodiments of this specification, lesions are divided into 9 categories (respectively, hepatocellular carcinoma, cholangiocarcinoma, metastasis, hemangioma, focal nodular hyperplasia, cyst, calcification, other malignant tumors, and other benign tumors), and organs are divided into 8 categories (respectively, liver, gallbladder, intrahepatic blood vessels, spleen, stomach, pancreas, right kidney, and inferior vena cava).

[0077] In the method provided in the embodiments of the present specification, after receiving multiple target images, the image processing model processes the multiple target images to generate object feature information and lesion feature information, wherein the object feature information can be understood as feature information used to generate object classification information, and the lesion feature information can be understood as feature information used to generate lesion classification information and lesion segmentation information. Specifically, the lesion feature information includes lesion sub-feature information and organ sub-feature information, the lesion sub-feature information is used to determine the lesion type, and the organ sub-feature information is used to determine the organ type.

[0078] In the method provided in the embodiment of this specification, although the object feature information can be used to generate the object classification information, in order to improve the prediction accuracy, the lesion feature information is fused with the object feature information to make the prediction more accurate. Based on this, the object feature information and the lesion feature information can be fused to obtain the object fusion feature information. The object fusion feature information is still the feature information used to predict the object classification information. Compared with the object feature information, it adds the relevant content of the lesion feature information, so that it can be more accurate in the subsequent prediction process.

[0079] Finally, the image processing model determines the object classification information based on the object fusion feature information, and determines the lesion classification information and lesion segmentation information based on the lesion feature information, thereby forming the final detection result.

[0080] In the method provided in the embodiment of this specification, the model structure of the image processing model is further explained. Specifically, the image processing model includes an image processing backbone network, an object branch network, a lesion branch network and a fusion network;

[0081] See also Figure 2 , Figure 2 FIG. 4 shows a schematic diagram of a model structure of an image processing model provided in an embodiment of the present specification. Figure 2 As shown, the image processing model includes an image processing backbone network, an object branch network, a lesion branch network and a fusion network.

[0082] Inputting the multiple target images into the image processing model to obtain the detection results corresponding to the target detection area, including S1042-S1048:

[0083] S1042: Input the multiple target images into the image processing backbone network, and extract feature information of the objects to be processed and feature information of the lesions to be processed in the image processing backbone network.

[0084] Multiple target images can form a 3D detection area for the target detection area. The multiple target images are input into the image processing backbone network for processing, and the feature information of the object to be processed and the feature information of the lesion to be processed generated by the image processing backbone network in the process of processing the multiple target images can be extracted.

[0085] It should be noted that the feature information of the object to be processed and the feature information of the lesion to be processed are both intermediate parameter features generated by the image processing model in the process of processing multiple target images. According to their respective uses, they are divided into the feature information of the object to be processed and the feature information of the lesion to be processed. If a parameter feature is used to generate both the object feature information and the lesion feature information, then the parameter feature belongs to both the feature information of the object to be processed and the feature information of the lesion to be processed.

[0086] Specifically, the image processing backbone network includes an image encoder and an image decoder;

[0087] Inputting the plurality of target images into the image processing backbone network, and extracting feature information of the objects to be processed and feature information of the lesions to be processed in the image processing backbone network, comprising:

[0088] Inputting the plurality of target images into the image encoder to obtain target image coding features and at least one multi-scale image coding feature;

[0089] Inputting the target image coding feature into the image decoder to obtain at least one first multi-scale image decoding feature, at least one second multi-scale image decoding feature and a target image decoding feature;

[0090] The at least one multiscale image encoding feature and the at least one first multiscale image decoding feature are determined as feature information of the object to be processed, and the at least one second multiscale image decoding feature and the target image decoding feature are determined as feature information of the lesion to be processed.

[0091] In the method provided in the embodiments of this specification, the nnU-Net framework is used to build an image processing model. The image processing backbone network includes an image encoder and an image decoder. In this embodiment, the Pixel Encoder of U-Net is used as the image encoder, and the feature pyramid network is used as the image decoder.

[0092] In the process of inputting multiple target images into the image processing model, the multiple target images are first encoded by the image encoder to obtain target image encoding features and at least one multi-scale image encoding feature. The target image encoding feature is the encoding feature output by the image encoder, and the at least one multi-scale image encoding feature is the encoding parameter feature generated by the image encoder in the process of processing the multiple target images.

[0093] The target image coding feature is input into the image decoder to obtain at least one first multi-scale image decoding feature, at least one second multi-scale image decoding feature and a final target image decoding feature generated by the image decoder during the decoding process.

[0094] At this time, the multiscale image encoding features and the first multiscale image decoding features are used as feature information of the object to be processed, and the second multiscale image decoding features and the target image decoding features are used as feature information of the lesion to be processed.

[0095] Wherein, the image encoder comprises a plurality of sequentially connected image encoding layers;

[0096] Inputting the plurality of target images into the image encoder to obtain target image coding features and at least one multi-scale image coding feature comprises:

[0097] Inputting the plurality of target images into the image encoder to obtain encoding features of the target images output by the image encoder;

[0098] At least one target image coding layer is determined among the multiple image coding layers, and multi-scale image coding features output by each target image coding layer are obtained.

[0099] It should be noted that the target image coding layer does not specifically refer to a certain image coding layer, but refers to the image coding layer from which image coding features need to be extracted. For example, if there are 6 image coding layers in total, and the multi-scale image coding features output by the 1st, 3rd, and 5th image coding layers are needed, then the 1st, 3rd, and 5th image coding layers are the target image coding layers. For another example, if there are 12 image coding layers in total, and the multi-scale image coding features output by the 9th, 10th, 11th, and 12th image coding layers are needed, then the 9th, 10th, 11th, and 12th image coding layers are the target image coding layers.

[0100] See also Figure 2 ,by Figure 2 The image processing backbone network shown in FIG. 1 is used as an example to explain the image encoder, which includes 6 image encoding layers. Multiple target images are input to the image encoder for processing, and the 5th and 6th image encoding layers encode the target images. Then, the target image encoding features output by the image encoder and the multi-scale image encoding features output by the 5th and 6th image encoding layers are obtained. and

[0101] In another specific implementation provided by the embodiments of this specification, the image decoder includes a plurality of sequentially connected image decoding layers;

[0102] Inputting the target image coding feature into the image decoder to obtain at least one first multi-scale image decoding feature and at least one second multi-scale image decoding feature, comprising:

[0103] determining at least one first image decoding layer and at least one second image decoding layer among the plurality of image decoding layers;

[0104] A first multi-scale image decoding feature output by the first image decoding layer is obtained, and a second multi-scale image decoding feature output by the second image decoding layer is obtained.

[0105] Similar to the target image coding layer mentioned above, the first image decoding layer and the second image decoding layer in the image decoding layer do not specifically refer to a certain image decoding layer, but refer to image decoding layers used to determine the first multi-scale image decoding features and the second multi-scale image decoding features.

[0106] See also Figure 2 ,by Figure 2 The image processing backbone network shown in FIG. 1 is used as an example to explain the image processing backbone network. The image decoder includes 6 image decoding layers. The target image coding features are input into the image decoder. It is determined that the output features of the 5th and 6th image decoding layers are the first multi-scale image decoding features, and the output features of the 2nd, 3rd, and 4th image decoding layers are the second multi-scale image decoding features. Then, the first multi-scale image decoding features output by the 5th and 6th image decoding layers are obtained. and And the second multi-scale image decoding features output by the 2nd, 3rd, and 4th image decoding layers and And the target image decoding features output by the final image decoder

[0107] by Figure 2 As shown in the figure, multi-scale image encoding features and The first multi-scale image decoding feature and Composition of feature information of the object to be processed, second multi-scale image decoding feature and And target image decoding features Form the characteristic information of the lesion to be processed.

[0108] S1044, inputting the feature information of the object to be processed into the object branch network to obtain the object feature information, inputting the feature information of the lesion to be processed into the lesion branch network to obtain the lesion feature information and the feature information of the image to be segmented.

[0109] The object branch network is used to generate object classification information for the object to be processed. After obtaining the feature information of the object to be processed in the above steps, the feature information of the object to be processed is input into the object branch network, processed in the object branch network, and the object feature information is obtained.

[0110] See also Figure 3 , Figure 3 FIG. 2 shows a data processing diagram of an object branch network provided by an embodiment of the present specification. Figure 3 As shown in the figure, the feature information of the object to be processed is iteratively input into the 4-layer Dual-path Transformer Block (DPB) for processing to obtain the object feature information. Dual-path Transformer Block is a special Transformer structure that optimizes performance and reduces computational complexity by decomposing the processing process into two parallel paths (usually called local path and global path), thereby improving processing efficiency and processing effect. These two paths usually focus on different types of features, the local path focuses more on fine-grained local features, and the global path focuses more on scene-level global features.

[0111] The lesion branch network is used to generate segmentation information and classification information for lesions. After obtaining the processed lesion feature information in the above steps, the processed lesion feature information is input into the lesion branch network, processed in the lesion branch network, and the lesion feature information and the feature information of the image to be segmented are obtained.

[0112] Specifically, the lesion feature information to be processed includes at least one second multi-scale image decoding feature and a target image decoding feature;

[0113] Inputting the feature information of the lesion to be processed into the lesion branch network to obtain the feature information of the lesion and the feature information of the image to be segmented, including:

[0114] Performing feature decoding on each second multi-scale image decoding feature to obtain lesion feature information;

[0115] The lesion feature information and the target image decoding feature are fused to generate feature information of the image to be segmented.

[0116] In the lesion branch network, lesion feature information and image feature information to be segmented are generated simultaneously. The lesion feature information is used to determine lesion classification information, and the image feature information to be segmented is used to determine lesion segmentation information.

[0117] See also Figure 4 , Figure 4 FIG. 2 shows a schematic diagram of data processing of a lesion branch network provided in an embodiment of the present specification. Figure 4 As shown in FIG. 1 , in the lesion branch network, a Transformer Decoder structure is included, 50 randomly initialized queries are input, and decoding is performed in the Transformer Decoder and the second multi-scale image decoding features. Figure 4 As shown, the second multi-scale image decoding feature and The 50 queries are input into the Transformer Decoder for processing, and the lesion feature information output by the Transformer Decoder is obtained. At the same time, in order to better segment the abnormal objects at the pixel level, the lesion feature information and the target image decoding feature output by the image decoder are combined. The image is fused to obtain feature information of the image to be segmented, which is used to perform pixel-level masking on abnormal objects in the detection area.

[0118] At this point, the object feature information generated in the object branch network is obtained, and the lesion feature information and the image feature information to be segmented generated in the lesion branch network are also obtained.

[0119] S1046: Input the object feature information and the lesion feature information into the fusion network to generate object fusion feature information.

[0120] The fusion network can be understood as a network used to fuse object feature information and lesion feature information together to generate object fusion feature information. Object fusion feature information enables the image processing model to fuse the global information of the object and the local information of the lesion, which can improve the detection ability of smaller abnormal objects.

[0121] In a specific implementation manner provided in the embodiments of this specification, the object feature information and the lesion feature information are input into the fusion network to generate object fusion feature information, including:

[0122] The object feature information and the lesion feature information are input into the fusion network, lesion sub-feature information in the lesion feature information is extracted in the fusion network, and the object feature information and the lesion sub-feature information are spliced ​​to generate object fusion feature information.

[0123] In practical applications, the lesion feature information includes lesion sub-feature information and organ sub-feature information. In order to make the lesion feature available for reference in the object feature, the lesion sub-feature information can be fused with the object feature information to generate object fusion feature information.

[0124] See also Figure 5 , Figure 5 FIG. 2 shows a schematic diagram of data processing of a fusion network provided in an embodiment of the present specification. Figure 5 As shown, lesion sub-feature information used to determine the lesion type is extracted from the lesion feature information, and the lesion sub-feature information is spliced ​​with the object feature information to obtain the object fusion feature information.

[0125] The object fusion feature information is the feature information used to finally determine the object classification information, which includes both the object feature information and the lesion sub-feature information, so that the final prediction result is more accurate.

[0126] S1048. Generate lesion classification information according to the lesion feature information, generate lesion segmentation information according to the feature information of the image to be segmented, generate object classification information according to the object fusion feature information, and generate a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0127] After the above steps, the lesion classification information, the feature information of the image to be segmented and the object fusion feature information can be obtained. Then, the corresponding lesion classification information, lesion segmentation information and object classification information can be generated according to these three information respectively. Finally, the final detection result is formed according to the lesion classification information, lesion segmentation information and object classification information.

[0128] In practical applications, three different classifiers are used to process different information. Specifically, in a specific implementation provided in this specification, lesion classification information is generated according to the lesion feature information, and lesion segmentation information is generated according to the feature information of the image to be segmented, including:

[0129] Inputting the lesion feature information into a lesion classifier to generate lesion classification information;

[0130] The feature information of the image to be segmented is input into a segmenter to generate lesion segmentation information.

[0131] Generating object classification information according to the object fusion feature information includes:

[0132] The object fusion feature information is input into an object classifier to generate object classification information.

[0133] In practical applications, the lesion feature information is input into the lesion classifier for processing, and the lesion classification information output by the lesion classifier can be obtained; the feature information of the image to be segmented is input into the segmentor for processing, and the lesion segmentation information output by the segmentor can be obtained; the object fusion feature information is input into the object classifier for processing, and the object classification information output by the object classifier can be obtained.

[0134] Through the image processing method provided in the embodiments of this specification, multiple target images are input into the image processing model. In the image processing model, object feature information and lesion feature information are generated based on each target image. At the same time, the object feature information and the lesion feature information are fused to generate object fusion feature information. When the model predicts the final object classification information, in addition to using the object feature information, it also refers to the lesion feature information, fuses the global information and the local information, and improves the detection capability of small lesions.

[0135] The image processing model provided in the embodiments of this specification is a pre-trained image processing model. Figure 6 , Figure 6 A flowchart of an image processing model training method provided in an embodiment of this specification is shown. Figure 6 As shown, the image processing model is trained and generated through the following steps:

[0136] Step 602: Acquire multiple sample target images and sample detection results corresponding to the multiple sample target images, wherein the sample detection results include sample object classification information, sample lesion classification information and sample lesion segmentation information.

[0137] Specifically, the training method of the image processing model provided in this specification uses supervised training, which includes training sample pairs, and the training sample pairs specifically include multiple sample target images for the target detection area, and sample detection results corresponding to the multiple sample target images. Among them, the multiple sample target images can form a three-dimensional image for the target detection area. The sample detection results specifically include sample object classification information, sample lesion classification information and sample lesion segmentation information.

[0138] The sample object classification information includes whether there are abnormal objects in the target detection area of ​​the detection object, the type of abnormal objects, and the probability of abnormal objects corresponding to each category. The sample lesion classification information can be understood as the lesion classification for the lesion corresponding to the target detection area, and the organ classification of the organ where the lesion is located. The sample lesion segmentation information can be understood as the mask information for abnormal objects in the target detection area.

[0139] In the method provided in the embodiments of the present specification, supervised training is performed on the image processing model through multiple sample target images and sample detection results corresponding to the multiple sample target images.

[0140] In a specific implementation provided in this specification, obtaining a plurality of sample target images and sample detection results corresponding to the plurality of sample target images includes:

[0141] Determine a sample detection object, and obtain a sample enhanced image and a sample target image corresponding to the sample detection object, wherein the sample enhanced image and the sample target image corresponding to the sample detection object correspond to the same target detection area of ​​the sample detection object;

[0142] Generating a reference sample detection result for a target detection area on the sample enhanced image;

[0143] The sample detection result is marked on the sample target image based on the reference sample detection result.

[0144] The sample enhanced image specifically refers to an image with enhanced contrast for the target detection area, and the sample target image specifically refers to a normal image for the target detection area. The sample enhanced image and the sample target image correspond to the same target detection area of ​​the same sample detection object.

[0145] For example, taking the sample detection object as Zhang San and the target detection area as the liver as an example, after the sample detection object is determined, the corresponding sample enhanced image and sample target image are acquired based on the target detection area of ​​the sample detection object.

[0146] In practical applications, the sample enhanced image is an image with enhanced contrast for the target detection area, and the detection result for the target detection area can be better obtained in the sample enhanced image. In the embodiment of this specification, the detection result for the target detection area of ​​the sample enhanced image becomes the reference sample detection result. Then, the reference sample detection result is matched to the sample target image by registration to obtain the sample detection result corresponding to the sample target image.

[0147] The process of generating reference sample detection results for the target detection area on the sample enhanced image is usually in the form of manual annotation, and manual annotation requires experienced technicians to perform manual annotation, which is time-consuming, labor-intensive and costly. In an optional implementation provided in the embodiments of this specification, a technician may annotate part of the sample enhanced image. A training sample pair of sample enhanced images and reference sample detection results is formed, and then a sample enhanced image annotation model is trained by the training sample pair. After the sample enhanced image annotation model is trained, the sample enhanced image that has not been manually annotated is input into the sample enhanced image annotation model for annotation, thereby generating a reference sample detection result of the sample enhanced image.

[0148] For example, in auxiliary medical scenarios, with the permission of the user, paired enhanced CT images and plain scan CT images with image quality that meets the requirements are obtained from the imaging department. Paired enhanced CT images and plain scan CT images are CT images of the same sample test object taken at the same time point and at the same angle. Enhanced CT images include arterial phase enhanced CT, venous phase enhanced CT, delayed phase enhanced CT, etc.

[0149] Experienced radiologists and experts were invited to mark the mask information of abnormal objects in the target detection area on some enhanced CT images, and refer to the pathological gold standard of the sample detection object to provide the lesion classification information and object classification information of the abnormal objects. The mask information, lesion classification information and object classification information constitute the reference sample detection results.

[0150] An enhanced CT model is trained based on the annotated enhanced CT images and the reference sample detection results, so that the enhanced CT model has the ability to predict and annotate the detection results based on the enhanced CT images. The unlabeled enhanced CT images are input into the enhanced CT model for processing, and the unlabeled enhanced CT images are processed by the enhanced CT model to generate the reference sample detection results corresponding to the unlabeled enhanced CT images.

[0151] In the aforementioned steps, since the enhanced CT image and the plain scan CT image are registered, the reference sample detection result on the enhanced CT image can be translated into the plain scan CT image as the sample detection result of the plain scan CT image.

[0152] After the above processing, the sample target image and the sample detection results corresponding to the sample image can be obtained. It should be noted that, among the sample target image and the sample detection results corresponding to the sample image, the first part is obtained by aligning the manually annotated sample enhanced image with the sample reference detection results, and the second part is obtained by aligning the reference detection results annotated by the model. In the subsequent processing, when the sample target image and the sample detection results corresponding to the sample image are divided into a training set and a test set, in order to ensure the accuracy of the training data, a test set is selected from the first part of the manually annotated data for subsequent testing of the model performance.

[0153] Step 604: Input the multiple sample target images into the image processing model to obtain the predicted detection results and training contrast loss values ​​output by the image processing model.

[0154] After obtaining the sample target image and the sample detection result corresponding to the sample target image, the image processing model can be trained. At this time, the image processing model is an image processing model that has not been trained yet, and it needs to be trained by the sample target image and the sample detection result corresponding to the sample target image.

[0155] During the training process, the image processing model will output the predicted detection results and training comparison loss values ​​generated based on the sample target image. In the specific implementation provided in this specification, the concept of contrast loss value is introduced, that is, a two-stage category-balanced contrast loss function is constructed, and the comparison loss function is processed in the object branch network and the lesion branch network of the image processing model respectively, and the training comparison loss value is obtained during the training process.

[0156] In a specific implementation provided in the examples of this specification, the image processing model includes an image processing backbone network, an object branch network, a lesion branch network and a fusion network;

[0157] Inputting the plurality of sample target images into an image processing model to obtain a predicted detection result and a training contrast loss value output by the image processing model includes:

[0158] Inputting the plurality of sample target images into an image processing backbone network, and extracting feature information of sample objects to be processed and feature information of sample lesions to be processed in the image processing backbone network;

[0159] Inputting the feature information of the sample to be processed object into the object branch network to obtain predicted object feature information, inputting the feature information of the sample to be processed lesion into the lesion branch network to obtain predicted lesion feature information and predicted feature information of the image to be segmented;

[0160] In the object branch network, an object comparison loss value is obtained according to the sample object classification information corresponding to the sample target image, and in the lesion branch network, a lesion comparison loss value is obtained according to the sample lesion classification information corresponding to the sample target image;

[0161] Inputting the sample object feature information and the predicted lesion feature information into the fusion network to generate sample object fusion feature information;

[0162] Generate first object classification information according to the sample object feature information, generate second object classification information according to the sample object fusion feature information, generate predicted lesion classification information according to the predicted lesion feature information, and generate predicted lesion segmentation information according to the predicted image feature information to be segmented;

[0163] A predicted detection result is generated according to the first object classification information, the second object classification information, the predicted lesion classification information and the predicted lesion segmentation information, and a training comparison loss value is generated according to the object comparison loss value and the lesion comparison loss value.

[0164] During the model training phase, the model structure of the image processing model is the same as the model structure of the above-mentioned model application phase. For the relevant content about the model structure and data processing flow of the image processing model, please refer to the relevant part of the above-mentioned model application phase, which will not be repeated here.

[0165] In the model training stage, the image processing model first processes multiple sample target images in the image processing backbone network. Extract the sample object feature information and sample lesion feature information generated in the image processing backbone network. Input the sample object feature information into the object branch network to obtain the predicted object feature information; input the sample lesion feature information into the lesion branch network to obtain the predicted lesion feature information and the predicted image feature information to be segmented. The data processing process of the image processing backbone network, object branch network and lesion branch network in the model training stage is the same as that in the above application stage, and will not be repeated here.

[0166] It should be noted that in the model training stage, a two-stage category-balanced contrast loss function is introduced. In the object branch network, on the basis of obtaining the feature information of the sample to be processed, whether the object to be detected has abnormal objects is introduced to obtain two types of contrast learning loss functions. In the lesion branch network, on the basis of obtaining the feature information of the sample lesion to be processed, a comparison learning loss function of several abnormal objects is introduced. Through the comparison learning method, the image processing model learns the differences between the feature information of the sample to be processed objects between each training batch, and the differences between the feature information of the sample lesions to be processed between each training batch. Through the comparison loss function, the object comparison loss value and the lesion comparison loss value can be obtained.

[0167] In another specific implementation provided by the embodiment of this specification, during the training process of the image processing model provided by the embodiment of this specification, if the batch size is too small, it will easily cause the problem of model collapse. On this basis, the training method of the embodiment of this specification uses a method of referring to historical sample object feature information. Specifically, the feature information of the sample object to be processed is input into the object branch network to obtain the predicted object feature information, including:

[0168] Obtain historical sample object feature information;

[0169] Inputting the historical object feature information and the sample to-be-processed object feature information into the object branch network to obtain predicted object feature information;

[0170] Accordingly, the feature information of the sample lesion to be processed is input into the lesion branch network to obtain the predicted lesion feature information and the predicted feature information of the image to be segmented, including:

[0171] Obtain historical sample lesion feature information;

[0172] The historical sample lesion feature information and the sample to-be-processed lesion feature information are input into the lesion branch network to obtain predicted lesion feature information and predicted image feature information to be segmented.

[0173] In this embodiment, the strategy of memory bank is introduced, and object feature storage blocks and lesion feature storage blocks are added to the object branch network and the lesion branch network respectively. Among them, the object feature storage block is used to store historical sample object feature information, and the lesion feature storage block is used to store historical sample lesion feature information. The capacity of the two feature storage blocks is the same. Due to the different data types in the classification problem considered at the same time, there will be a long tail problem of certain information classification, that is, the problem of small amount of data of certain classification information. In order to solve the long tail problem of classification, the number of features of each category stored in the two feature storage blocks is balanced. In the process of updating each feature storage block, the number of features stored in each category must also be considered to ensure that the number of features of each category remains balanced.

[0174] In addition to introducing the contrast loss function, the training method provided in the embodiments of this specification will also generate first object classification information based on the sample object feature information, and generate second object classification information based on the sample object fusion feature information. That is, during the model training process, two object classification information will be generated, one of which is the first object classification information generated based on the sample object feature information, and the other is the second object classification information generated based on the sample object feature information and the predicted lesion feature information. Both of them will be used as the prediction results of the model in the model training stage and used for the subsequent calculation of the model loss value.

[0175] At the same time, similar to the above-mentioned application stage, in the model training stage, predicted lesion classification information is generated based on the predicted lesion feature information, and predicted lesion segmentation information is generated based on the predicted feature information of the image to be segmented.

[0176] Finally, the first object classification information, the second object classification information, the predicted lesion classification information and the predicted lesion segmentation information are determined to generate a predicted detection result. The object comparison loss value and the lesion comparison loss value are determined as the training comparison loss value.

[0177] Step 606: Calculate a predicted loss value according to the predicted detection result and the sample detection result.

[0178] After obtaining the predicted test results, the predicted test results are compared with the sample test results to calculate the predicted loss value. At this time, the image processing model is not a trained model. By calculating the predicted loss value, the difference between the predicted results and the sample results is determined, so as to further adjust the parameters of the image processing model and realize the training of the image processing model.

[0179] Specifically, in a specific implementation provided in this specification, calculating the predicted loss value according to the predicted detection result and the sample detection result includes:

[0180] Calculate a first object loss value according to the first object classification information and the sample object classification information;

[0181] Calculating a second object loss value according to the second object classification information and the sample object classification information;

[0182] Calculating a lesion classification loss value according to the predicted lesion classification information and the sample lesion classification information;

[0183] A lesion segmentation loss value is calculated according to the predicted lesion segmentation information and the sample lesion segmentation information.

[0184] In practical applications, the predicted detection results include the first object classification information and the second object classification information, both of which are calculated with the sample object classification information to calculate the loss value. Specifically, the first object loss value is calculated by the first object classification information and the sample object classification information, and the second object loss value is calculated by the second object classification information and the sample object classification information. Similarly, the lesion classification loss value is calculated based on the predicted lesion classification information and the sample lesion classification information, and the lesion segmentation loss value is calculated based on the predicted lesion segmentation information and the predicted lesion classification information. There are many ways to calculate the model loss value, such as the cross entropy loss function, the maximum loss function, the average loss function, etc. In this specification, the specific method of the loss function is not limited, and the actual application shall prevail.

[0185] Step 608: Adjust the model parameters of the image processing model according to the prediction loss value and the training comparison loss value, and continue to train the image processing model until the model training stop condition is reached.

[0186] After obtaining the prediction loss value and the training comparison loss value, the model parameters of the image processing model can be adjusted by backpropagation based on these two types of loss values.

[0187] Specifically, in a specific implementation provided in the embodiments of this specification, adjusting the model parameters of the image processing model according to the prediction loss value and the training contrast loss value includes:

[0188] Calculating a model loss value according to the first object loss value, the second object loss value, the lesion classification loss value, the lesion segmentation loss value, the object comparison loss value, and the lesion comparison loss value;

[0189] The model parameters of the image processing model are adjusted according to the model loss value.

[0190] In practical applications, the prediction loss value and the training comparison loss value are fused to obtain a new loss value. Furthermore, according to the preset loss weight, the first object loss value, the second object loss value, the lesion classification loss value, the lesion segmentation loss value, the object comparison loss value and the lesion comparison loss value are fused to obtain a new model loss value. The model parameters of the image processing model are further adjusted according to the model loss value until the model training stop condition is reached to obtain a trained image processing model.

[0191] The training method of the image processing model provided in the embodiment of this specification proposes to use the sample enhanced image to train a sample enhanced image annotation model through partial manual annotation, so that the sample enhanced image annotation model can annotate the sample enhanced image that has not been manually annotated, saving annotation time. In addition, by aligning the sample enhanced image and the sample target image, the reference sample detection result on the sample enhanced image is transferred to the sample target image to obtain the sample detection result of the sample target image, which solves the problem of difficult and missing annotation on the sample target image and reduces the annotation cost.

[0192] In addition, during the model training process, a two-stage category-balanced contrastive learning loss function was introduced to enhance the differential diagnosis ability of the image processing model. The classification accuracy can be improved in both the object branch network and the lesion branch network, especially for some long-tail categories with less feature data, and it has better classification and recognition accuracy.

[0193] The following combination Figure 7, taking the application of the image processing method provided in this specification in the processing of CT images as an example, the image processing method is further explained. Figure 7 A processing flow chart of a CT image processing method provided in an embodiment of the present specification is shown, which specifically includes the following steps.

[0194] Step 702: Receive a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether there is an abnormal object in the target detection area.

[0195] Step 704: Input the multiple plain scan CT images into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0196] It should be noted that the implementation method of step 702 to step 704 is the same as the implementation method of the above-mentioned step 102 to step 104, and will not be repeated in the embodiments of this specification.

[0197] Specifically, in the method provided in this embodiment, the CT image processing task of detecting whether there is an abnormal object in the target detection area is taken as an example for further explanation. Based on this, the CT image processing task includes a plain scan CT image of the target detection area, and the plain scan CT image is input into the image processing model for identification. The image processing model can identify whether there is an abnormal object in the plain scan CT image, and the Mask information of the abnormal object if there is an abnormal object. At the same time, it can also predict the object classification information of the object to be detected, that is, whether there is an abnormal object in the target detection area of ​​the object to be detected, the type of abnormal object and other information. Through this method, the problem that the current application cannot detect the target detection area based on the plain scan CT image, the detection effect is poor, and the accuracy is low.

[0198] See also Figure 8 , Figure 8 A flowchart of a computer-aided diagnosis method for cancer provided in an embodiment of this specification is shown, which specifically includes:

[0199] Step 802: Receive a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether a tumor exists in the target detection area.

[0200] Step 804: Input the multiple plain scan CT images into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0201] The computer-aided diagnosis method for cancer provided in this embodiment can be applied to screening whether there are tumors in each target detection area, including but not limited to the pancreas, esophagus, liver, lungs, breast, intestines, stomach, systemic lymph nodes and other organs of the whole body. The test results include user classification information of the user to be tested (i.e., whether there is a tumor, whether the tumor is a malignant tumor, etc.), lesion classification information (what type of tumor the tumor is) and lesion segmentation information (mask information marking the tumor on the CT image). The method provided in the embodiments of this specification can provide guiding suggestions for doctors, assist in improving the accuracy of doctors' diagnosis, and provide data support for doctors to give diagnostic results.

[0202] See also Fig. 9 , Fig. 9 A flowchart of a computer-aided diagnosis method for liver cancer provided in an embodiment of this specification is shown, which specifically includes:

[0203] Step 902: Receive a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a liver region, and the CT image processing task is used to detect whether a tumor exists in the liver region.

[0204] Step 904: Input the multiple plain scan CT images into an image processing model to obtain a detection result corresponding to the liver region, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0205] The computer-aided diagnosis method for liver cancer provided in this embodiment can be used to screen for diseases such as hepatocellular carcinoma, cholangiocarcinoma, metastases, hemangiomas, focal nodular hyperplasia, cysts, calcifications, etc., to provide guidance and suggestions for doctors, to help improve the accuracy of doctors' diagnosis, and to provide data support for doctors to give diagnostic results.

[0206] See also Fig.10 , Fig.10 The architecture diagram of a computer-aided diagnosis system for cancer provided in one embodiment of the present specification is shown. The computer-aided diagnosis system for cancer may include a client 100 and a server 200;

[0207] The client 100 is used to send a CT image processing task to the server 200, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether a tumor exists in the target detection area;

[0208] The server 200 is used to input the multiple plain scan CT images into the image processing model to obtain the detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates the detection result according to the object classification information, the lesion classification information and the lesion segmentation information; and sends the detection result corresponding to the target detection area to the client 100;

[0209] The client 100 is also used to receive the detection result sent by the server 200.

[0210] The computer-aided diagnosis system for cancer may include multiple clients 100 and a server 200, wherein the client 100 may be referred to as a terminal-side device and the server 200 may be referred to as a cloud-side device. A communication connection may be established between multiple clients 100 through the server 200. In the computer-aided diagnosis scenario for cancer, the server 200 is used to provide computer-aided diagnosis services for cancer between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver, respectively, and achieve communication through the server 200.

[0211] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the computer-aided diagnosis scenario of cancer, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a detection result based on the data stream and pushes the detection result to other clients that have established communication.

[0212] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0213] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device, and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0214] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that provide support for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0215] It is worth noting that the computer-aided diagnosis method for cancer provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the computer-aided diagnosis method for cancer provided in the embodiments of this specification. In other embodiments, the computer-aided diagnosis method for cancer provided in the embodiments of this specification may also be jointly executed by the client and the server.

[0216] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Fig.11 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0217] The receiving module 1102 is configured to receive an image processing task, wherein the image processing task carries a plurality of target images corresponding to a target detection area, and the image processing task is used to detect whether there is an abnormal object in the target detection area.

[0218] The detection module 1104 is configured to input the multiple target images into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each target image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

[0219] Optionally, the image processing model includes an image processing backbone network, an object branch network, a lesion branch network and a fusion network;

[0220] The detection module 1104 is further configured to:

[0221] Inputting the plurality of target images into the image processing backbone network, and extracting feature information of the objects to be processed and feature information of the lesions to be processed in the image processing backbone network;

[0222] Inputting the feature information of the object to be processed into the object branch network to obtain the object feature information, inputting the feature information of the lesion to be processed into the lesion branch network to obtain the lesion feature information and the feature information of the image to be segmented;

[0223] Inputting the object feature information and the lesion feature information into the fusion network to generate object fusion feature information;

[0224] Lesion classification information is generated according to the lesion feature information, lesion segmentation information is generated according to the feature information of the image to be segmented, object classification information is generated according to the object fusion feature information, and a detection result is generated according to the object classification information, the lesion classification information and the lesion segmentation information.

[0225] Optionally, the image processing backbone network includes an image encoder and an image decoder;

[0226] The detection module 1104 is further configured to:

[0227] Inputting the plurality of target images into the image encoder to obtain target image coding features and at least one multi-scale image coding feature;

[0228] Inputting the target image coding feature into the image decoder to obtain at least one first multi-scale image decoding feature, at least one second multi-scale image decoding feature and a target image decoding feature;

[0229] The at least one multiscale image encoding feature and the at least one first multiscale image decoding feature are determined as feature information of the object to be processed, and the at least one second multiscale image decoding feature and the target image decoding feature are determined as feature information of the lesion to be processed.

[0230] Optionally, the image encoder comprises a plurality of sequentially connected image coding layers;

[0231] The detection module 1104 is further configured to:

[0232] Inputting the plurality of target images into the image encoder to obtain encoding features of the target images output by the image encoder;

[0233] At least one target image coding layer is determined among the multiple image coding layers, and multi-scale image coding features output by each target image coding layer are obtained.

[0234] Optionally, the image decoder comprises a plurality of sequentially connected image decoding layers;

[0235] The detection module 1104 is further configured to:

[0236] determining at least one first image decoding layer and at least one second image decoding layer among the plurality of image decoding layers;

[0237] A first multi-scale image decoding feature output by the first image decoding layer is obtained, and a second multi-scale image decoding feature output by the second image decoding layer is obtained.

[0238] Optionally, the lesion feature information to be processed includes at least one second multi-scale image decoding feature and a target image decoding feature;

[0239] The detection module 1104 is further configured to:

[0240] Performing feature decoding on each second multi-scale image decoding feature to obtain lesion feature information;

[0241] The lesion feature information and the target image decoding feature are fused to generate feature information of the image to be segmented.

[0242] Optionally, the detection module 1104 is further configured to:

[0243] Inputting the lesion feature information into a lesion classifier to generate lesion classification information;

[0244] The feature information of the image to be segmented is input into a segmenter to generate lesion segmentation information.

[0245] Optionally, the detection module 1104 is further configured to:

[0246] The object feature information and the lesion feature information are input into the fusion network, lesion sub-feature information in the lesion feature information is extracted in the fusion network, and the object feature information and the lesion sub-feature information are spliced ​​to generate object fusion feature information.

[0247] Optionally, the detection module 1104 is further configured to:

[0248] The object fusion feature information is input into an object classifier to generate object classification information.

[0249] Optionally, the device further comprises a training module configured to:

[0250] Acquire multiple sample target images and sample detection results corresponding to the multiple sample target images, wherein the sample detection results include sample object classification information, sample lesion classification information and sample lesion segmentation information;

[0251] Inputting the plurality of sample target images into an image processing model to obtain a predicted detection result and a training contrast loss value output by the image processing model;

[0252] Calculate the predicted loss value according to the predicted detection result and the sample detection result;

[0253] The model parameters of the image processing model are adjusted according to the predicted loss value and the training contrast loss value, and the image processing model is continuously trained until a model training stop condition is reached.

[0254] Optionally, the training module is further configured to:

[0255] Determine a sample detection object, and obtain a sample enhanced image and a sample target image corresponding to the sample detection object, wherein the sample enhanced image and the sample target image corresponding to the sample detection object correspond to the same target detection area of ​​the sample detection object;

[0256] Generating a reference sample detection result for a target detection area on the sample enhanced image;

[0257] The sample detection result is marked on the sample target image based on the reference sample detection result.

[0258] Optionally, the image processing model includes an image processing backbone network, an object branch network, a lesion branch network and a fusion network;

[0259] The training module is further configured as follows:

[0260] Inputting the plurality of sample target images into an image processing model to obtain a predicted detection result and a training contrast loss value output by the image processing model includes:

[0261] Inputting the plurality of sample target images into an image processing backbone network, and extracting feature information of sample objects to be processed and feature information of sample lesions to be processed in the image processing backbone network;

[0262] Inputting the feature information of the sample to be processed object into the object branch network to obtain predicted object feature information, inputting the feature information of the sample to be processed lesion into the lesion branch network to obtain predicted lesion feature information and predicted feature information of the image to be segmented;

[0263] In the object branch network, an object comparison loss value is obtained according to the sample object classification information corresponding to the sample target image, and in the lesion branch network, a lesion comparison loss value is obtained according to the sample lesion classification information corresponding to the sample target image;

[0264] Inputting the sample object feature information and the predicted lesion feature information into the fusion network to generate sample object fusion feature information;

[0265] Generate first object classification information according to the sample object feature information, generate second object classification information according to the sample object fusion feature information, generate predicted lesion classification information according to the predicted lesion feature information, and generate predicted lesion segmentation information according to the predicted image feature information to be segmented;

[0266] A predicted detection result is generated according to the first object classification information, the second object classification information, the predicted lesion classification information and the predicted lesion segmentation information, and a training comparison loss value is generated according to the object comparison loss value and the lesion comparison loss value.

[0267] Optionally, the training module is further configured to:

[0268] Calculating a first object loss value according to the first object classification information and the sample object classification information;

[0269] Calculating a second object loss value according to the second object classification information and the sample object classification information;

[0270] Calculating a lesion classification loss value according to the predicted lesion classification information and the sample lesion classification information;

[0271] A lesion segmentation loss value is calculated according to the predicted lesion segmentation information and the sample lesion segmentation information.

[0272] Optionally, the training module is further configured to:

[0273] Obtain historical sample object feature information;

[0274] Inputting the historical object feature information and the sample to-be-processed object feature information into the object branch network to obtain predicted object feature information;

[0275] Obtain historical sample lesion feature information;

[0276] The historical sample lesion feature information and the sample to-be-processed lesion feature information are input into the lesion branch network to obtain predicted lesion feature information and predicted image feature information to be segmented.

[0277] Optionally, the training module is further configured to:

[0278] Calculating a model loss value according to the first object loss value, the second object loss value, the lesion classification loss value, the lesion segmentation loss value, the object comparison loss value, and the lesion comparison loss value;

[0279] The model parameters of the image processing model are adjusted according to the model loss value.

[0280] Through the image processing device provided in the embodiments of this specification, multiple target images are input into the image processing model. In the image processing model, object feature information and lesion feature information are generated based on each target image. At the same time, the object feature information and the lesion feature information are fused to generate object fusion feature information. When the model predicts the final object classification information, in addition to using the object feature information, it also refers to the lesion feature information, fuses the global information and the local information, and improves the detection capability of small lesions.

[0281] In addition, the training module provided in the embodiment of this specification proposes to use the sample enhanced image to train a sample enhanced image annotation model through partial manual annotation, so that the sample enhanced image annotation model can annotate the sample enhanced image that has not been manually annotated, saving annotation time. In addition, by aligning the sample enhanced image and the sample target image, the reference sample detection result on the sample enhanced image is transferred to the sample target image to obtain the sample detection result of the sample target image, which solves the problem of difficult and missing annotation on the sample target image and reduces the annotation cost.

[0282] At the same time, during the model training process, a two-stage category-balanced contrastive learning loss function was introduced to enhance the differential diagnosis ability of the image processing model, and to improve the classification accuracy in both the object branch network and the lesion branch network, especially for some long-tail categories with less feature data, with better classification and recognition accuracy.

[0283] The above is a schematic scheme of an image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0284] Fig.12 The block diagram of a computing device 1200 according to one embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0285] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0286] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0287] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 may also be a mobile or stationary server.

[0288] Among them, the processor 1220 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer.

[0289] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer.

[0290] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer.

[0291] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer embodiments.

[0292] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing method, CT image processing method, computer-aided diagnosis method for cancer, or computer-aided diagnosis method for liver cancer.

[0293] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned image processing method, CT image processing method, computer-aided diagnosis method for cancer or computer-aided diagnosis method for liver cancer belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned image processing method, CT image processing method, computer-aided diagnosis method for cancer or computer-aided diagnosis method for liver cancer.

[0294] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0295] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0296] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0297] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0298] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: receiving an image processing task, wherein the image processing task carries a plurality of target images corresponding to a target detection area, and the image processing task is used to detect whether there is an abnormal object in the target detection area; The multiple target images are input into an image processing model to obtain detection results corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each target image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates detection results according to the object classification information, the lesion classification information and the lesion segmentation information.

2. The method according to claim 1, wherein the image processing model comprises an image processing backbone network, an object branch network, a lesion branch network and a fusion network; Inputting the multiple target images into an image processing model to obtain detection results corresponding to the target detection area includes: Inputting the plurality of target images into the image processing backbone network, and extracting feature information of the objects to be processed and feature information of the lesions to be processed in the image processing backbone network; Inputting the feature information of the object to be processed into the object branch network to obtain the object feature information, inputting the feature information of the lesion to be processed into the lesion branch network to obtain the lesion feature information and the feature information of the image to be segmented; Inputting the object feature information and the lesion feature information into the fusion network to generate object fusion feature information; Lesion classification information is generated according to the lesion feature information, lesion segmentation information is generated according to the feature information of the image to be segmented, object classification information is generated according to the object fusion feature information, and a detection result is generated according to the object classification information, the lesion classification information and the lesion segmentation information.

3. The method according to claim 2, wherein the image processing backbone network comprises an image encoder and an image decoder; Inputting the plurality of target images into the image processing backbone network, and extracting feature information of the objects to be processed and feature information of the lesions to be processed in the image processing backbone network, comprising: Inputting the plurality of target images into the image encoder to obtain target image coding features and at least one multi-scale image coding feature; Inputting the target image coding feature into the image decoder to obtain at least one first multi-scale image decoding feature, at least one second multi-scale image decoding feature and a target image decoding feature; The at least one multiscale image encoding feature and the at least one first multiscale image decoding feature are determined as feature information of the object to be processed, and the at least one second multiscale image decoding feature and the target image decoding feature are determined as feature information of the lesion to be processed.

4. The method of claim 3, wherein the image encoder comprises a plurality of sequentially connected image coding layers; Inputting the plurality of target images into the image encoder to obtain target image coding features and at least one multi-scale image coding feature comprises: Inputting the plurality of target images into the image encoder to obtain encoding features of the target images output by the image encoder; At least one target image coding layer is determined among the multiple image coding layers, and multi-scale image coding features output by each target image coding layer are obtained.

5. The method of claim 3, wherein the image decoder comprises a plurality of sequentially connected image decoding layers; Inputting the target image coding feature into the image decoder to obtain at least one first multi-scale image decoding feature and at least one second multi-scale image decoding feature, comprising: determining at least one first image decoding layer and at least one second image decoding layer among the plurality of image decoding layers; A first multi-scale image decoding feature output by the first image decoding layer is obtained, and a second multi-scale image decoding feature output by the second image decoding layer is obtained.

6. The method according to claim 2, wherein the feature information of the lesion to be processed comprises at least one second multi-scale image decoding feature and a target image decoding feature; Inputting the feature information of the lesion to be processed into the lesion branch network to obtain the feature information of the lesion and the feature information of the image to be segmented, including: Performing feature decoding on each second multi-scale image decoding feature to obtain lesion feature information; The lesion feature information and the target image decoding feature are fused to generate feature information of the image to be segmented.

7. The method according to claim 6, generating lesion classification information according to the lesion feature information, and generating lesion segmentation information according to the feature information of the image to be segmented, comprising: Inputting the lesion feature information into a lesion classifier to generate lesion classification information; The feature information of the image to be segmented is input into a segmenter to generate lesion segmentation information.

8. The method according to claim 2, inputting the object feature information and the lesion feature information into the fusion network to generate object fusion feature information, comprising: The object feature information and the lesion feature information are input into the fusion network, lesion sub-feature information in the lesion feature information is extracted in the fusion network, and the object feature information and the lesion sub-feature information are spliced ​​to generate object fusion feature information.

9. The method according to claim 8, generating object classification information according to the object fusion feature information, comprising: The object fusion feature information is input into an object classifier to generate object classification information.

10. The method according to claim 1, wherein the image processing model is trained and generated by the following steps: Obtain multiple sample target images and sample detection results corresponding to the multiple sample target images, wherein: The sample detection results include sample object classification information, sample lesion classification information and sample lesion segmentation information; Inputting the plurality of sample target images into an image processing model to obtain a predicted detection result and a training contrast loss value output by the image processing model; Calculate the predicted loss value according to the predicted detection result and the sample detection result; The model parameters of the image processing model are adjusted according to the predicted loss value and the training contrast loss value, and the image processing model is continuously trained until a model training stop condition is reached.

11. The method according to claim 10, wherein obtaining a plurality of sample target images and sample detection results corresponding to the plurality of sample target images comprises: Determine a sample detection object, and obtain a sample enhanced image and a sample target image corresponding to the sample detection object, wherein the sample enhanced image and the sample target image corresponding to the sample detection object correspond to the same target detection area of ​​the sample detection object; Generating a reference sample detection result for a target detection area on the sample enhanced image; The sample detection result is marked on the sample target image based on the reference sample detection result.

12. The method according to claim 11, wherein the image processing model comprises an image processing backbone network, an object branch network, a lesion branch network and a fusion network; Inputting the plurality of sample target images into an image processing model to obtain a predicted detection result and a training contrast loss value output by the image processing model includes: Inputting the plurality of sample target images into an image processing backbone network, and extracting feature information of sample objects to be processed and feature information of sample lesions to be processed in the image processing backbone network; Inputting the feature information of the sample to be processed object into the object branch network to obtain predicted object feature information, inputting the feature information of the sample to be processed lesion into the lesion branch network to obtain predicted lesion feature information and predicted feature information of the image to be segmented; In the object branch network, an object comparison loss value is obtained according to the sample object classification information corresponding to the sample target image, and in the lesion branch network, a lesion comparison loss value is obtained according to the sample lesion classification information corresponding to the sample target image; Inputting the sample object feature information and the predicted lesion feature information into the fusion network to generate sample object fusion feature information; Generate first object classification information according to the sample object feature information, generate second object classification information according to the sample object fusion feature information, generate predicted lesion classification information according to the predicted lesion feature information, and generate predicted lesion segmentation information according to the predicted image feature information to be segmented; A predicted detection result is generated according to the first object classification information, the second object classification information, the predicted lesion classification information and the predicted lesion segmentation information, and a training comparison loss value is generated according to the object comparison loss value and the lesion comparison loss value.

13. The method according to claim 12, wherein the predicted loss value is calculated according to the predicted detection result and the sample detection result, comprising: Calculate a first object loss value according to the first object classification information and the sample object classification information; Calculating a second object loss value according to the second object classification information and the sample object classification information; Calculating a lesion classification loss value according to the predicted lesion classification information and the sample lesion classification information; A lesion segmentation loss value is calculated according to the predicted lesion segmentation information and the sample lesion segmentation information.

14. The method according to claim 12, inputting the feature information of the sample to be processed object into the object branch network to obtain the feature information of the predicted object, comprising: Obtain historical sample object feature information; Inputting the historical object feature information and the sample to-be-processed object feature information into the object branch network to obtain predicted object feature information; Accordingly, the feature information of the sample lesion to be processed is input into the lesion branch network to obtain the predicted lesion feature information and the predicted feature information of the image to be segmented, including: Obtain historical sample lesion feature information; The historical sample lesion feature information and the sample to-be-processed lesion feature information are input into the lesion branch network to obtain predicted lesion feature information and predicted image feature information to be segmented.

15. The method of claim 12, wherein adjusting a model parameter of an image processing model according to the prediction loss value and the training contrast loss value comprises: Calculating a model loss value according to the first object loss value, the second object loss value, the lesion classification loss value, the lesion segmentation loss value, the object comparison loss value, and the lesion comparison loss value; The model parameters of the image processing model are adjusted according to the model loss value.

16. A CT image processing method, comprising: receiving a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether there is an abnormal object in the target detection area; The multiple plain scan CT images are input into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

17. A computer-aided diagnosis method for cancer, comprising: receiving a CT image processing task, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether a tumor exists in the target detection area; The multiple plain scan CT images are input into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

18. A computer-aided diagnosis method for liver cancer, comprising: Receive CT image processing tasks, where: The CT image processing task carries a plurality of plain scan CT images corresponding to the liver region, and the CT image processing task is used to detect whether a tumor exists in the liver region; The multiple plain scan CT images are input into an image processing model to obtain a detection result corresponding to the liver area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

19. A computer-aided diagnosis system for cancer, comprising a client and a server, wherein: The client is used to send a CT image processing task to the server, wherein the CT image processing task carries a plurality of plain scan CT images corresponding to a target detection area, and the CT image processing task is used to detect whether a tumor exists in the target detection area; The server is used to input the multiple plain scan CT images into an image processing model to obtain a detection result corresponding to the target detection area, wherein the image processing model generates object feature information and lesion feature information based on each plain scan CT image, generates object fusion feature information according to the object feature information and the lesion feature information, determines object classification information according to the object fusion feature information, determines lesion classification information and lesion segmentation information according to the lesion feature information, and generates a detection result according to the object classification information, the lesion classification information and the lesion segmentation information.

20. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 18 are implemented.

21. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 18.

22. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 18.

Citation Information

Patent Citations

  • Image processing method and training method of image processing model

    CN117094959A

  • Image detection method and device based on multispectral image fusion, medium and equipment

    CN118155036A

  • Training method of image processing model and image processing method

    CN118247284A

  • Image detection method and apparatus, and device, medium and program

    WO2022179083A1

Cited By

  • Flat-scanning CT (Computed Tomography) image lesion detection method based on back projection data and related equipment

    CN119251222A

  • Plain scan CT image lesion detection method and related equipment based on back-projection data

    CN119251222B

  • Image processing model training method and image processing method

    CN120746925A

  • Image processing method and CT image processing method

    CN121121387A

  • Medical imaging processing method and system

    TWI901468B