Method for training target image processing model and image processing method

Abstract

Embodiments of the present description provide a method for training a target image processing model and an image processing method. The method for training a target image processing model comprises: determining an initial image and an enhanced image of a target object; fusing the initial image and the enhanced image to obtain a fused image, and masking the initial image to obtain a masked image; inputting the fused image into a reference image processing model, and using the reference image processing model to perform encoding and decoding processing on the fused image to obtain an initial decoded image feature and an initial decoded classification feature; inputting the masked image into a target image processing model, and using the target image processing model to perform encoding and decoding processing on the masked image to obtain a target decoded image feature and a target decoded classification feature; and, on the basis of the initial decoded image feature, the initial decoded classification feature, the target decoded image feature, and the target decoded classification feature, training the target image processing model. The accuracy of detecting and segmenting the target object by using the initial image is improved.

Description

Target image processing model training method and image processing method

[0001] The present application claims priority to the Chinese patent application No. 202410843483.5, filed on June 26, 2024, and titled "Target image processing model training method and image processing method", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present specification relate to the technical field of computer technology, in particular to a target image processing model training method and an image processing method; one or more embodiments of the present specification also relate to a target image processing model training device, an image processing method, a computing device, a computer-readable storage medium, and a computer program product. BACKGROUND

[0003] With the popularization of breast cancer screening, more and more people receive non-contrast computed tomography (NCCT) examination. However, the results of NCCT examination may not be as accurate as contrast-enhanced computed tomography (CECT) examination, that is, the use of non-contrast computed tomography (NCCT) for examination may affect early diagnosis and treatment.

[0004] Therefore, there is an urgent need for a method to improve the diagnostic effect of non-contrast computed tomography (NCCT) examination and improve the detection sensitivity and specificity of tumors in non-contrast computed tomography (NCCT) data. SUMMARY

[0005] Therefore, embodiments of the present specification provide a target image processing model training method. One or more embodiments of the present specification also relate to a target image processing model training device, an image processing method, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defect that the examination results obtained by using non-contrast computed tomography (NCCT) are inaccurate in the prior art.

[0006] According to a first aspect of embodiments of the present specification, a target image processing model training method is provided, comprising:

[0007] determining an initial image and an enhanced image of a target object;

[0008] fusing the initial image and the enhanced image to obtain a fused image, and performing mask processing on the initial image to obtain a mask image;

[0009] input the fusion image into a reference image processing model, perform encoding and decoding processing on the fusion image by using the reference image processing model, and obtain initial decoded image features and initial decoded classification features;

[0010] input the mask image into a target image processing model, perform encoding and decoding processing on the mask image by using the target image processing model, and obtain target decoded image features and target decoded classification features;

[0011] train the target image processing model according to the initial decoded image features, the initial decoded classification features, the target decoded image features, and the target decoded classification features.

[0012] According to a second aspect of an embodiment of the present specification, an image processing method is provided, including:

[0013] determining a target image, and inputting the target image into a target image processing model;

[0014] performing encoding and decoding processing on the target image by using the target image processing model, and obtaining target image features and target classification features of the target image;

[0015] obtaining a segmentation image corresponding to the target image by using the target image features;

[0016] obtaining a classification result corresponding to the target image by using the target classification features.

[0017] According to a third aspect of an embodiment of the present specification, a computer-aided diagnosis method for cancer is provided, including:

[0018] determining a CT image of a target detection region;

[0019] inputting the CT image into a CT image processing model, processing the CT image by using the CT image processing model, and obtaining a CT segmentation image and a CT classification result corresponding to the CT image, wherein the CT image processing model is obtained by training the target image processing model according to the target image processing model training method;

[0020] obtaining a detection result of whether a tumor exists in the target detection region according to the CT segmentation image and the CT classification result.

[0021] According to a fourth aspect of an embodiment of the present specification, a computer-aided diagnosis method for breast cancer is provided, including:

[0022] determining a CT image of a breast region;

[0023] The CT image is input into the CT image processing model, and the CT image is processed by the CT image processing model to obtain the CT segmentation image and CT classification result corresponding to the CT image. The CT image processing model is trained by the above-mentioned target image processing model training method.

[0024] Based on the CT segmentation image and the CT classification result, the detection result of whether there is a tumor in the breast region is obtained.

[0025] According to a fifth aspect of the embodiments of this specification, another image processing method is provided, applied to a client of a medical system, comprising:

[0026] In response to a user's selection operation on the user interface of the client, a medical image is determined;

[0027] The medical image is sent to the server of the medical system, and the segmented image and classification result corresponding to the medical image are received from the server. The segmented image and classification result corresponding to the medical image are obtained by processing the medical image according to the target image processing model. The target image processing model is trained by the above-mentioned target image processing model training method.

[0028] The segmented image and classification results corresponding to the medical image are displayed to the user through the user interface.

[0029] According to a sixth aspect of the embodiments of this specification, a computer-aided diagnosis system for cancer is provided, comprising a client and a server, wherein,

[0030] The client is used to send CT images of the target detection area to the server;

[0031] The server is used to input the CT image into a CT image processing model, process the CT image using the CT image processing model, obtain the segmentation image and classification result corresponding to the CT image, and obtain the detection result of whether there is a tumor in the target detection area based on the CT segmentation image and the CT classification result, and return the detection result to the client. The CT image processing model is trained using the above-mentioned target image processing model training method.

[0032] According to a seventh aspect of the embodiments of this specification, a target image processing model training apparatus is provided, comprising:

[0033] The image determination module is configured to determine the initial image and the enhanced image of the target object;

[0034] The image obtaining module is configured to fuse the initial image and the enhanced image to obtain a fused image, and to perform mask processing on the initial image to obtain a mask image.

[0035] The initial feature obtaining module is configured to input the fused image into a reference image processing model, and to perform encoding and decoding processing on the fused image by using the reference image processing model to obtain initial decoded image features and initial decoded classification features.

[0036] The target feature obtaining module is configured to input the mask image into a target image processing model, and to perform encoding and decoding processing on the mask image by using the target image processing model to obtain target decoded image features and target decoded classification features.

[0037] The model training obtaining module is configured to train the target image processing model according to the initial decoded image features, the initial decoded classification features, the target decoded image features, and the target decoded classification features.

[0038] According to an eighth aspect of an embodiment of the present specification, an image processing apparatus is provided, including:

[0039] The determining module is configured to determine a target image, and to input the target image into a target image processing model.

[0040] The feature obtaining module is configured to perform encoding and decoding processing on the target image by using the target image processing model to obtain target image features and target classification features of the target image.

[0041] The image obtaining module is configured to obtain a segmentation image corresponding to the target image by using the target image features.

[0042] The result obtaining module is configured to obtain a classification result corresponding to the target image by using the target classification features.

[0043] According to a ninth aspect of an embodiment of the present specification, a computing device is provided, including:

[0044] a memory and a processor;

[0045] The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the above-mentioned target image processing model training method and image processing method.

[0046] According to a tenth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, which, when executed by a processor, implement the steps of the above-mentioned target image processing model training method and image processing method.

[0047] According to an eleventh aspect of the embodiments of the present specification, a computer program product is provided, which includes computer programs / instructions, which, when executed by a processor, implement the steps of the above-mentioned target image processing model training method and image processing method.

[0048] The one or more embodiments of the present specification provide a target image processing model training method. In the case of fusing an initial image and an enhanced image into a reference image processing model, the initial decoding image features and the initial decoding classification features containing rich information of a target object can be obtained by using the reference image processing model. In the case of inputting a mask image corresponding to the initial image into the target image processing model to obtain target decoding image features and target decoding classification features, and training the target image processing model by using the initial decoding image features, the initial decoding classification features, the target decoding image features, and the target decoding classification features, when the target image processing model processes the mask image to restore the features of the mask image, the initial decoding image features and the initial decoding classification features can be referred to for restoration, so that the detailed feature-level information (i.e., the initial decoding image features and the initial decoding classification features) of the target object can be better captured, the knowledge learned from the reference image processing model can be transferred to the target image processing model, and the accuracy of the target image processing model in detecting and segmenting the target object by using the initial image can be improved in the subsequent case of inputting the initial image of the target object into the target image processing model. BRIEF DESCRIPTION OF DRAWINGS

[0049] FIG. 1 is a scene schematic diagram of an image processing method according to an embodiment of the present specification;

[0050] FIG. 2 is a flowchart of a target image processing model training method according to an embodiment of the present specification;

[0051] FIG. 3 is a flowchart of an image processing method according to an embodiment of the present specification;

[0052] FIG. 4 is a flowchart of a computer-aided diagnosis method for breast cancer according to an embodiment of the present specification;

[0053] FIG. 5 is a flowchart of an image processing method applied to a client of a medical system according to an embodiment of the present specification;

[0054] FIG. 6 is a structural framework diagram of a target image processing model training method provided by an embodiment of the present specification, applied to a medical image diagnosis scenario;

[0055] FIG. 7 is a structural schematic diagram of a target image processing model training device provided by an embodiment of the present specification;

[0056] FIG. 8 is a structural schematic diagram of an image processing device provided by an embodiment of the present specification;

[0057] FIG. 9 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0058] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it is understood that the present specification will encompass numerous variations beyond those described in the detailed description.

[0059] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0060] It will be understood that, although the terms first, second, etc. can be employed in describing various information, such information should not be limited by these terms. These terms are only used to differentiate one piece of information from another. For example, a first can be termed a second, and, similarly, a second can be termed a first, without departing from the scope of one or more embodiments of the present specification. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining".

[0061] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0062] First, the nomenclature terms related to one or more embodiments of the present specification are explained.

[0063] Knowledge distillation: a model compression technique that aims to transfer the knowledge learned by a complex model (teacher model) to a lightweight model (student model) to achieve model compression and acceleration. This method optimizes the student model by minimizing the difference between the outputs of the teacher model and the student model.

[0064] FKD: Feature-level Knowledge Distillation, a technique for model compression and knowledge transfer in the field of deep learning. It transfers the implicit knowledge of the teacher model by making the intermediate layer features of the student model close to the corresponding layer features of the teacher model.

[0065] Masked image modeling: a self-supervised learning method, specifically, the algorithm randomly "masks" part of the input image, and then asks the model to predict the content of the masked part. This method encourages the model to understand and learn the global context and local details of the image, thereby improving its comprehensive understanding of the image content.

[0066] In the present specification, a target image processing model training method is provided, and the present specification also relates to a target image processing model training device, an image processing method, an image processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0067] Referring to FIG. 1, FIG. 1 shows a scene schematic diagram of an image processing method according to an embodiment of the present specification.

[0068] The image processing method provided by the embodiments of the present specification can be applied to different scenes. For example, when the image processing method is applied to an autonomous driving scene, detection and recognition of pedestrians, vehicles, or traffic signs in traffic images can be achieved. When the image processing method is applied to a medical scene, specifically for medical image analysis, detection and recognition of organs, tumors, or lesion areas in medical images can be achieved.

[0069] Taking the image processing method provided by the embodiments of the present specification as an example, the detection of tumors in plain CT images is described in detail.

[0070] Specifically, the image processing method is implemented by an application-side device 102 and a server 104. The application-side device 102 is configured to send a plain CT image to the server 104, such as a breast plain CT image.

[0071] The target image processing model is trained in the server 104, as shown in FIG. 1, which includes an encoding layer, a decoding layer and a full connection layer; thus, when the server 104 receives the plain CT image sent by the terminal device 102, the plain CT image is input into the target image processing model, the encoding layer and the decoding layer of the target image processing model are used to encode and decode the plain CT image, the target image features and the classification features corresponding to each decoding layer are obtained, the segmentation image corresponding to the plain CT image is obtained through the image features, the segmentation image is an image with different gray values indicating different regions, for example, 0 represents background, 1 represents organ, and 2 represents tumor; the key classification features corresponding to each decoding layer are obtained by convolution and pooling processing of the classification features corresponding to each decoding layer, the target classification features obtained by splicing the key classification features are input into the full connection layer, and thus the classification result corresponding to the plain CT image is obtained, such as 0 representing non-cancer and 1 representing cancer; the obtained segmentation image and classification result are returned to the terminal device 102.

[0072] Specifically, the training steps of the target image processing model are as follows: determining the initial image and the enhanced image of the target object; fusing the initial image and the enhanced image to obtain a fused image, and performing mask processing on the initial image to obtain a mask image; inputting the fused image into a reference image processing model, using the reference image processing model to perform encoding and decoding processing on the fused image to obtain initial decoding image features and initial decoding classification features; inputting the mask image into the target image processing model, using the target image processing model to perform encoding and decoding processing on the mask image to obtain target decoding image features and target decoding classification features; training the target image processing model according to the initial decoding image features, the initial decoding classification features, the target decoding image features and the target decoding classification features.

[0073] The terminal-side device 102 can include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5) application, or a light application (also known as a small program, a lightweight application), or a cloud application, and the like, which can be developed based on a software development kit (SDK) of a corresponding service provided by the server, such as a real-time communication (RTC) SDK, and the like. The terminal-side device can be deployed in an electronic device, and needs to be run in dependence on the device or some APP in the device, and the like. The electronic device can have a display screen and support information browsing, and the like, and can be a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant communication tools, mailbox clients, social platform software, and the like.

[0074] The server 104 can be understood as a server providing various services, including a physical server, a cloud server, for example, a server providing communication services for multiple clients, for example, a server for background training supporting a model used on a client, for example, a server processing data sent by a client, and the like. It should be noted that the server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server 104 can also be a server of a distributed system, or a server combined with a blockchain. The server 104 can also be a cloud server of cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and the like basic cloud computing services, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0075] It should be noted that the image processing method provided in the embodiments of the present specification can be executed by the server 104, and in other embodiments of the present specification, the target image processing model can be deployed in the terminal-side device 102, so that the terminal-side device 102 can also have similar functions as the server 104, thereby executing the image processing method provided in the embodiments of the present specification; in other embodiments, the image processing method provided in the embodiments of the present specification can also be executed by the terminal-side device 102 and the server 104 together.

[0076] The image processing method provided by the embodiments of the present specification can input an image into a target image processing model, and obtain target image features and target classification features of the image by using the target image processing model. In the case that the target image processing model is trained by using features containing enhanced image information, the target image processing model can better capture detailed feature level information of a target object in the image, so as to obtain more accurate segmentation images and classification results.

[0077] Referring to FIG. 2, FIG. 2 shows a flowchart of a target image processing model training method provided by an embodiment of the present specification, which specifically includes the following steps.

[0078] Step 202: determining an initial image and an enhanced image of a target object.

[0079] The target object can be understood as a region, object or feature in the image that has a specific meaning or interest, which is a key element in image analysis, image processing and image understanding tasks; the target object is understood differently in different application scenarios, such as in a traffic scenario, the target object can be understood as a pedestrian, a vehicle or a traffic sign; in a logistics scenario, the target object can be understood as a shelf, a cargo or an electronic tag; in a medical scenario, the target object can be understood as an organ, a tumor, etc.

[0080] The initial image can be understood as an original image obtained by collecting (such as shooting or scanning) the target object, which can be directly obtained by a camera, a scanner or other image capturing devices; the enhanced image can be understood as an image that can highlight the target object, which can be obtained by adjusting the initial image.

[0081] For example, in a medical scenario, the initial image can be a plain CT image, and in the case that the target object is a tumor, the enhanced image can be an enhanced CT image. The enhanced CT image can more clearly show the blood vessel structure, the tumor, the inflammation area or other pathological changes than the plain CT image, because the contrast between them and the surrounding normal tissue is improved. In a logistics scenario, the initial image is a shelf image collected by an inspection robot, and in the case that the target object is a cargo, the enhanced image can be an image obtained by sharpening the initial image, so that the cargo on the shelf can be more clearly displayed.

[0082] Specifically, the initial image and the enhanced image of the target object are determined, and the enhanced image contains more target object related privileged information than the initial image.

[0083] Step 204: fusing the initial image and the enhanced image to obtain a fused image, and performing mask processing on the initial image to obtain a mask image.

[0084] The fusion image can be understood as an image obtained by splicing fusion, weighted fusion, Poisson fusion, etc. of the initial image and the enhanced image. The mask image can be understood as an image obtained by performing mask processing on the initial image, and randomly masking part of the image pixels in the initial image.

[0085] Specifically, by splicing and fusing the initial image and the enhanced image, a spliced and fused fusion image is obtained, so that when the subsequent reference image processing model is processed for the fusion image, the complementary information of the multi-modal image can be utilized to enrich the feature learning of the model and enhance the generalization ability, thereby providing strong support for the subsequent knowledge distillation process.

[0086] For example, by splicing and fusing the plain CT image and the enhanced CT image at the channel level and inputting the reference image processing model, the reference image processing model can better understand the complex features in the plain CT image, so that when the reference image processing model (as a teacher model) transfers the feature-level information to the target image processing model (as a student model), the target image processing model can also perform reasonable processing on the plain CT image when the input data of the target image processing model is the plain CT image, thereby improving the accuracy and effectiveness of detection and segmentation based on the plain CT image.

[0087] By inputting the mask image into the target image processing model, the features based on the mask image can be recovered more according to the corresponding features of the reference image processing model, so that detailed feature-level information of the target object is obtained from the reference image processing model, and the distillation constraint is strengthened.

[0088] In an embodiment of the present specification, when the target image processing model needs to learn from the reference image processing model and obtain detailed feature-level information of the target object from the reference image processing model, the reference image processing model can process the image containing the target object to obtain detailed feature-level information of the target object. The training steps of the reference image processing model are as follows:

[0089] Determine the initial image sample, the enhanced image sample, the image label, and the classification label of the target object sample.

[0090] Fuse the initial image sample and the enhanced image sample to obtain a fusion image sample, and input the fusion image sample into the reference image processing model.

[0091] Obtain the predicted segmentation image and the predicted classification result corresponding to the initial image sample by using the reference image processing model.

[0092] According to the predicted segmentation image, the image label, the predicted classification result, and the classification label, the reference image processing model is trained.

[0093] In an embodiment of the present specification, the obtaining, by the reference image processing model, the predicted segmentation image and the predicted classification result corresponding to the initial image sample comprises:

[0094] The fusion image sample is encoded and decoded by using the reference image processing model to obtain a target decoded image sample feature and a target decoded classification sample feature.

[0095] The predicted segmentation image corresponding to the fusion image sample is obtained by using the target decoded image sample feature.

[0096] The predicted classification result corresponding to the fusion image sample is obtained by using the target decoded classification sample feature.

[0097] In an embodiment of the present specification, the reference image processing model comprises an encoder and a decoder, the encoder is constituted by a plurality of encoding layers, and the decoder is constituted by a plurality of decoding layers.

[0098] The fusion image sample is encoded and decoded by using the reference image processing model to obtain a target decoded image sample feature and a target decoded classification sample feature, comprising:

[0099] The fusion image sample is encoded by using the plurality of encoding layers to obtain an initial encoded image sample feature.

[0100] The initial encoded image sample feature is decoded by using the plurality of decoding layers to obtain an initial decoded image sample feature corresponding to each decoding layer in the plurality of decoding layers.

[0101] The initial decoded image sample feature corresponding to the final decoding layer in the plurality of decoding layers is determined as the target decoded classification sample feature, or according to an image classification task, the initial decoded image sample feature corresponding to each decoding layer in the plurality of decoding layers is subjected to convolution and pooling processing to obtain a key image sample feature corresponding to each decoding layer, and the key image sample features corresponding to the decoding layers are fused to obtain the target decoded classification sample feature.

[0102] Specifically, the training process of the reference image processing model is described in detail: during the training of the reference image processing model, the initial image sample and the enhanced image sample of the target object sample are fused to obtain a fused image sample, and the fused image sample is input into the reference image processing model as input; the network architecture of the reference image processing model is a multi-task network architecture, including an encoder-decoder structure with a skip connection; the reference image processing model includes two branches of specific tasks, one is a segmentation branch for obtaining a segmentation image, and the other is a classification branch for obtaining a classification result.

[0103] The encoder and the decoder of the reference image processing model are used to perform encoding and decoding processing on the fused image sample, that is, the fused image sample is encoded by multiple encoding layers in the encoder to obtain initial encoded image sample features, and the initial encoded image sample features are decoded by multiple decoding layers in the decoder to obtain initial decoded image sample features corresponding to each decoding layer in the multiple decoding layers.

[0104] In actual application, for the classification branch, the initial decoded image sample features corresponding to the last decoding layer in the multiple decoding layers can be determined as target decoded classification sample features; or in order to more accurately obtain the classification result, the corresponding initial decoded image sample features are obtained from each decoding layer, and convolution processing is performed on these features, representative key image sample features are extracted from the features after convolution processing by applying global large pooling, and the key image sample features corresponding to each decoding layer are spliced and fused to obtain target decoded classification sample features; the target decoded classification sample features are input into a fully connected layer, and the fully connected layer is used to obtain the predicted classification result of the initial image sample.

[0105] For the segmentation branch, high-level features of the fused image sample are captured in the encoder to obtain a highly abstract representation of the content of the fused image sample; the decoder is responsible for restoring the abstract feature map generated by the encoder to the spatial size of the original image. In the case of a skip connection encoder-decoder structure of the reference image processing model, each layer of the encoder is directly connected to the corresponding decoding layer (usually the same scale or close to the scale), and the feature map in the encoding stage is directly transmitted, thereby preserving the detail information of the original image, which helps to reconstruct finer boundaries and textures in the final output. Through the processing of the encoder and the decoder, the features output by the decoder after being processed by multiple decoding layers are determined as target decoded image sample features, so that the predicted segmentation image corresponding to the initial image sample is obtained by using the target decoded image sample features. The predicted segmentation image uses different gray values to represent different objects to achieve the segmentation of different objects.

[0106] In the case of obtaining the predicted segmentation image and the predicted classification result, a segmentation loss function of the segmentation branch is calculated by the predicted segmentation image and the image label, and a classification loss function of the classification branch is calculated by the predicted classification result and the classification label; according to the segmentation loss function and the classification loss function, the reference image processing model is trained, so that a reference image processing model with higher accuracy is obtained by training with pairs of initial image samples and enhanced image samples. The reference image processing model benefits from a larger training data set and can capture rich information present in the enhanced image.

[0107] In the case of improving the accuracy of the segmentation image and the classification result corresponding to the initial image, the knowledge distillation technology is used to transfer the knowledge learned from the reference image processing model (teacher model) to the lower-accuracy target image processing model (student model) trained using the initial image sample. The model structure of the target image processing model is consistent with that of the reference image processing model, thereby improving the accuracy of the segmentation image and the classification result obtained by the target image processing model based on the initial image.

[0108] Step 206: input the fusion image into the reference image processing model, and perform encoding and decoding processing on the fusion image using the reference image processing model to obtain initial decoding image features and initial decoding classification features.

[0109] The initial decoding image features can be understood as decoding features obtained by the decoding layer of the reference image processing model, which are used to generate the segmentation image corresponding to the initial image. The initial decoding image features can be the features corresponding to each decoding layer for generating the segmentation image corresponding to the initial image, or the features corresponding to the last decoding layer after passing through multiple decoding layers for generating the segmentation image corresponding to the initial image. The initial decoding classification features can be understood as features obtained by the decoding layer of the reference image processing model, which are used to classify the initial image. The initial decoding classification features can be the features corresponding to each decoding layer for classifying the initial image, or the features corresponding to the last decoding layer after passing through multiple decoding layers for classifying the initial image.

[0110] Specifically, by inputting the fusion image into the reference image processing model, initial decoding image features and initial decoding classification features corresponding to the fusion image are obtained through the encoding and decoding processing of the reference image processing model. The initial decoding image features are suitable for generating or understanding image segmentation, which refers to dividing an image into multiple regions, each region corresponding to a different object or object category in the image. Therefore, such features help to identify and separate different objects or regions in the image. The initial decoding classification features focus on classifying the entire image or the main body in the image, i.e., identifying which category or attribute the image belongs to. They are the keys for the model to judge the image content, such as determining whether an image is an organ, a tumor, etc.

[0111] In one or more embodiments of the present specification, the reference image processing model includes an encoder and a decoder, the encoder is composed of multiple encoding layers, and the decoder is composed of multiple decoding layers. Through the processing of multiple decoding layers, initial decoding image features corresponding to multiple decoding layers can be obtained, so as to determine initial decoding classification features according to the initial decoding image features corresponding to multiple decoding layers. The specific implementation is as follows:

[0112] The encoding and decoding processing of the fusion image by the reference image processing model to obtain initial decoding image features and initial decoding classification features includes:

[0113] Encoding processing of the fusion image by the multiple encoding layers to obtain initial encoding image features;

[0114] Decoding processing of the initial encoding image features by the multiple decoding layers to obtain initial decoding image features corresponding to the multiple decoding layers, and determining initial decoding classification features according to the initial decoding image features corresponding to the multiple decoding layers.

[0115] Wherein, the initial encoding image features can be understood as compact and information-rich feature representations obtained by encoding processing of the fusion image by multiple encoding layers; the initial decoding image features can be understood as image representations of different abstraction levels output by the decoding layers; and the initial decoding classification features can be understood as features related to image classification determined according to the initial decoding image features corresponding to the decoding layers.

[0116] Specifically, the fusion image is processed using multiple encoding layers. The encoding layer usually involves gradually reducing the spatial resolution of the image while increasing the abstraction level of the feature representation. This process can be regarded as feature extraction of the fusion image. Through a series of dimension reduction operations such as convolution and pooling, the fusion image is converted into a set of compact and information-rich feature representations, i.e., initial encoding image features.

[0117] The initial encoded image features obtained by the encoding layer are decoded by using multiple decoding layers. The role of the decoding layer is to reversely decode and up-sample the highly compressed feature information step by step, and restore the spatial structure close to the initial image. In this process, according to the initial encoded image features obtained by encoding, the partial or complete structure of the image is reconstructed through deconvolution, up-sampling and other operations to generate initial decoding image features corresponding to multiple decoding layers. The output of each decoding layer can be regarded as an image representation at different abstraction levels. The closer to the output end of the decoding layer, the closer the feature is to the original spatial structure of the image.

[0118] Further analysis is performed on the multiple initial decoding image features corresponding to the multiple decoding layers. In actual application, the initial decoding image features after decoding are analyzed, and the most relevant part for image classification is extracted, which can include the combination of global or local features, and is used for category judgment of the initial image.

[0119] The target image processing model training method provided by the embodiments of the present specification can obtain more refined initial decoding image features corresponding to each decoding layer through the encoding and decoding processing of the fusion image by using multiple encoding layers and multiple decoding layers. Therefore, when the initial decoding image features are used to determine the initial decoding classification features, initial decoding classification features containing rich information can also be obtained, which facilitates to improve the performance of the trained target image processing model and improve the accuracy of image segmentation and image classification.

[0120] In one or more embodiments of the present specification, different processing can be performed on the initial decoding image features according to actual needs, so as to obtain initial decoding classification features meeting different needs. The specific implementation modes are as follows:

[0121] The initial decoding classification features are determined according to the initial decoding image features corresponding to the multiple decoding layers, and the initial decoding classification features are determined according to the initial decoding image features corresponding to the multiple decoding layers.

[0122] The initial decoding image features corresponding to the final decoding layer in the multiple decoding layers are determined as the initial decoding classification features; or

[0123] According to the image classification task, the initial decoding image features corresponding to each decoding layer in the multiple decoding layers are subjected to convolution and pooling processing to obtain first key decoding image features corresponding to the decoding layers, and the first key decoding image features corresponding to the decoding layers are fused to obtain initial decoding classification features.

[0124] Among them, the final decoding layer can be understood as the last decoding layer in the plurality of decoding layers, and in the case that each decoding layer is processed based on the features output by the previous decoding layer, the features output by the final decoding layer are the final product of the entire decoding process, which contains the key information required by the model for in-depth understanding and prediction of the fusion image. The first key decoding image feature can be understood as a feature that is representative for image classification in the initial decoding image feature corresponding to each decoding layer.

[0125] The image classification task can be understood as a task of classifying images into different types of images according to the target object in the image; for example, in a medical scenario, the goal of the image classification task can be to divide images into images containing tumors and images not containing tumors according to the target object in the image; in a logistics scenario, the goal of the image classification task can be to divide images into images with normal goods display and images with abnormal goods display according to the target object in the image.

[0126] Specifically, in the case of wanting to simplify the calculation and improve the efficiency, the feature output by the decoder after being processed by the plurality of decoding layers, i.e., the initial decoding image feature output by the last decoding layer, can be determined as the initial decoding classification feature.

[0127] In the case of wanting to obtain more accurate classification results, the initial decoding image feature corresponding to each decoding layer can be extracted according to the image classification task, i.e., the initial decoding image feature corresponding to each decoding layer containing different hierarchical scales is obtained, and the initial decoding image feature corresponding to each decoding layer is further refined through convolution processing to obtain features meaningful for the image classification task, the spatial dimension is reduced through the pooling operation (the global larger pooling is applied in the embodiments of the present specification), and the most important information in each initial decoding image feature after convolution is retained, and the parameter quantity of the subsequent fully connected layer can also be reduced through the pooling operation, thereby reducing the risk of overfitting and reducing the computational burden; the first key decoding image feature corresponding to each decoding layer is obtained, and the initial decoding classification feature is obtained by splicing and fusing the first key decoding image feature corresponding to each decoding layer.

[0128] The target image processing model training method provided by the embodiments of the present specification can obtain initial decoding classification features meeting different requirements according to actual needs, and in the case of fusing the first key decoding image feature corresponding to each decoding layer to obtain the initial decoding classification feature, the context information of the overall level can be integrated while capturing the details of the local level, thereby obtaining more accurate results.

[0129] In one or more embodiments of the present specification, in a case where the initial decoding image feature and the initial decoding classification feature are obtained, an initial segmentation image corresponding to the image segmentation task is obtained, and an initial classification result corresponding to the image classification task is obtained. The specific implementation is as described below.

[0130] After the fusion image is input into the reference image processing model and the initial decoding image feature and the initial decoding classification feature are obtained by using the reference image processing model to perform encoding and decoding processing on the fusion image, the method further includes:

[0131] According to the image segmentation task, the initial decoding image feature is used to obtain an initial segmentation image corresponding to the initial image.

[0132] According to the image classification task, the initial decoding classification feature is used to obtain an initial classification result corresponding to the initial image.

[0133] The image segmentation task can be understood as a task of segmenting different objects or different regions in an image. The image segmentation process can be regarded as classifying each pixel in the image and assigning a label to each pixel to represent the class or object to which it belongs. For example, in the case where the target image processing model is applied to the medical field in the embodiments of the present specification, the background in the image is marked as 0, the organ is marked as 1, and the tumor is marked as 2.

[0134] The initial segmentation image can be understood as an image obtained by segmenting different objects in the initial image. The initial classification result can be understood as a classification result determined according to the target object in the initial image. For example, in the medical field, the initial segmentation image is an image with different gray values representing the background (0), the organ (1), and the tumor (2). The initial classification result includes two types: one is cancer and the other is non-cancer.

[0135] In actual applications, according to specific downstream tasks, the initial decoding image feature can be used to obtain an initial segmentation image of the initial image corresponding to the image segmentation task, and the initial decoding classification feature can be used to obtain an initial classification result of the initial image corresponding to the image classification task.

[0136] The target image processing model training method provided by the embodiments of the present specification can not only use the initial decoding image feature to implement the pixel-level image segmentation task, but also use the initial decoding classification feature to perform the image-level image classification task, thereby obtaining various results related to the initial image.

[0137] Step 208: inputting the mask image into a target image processing model, performing encoding and decoding processing on the mask image by using the target image processing model, and obtaining target decoding image features and target decoding classification features.

[0138] The target image processing model has the same network architecture as the reference image processing model, that is, the target image processing model is also a multi-task network architecture including an encoder-decoder structure with a skip connection.

[0139] In a similar manner to obtaining the initial decoding image features and the initial decoding classification features by using the reference image processing model, the target image processing model obtains the target decoding image features and the target decoding classification features.

[0140] The target decoding image features can be understood as decoding features obtained by using a decoding layer of the target image processing model, which are used to generate features of a segmentation image corresponding to the mask image. The target decoding classification features can be understood as features obtained by using the decoding layer of the target image processing model, which are used to classify the mask image.

[0141] In one or more embodiments of the present specification, the target image processing model includes an encoder and a decoder, the encoder is composed of a plurality of encoding layers, and the decoder is composed of a plurality of decoding layers.

[0142] The target image processing model is used to perform encoding and decoding processing on the mask image to obtain target decoding image features and target decoding classification features, including:

[0143] The mask image is encoded by using the plurality of encoding layers to obtain target encoding image features.

[0144] The target encoding image features are decoded by using the plurality of decoding layers to obtain target decoding image features corresponding to the plurality of decoding layers, and target decoding classification features are determined according to the target decoding image features corresponding to the plurality of decoding layers.

[0145] In one or more embodiments of the present specification, the target decoding classification features are determined according to the target decoding image features corresponding to the plurality of decoding layers, including:

[0146] The target decoding image features corresponding to a final decoding layer in the plurality of decoding layers are determined as the target decoding classification features; or

[0147] According to the image classification task, the target decoding image features corresponding to each decoding layer in the plurality of decoding layers are subjected to convolution and pooling processing to obtain second key decoding image features corresponding to each decoding layer, and the second key decoding image features corresponding to each decoding layer are fused to obtain target decoding classification features.

[0148] In one or more embodiments of the present specification, after the target image processing model is input with the mask image, the target image processing model is used to perform encoding and decoding processing on the mask image to obtain target decoding image features and target decoding classification features, the method further comprises:

[0149] According to the image segmentation task, the target decoding image features are used to obtain a target segmentation result corresponding to the initial image;

[0150] According to the image classification task, the target decoding classification features are used to obtain a target classification result corresponding to the initial image.

[0151] The specific implementation is similar to the process of the reference image processing model processing the fused image, which will not be described here.

[0152] Step 210: training the target image processing model according to the initial decoding image features, the initial decoding classification features, the target decoding image features, and the target decoding classification features.

[0153] Specifically, when training the target image processing model, in order to extract privileged information from the reference image processing model, the feature-level knowledge distillation (FKD) loss is used to minimize the intermediate features of the reference image processing model and the target image processing model, i.e., to minimize the difference between the initial decoding image features and the target decoding image features, and to minimize the difference between the initial decoding classification features and the target decoding classification features, so as to guide the target image processing model to inherit and enhance the knowledge in the image from the reference image processing model.

[0154] In one or more embodiments of the present specification, the target image processing model is trained by calculating the loss function between the initial decoding image features and the target decoding image features, and calculating the loss function between the initial decoding classification features and the target decoding classification features through feature-level knowledge distillation. The specific implementation is as follows:

[0155] The target image processing model is trained according to the initial decoding image features, the initial decoding classification features, the target decoding image features, and the target decoding classification features, comprising:

[0156] According to the initial decoding image features and the target decoding image features, a segmentation loss function is obtained;

[0157] obtaining a classification loss function according to the initial decoding classification feature and the target decoding classification feature;

[0158] training the target image processing model according to the segmentation loss function and the classification loss function.

[0159] The segmentation loss function and the classification loss function can be measured by feature similarity, such as cosine similarity or mean square error.

[0160] In actual application, since the image segmentation task needs to capture fine-grained local features, the similarity of the initial decoding image features corresponding to each decoding layer of the reference image processing model and the target decoding image features corresponding to each decoding layer of the target image processing model is compared to obtain the segmentation loss function of each decoding layer, so that the target image processing model (student model) can better learn different scale and detail information.

[0161] In the case of image classification task paying more attention to global features, the features output by the decoder at the end of the processing of multiple decoding layers in the reference image processing model can be used as the initial decoding classification feature, and the features output by the decoder at the end of the processing of multiple decoding layers in the target image processing model can be used as the target decoding classification feature, and the similarity comparison on the final feature representation (the features output by the decoder at the end of the processing) can be performed to obtain the classification loss function.

[0162] The target image processing model is trained by the segmentation loss function of the image segmentation task and the classification loss function of the image classification task.

[0163] The target image processing model training method provided by the embodiments of the present specification can obtain the segmentation loss function of the initial decoding image feature and the target decoding image feature on each decoding layer in the case of image classification task needing to guide layer by layer to obtain multi-scale information and ensure the learning of detail information, and can obtain the classification loss function through the final feature representation in the case of image classification task being able to effectively express global semantic information, so as to simplify the calculation and improve the efficiency.

[0164] In one or more embodiments of the present specification, by introducing an additional reconstruction branch in the target image processing model, the target decoding image feature corresponding to the decoding layer is used to reconstruct an enhanced image, and the distillation process is further strengthened, so as to obtain the missing enhanced image information in the input of the target image processing model. The specific implementation manner is as follows:

[0165] Before the training of the target image processing model according to the initial decoding image feature, the initial decoding classification feature, the target decoding image feature and the target decoding classification feature, the method further comprises:

[0166] According to the image prediction task, the target decoding image feature is subjected to convolution and up-sampling processing to obtain a predicted decoding image feature, and a predicted enhanced image is obtained according to the predicted decoding image feature.

[0167] The training of the target image processing model according to the initial decoding image feature, the initial decoding classification feature, the target decoding image feature, the target decoding classification feature includes:

[0168] The training of the target image processing model according to the initial decoding image feature, the initial decoding classification feature, the target decoding image feature, the target decoding classification feature, the predicted enhanced image and the enhanced image.

[0169] The predicted decoding image feature can be understood as a feature containing specific details and structural information required for generating the predicted enhanced image.

[0170] Specifically, the target decoding image feature is further refined and combined by performing convolution processing on the target decoding image feature corresponding to each decoding layer, and the resolution of the feature map is increased and the size of the feature map is enlarged by using the up-sampling operation, so as to approach or restore the size of the original image, to prepare for generating a high-resolution predicted image, and to restore the spatial details of the image to obtain the predicted decoding image feature. The predicted decoding image feature is used to generate a predicted enhanced image, which should contain more information related to the target object than the initial image.

[0171] In the case where the predicted enhanced image is obtained, the target image processing model is trained by using the initial decoding image feature, the initial decoding classification feature, the target decoding image feature, the target decoding classification feature, the predicted enhanced image and the enhanced image.

[0172] That is, in the case where the reconstruction branch is additionally added, the training of the target image processing model is based on the above-mentioned training of the target image processing model, and the training of the target image processing model by the predicted enhanced image and the enhanced image is additionally added.

[0173] The target image processing model training method provided by the embodiments of the present specification integrates an additional reconstruction branch into the target image processing model, obtains a predicted enhanced image by using the additional reconstruction branch, and further enhances the distillation process by the predicted enhanced image and the enhanced image, so that the target image processing model can implicitly acquire knowledge from the enhanced image.

[0174] In one or more embodiments of the present specification, in the case of obtaining a predicted enhanced image, when training the target image processing model, a reconstruction loss function is obtained on the basis of obtaining a segmentation loss function and a classification loss function, so as to train the target image processing model according to the three loss functions. The specific implementation is as follows:

[0175] The target image processing model is trained according to the initial decoded image feature, the initial decoded classification feature, the target decoded image feature, the target decoded classification feature, the predicted enhanced image, and the enhanced image.

[0176] A segmentation loss function is obtained according to the initial decoded image feature and the target decoded image feature.

[0177] A segmentation loss function is obtained according to the initial decoded classification feature and the target decoded classification feature.

[0178] A reconstruction loss function is obtained according to the predicted enhanced image and the enhanced image.

[0179] The target image processing model is trained according to the segmentation loss function, the classification loss function, and the reconstruction loss function.

[0180] Specifically, because there are pairs of initial images and enhanced images for the same target image, when the mask image corresponding to the initial image is input into the target image processing model, the similarity between the obtained predicted enhanced image and the enhanced image (the enhanced image paired with the initial image) is calculated to obtain the reconstruction loss function corresponding to the image prediction task.

[0181] In actual application, on the basis of obtaining the segmentation loss function and the classification loss function, the reconstruction loss function is obtained, so as to train the target image processing model by using the segmentation loss function, the classification loss function, and the reconstruction loss function.

[0182] The target image processing model provided by the embodiments of the present specification can make the target image processing model learn and capture more fine-grained features by reconstructing the enhanced image and using the reconstruction loss function of the predicted enhanced image and the enhanced image obtained by reconstruction. These features may not be obvious in the initial image, and in this way, the target image processing model can more deeply understand the complex features in the initial image. In addition, the reconstruction branch provides a powerful additional constraint, prompting the target image processing model not only to match the output or feature representation of the reference image processing model, but also to actually generate high-quality enhanced images. Through this multi-task learning strategy, the generalization ability and prediction performance of the target image processing model can be improved.

[0183] In one or more embodiments of the present specification, the initial image and the enhanced image of the target object can be obtained by the client, and in the case of training the target image processing model, the target image processing model can be deployed on the client according to the actual needs of the client, or the model interface information for the client to interact with the target image processing model can be provided to the client. The specific implementation is as follows:

[0184] The initial image and the enhanced image of the target object are determined, comprising:

[0185] The initial image and the enhanced image of the target object sent by the client are received.

[0186] After training the target image processing model, the method further comprises:

[0187] The target image processing model or the model interface information corresponding to the target image processing model is sent to the client.

[0188] The model interface information can be understood as containing detailed information on how to interact with the target image processing model, including data format, calling method, configuration options, and error handling, so as to ensure that the target image processing model can be effectively and accurately integrated into various applications and services.

[0189] Specifically, the client can send the initial image and the enhanced image of the target object to the server, and the server can process the initial image and the enhanced image of the target object by using the above-mentioned target image processing model training method to train the target image processing model.

[0190] In the case that the server obtains the target image processing model, the trained target image processing model can be sent to the client to deploy the target image processing model locally on the client; or to save the computing resources of the client, the target image processing model can be deployed on the server (which can also be a cloud server), and the model interface information corresponding to the target image processing model can be sent to the client to facilitate the client to interact with the target image processing model through the model interface information. For example, in the case that the client sends a plain CT image to the server, the server returns the segmentation image and the classification result corresponding to the plain CT image to the client.

[0191] The target image processing model training method provided by the embodiments of the present specification can deploy the target image processing model on the client, or use the server as a model providing platform to provide model services for the client, thereby saving the computing resources of the client.

[0192] The target image processing model training method provided by the embodiments of the present specification trains a reference image processing model with higher accuracy by using pairs of initial images and enhanced images, the reference image processing model captures rich information present in the enhanced images, and uses a knowledge distillation technique to transfer the knowledge learned from the reference image processing model to a target image processing model with lower accuracy trained using the mask image corresponding to the initial image, so as to realize the transmission of the privileged information of the reference image processing model to the target image processing model, and on the basis of training using the mask image, the distillation constraint can be enhanced and the robustness of the target image processing model can be improved, and the accuracy of detection and segmentation of the target object using the initial image can be improved.

[0193] Referring to FIG. 3, FIG. 3 shows a flowchart of an image processing method according to an embodiment of the present specification, which specifically includes the following steps.

[0194] Step 302: Determine a target image, and input the target image into a target image processing model.

[0195] Specifically, the target image processing model is obtained by training using the target image processing model training method.

[0196] Step 304: Perform encoding and decoding processing on the target image using the target image processing model to obtain target image features of the target image and target classification features.

[0197] The target image can be understood as an initial image (i.e., the initial image in the above embodiments) containing a target object. In the medical field, the target image can be understood as a plain CT image.

[0198] In actual applications, the initial image containing the target object without processing is input into the target image processing model, and in the case that the target image processing model includes an encoder and a decoder, the target image is subjected to encoding and decoding processing to obtain target image features for image segmentation and target classification features for image classification.

[0199] In one or more embodiments of the present specification, the target image processing model includes an encoder and a decoder, the encoder is composed of multiple encoding layers, and the decoder is composed of multiple decoding layers; the encoding image features obtained by encoding are subjected to decoding processing using multiple decoding layers to obtain target image features corresponding to each decoding layer, so as to further obtain target classification features. The specific implementation is as follows:

[0200] The encoding and decoding processing of the target image using the target image processing model to obtain target image features of the target image and target classification features includes:

[0201] encoding processing on the target image by using the plurality of encoding layers to obtain an encoded image feature;

[0202] decoding processing on the encoded image feature by using the plurality of decoding layers to obtain a target image feature corresponding to each decoding layer, and determining a target classification feature according to the target image features corresponding to the plurality of decoding layers.

[0203] The encoded image feature can be understood as a compact and information-rich feature representation obtained by encoding processing on the target image by using the plurality of encoding layers. The target image feature can be understood as an image representation at different levels of abstraction output by the decoding layer. The target classification feature can be understood as a feature related to image classification determined according to the target image features corresponding to the decoding layers.

[0204] Specifically, the target image is converted into a set of compact and information-rich feature representations, i.e., encoded image features, by using the plurality of encoding layers to perform encoding processing on the target image. The encoded image features obtained by the encoding layers are decoded by using the plurality of decoding layers to restore the spatial structure close to the target image. In this process, each decoding layer corresponds to a target image feature, and the output of each decoding layer can be regarded as an image representation at different levels of abstraction. The decoding layer closer to the output end, the feature is closer to the original spatial structure of the image.

[0205] In actual application, the target image features after decoding are analyzed, and the most relevant part to image classification is extracted, which can include a combination of global or local features, and is used for class judgment of the initial image.

[0206] The image processing method provided by the embodiments of the present specification can obtain the target image features corresponding to each decoding layer with finer granularity by encoding and decoding processing on the target image by using the plurality of encoding layers and the plurality of decoding layers, so that the target classification feature also contains rich information about the target object when the target classification feature is determined by using the target decoding image feature.

[0207] In one or more embodiments of the present specification, different processing can be performed on the target image features according to actual needs, so as to obtain target classification features meeting different needs. The specific implementation modes are as follows:

[0208] The determining of the target classification feature according to the target image features corresponding to the plurality of decoding layers comprises:

[0209] The target image feature corresponding to the final decoding layer in the plurality of decoding layers is determined as the target classification feature; or

[0210] According to the image classification task, the target image features corresponding to each decoding layer in the plurality of decoding layers are subjected to convolution and pooling processing to obtain key image features corresponding to each decoding layer, and the key image features corresponding to each decoding layer are fused to obtain target classification features.

[0211] For specific implementation, refer to the above embodiments, which will not be repeated here.

[0212] In actual application, features in each decoding layer are extracted, convolution is performed on the extracted features, and representative features (i.e., key image features) are extracted from the features after convolution by applying global large pooling. These representative features are spliced, and the spliced features (i.e., target classification features) are input into a fully connected layer to obtain a classification result.

[0213] The target image processing model training method provided by the embodiments of the present specification is beneficial to fine-grained segmentation and classification when extracting features in each decoding layer. Through this splicing method, local details and integrated contextual information can be captured, thereby improving the overall performance and interpretability of the target image processing model.

[0214] Step 306: obtaining a segmentation image corresponding to the target image by using the target image features.

[0215] Step 308: obtaining a classification result corresponding to the target image by using the target classification features.

[0216] The image processing method provided by the embodiments of the present specification, when the target image processing model is obtained by the above target image processing model training method, the target image processing model can obtain target image features and target classification features containing rich information of the target image, and the target image processing model can improve the accuracy of the segmentation image and the classification result corresponding to the target image.

[0217] The embodiments of the present specification also provide a computer-aided diagnosis method for cancer, specifically comprising:

[0218] determining a CT image of a target detection region;

[0219] inputting the CT image into a CT image processing model, processing the CT image by using the CT image processing model to obtain a CT segmentation image and a CT classification result corresponding to the CT image, wherein the CT image processing model is obtained by training the above target image processing model training method;

[0220] obtaining a detection result of whether the target detection region has a tumor according to the CT segmentation image and the CT classification result.

[0221] Specifically, the target detection region can be a region to be detected, such as a certain organ region of a human body, and the CT image can be a CT image corresponding to the target detection region, including a plain CT image and an enhanced CT image.

[0222] The CT image is input into the CT image processing model. In the case where the CT image processing model is trained by the above-mentioned target image processing model training method, the CT segmentation image and the CT classification result corresponding to the CT image can be obtained through the CT image processing model, so that the detection result of whether the target detection region has a tumor is obtained according to the CT segmentation image and the CT classification result.

[0223] In actual application, the computer-aided diagnosis method of cancer is realized by a computer, and the image processing method related to diagnosis is implemented by a device with image processing capability such as a computer, in order to improve the accuracy of image processing, facilitate image recognition, storage and transmission, and provide a detection result with a probability value by the computer, which can provide a reference for medical personnel to accurately diagnose diseases and develop treatment plans.

[0224] That is, according to the above content, the detection result obtained according to the CT segmentation image and the CT classification result can be understood as a probability value of whether the target detection region has a tumor. Medical personnel can use the probability value as a reference for diagnosing diseases. For example, in the case where the probability value is greater than a preset threshold (for example, the preset threshold is 80%), medical personnel can increase the probability of diagnosing that the target detection region has a tumor, so as to reduce the risk of misdiagnosis.

[0225] The computer-aided diagnosis method of cancer provided by the embodiments of the present specification can obtain accurate CT segmentation images and CT classification results by using the CT image processing model in the case where the CT image processing model is obtained by the above-mentioned target image processing model training method, so as to improve the accuracy of the detection result of whether the target detection region has a tumor according to the CT segmentation image and the CT classification result.

[0226] Referring to FIG. 4, FIG. 4 shows a flowchart of a computer-aided diagnosis method of breast cancer according to an embodiment of the present specification, which specifically includes the following steps.

[0227] Step 402: Determine a CT image of a breast region.

[0228] Step 404: Input the CT image into a CT image processing model, and process the CT image by using the CT image processing model to obtain a CT segmentation image and a CT classification result corresponding to the CT image, wherein the CT image processing model is trained by the above-mentioned target image processing model training method.

[0229] Step 406: Based on the CT segmentation image and the CT classification result, obtain the detection result of whether there is a tumor in the breast region.

[0230] Taking the breast region as the target detection area as an example, the computer-aided diagnosis method for breast cancer is explained. Specifically, when applied to medical imaging diagnosis scenarios, the target detection area can be the breast region. In this case, the CT image can be a plain CT image or an enhanced image containing organs or tumors. Correspondingly, the CT segmentation image can be a CT segmentation image that uses 0 to represent the background, 1 to represent the organ, and 2 to represent the tumor. This CT segmentation image uses different gray values ​​to represent different objects. The CT classification result can be a CT classification result that uses 0 to represent non-cancer and 1 to represent cancer.

[0231] In practical applications, when CT segmentation images and CT classification results are obtained, the results of these two parts can be combined to obtain the detection results for the breast region. These detection results are used to detect whether there is a tumor in the breast region.

[0232] For specific implementation details, please refer to the above embodiments, which will not be repeated here.

[0233] The computer-aided diagnostic method for breast cancer provided in the embodiments of this specification can process CT images of the breast region using a CT image processing model. In plain CT scans used for tumor screening and opportunistic detection, it can improve the accuracy of plain CT scans, thereby improving the diagnostic effect for breast cancer.

[0234] Referring to Figure 5, Figure 5 shows a flowchart of an image processing method for a client application in a medical system according to an embodiment of this specification, specifically including the following steps.

[0235] Step 502: In response to the user's selection operation on the user interface of the client, determine the medical image;

[0236] In practical applications, users (usually medical professionals) upload medical images by clicking on the user interface of the client, such as clicking the upload button on the user interface, or by directly selecting a medical image in the user interface; the medical image can be a plain CT scan of the breast.

[0237] Step 504: Send the medical image to the server of the medical system, and receive the segmented image and classification result corresponding to the medical image returned by the server. The segmented image and classification result corresponding to the medical image are obtained by processing the medical image according to the target image processing model. The target image processing model is trained by the above-mentioned target image processing model training method.

[0238] Specifically, the medical image is sent to a service end of a medical system, a target image processing model is deployed at the service end of the medical system, the medical image is processed by using the target image processing model, a segmentation image corresponding to the medical image and a classification result are obtained, and the segmentation image corresponding to the medical image and the classification result are returned to the client.

[0239] Step 506: The segmentation image corresponding to the medical image and the classification result are displayed to the user through the user interaction interface.

[0240] Specifically, in a case where the segmentation image corresponding to the medical image and the classification result sent by the service end are received, the segmentation image and the classification result are displayed to the user through the user interaction interface, so that the user determines the information processing result of the medical image through the segmentation image and the classification result.

[0241] The image processing method provided by the embodiments of the present specification realizes automatic processing of the medical image through interaction between the client and the service end, and automatic processing and instant feedback can greatly shorten the time from image uploading to result analysis, accelerate the entire diagnosis process, and improve the diagnosis accuracy by using the target image processing model; in a region where plain CT examination is used as a preferred scheme for tumor screening, the method can improve the accuracy of plain CT examination, thereby better serving public health.

[0242] The embodiments of the present specification also provide a computer-aided diagnosis system for cancer, comprising a client and a service end, wherein,

[0243] The client is configured to send a CT image of a target detection region to the service end.

[0244] The service end is configured to input the CT image into a CT image processing model, process the CT image by using the CT image processing model, obtain a segmentation image corresponding to the CT image and a classification result, and obtain a detection result of whether a tumor exists in the target detection region according to the CT segmentation image and the CT classification result, and return the detection result to the client, wherein the CT image processing model is obtained by training the target image processing model.

[0245] The above medical image is taken as an example of the CT image of the target detection region, and specific implementation can be referred to the above embodiments, which will not be described here.

[0246] The computer-aided diagnosis system for cancer provided by the embodiments of the present specification realizes automatic processing of the CT image of the target detection region through interaction between the client and the service end, accelerates the entire diagnosis process, and improves the accuracy of the detection result by using the CT image processing model.

[0247] Referring to FIG. 6, FIG. 6 shows a structural framework diagram of a target image processing model training method provided by an embodiment of the present specification, applied to a medical image diagnosis scenario.

[0248] In practical applications, the enhanced CT contains the understanding of additional information that can enhance the diagnostic performance. However, the AI (Artificial Intelligence) model learned only by relying on the enhanced CT cannot be generalized and generalized to the plain CT, and cannot improve and enhance the diagnostic performance for the plain CT images which are widely used in conventional clinical practice; the knowledge obtained from the enhanced CT is transferred to the plain CT image.

[0249] Specifically, a higher-accuracy teacher model (i.e., the reference image processing model in the above embodiment) is trained using paired plain CT and enhanced CT images, which benefits from a larger training dataset and captures rich information present in the enhanced CT images. However, due to the inherent differences between the two CT phases, it cannot be directly generalized to the plain CT image. Therefore, the knowledge distillation technique is used to transfer the knowledge learned from the teacher model to a lower-accuracy student model (i.e., the target image processing model in the above embodiment) trained using the plain CT image.

[0250] By extracting knowledge from the enhanced CT and the corresponding teacher model of the plain CT, the diagnostic performance of the student model corresponding to the plain CT is improved, and in order to better capture detailed feature-level information through the distillation process, masked image modeling is integrated into the knowledge distillation process, i.e., by masking the image blocks input to the student model and recovering the features from the teacher model, the distillation constraint can be strengthened.

[0251] In addition, by introducing an additional reconstruction branch in the student model, the enhanced CT image is reconstructed using the features obtained by the decoder, which further strengthens the distillation process, so that the student model can to some extent supplement the missing enhanced CT information in the input.

[0252] Specifically, in the architecture diagram as shown in FIG. 6, the teacher model and the student model are included, and the architecture diagram is described in detail in the following stages.

[0253] Teacher model training stage: In practical applications, a multi-phase CT (the input of the teacher network is both enhanced CT and plain CT, so it is a multi-phase CT) teacher model is constructed, which is used for plain CT and enhanced CT knowledge modeling, and assists the learning of the single-phase CT (the input of the student model is the plain CT) student model.

[0254] Obtain a pair of plain scan CT and enhanced CT, input the pair of plain scan CT and enhanced CT into the teacher model after splicing, the target is to enable the teacher model to guide the learning process of the robust student model, and improve the prediction of the student model on the plain scan CT image.

[0255] The teacher model is a U-net structure, including two task-specific branches for image segmentation task and image classification task. Specifically, the features in the input image (spliced plain scan CT and enhanced CT) are extracted by using an encoder, and the encoder and decoder are connected using skip connections, that is, the features of the encoding layer are combined with the up-sampling features of the corresponding decoding layer, the fine spatial information retained in the encoder can be injected into the decoder to generate an accurate segmentation image.

[0256] The features in each decoding layer are extracted, and the extracted features are subjected to convolution processing, and representative features are extracted from the convolution-processed features by applying global large pooling, and these representative features are spliced into a fully connected layer to obtain a classification result. The segmentation loss function of the generated segmentation image and the segmentation image label is calculated, and the classification loss function of the obtained classification result and the classification label is calculated, so as to train the teacher network according to the image loss function and the classification loss function.

[0257] The goal of multi-task learning of the teacher model is to minimize the following loss function:

[0258] Wherein, And represent the segmentation loss function and the classification loss function respectively, and λ1 and λ2 are hyperparameters used to balance the importance of segmentation and classification tasks; denotes the segmentation image label corresponding to the enhanced CT and the plain scan CT; denotes the classification label of the entire input; And are the predicted segmentation image and the classification result respectively; it should be noted that for the pair of inputs, share the same value because they represent the global label of the same patient (such as the same breast of the patient), and similarly, also share the same value.

[0259] Student model training stage: in order to extract privileged information (information contained in the enhanced CT and not contained in the plain scan CT) from the teacher network, the constraint of the student model learning from the teacher model is applied to the intermediate features by using the feature-level KD loss (denoted as ) to guide the student model to inherit the contrast enhancement knowledge from the teacher model.

[0260] From the training set of training the teacher network, a plain CT is obtained, and a mask processing is performed, and the masked mask CT is input into the student model; the student model and the teacher network are the same encoder-decoder structure; when the decoder of the student model performs upsampling on the features extracted by the encoder, the features of the decoder in the teacher network are used for guidance, that is, in the case that the decoder of the teacher network is decoder 1 and the decoder of the student model is decoder 2, for the segmentation branch, the similarity of the features extracted by each layer of decoder 1 and the features extracted by each layer of decoder 2 is calculated.

[0261] Specifically, taking the decoder containing 5 layers as an example, the similarity of the features extracted by the first layer of decoder 1 and the features extracted by the first layer of decoder 2 is calculated, and the features extracted by the second layer of decoder 2 are obtained according to the features extracted by the first layer of decoder 2, the similarity of the features extracted by the second layer of decoder 1 and the features extracted by the second layer of decoder 2 is calculated, and so on, until the similarity of the features extracted by the fifth layer of decoder 1 and the features extracted by the fifth layer of decoder 2 is calculated, so as to better guide the student model to learn different scales and detailed information, capture fine-grained local features, and generate more accurate segmentation images by comparing the similarity at each decoder layer.

[0262] The features finally output by the decoder after being processed by multiple decoder layers are referred to as final features, and the similarity between the final feature representation of the classification branch extracted by the teacher network and the final feature representation of the classification branch extracted by the teacher network is calculated, so as to obtain the classification result consistent with the classification result output by the teacher network.

[0263] By comparing the similarity of the features of each decoder layer through the segmentation branch and comparing the similarity of the final features of the decoder through the classification branch, the feature-level knowledge distillation between the teacher network and the student model is realized; and an additional reconstruction branch (containing convolution operation) is integrated into the student model, and the features of each decoder layer of the student model are convolved and upsampled to obtain a reconstruction enhanced CT image; the loss function of the reconstruction enhanced CT image and the corresponding enhanced CT image is calculated, the student model is adjusted, and the student model is trained.

[0264] In actual application, when training the student model, the plain CT can be input into the student model, but when the plain CT and the enhanced CT show similar appearance, the extraction of knowledge from the teacher model to the student model may still lack effectiveness when calculating the loss function, so the masked mask CT is input into the student model.

[0265] In the case of inputting the masked CT into the student model after masking, the features of these masks are recovered from the teacher model, and the knowledge extraction loss forces the teacher model to extract valuable information, thereby setting a strong constraint for knowledge extraction.

[0266] Therefore, the specific training target can be expressed as:

[0267] wherein X NC represents the plain scan image, represents the masked plain scan image, Θ T represents the parameters corresponding to the teacher model, Θ S represents the parameters corresponding to the student model. and represent the feature representations extracted by the i-th decoding layer (a total of n decoding layers) of the teacher (T) and student (S) models, respectively, for the segmentation task. and represent the final feature representations (before the fully connected layer) extracted by the teacher (T) and student (S) models for the classification task, respectively. is a measure of similarity between two features, such as cosine similarity or mean squared error (MSE).

[0268] Student model inference stage: In the inference stage, the student network can use the plain scan CT image for prediction (e.g., cancer detection [cancer vs. non-cancer], tumor segmentation).

[0269] Input the plain scan CT into the student model, and use the encoder-decoder structure of the student model to obtain a segmentation image, where the segmentation image is a grayscale image consistent with the size of the plain scan CT (the segmentation image has 2 or 3 values: 0 represents background, 1 organ, and 2 represents malignant tumor; if no tumor is detected / segmented, it is only 0 and 1 two values).

[0270] By extracting the features in each decoding layer, performing convolution on the extracted features, applying global max pooling to extract representative features from the convolution-processed features, and inputting these representative features into the fully connected layer, the classification result is obtained.

[0271] The image processing method provided by the embodiments of the present specification proposes a novel knowledge distillation framework, which transfers the knowledge from a high-accuracy teacher model trained by paired enhanced CT and plain CT images to a low-accuracy student model trained by plain CT images; the student model is encouraged to imitate the behavior of the teacher network to minimize the difference between their outputs; the diagnostic effect is improved by transferring the privileged knowledge of the teacher network to the student model; the mask image modeling is incorporated into the knowledge distillation, focusing on distilling knowledge from the unmasked CT image blocks to strengthen the constraints and improve the robustness; the performance of breast cancer detection and segmentation based on plain CT is significantly improved.

[0272] Corresponding to the method embodiments described above, the present specification also provides target image processing model training device embodiments. FIG. 7 shows a structural schematic diagram of a target image processing model training device according to an embodiment of the present specification. As shown in FIG. 7, the device comprises:

[0273] An image determination module 702 configured to determine an initial image and an enhanced image of a target object;

[0274] An image obtaining module 704 configured to fuse the initial image and the enhanced image to obtain a fused image, and to perform mask processing on the initial image to obtain a mask image;

[0275] An initial feature obtaining module 706 configured to input the fused image into a reference image processing model, and to perform encoding and decoding processing on the fused image by using the reference image processing model to obtain initial decoding image features and initial decoding classification features;

[0276] A target feature obtaining module 708 configured to input the mask image into a target image processing model, and to perform encoding and decoding processing on the mask image by using the target image processing model to obtain target decoding image features and target decoding classification features;

[0277] A model training obtaining module 710 configured to train the target image processing model according to the initial decoding image features, the initial decoding classification features, the target decoding image features, and the target decoding classification features.

[0278] Optionally, the initial feature obtaining module 706 is further configured to:

[0279] perform encoding processing on the fused image by using the plurality of encoding layers to obtain initial encoding image features;

[0280] decode the initial encoded image features by using the plurality of decoding layers to obtain initial decoded image features corresponding to the plurality of decoding layers, and determine initial decoded classification features according to the initial decoded image features corresponding to the plurality of decoding layers.

[0281] Optionally, the initial feature obtaining module 706 is further configured to:

[0282] determine the initial decoded image features corresponding to the final decoding layer in the plurality of decoding layers as the initial decoded classification features; or

[0283] perform convolution and pooling processing on the initial decoded image features corresponding to each decoding layer in the plurality of decoding layers according to the image classification task to obtain first key decoded image features corresponding to the each decoding layer, and fuse the first key decoded image features corresponding to the each decoding layer to obtain the initial decoded classification features.

[0284] Optionally, the target feature obtaining module 708 is further configured to:

[0285] encode the mask image by using the plurality of encoding layers to obtain target encoded image features;

[0286] decode the target encoded image features by using the plurality of decoding layers to obtain target decoded image features corresponding to the plurality of decoding layers, and determine target decoded classification features according to the target decoded image features corresponding to the plurality of decoding layers.

[0287] Optionally, the target feature obtaining module 708 is further configured to:

[0288] determine the target decoded image features corresponding to the final decoding layer in the plurality of decoding layers as the target decoded classification features; or

[0289] perform convolution and pooling processing on the target decoded image features corresponding to each decoding layer in the plurality of decoding layers according to the image classification task to obtain second key decoded image features corresponding to the each decoding layer, and fuse the second key decoded image features corresponding to the each decoding layer to obtain the target decoded classification features.

[0290] Optionally, the model training obtaining module 710 is further configured to:

[0291] obtain a segmentation loss function according to the initial decoded image features and the target decoded image features;

[0292] obtain a classification loss function according to the initial decoded classification features and the target decoded classification features;

[0293] According to the segmentation loss function and the classification loss function, the target image processing model is trained.

[0294] The apparatus further includes:

[0295] The reconstruction module is configured to perform convolution and up-sampling processing on the target decoded image feature according to an image prediction task to obtain a predicted decoded image feature, and obtain a predicted enhanced image according to the predicted decoded image feature.

[0296] Optionally, the model training obtaining module 710 is further configured to:

[0297] According to the initial decoded image feature, the initial decoded classification feature, the target decoded image feature, the target decoded classification feature, the predicted enhanced image, and the enhanced image, the target image processing model is trained.

[0298] Optionally, the model training obtaining module 710 is further configured to:

[0299] According to the initial decoded image feature and the target decoded image feature, a segmentation loss function is obtained.

[0300] According to the initial decoded classification feature and the target decoded classification feature, a classification loss function is obtained.

[0301] According to the predicted enhanced image and the enhanced image, a reconstruction loss function is obtained.

[0302] According to the segmentation loss function, the classification loss function, and the reconstruction loss function, the target image processing model is trained.

[0303] The apparatus further includes:

[0304] The initial result obtaining module is configured to obtain an initial segmentation image corresponding to the initial image according to an image segmentation task by using the initial decoded image feature, and obtain an initial classification result corresponding to the initial image according to an image classification task by using the initial decoded classification feature.

[0305] The apparatus further includes:

[0306] The target result obtaining module is configured to obtain a target segmentation result corresponding to the initial image according to an image segmentation task by using the target decoded image feature, and obtain a target classification result corresponding to the initial image according to an image classification task by using the target decoded classification feature.

[0307] The apparatus further includes:

[0308] The reference model training module is configured to determine an initial image sample, an enhanced image sample, an image label, and a classification label of a target object sample; fuse the initial image sample and the enhanced image sample to obtain a fused image sample, and input the fused image sample into a reference image processing model; obtain a predicted segmentation image corresponding to the initial image sample and a predicted classification result by using the reference image processing model; and train the reference image processing model according to the predicted segmentation image, the image label, the predicted classification result, and the classification label.

[0309] Optionally, the image determination module 702 is further configured to:

[0310] receive an initial image and an enhanced image of a target object sent by a client.

[0311] The apparatus further includes:

[0312] The sending module is configured to send the target image processing model or model interface information corresponding to the target image processing model to the client.

[0313] One embodiment of the present specification provides a target image processing model training apparatus. In the case of fusing an initial image and an enhanced image and inputting the same into a reference image processing model, the reference image processing model can obtain initial decoding image features and initial decoding classification features containing rich information of a target object. In the case of inputting a mask image corresponding to the initial image into a target image processing model, obtaining target decoding image features and target decoding classification features, and training the target image processing model by using the initial decoding image features, the initial decoding classification features, the target decoding image features, and the target decoding classification features, the target image processing model can refer to the initial decoding image features and the initial decoding classification features for recovery when the target image processing model processes the mask image to restore features, so as to better capture detailed feature-level information (i.e., the initial decoding image features and the initial decoding classification features) of the target object, realize the transfer of knowledge learned from the reference image processing model to the target image processing model, and improve the accuracy of the target image processing model in detecting and segmenting the target object by using the initial image in subsequent cases.

[0314] The above is a schematic scheme of a target image processing model training apparatus according to an embodiment of the present specification. It should be noted that the technical scheme of the target image processing model training apparatus belongs to the same concept as the technical scheme of the target image processing model training method described above, and the details of the technical scheme of the target image processing model training apparatus that are not described in detail can be referred to the description of the technical scheme of the target image processing model training method.

[0315] Corresponding to the method embodiments described above, the specification also provides image processing device embodiments. FIG. 8 shows a structural schematic diagram of an image processing device according to an embodiment of the specification. As shown in FIG. 8, the device includes:

[0316] The determining module 802 is configured to determine a target image, and input the target image into a target image processing model.

[0317] The feature obtaining module 804 is configured to perform encoding and decoding processing on the target image by using the target image processing model, and obtain target image features and target classification features of the target image.

[0318] The image obtaining module 806 is configured to obtain a segmentation image corresponding to the target image by using the target image features.

[0319] The result obtaining module 808 is configured to obtain a classification result corresponding to the target image by using the target classification features.

[0320] Optionally, the feature obtaining module 804 is further configured to:

[0321] perform encoding processing on the target image by using the multiple encoding layers to obtain encoded image features;

[0322] perform decoding processing on the encoded image features by using the multiple decoding layers to obtain target image features corresponding to the multiple decoding layers, and determine target classification features according to the target image features corresponding to the multiple decoding layers.

[0323] Optionally, the feature obtaining module 804 is further configured to:

[0324] determine the target image features corresponding to the final decoding layer in the multiple decoding layers as the target classification features; or

[0325] perform convolution and pooling processing on the target image features corresponding to each decoding layer in the multiple decoding layers according to an image classification task to obtain key image features corresponding to the each decoding layer, and fuse the key image features corresponding to the each decoding layer to obtain the target classification features.

[0326] The image processing device provided by the embodiments of the specification can obtain target image features and target classification features containing rich information of a target image when the target image processing model is obtained by using the target image processing model training method described above, and further improve the accuracy of a segmentation image and a classification result corresponding to the target image by using the target image processing model.

[0327] The above is a schematic scheme of the image processing device of the embodiment. It should be noted that the technical scheme of the image processing device belongs to the same concept as the technical scheme of the image processing method described above, and the details of the technical scheme of the image processing device that are not described in detail can be referred to the description of the technical scheme of the image processing method.

[0328] FIG. 9 shows a structural block diagram of a computing device 900 according to an embodiment of the present specification. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 through a bus 930, and a database 950 is used to store data.

[0329] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 can include one or more of any type of network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC).

[0330] In an embodiment of the present specification, the above-mentioned components of the computing device 900 and other components not shown in FIG. 9 can also be connected to each other, for example, through a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 9 is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0331] The computing device 900 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 can also be a mobile or stationary server.

[0332] The processor 920 is configured to execute instructions of a computer program to implement the steps of the above-mentioned target image processing model training method.

[0333] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments. In particular, the computing device embodiment is basically similar to the target image processing model training method and the multiple image processing method embodiments, and thus the description is relatively simple, and the relevant parts can be referred to the description of the target image processing model training method and the multiple image processing method embodiments.

[0334] An embodiment of the specification further provides a computer-readable storage medium storing computer programs / instructions, which are executed by a processor to implement the steps of the above-mentioned target image processing model training method and the multiple image processing method.

[0335] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments mainly describes the difference from other embodiments. In particular, the computer-readable storage medium embodiment is basically similar to the target image processing model training method and the multiple image processing method embodiments, and thus the description is relatively simple, and the relevant parts can be referred to the description of the target image processing model training method and the multiple image processing method embodiments.

[0336] An embodiment of the specification further provides a computer program product including computer programs / instructions, which are executed by a processor to implement the steps of the above-mentioned target image processing model training method and the multiple image processing method.

[0337] The above is a schematic scheme of a computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the target image processing model training method and the plurality of image processing methods described above belong to the same concept, and the technical scheme of the computer program product is not described in detail. The contents of the description can be seen from the description of the target image processing model training method and the plurality of image processing methods.

[0338] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0339] The computer instructions include computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the contents included in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice, for example, in some regions, according to the patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0340] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the embodiments of the present specification.

[0341] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be seen from the related description of other embodiments.

[0342] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. Alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a target image processing model, comprising: Determine the initial image and enhanced image of the target object; The initial image and the enhanced image are fused to obtain a fused image, and the initial image is masked to obtain a masked image; The fused image is input into a reference image processing model, and the reference image processing model is used to encode and decode the fused image to obtain initial decoded image features and initial decoded classification features. The masked image is input into the target image processing model, and the target image processing model is used to encode and decode the masked image to obtain target decoded image features and target decoded classification features; The target image processing model is trained based on the initial decoded image features, the initial decoded classification features, the target decoded image features, and the target decoded classification features.

2. The target image processing model training method according to claim 1, wherein the reference image processing model includes an encoder and a decoder, the encoder is composed of multiple encoding layers, and the decoder is composed of multiple decoding layers; The step of using the reference image processing model to encode and decode the fused image to obtain initial decoded image features and initial decoded classification features includes: The fused image is encoded using the multiple coding layers to obtain initial coded image features; The initial encoded image features are decoded using the multiple decoding layers to obtain the initial decoded image features corresponding to the multiple decoding layers, and the initial decoded classification features are determined based on the initial decoded image features corresponding to the multiple decoding layers.

3. The target image processing model training method according to claim 2, wherein determining the initial decoding classification features based on the initial decoded image features corresponding to the plurality of decoding layers includes: The initial decoded image features corresponding to the final decoding layer among the multiple decoding layers are determined as the initial decoding classification features; or, According to the image classification task, the initial decoded image features corresponding to each of the multiple decoding layers are subjected to convolution and pooling to obtain the first key decoded image features corresponding to each decoding layer. The first key decoded image features corresponding to each decoding layer are then fused to obtain the initial decoding classification features.

4. The target image processing model training method according to claim 1, wherein the target image processing model includes an encoder and a decoder, the encoder is composed of multiple encoding layers, and the decoder is composed of multiple decoding layers; The step of using the target image processing model to encode and decode the mask image to obtain target decoded image features and target decoded classification features includes: The mask image is encoded using the multiple encoding layers to obtain the target encoded image features; The target encoded image features are decoded using the multiple decoding layers to obtain the target decoded image features corresponding to the multiple decoding layers, and the target decoding classification features are determined based on the target decoded image features corresponding to the multiple decoding layers.

5. The target image processing model training method according to claim 4, wherein determining the target decoding classification features based on the target decoded image features corresponding to the plurality of decoding layers includes: The target decoded image features corresponding to the final decoding layer among the multiple decoding layers are determined as target decoding classification features; or, According to the image classification task, the target decoded image features corresponding to each of the multiple decoding layers are subjected to convolution and pooling processing to obtain the second key decoded image features corresponding to each decoding layer. The second key decoded image features corresponding to each decoding layer are then fused to obtain the target decoding classification features.

6. The target image processing model training method according to any one of claims 1-5, wherein training the target image processing model based on the initial decoded image features, the initial decoded classification features, the target decoded image features, and the target decoded classification features comprises: Based on the initial decoded image features and the target decoded image features, a segmentation loss function is obtained; Based on the initial decoding classification features and the target decoding classification features, a classification loss function is obtained; The target image processing model is trained based on the segmentation loss function and the classification loss function.

7. The target image processing model training method according to any one of claims 1-5, wherein before training the target image processing model based on the initial decoded image features, the initial decoded classification features, the target decoded image features, and the target decoded classification features, the method further includes: According to the image prediction task, the target decoded image features are convolved and upsampled to obtain the predicted decoded image features, and the predicted enhanced image is obtained based on the predicted decoded image features. The step of training the target image processing model based on the initial decoded image features, the initial decoded classification features, the target decoded image features, and the target decoded classification features includes: Based on the initial decoded image features and the target decoded image features, a segmentation loss function is obtained; Based on the initial decoded classification features and the target decoded classification features, a classification loss function is obtained; Based on the predicted enhanced image and the enhanced image, a reconstruction loss function is obtained; The target image processing model is trained based on the segmentation loss function, the classification loss function, and the reconstruction loss function.

8. The target image processing model training method according to claim 1, further comprising, after inputting the fused image into a reference image processing model and using the reference image processing model to perform encoding and decoding processing on the fused image to obtain initial decoded image features and initial decoded classification features: Based on the image segmentation task, the initial segmented image corresponding to the initial image is obtained using the features of the initial decoded image; Based on the image classification task, the initial classification result corresponding to the initial image is obtained by utilizing the initial decoding classification features.

9. The target image processing model training method according to claim 1, further comprising, after inputting the mask image into the target image processing model and using the target image processing model to perform encoding and decoding processing on the mask image to obtain target decoded image features and target decoded classification features: Based on the image segmentation task, the target segmentation result corresponding to the initial image is obtained by utilizing the target decoded image features; Based on the image classification task, the target classification result corresponding to the initial image is obtained by using the target decoding classification features.

10. The target image processing model training method according to claim 1, wherein the training steps of the reference image processing model are as follows: Determine the initial image sample, enhanced image sample, image label, and classification label of the target object sample; The initial image sample and the enhanced image sample are fused to obtain a fused image sample, and the fused image sample is input into a reference image processing model; Using the reference image processing model, the predicted segmentation image and the predicted classification result corresponding to the initial image sample are obtained; The reference image processing model is trained based on the predicted segmented image, the image label, the predicted classification result, and the classification label.

11. The target image processing model training method according to claim 1, wherein determining the initial image and enhanced image of the target object includes: Receive the initial and enhanced images of the target object sent by the client; After training the target image processing model, the method further includes: The target image processing model or the model interface information corresponding to the target image processing model is sent to the client.

12. An image processing method, comprising: Determine the target image and input the target image into the target image processing model; The target image is encoded and decoded using the target image processing model to obtain the target image features and target classification features. Using the features of the target image, a segmented image corresponding to the target image is obtained; Using the target classification features, the classification result corresponding to the target image is obtained.

13. The image processing method according to claim 12, wherein the target image processing model includes an encoder and a decoder, the encoder being composed of multiple encoding layers, and the decoder being composed of multiple decoding layers; The step of using the target image processing model to encode and decode the target image to obtain the target image features and target classification features includes: The target image is encoded using the multiple encoding layers to obtain encoded image features; The encoded image features are decoded using the multiple decoding layers to obtain the target image features corresponding to the multiple decoding layers, and the target classification features are determined based on the target image features corresponding to the multiple decoding layers.

14. The image processing method according to claim 13, wherein determining the target classification features based on the target image features corresponding to the plurality of decoding layers includes: The target image features corresponding to the final decoding layer among the multiple decoding layers are determined as target classification features; or, According to the image classification task, the target image features corresponding to each decoding layer in the multiple decoding layers are subjected to convolution and pooling processing to obtain the key image features corresponding to each decoding layer. The key image features corresponding to each decoding layer are then fused to obtain the target classification features.

15. A computer-aided diagnostic method for cancer, comprising: CT images that identify the target detection area; The CT image is input into a CT image processing model, and the CT image is processed using the CT image processing model to obtain a CT segmentation image and CT classification result corresponding to the CT image. The CT image processing model is trained by any one of the target image processing model training methods of claims 1-10. Based on the CT segmentation image and the CT classification result, a detection result is obtained to determine whether a tumor exists in the target detection area.

16. A computer-aided diagnostic method for breast cancer, comprising: Determine CT images of the breast region; The CT image is input into a CT image processing model, and the CT image is processed using the CT image processing model to obtain a CT segmentation image and CT classification result corresponding to the CT image. The CT image processing model is trained by any one of the target image processing model training methods of claims 1-10. Based on the CT segmentation image and the CT classification result, the detection result of whether there is a tumor in the breast region is obtained.

17. A computer-aided diagnosis system for cancer, comprising a client and a server, wherein, The client is used to send CT images of the target detection area to the server; The server is used to input the CT image into a CT image processing model, process the CT image using the CT image processing model, obtain the segmentation image and classification result corresponding to the CT image, and obtain the detection result of whether there is a tumor in the target detection area based on the CT segmentation image and the CT classification result, and return the detection result to the client. The CT image processing model is trained by any one of the target image processing model training methods of claims 1-12.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-16.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-16.

Images