Training of image processing model, image processing, computer-aided diagnosis method

Abstract

This application provides a method for training an image processing model, image processing, and computer-aided diagnosis, relating to the field of image processing technology. The training method includes: acquiring a first sample image and entity description information associated with the first sample image, wherein the entity description information is used to describe target entities in the first sample image; performing self-supervised training on an initial network based on the first sample image to obtain an intermediate network, wherein the intermediate network is used at least to extract structural features of the target entities; and performing target training on the intermediate network based on the first sample image and the entity description information to obtain an image processing model. The technical solution of this application embodiment can improve the ability of the image processing model to represent target entities in an image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for training an image processing model, image processing, and computer-aided diagnosis. Background Technology

[0002] Non-contrast computed tomography (NCCT) images, or simply NCCT images, are widely used in the medical field due to their lack of the need for contrast agents and high safety. Image processing models for these images typically rely on extensive supervised annotation for training. However, these models are limited by the annotation system and have weak generalization ability, resulting in insufficient accuracy in processing NCCT images. Therefore, improving the ability of image processing models to represent NCCT images without relying on additional input is an urgent technical problem to be solved. Summary of the Invention

[0003] This application provides a method for training an image processing model, image processing, and computer-aided diagnosis, in order to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a training method for an image processing model, comprising: acquiring a first sample image and entity description information associated with the first sample image, wherein the entity description information is used to describe a target entity in the first sample image; performing self-supervised training on an initial network based on the first sample image to obtain an intermediate network, wherein the intermediate network is at least used to extract structural features of the target entity; performing target training on the intermediate network based on the first sample image and the entity description information, and obtaining an image processing model based on the trained intermediate network.

[0005] Secondly, embodiments of this application provide an image processing method, comprising: acquiring an image to be processed containing a target entity; inputting the image to be processed into a trained image processing model, and acquiring a visual representation of the target entity output by the image processing model, wherein the image processing model is trained based on the method described in the first aspect.

[0006] Thirdly, embodiments of this application provide a computer-aided diagnostic method, comprising: acquiring a non-enhanced computed tomography (CT) image containing a screening entity; inputting the CT image into an image processing model to obtain a visual representation of the screening entity output by the image processing model, wherein the image processing model is trained based on the method described in the first aspect.

[0007] Fourthly, embodiments of this application provide a computer-aided diagnostic system, comprising: a client, configured to send an anomaly screening request, the anomaly screening request carrying a non-enhanced computed tomography (CT) image containing a screening entity, the anomaly screening request being used to detect whether the screening entity has an anomaly; a server, configured to receive the anomaly screening request from the client, input the non-enhanced CT image into an image processing model, obtain a visual representation of the screening entity output by the image processing model, detect whether the screening entity has an anomaly based on the visual representation, and send a detection result to the client, wherein the image processing model is trained based on the method described in the first aspect, and the detection result is used to characterize whether the screening entity has an anomaly; the client is further configured to output the detection result.

[0008] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0009] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0010] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of this application.

[0011] In the technical solution provided in this application embodiment, a first sample image and entity description information describing the target entity in the first sample image are acquired. An intermediate network is obtained by performing self-supervised training on an initial network based on the first sample image. Then, the intermediate network is trained on a target based on the first sample image and the entity description information to obtain an image processing model. Thus, through self-supervised training, the network not only initially acquires the ability to extract the structural features of the target entity, but also eliminates the need for manual annotation, reducing training costs. By performing target training, various auxiliary knowledge from the entity description information can be integrated into the image processing model, enabling the visual representation output by the image processing model to incorporate relevant auxiliary knowledge. This significantly improves the quality and richness of the visual representation, thereby enhancing the image processing model's ability to represent target entities in images.

[0012] In medical imaging scenarios, the trained image processing model only needs to input NCCT images during the inference phase to obtain high-quality visual representations of organs and lesions in NCCT images, without relying on additional descriptive information or annotations. This achieves analysis performance close to that assisted by multimodal information with single-modal input, improving lesion detection rate and organ localization accuracy, and overcoming the shortcomings of existing image processing models, such as weak generalization ability due to reliance on supervised annotations.

[0013] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0015] Figure 1 A schematic diagram of NCCT image processing in related technologies is shown; Figure 2 An architectural diagram of the image processing system 200 provided in an embodiment of this application is shown; Figure 3 An architecture diagram of the computer-aided diagnostic system 300 provided in an embodiment of this application is shown; Figure 4 A flowchart of a training method 400 for an image processing model provided in an embodiment of this application is shown; Figure 5 This illustration shows a first schematic diagram of the training process of the image processing model provided in an embodiment of this application; Figure 6 This illustration shows a second schematic diagram of the training process of the image processing model provided in an embodiment of this application; Figure 7 A flowchart of the image processing method 700 provided in an embodiment of this application is shown; Figure 8 A flowchart of a computer-aided diagnostic method 800 provided in an embodiment of this application is shown; Figure 9 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0016] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0017] First, the terms used in the embodiments of this application are explained as follows: CT: is an imaging technique that uses X-ray beams to scan layers of the human body and reconstruct cross-sectional images using a computer. Its data is usually stored in the form of a three-dimensional voxel grid, where each voxel represents a volume unit in space and contains a specific Hounsfield Unit to reflect tissue density.

[0018] Lesion: refers to an abnormal area in the anatomical structure or tissue formed due to pathological changes, and is the core focus of diagnosis and treatment planning.

[0019] Segmentation Mask: Also known as a mask, it is a binary or multi-labeled matrix with the same spatial dimensions as the original image. It is used to accurately delineate the boundaries of objects (such as lesion structures or organ structures), separate the target region from the background, and provide standardized annotation information for quantitative analysis and subsequent deep model training.

[0020] The Tumor-Node-Metastasis Staging System (TNM staging) is a standardized staging system used to describe the anatomical extent of malignant tumors.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0022] While noncontrast-guided CT (NCCT) images require no contrast agents and are highly safe, their soft tissue contrast is very limited because they rely solely on inherent tissue attenuation differences to characterize anatomical structures. This results in insufficient visual information provided by NCCT images, making accurate identification and characterization of lesions (especially tumors) difficult. Therefore, as... Figure 1As shown, medical personnel need to combine contrast-enhanced computed tomography (CECT) images corresponding to NCCT images, radiology reports, and pathology reports to identify and characterize lesions. However, since the generation of radiology and pathology reports often lags behind that of NCCT images, the efficiency of lesion identification is low. To address this technical problem, related technologies typically employ supervised annotation for model training, that is, introducing clinically relevant annotation signals (such as organ segmentation) to train the model, enabling it to acquire clinical knowledge relevant to a specific task. However, its performance is severely limited by the pre-set annotation system and the specific task classification system, resulting in weak model generalization ability and difficulty in adapting to diverse downstream tasks.

[0023] In view of this, embodiments of this application provide a method for training an image processing model, image processing, and computer-aided diagnosis, aiming to improve the ability of the image processing model to represent target entities in an image without requiring additional input. A detailed description follows.

[0024] To facilitate understanding, let's first combine Figure 2 This paper introduces the application scenarios of the technical solution presented in this application. Figure 2 This diagram illustrates the architecture of an image processing system 200 provided in an embodiment of this application. Figure 2 As shown, the image processing system 200 includes a client 201 and a server 202.

[0025] The client 201 is used to send image processing requests to the server 202. For example, a user can input natural language commands or upload a target image to be processed based on the user interface provided by the client 201, thereby triggering the image processing request. The target image includes the target entity.

[0026] The server 202 receives the image processing request and inputs the target image into the image processing model to obtain the visual representation of the target entity in the target image output by the image processing model.

[0027] For example, server 202 can send a visual representation of the target entity to client 201, and client 201 can output the visual representation of the target entity, so that users can intuitively view the status of the target entity, or perform subsequent data analysis or task processing based on the visual representation.

[0028] In this embodiment, the image processing model is trained based on method 400. In method 400, an initial network to be trained is self-supervised based on the acquired sample images to obtain an intermediate network; and the intermediate network is target-trained based on the first sample image and the entity description information associated with the first sample image, and the image processing model is obtained based on the trained intermediate network.

[0029] The image processing system 200 can be applied to various scenarios, including but not limited to autonomous driving, industrial quality inspection, remote sensing monitoring, smart retail, and medical diagnosis.

[0030] For example, in autonomous driving scenarios, image processing requests can be real-time traffic analysis requests, with target entities such as pedestrians, vehicles, or traffic signs on the road. Visual features can be used by users (or autonomous driving systems) to accurately identify the outlines and locations of obstacles, thereby enabling safe path planning and obstacle avoidance decisions. Similarly, in industrial quality inspection scenarios, image processing requests can be product defect detection requests, with target entities such as the product and its surface defects. Visual features can be used by users (or industrial quality inspection systems) to quickly locate defective areas and determine product quality levels, thereby improving production line inspection efficiency and accuracy. Furthermore, in smart retail scenarios, image processing requests can be product recognition requests, with target entities such as products on shelves. Visual features can be used by users (or smart retail systems) to quickly count the quantity and location of displayed products, thus assisting in inventory management.

[0031] In method 400 of this embodiment, self-supervised training enables the network to initially extract structural features of target entities in images without relying on manual annotation, thus reducing training costs. By training the target entity, various auxiliary knowledge from entity description information can be integrated into the image processing model, allowing the visual representation output by the model to incorporate relevant auxiliary knowledge. This significantly improves the quality and richness of the visual representation, thereby enhancing the image processing model's ability to represent target entities in images.

[0032] The following is combined Figure 3 This paper introduces the application of the technical solution of this application in medical diagnostic scenarios. Figure 3 This diagram illustrates the architecture of a computer-aided diagnostic system 300 provided in an embodiment of this application. Figure 3 As shown, the computer-aided diagnostic system 300 includes a client 301 and a server 302.

[0033] Client 301 is used to send an anomaly screening request to server 302. The anomaly screening request carries an NCCT image containing the entity to be screened, and is used to detect whether the entity has any anomalies. For example, a user can trigger the anomaly screening request by entering natural language commands or uploading an NCCT image containing the entity to be screened through the user interface provided by client 301.

[0034] Server 302 receives and responds to anomaly screening requests from client 301, inputs the NCCT image carried in the request into an image processing model, and obtains a visual representation of the entity to be screened, output by the image processing model. The visual representation identifies the spatial distribution region, contour information, or boundary range of the entity to be screened in the NCCT image. Server 302 further detects whether the entity to be screened has anomalies based on the visual representation. For example, server 302 can input the visual representation into anomaly detection model, obtain the detection result output by the anomaly detection model, and send the detection result to client 301. This detection result indicates whether the entity to be screened has anomalies.

[0035] Client 301 is also used to output test results, allowing users (such as doctors) to intuitively view whether there are abnormalities in the screened entities, or to make subsequent clinical analysis or diagnostic decisions based on the test results.

[0036] This application embodiment trains an image processing model using method 400, enabling the model to obtain high-quality visual representations of organs and lesions in NCCT images by inputting only a single target image during the inference phase, without relying on additional descriptive information or annotations. This achieves analysis performance close to that assisted by multimodal information with single-modal input, improving lesion detection rate and organ localization accuracy, and overcoming the shortcomings of existing image processing models, such as weak generalization ability due to reliance on supervised annotations.

[0037] It is understood that the other components of the aforementioned image processing system, computer-aided diagnostic system, client, and server can employ various technical solutions now and in the future known to those skilled in the art, and will not be described in detail here. Furthermore, the deployment form or other configurations of the aforementioned image processing system, computer-aided diagnostic system, client, and server may differ in different application scenarios, and the embodiments of this application do not specifically limit them in this regard.

[0038] For example, a connection is established between the client and the server via a network. The network provides the medium for the communication link between the client and the server. Networks can include various connection types, such as wired, wireless communication links, or fiber optic cables. Data transmitted by the client can be processed such as encoding, transcoding, and compression before being sent to the server. There can be multiple clients; users can interact with the server through their clients to receive data sent by other clients or send data to other clients.

[0039] For example, the client can be hardware, such as mobile phones, personal computers, tablets, wearable devices, and other electronic devices; it can also be a browser, application (APP), web application, mini-program, cloud application, or microservice deployed on the electronic device. The server can include servers that provide various services, such as servers that provide communication services to clients, servers that support models used on clients, and servers that process data sent by clients. The server can be deployed as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server, which can be a physical server or a cloud server.

[0040] It should be noted that the application scenarios or examples provided in this application are for ease of understanding, and this application does not specifically limit the application of the technical solutions. Furthermore, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0041] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0042] Figure 4 A flowchart illustrating a training method for an image processing model provided in an embodiment of this application is shown. This method can be applied to a training device, which may be... Figure 2 or Figure 3 The server in the context can also be a different device. For example... Figure 4 As shown, the method may include steps S401, S402 and S403.

[0043] Step S401: Obtain the first sample image and the entity description information associated with the first sample image. The entity description information is used to describe the target entity in the first sample image.

[0044] The first sample image can be an image acquired based on a specific image acquisition mode, which can be flexibly set as needed in practical applications; this invention does not impose any limitations on this. The first sample image can be a two-dimensional image or three-dimensional volume data. Entity description information is auxiliary information used to describe the target entities in the first sample image. Entity description information can include at least one of the following: image-based information, text-based information, and labeled information. The target entity is the object in the first sample image that needs to be identified, analyzed, or represented. The target entity can include the abnormal entity to be detected (or referred to as the first type of entity) and the reference entity related to the abnormal entity (or referred to as the second type of entity).

[0045] As an example, in a medical diagnostic scenario, the first sample image can be an NCCT slice image or a 3D volumetric NCCT image. Target entities can include lesion structures (i.e., abnormal entities) and organ structures associated with the lesion structures (i.e., reference entities). Entity description information can include CECT images, text-based radiology and pathology reports, and annotated segmentation maps. The radiology report includes structured descriptive text of the imaging findings written by a radiologist. The pathology report contains structured pathological diagnostic text with TNM staging information, such as pancreatic cancer, T3N1M0. The annotated segmentation map can be in binary mask form, such as lesion structures as 1 and organ structures as 0. This means that NCCT and CECT images acquired by the same user in the same examination process, along with the corresponding radiology and pathology reports, can be obtained from the medical system, along with annotated segmentation maps annotated by experts for the NCCT images.

[0046] As another example, in a product defect detection scenario, the first sample image can be a visible light image of the product surface taken by an industrial camera under standard lighting conditions. Entity description information can include the product's infrared image, product inspection report, product analysis report, and annotated segmentation map of the visible light image. The product inspection report can include an evaluation of the product's overall appearance and descriptive text for suspected defect areas; for example, a linear mark approximately 5mm long exists on the shell surface, suspected to be a scratch, located in the upper left corner of the shell. The product analysis report can include descriptive text such as the defect type, severity level, and impact on product quality assessment; for example, a scratch with a depth of approximately 30 micrometers that does not penetrate the oxide layer is determined to be a minor defect that does not affect the product's lifespan. The annotated segmentation map can be in binary mask form, such as a defect area being 1 and the product itself being 0. That is, for the same product, images of different modalities of the product, the associated product inspection report and product analysis report, and annotated segmentation maps marked by quality inspectors can be obtained.

[0047] Step S402: Perform self-supervised training on the initial network based on the first sample image to obtain an intermediate network. The intermediate network is used to extract the structural features of the target entity.

[0048] The initial network is the neural network model to be trained. Its structure can be customized according to actual needs, such as using architectures like Convolutional Neural Networks (CNNs) or Vision Transformers. Appropriate network structures can be selected based on the specific application; this application does not impose any limitations on this. Self-supervised training refers to training using the information inherent in the first sample image itself, without requiring external labeled data. Self-supervised training methods can include Masked Image Modeling (MIM) and contrastive learning. In this embodiment, self-supervised training is at least used to enable the initial network to learn to extract structural features of the target entity; that is, the intermediate network obtained through training possesses the ability to extract structural features of the target entity, laying the foundation for subsequent target training.

[0049] Step S403: Based on the first sample image and entity description information, perform target training on the intermediate network, and obtain the image processing model based on the trained intermediate network.

[0050] In this context, target training refers to further training the intermediate network using the first sample image and its associated entity descriptions. Target training can employ knowledge distillation, allowing the intermediate network to learn the soft labels output by the teacher network; contrastive learning, enabling the intermediate network to learn to align visual features with textual features; or multi-task learning, allowing the intermediate network to learn multiple auxiliary tasks simultaneously. During target training, the intermediate network can learn various auxiliary knowledge from the entity descriptions. This enables the trained image processing model to process the input target image and output a visual representation of the target entities in the target image, which integrates relevant auxiliary knowledge.

[0051] In the technical solution provided in this application embodiment, a first sample image and entity description information for describing target entities in the first sample image are acquired. An intermediate network is obtained by performing self-supervised training on an initial network based on the first sample image. The intermediate network is then trained on a target based on the first sample image and entity description information. An image processing model is obtained from the trained intermediate network. Thus, through self-supervised training, the network not only initially acquires the ability to extract structural features of target entities in the image, but also eliminates the need for manual annotation, reducing training costs. By performing target training, various auxiliary knowledge from the entity description information can be integrated into the image processing model, enabling the visual representation output by the image processing model to incorporate relevant auxiliary knowledge. This significantly improves the quality and richness of the visual representation, thereby enhancing the image processing model's representation ability of the first modality image.

[0052] In some implementations, the first sample image includes a first image patch and a second image patch. Accordingly, the aforementioned self-supervised training of the initial network based on the first sample image to obtain an intermediate network may include: masking the first image patch with a preset image patch to obtain a masked first sample image; inputting the masked first sample image into the initial network to obtain a reconstructed image, the reconstructed image including the second image patch and the reconstruction result of the first image patch; and training the initial network using a first loss value to obtain an intermediate network, the first loss value including the loss value between the reconstructed image and the first sample image.

[0053] In some implementations, such as Figure 5As shown, self-supervised training is denoted as Q0. The initial network may include an image encoder (denoted as Enc) and an image decoder (denoted as Dnc). The image encoder performs multi-layer convolution or self-attention processing on the input image, outputting encoded features at at least one scale (for ease of distinction, the encoded features in Q0 can be referred to as the third encoded features). The image decoder performs upsampling and convolution processing on the encoded features, outputting a prediction result with the same size as the first sample image (in Q0, this prediction result is the reconstructed image). The first sample image can be divided into multiple image blocks according to a preset size. Multiple image blocks are randomly selected from these multiple image blocks as the first image block, or image blocks matching the structured occlusion method (e.g., grid, strip, etc.) are selected from these multiple image blocks as the first image block, and image blocks other than the first image block are determined as the second image block. The first image block is occluded using the preset image blocks to obtain the occluded first sample image. The occluded first sample image is input into the image encoder of the initial network, and the image encoder extracts features from the occluded first sample image to obtain the third encoded features. The image decoder of the initial network is used to reconstruct the third encoded features to obtain a reconstructed image. A first loss value is determined to reflect the difference between the reconstructed image and the first sample image. This first loss value is used to update the network parameters of the initial network, and the next training round is performed until the first training termination condition is met, resulting in a trained intermediate network.

[0054] In other words, the first image patch can be a series of consecutive image patches or a series of discrete image patches. The preset image patch is a template used to cover the first image patch. The preset image patch can be a rectangular block of a fixed size or any other arbitrary shape. The pixel values ​​of the preset image patch can be zero or other feature values. The masking process hides the visual information of the first image patch, allowing the network to learn to predict the content of the masked area based on contextual information. The first loss value can be any of the mean squared error (MSE) loss, L1 loss, or perceptual loss. The first training termination condition can be reaching a preset number of training rounds or the first loss value being less than a loss threshold.

[0055] For example, the first sample image is divided into multiple 16×16 pixel image blocks. 50% of these blocks are randomly selected as the first image block for masking, and the remaining 50% are retained as the second image block. The masked first sample image is input into the image encoder of the initial network for feature extraction, and the image decoder of the initial network predicts the content of the masked first image block, outputting a reconstructed image. The mean squared error between the reconstructed image and the original first sample image is calculated as the first loss value, and this loss value is used to update the parameters of the initial network.

[0056] Therefore, by masking the first sample image, the network learns to predict the content of the masked area based on the image context information, thus enabling the network to grasp the inherent structural features of the first sample image. This self-supervised training method eliminates the need for manually labeled data, reducing training costs, and enables the network to extract basic structural features of target entities, laying the foundation for subsequent target training.

[0057] To integrate various auxiliary knowledge from entity description information into the image processing model, thereby achieving lightweight inference, in some implementations, target training includes multiple training stages. Accordingly, the aforementioned target training of the intermediate network based on the first sample image and entity description information to obtain the image processing model may include: For any training stage: input the first sample image into the intermediate network corresponding to the training stage to obtain the first prediction result of the first sample image, which is used to characterize the content of the first sample image; input the entity description information corresponding to the training stage into the auxiliary network corresponding to the training stage to obtain the auxiliary features corresponding to the entity description information; train the intermediate network corresponding to the training stage using the first prediction result and the auxiliary features; if the training stage is the last training stage, obtain the image processing model based on the trained intermediate network.

[0058] The target training can include multiple sequentially executed training phases, each introducing different entity description information, enabling the intermediate network to gradually learn different types of knowledge. Accordingly, for any training phase, a first sample image can be input into the intermediate network to be trained for that phase to obtain a first prediction result for the first sample image. The entity description information corresponding to that training phase is synchronously or asynchronously input into the auxiliary network corresponding to that phase to obtain auxiliary features corresponding to the entity description information. Based on the first prediction result and auxiliary features, a target loss value is determined, and the network parameters of the intermediate network corresponding to that training phase are updated according to the target loss value until a second training termination condition is met, resulting in the intermediate network that has completed training for that phase. If this is the last training phase, the image decoders in each auxiliary network and the trained intermediate network are discarded, and the image encoder in the trained intermediate network is determined as the image processing model.

[0059] The auxiliary network varies depending on the entity description information, and the second training termination condition can be the same as or different from the aforementioned first training termination condition. For example, both the first and second training termination conditions can be the number of preset training rounds; or, for instance, the first training termination condition is the number of preset training rounds, and the second training termination condition is that the target loss value is less than a loss value threshold.

[0060] Therefore, by gradually introducing different entity description information in multiple training stages, the image encoder can gradually integrate various auxiliary knowledge, thereby improving the visual representation quality of the trained image processing model. Furthermore, by retaining only the image encoder and discarding the auxiliary networks and image decoder after training, the model achieves multi-source collaboration during training and efficient single-modal execution during inference.

[0061] In order to extract effective feature representations and segmentation results from the first sample image to complete target training, in some embodiments, the aforementioned first prediction result includes first encoded features and a predicted segmentation result for the target entity. Accordingly, the aforementioned inputting the first sample image into the intermediate network corresponding to the training stage to obtain the first prediction result of the first sample image may include: inputting the first sample image into the image encoder in the intermediate network corresponding to the training stage, and using the image encoder to extract features from the first sample image to obtain the first encoded features; and using the image decoder in the intermediate network corresponding to the training stage to decode the first encoded features to obtain the predicted segmentation result.

[0062] For any training stage, the first sample image is input into the image encoder in the intermediate network corresponding to that training stage. The image encoder extracts features from the first sample image and outputs the first encoded features. These first encoded features are then input into the image decoder in the intermediate network corresponding to that training stage. The image decoder decodes the first encoded features and outputs the predicted segmentation result. The first encoded features can be a single-scale feature map or a multi-scale feature map. When using multi-scale feature maps, it can include multiple layers of feature maps from high resolution to low resolution. Shallow feature maps have high resolution and contain rich spatial detail information. Deep feature maps have low resolution and contain abstract semantic information. The predicted segmentation result can be a multi-channel probability map, where each channel corresponds to a category (e.g., background, anomalous object, reference object), and the value of each pixel represents the probability that the pixel belongs to that category. The predicted segmentation result can also be a binary segmentation mask used to identify the region where the target entity is located.

[0063] For example, in a medical diagnostic scenario, the first sample image is an NCCT image. The NCCT image is input to an image encoder, which, through multi-layer convolution and downsampling operations, outputs six feature maps at different scales, constituting the first encoded features. These first encoded features are then input to an image decoder, which, through upsampling and skip connection operations, outputs a 3-channel segmentation probability map of the same size as the first sample image, corresponding to the three categories: background, colorectal region (i.e., reference object), and lesion (i.e., abnormal object). It should be noted that the specific operation process of the image encoder and image decoder is not limited to this and can be configured as needed in practical applications.

[0064] Therefore, by extracting the first encoded features containing multi-scale feature maps through the image encoder, both spatial details and semantic information are taken into account; by decoding the first encoded features into the predicted segmentation result through the image decoder, the localization and shape description of the target entity are realized; that is, by obtaining the first encoded features and the predicted segmentation result, an effective data foundation is provided for completing the training of the corresponding training stage.

[0065] In some implementations, the target training may include multiple training phases that are executed sequentially as a first training phase, a second training phase, and a third training phase. For ease of distinction, such as... Figure 5 and Figure 6 As shown, let the aforementioned self-supervised training be denoted as Q0, the first execution stage as Q1, the second training stage as Q2, and the third training stage as Q3. Then, by executing Q0, Q1, Q2, and Q3 sequentially, the image processing model is obtained. The differences between the three training stages are described in detail below.

[0066] like Figure 5 As shown, the first training phase includes the first training process (denoted as...). In the case of the first training phase, the aforementioned predicted segmentation result includes a first predicted segmentation map for the target entity, entity description information including a second sample image and a labeled segmentation map for the target entity, the contrast of the target entity in the second sample image being higher than its contrast in the first sample image, the auxiliary network including a teacher model, and auxiliary features including second encoded features and a second predicted segmentation map for the target entity. Correspondingly, the aforementioned inputting the entity description information corresponding to the training phase into the auxiliary network corresponding to the training phase to obtain the auxiliary features corresponding to the entity description information may include: inputting the second training sample into the image encoder of the teacher model, and using the image encoder of the teacher model to extract features from the second sample image to obtain the second encoded features; and using the image decoder of the teacher model to decode the second encoded features to obtain the second predicted segmentation map.

[0067] For example, in medical diagnostic scenarios, the comparison between lesion structures and organ structures in CECT images. SpendThe contrast of the lesion and organ structures is higher than that in NCCT images. To transfer the enhanced contrast information to the intermediate network, a second sample image and a labeled segmentation map of the target entity are introduced as entity description information in the first training phase. The teacher model is used to extract the second encoded features and the second predicted segmentation map from the second sample image. Distillation training enables the intermediate network to learn the enhanced contrast information in the second sample image. That is, in the first training phase, the first sample image is input into the image encoder of the intermediate network corresponding to the first training phase for feature extraction, and the first encoded features are output. The first encoded features are input into the image decoder of the intermediate network corresponding to the first training phase for decoding, and the first predicted segmentation map is output. The second sample image is input into the image encoder (denoted as Ece) of the teacher model for feature extraction, and the second encoded features are output. The second encoded features are input into the image decoder (denoted as Dce) of the teacher model for decoding, and the second predicted segmentation map is output.

[0068] The teacher model can be fully pre-trained on a relevant task involving the second modality image. This teacher model outputs high-quality second encoded features and a second predicted segmentation map. The parameters of the teacher model's image encoder and decoder are frozen during the first training phase and do not participate in gradient updates. Both the first and second predicted segmentation maps can be multi-channel probability maps, where the value of each pixel represents the probability that the pixel belongs to the background, anomaly, or reference object. The labeled segmentation map is segmentation data manually annotated by experts. This labeled segmentation map is used to identify the precise location and shape of the target entity in the first sample image, serving as the standard answer for training.

[0069] Understandably, each training phase includes multiple training rounds. In the first training phase, one training round corresponds to one first training process. In the first training round of the first training phase, the corresponding intermediate network is the intermediate network obtained through self-supervised training. In the second training round of the first training phase, the corresponding intermediate network is the intermediate network obtained after executing the first training round, and so on.

[0070] Therefore, by using the teacher model to extract the second coding features and the second predicted segmentation map of the second sample image as reference signals, the intermediate network can learn the structural and morphological knowledge of the target entity in the second sample image, thereby improving the intermediate network's ability to represent the target entity.

[0071] Furthermore, in the first training phase, the aforementioned intermediate network corresponding to the training phase is trained using the first prediction result and auxiliary features, including: determining a second loss value based on the first predicted segmentation map and the labeled segmentation map; determining a third loss value based on the first predicted segmentation map and the second predicted segmentation map; determining a fourth loss value based on the first encoding feature, the first predicted segmentation map, the second encoding feature, and the second predicted segmentation map; and training the intermediate network corresponding to the first training phase based on the second loss value, the third loss value, and the fourth loss value.

[0072] In some implementations, a second loss value can be determined based on the difference between the first predicted segmentation map and the labeled segmentation map. This second loss value can be Dice loss, cross-entropy loss, or a combination of both, and is used to enable the intermediate network to learn basic segmentation capabilities. A third loss value can be determined based on the difference between the first and second predicted segmentation maps. This third loss value can be a Kullback-Leibler (KL) divergence loss, a mean squared error loss, or an L1 loss (absolute error loss), and is used to make the segmentation results of the intermediate network approximate the segmentation results of the teacher model, learning the prediction knowledge in the teacher model. Furthermore, a fourth loss value can be determined based on the first encoded features, the first predicted segmentation map, the second encoded features, and the second predicted segmentation map. This fourth loss value is used to enable the intermediate network to learn the topological relationship between abnormal objects and reference objects. The second, third, and fourth loss values ​​are added together to obtain a target loss value, which is then used to train the intermediate network corresponding to the first training phase. The process for determining the fourth loss value is described in detail below.

[0073] Therefore, by determining the second loss value, the intermediate network can learn basic segmentation capabilities; by determining the third loss value, the intermediate network can learn the prediction details and uncertainty estimates of the teacher model; and by determining the fourth loss value, the intermediate network can learn the topological relationship between outliers and reference objects. Joint optimization of these three factors enables the intermediate network to simultaneously improve representation quality across multiple dimensions.

[0074] In some implementations, determining the fourth loss value based on the first coding feature, the first predicted segmentation map, the second coding feature, and the second predicted segmentation map may include: determining a first offset feature of the abnormal object relative to a reference object based on the first coding feature and the first predicted segmentation map; determining a second offset feature of the abnormal object relative to the reference object based on the second coding feature and the second predicted segmentation map; and determining the fourth loss value based on the first offset feature and the second offset feature.

[0075] As mentioned earlier, the first and second predicted segmentation maps can be multi-channel probability maps, where the value of each pixel represents the probability that the pixel belongs to the background, an abnormal object, or a reference object. Taking the example of the first and second encoded features including multi-scale feature maps, the scale and number of feature maps included in the first encoded features are the same as those included in the second encoded features. Accordingly, for any scale in the multi-scale, based on the feature map corresponding to that scale in the first encoded features and the first predicted segmentation map, the object features of the abnormal object in the first sample image (denoted as the fourth object feature for ease of description and differentiation) and the object features of the reference object (denoted as the fifth object feature for ease of description and differentiation) can be determined. Based on the fourth and fifth object features, the first offset feature of the abnormal object relative to the reference object is determined. Based on the feature map corresponding to that scale in the second encoded features and the second predicted segmentation map, the object features of the abnormal object in the second sample image (denoted as the sixth object feature for ease of description and differentiation) and the object features of the reference object (denoted as the seventh object feature for ease of description and differentiation) are determined. Based on the sixth and seventh object features, the second offset feature of the abnormal object relative to the reference object is determined. The fourth loss value is determined based on the first and second offset features.

[0076] More specifically, a first probability map of the channel corresponding to the abnormal object and a second probability map of the channel corresponding to the reference object can be obtained from the first predicted segmentation map. For any scale, the spatial resolution of the first probability map is adjusted to the spatial resolution corresponding to that scale to obtain a first abnormal object mask. The spatial resolution of the second probability map is adjusted to the spatial resolution corresponding to that scale to obtain a first reference object mask. For any spatial location (i.e., pixel) of the feature map at that scale in the first encoded features, the feature vector of that spatial location is multiplied by the probability of the corresponding location in the first abnormal object mask to obtain a new first feature map. The feature vectors of each spatial location in this new first feature map are summed to obtain a first cumulative feature vector. The probabilities in the first abnormal object mask are summed to obtain a first cumulative probability. The first cumulative feature vector is divided by the first cumulative probability to obtain the fourth object feature of the abnormal object in the first sample image. Furthermore, for any spatial location of the feature map at that scale in the first encoded features, the feature vector of that spatial location is multiplied by the probability of the corresponding location in the first reference object mask to obtain a new second feature map. The feature vectors of each spatial location in this new second feature map are summed to obtain a second cumulative feature vector. The probabilities in the first reference object mask are summed to obtain the second cumulative probability. The second cumulative feature vector is divided by the second cumulative probability to obtain the fifth object feature of the reference object in the first sample image. The fourth object feature is subtracted from the fifth object feature to obtain the first offset feature of the anomalous object relative to the reference object at this scale. Similarly, the sixth object feature of the anomalous object in the second sample image and the seventh object feature of the reference object in the second sample image can be obtained. The sixth object feature is subtracted from the seventh object feature to obtain the second offset feature of the anomalous object relative to the reference object at this scale. The similarity (e.g., cosine similarity) between the first and second offset features is determined, and the difference between 1 and this similarity is used as the first sub-loss corresponding to this scale. The average of the first sub-losses corresponding to each scale is determined as the fourth loss value.

[0077] In this diagram, any probability in the first probability map represents the probability that the corresponding spatial location belongs to an anomalous object, and any probability in the second probability map represents the probability that the corresponding spatial location belongs to a reference object. The first offset feature can also be called the first direction vector, and the second offset feature can also be called the second direction vector. Each offset feature represents the direction and magnitude of feature change from the reference object to the anomalous object, capturing the comparative differences between the anomalous object and the reference object.

[0078] For example, if the spatial resolution of the first sample image is 512*512, then the spatial resolution of the first predicted segmentation map is also 512*512, and the spatial resolutions of the first probability map and the second probability map are also 512*512. If the spatial resolution corresponding to the current scale is 32*32, that is, the spatial resolution of the feature map at the current scale is 32*32, then the first probability map and the second probability map can be reduced to 32*32 respectively by means of nearest neighbor interpolation, to obtain the first abnormal object mask and the first reference object mask, and then subsequent processing can be performed.

[0079] Let the first object feature be denoted as The second object feature is denoted as The third object feature is denoted as The fourth object feature is denoted as Then the first offset feature = - Second offset feature = - The first loss =1-cos( , ), where cos represents cosine similarity, and the smaller the first sub-loss, the closer the first offset feature of the student model is to the second offset feature of the teacher model.

[0080] Furthermore, the aforementioned second loss value is denoted as... The third loss value is denoted as So, the target loss value in the first training phase .

[0081] Therefore, by determining the offset features of the anomalous object relative to the reference object, and by making the offset features of the student model approximate those of the teacher model, the intermediate network can learn the relative positional relationship and feature contrast differences between the anomalous object and the reference object. Even if the boundary of the anomalous object is unclear in the image of the first modality, the intermediate network can accurately locate the anomalous object by capturing the feature differences between the anomalous object and the surrounding reference objects.

[0082] In the case of the second training phase, to prevent the network from forgetting the knowledge learned in previous training stages, such as... Figure 5As shown, the second training phase includes a first training process and a second training process (denoted as q2) executed sequentially. The first training process is the same as the first training phase, while the second training process is a new addition to the second training phase, used to introduce surface-level text analysis for visual-semantic space alignment. In the second training process, the predicted segmentation result includes a first predicted mask of the reference object, and the auxiliary network includes a first post-processing module and a first text processing model. Accordingly, in the second training process, the aforementioned input of entity description information corresponding to the training phase into the auxiliary network corresponding to the training phase to obtain auxiliary features includes: using a first post-processing module to obtain the first object feature of the reference object from the first encoding features according to the first prediction mask; using the first post-processing module to perform attention calculation on the first object feature and multiple preset category features of each category to obtain multiple target category features, where each target category feature represents the visual representation of the reference object in the corresponding category; using a first text processing model to extract features from multiple descriptive texts to obtain the first semantic features corresponding to each of the multiple objects, where the multiple objects include the reference object and the associated objects of the reference object, and the multiple descriptive texts include entity description information and descriptive text used to describe the associated objects; wherein, the auxiliary features include multiple target category features and the first semantic features corresponding to each of the multiple objects.

[0083] like Figure 5As shown, before entering the second training stage, the parameters of the first 1 / 3 of the image encoder layers in the intermediate network corresponding to the second training stage can be configured as non-updateable, i.e., frozen, to train the other network layers of the image encoder. Then, the training operation of the second training stage is executed, i.e., the first training process is executed first, and the second training process is executed after the first training process is completed. In the second training process, the first sample image is input into the image encoder of the current intermediate network for feature extraction to obtain the first encoded feature, which includes feature maps of multiple scales; the image decoder of the intermediate network is used to decode the first encoded feature to obtain the first prediction mask of the reference object. The first prediction mask is a single-channel probability map, and the value of each pixel represents the probability that the pixel belongs to the reference object. Then, through the first post-processing module, for any scale among the multiple scales, the first prediction mask is scaled to the spatial resolution corresponding to that scale to obtain a new first prediction mask. For any spatial location of the feature map of that scale in the first encoded feature, the feature vector of that spatial location is multiplied by the probability of the corresponding location in the new first prediction mask to obtain a new third feature map. The feature vectors of each spatial location in the new third feature map are summed to obtain the third cumulative feature vector. The probabilities in the new first prediction mask are summed to obtain the third cumulative probability. The third cumulative feature vector is divided by the third cumulative probability to obtain the sub-object features of the reference object at this scale. The sub-feature objects at each scale are concatenated or fused to obtain the first object features of the reference object. Preset category features of multiple categories are used as learnable query tokens, and attention is calculated between the query tokens and the first object features to obtain multiple target category features. At the same time, a first report containing descriptive text of multiple objects is input into the first text processing model, and the first semantic features corresponding to the multiple objects are output. The number of preset category features is consistent with the number of multiple objects in the first report. The specific process of attention calculation can be referred to in related technologies, and will not be detailed in this application. That is to say, in the second training process, the entity description information can be the descriptive text of the reference object (which can be called the first descriptive text for easy distinction), and the first descriptive text is used to describe the structure of the reference object.

[0084] For example, in a medical diagnostic scenario, the first report can be a radiology report corresponding to an NCCT image. Since multiple organ structures can be examined during a single examination, these multiple organ structures include a reference organ structure, and the other organ structures besides the reference organ structure are considered as associated objects. The radiology report includes descriptive text of the examined organ structures, with each descriptive text describing the size, shape, density, boundaries, etc., of the corresponding organ structure. A first text processing model can extract features from the multiple descriptive texts in the radiology report and output first semantic features corresponding to each of the multiple organ structures.

[0085] Therefore, by determining the first object feature of the reference object, extracting the target category feature of the reference object from the first object feature based on attention calculation, and extracting the first semantic feature from multiple descriptive texts, the alignment of visual features and semantic features can be achieved based on subsequent loss calculation, thereby enabling the intermediate network to learn to understand the high-level semantic information in the descriptive text and enhance the semantic richness of the visual representation.

[0086] Furthermore, in the case of a second training phase, training of the second training process is performed using at least one first sample image. Accordingly, the aforementioned training of the intermediate network corresponding to this training phase using the first prediction result and auxiliary features may include: for any first sample image, aggregating multiple target category features corresponding to the first sample image to obtain aggregated category features, and aggregating multiple first semantic features corresponding to the first sample image to obtain aggregated semantic features; for any aggregated category feature, combining the aggregated category feature with each aggregated semantic feature to obtain multiple first feature combinations; for any first feature combination, determining the fifth loss value of the aggregated category feature and the aggregated semantic feature in the first feature combination; and training the intermediate network corresponding to the second training phase based on each fifth loss value and the sixth loss value obtained from performing the first training process.

[0087] In the second training process, B first sample images are used for training, where B is an integer greater than or equal to 1. Correspondingly, the entity description information includes B first reports corresponding one-to-one with the B first sample images. That is, there are B sample pairs in the second training process, each sample pair including a first sample image and its corresponding first report. For any sample pair, multiple target category features corresponding to the first sample image in the sample pair can be aggregated using methods such as average pooling, max pooling, or weighted summation to obtain aggregated category features. Similarly, multiple first semantic features of the first report in the sample pair are aggregated using the same aggregation method (e.g., average pooling) to obtain aggregated semantic features. The B aggregated category features can be denoted as the aggregated category feature set V = { }, the B aggregate semantic features can be denoted as the aggregate semantic feature set T={ For any aggregated category feature in the aggregated category feature set, the target category feature is combined with each aggregated semantic feature in the aggregated semantic feature set to obtain multiple first feature combinations. If the aggregated category feature and aggregated semantic feature in any first feature combination come from the same sample pair, the first feature combination can be called a positive first feature combination; otherwise, it can be called a negative first feature combination. For any first feature combination, the similarity (e.g., cosine similarity or inner product) between the aggregated category feature and the aggregated semantic feature in the first feature combination is used as the fifth loss value. The fifth loss value is used to shorten the distance between two aggregated features in a positive first feature combination and widen the distance between two aggregated features in a negative first feature combination. Furthermore, for any aggregated category feature, based on the fifth loss value of the positive first feature combination to which the aggregated category feature belongs, and the fifth loss values ​​of each first feature combination to which the aggregated category feature belongs, the first visual-to-text alignment probability corresponding to the aggregated category feature is determined (this can also be called the first alignment probability of the sample pair corresponding to the aggregated category feature, representing the probability that the model considers the first sample image in the sample pair to match the first report). Determine a first probability distribution containing the first alignment probability, which includes B probabilities, where the i-th probability is the first alignment probability, and i represents the position of the sample pair corresponding to the aggregated category feature among the B sample pairs. Generate a first one-hot encoding corresponding to the aggregated category feature, which is a vector of length B, where the i-th bit is 1 and the other bits are 0. Determine a first cross-entropy loss between the first alignment probability and the first one-hot encoding. For any aggregated semantic feature, based on the fifth loss value of the positive first feature combination to which the aggregated semantic feature belongs, i.e., the fifth loss value of each first feature combination to which the aggregated semantic feature belongs, determine the second text-to-visual alignment probability (which can also be called the second alignment probability of the sample pair corresponding to the aggregated semantic feature) corresponding to the aggregated semantic feature. Determine a second probability distribution containing the second alignment probability, where the i-th probability is the second alignment probability, and i represents the position of the sample pair corresponding to the aggregated semantic feature among the B sample pairs. Generate a second one-hot encoding of the aggregated semantic feature, which is a vector of length B, where the i-th bit is 1 and the other bits are 0. Determine the second cross-entropy loss between the second alignment probability and the second one-hot encoding. For any sample pair, add the first cross-entropy loss and the second cross-entropy loss corresponding to that sample pair to obtain the first target cross-entropy loss. Multiply the average of the first target cross-entropy losses for all sample pairs by a preset parameter to obtain the first semantic contrast loss. Add this first semantic contrast loss to the sixth loss value obtained from performing the first training process to obtain the target loss value for the second training stage. Train the intermediate network for the second training stage using the target loss value.

[0088] Using the inner product as the fifth loss value, the first alignment probability can be expressed as: .

[0089] in, , and and It is an integer. Let exp represent the first alignment probability of the i-th aggregated category feature, and let exp represent the exponent. This represents the fifth loss value of the positive first feature combination to which the i-th aggregated category feature belongs. Indicates the i-th aggregated category feature and the i-th aggregated category feature. The fifth loss value is the first feature combination composed of aggregated text features. This is a learnable temperature parameter.

[0090] The second alignment probability can be expressed as: .

[0091] in, This represents the second alignment probability of the i-th aggregated semantic feature. The fifth loss value represents the positive first feature combination containing the i-th aggregated semantic feature. Indicates the i-th aggregate semantic feature and the i-th... The fifth loss value is the first feature combination composed of aggregated category features. It should be noted that... The values ​​are equal.

[0092] The first semantic contrast loss can be expressed as: .

[0093] in, This represents the first semantic contrast loss. The value of the i-th bit (i.e., 1) in the first one-hot encoding corresponding to the aggregated category feature in the i-th sample pair is omitted in the formula for the first semantic contrast loss because the values ​​of other bits in the one-hot encoding are zero. This represents the value (i.e., 1) of the i-th bit in the second one-hot encoding corresponding to the aggregated semantic feature in the i-th sample pair. Since the values ​​of other bits in this one-hot encoding are zero, the corresponding terms are omitted in the formula for the first semantic contrast loss. This represents the cross-entropy between the first probability distribution of the aggregated category features in the i-th sample pair and its first one-hot encoding. The cross-entropy represents the relationship between the second probability distribution of the aggregated semantic features in the i-th sample pair and its second one-hot encoding. That is, the preset parameters.

[0094] Since the calculation process for the sixth loss value is the same as that for the target loss in the first training phase, the sixth loss value will be denoted as [value here]. ,So = ,in, This represents the target loss value for the second training phase.

[0095] Therefore, by determining the fifth loss value of the first feature combination, the distance between the aggregated visual features and aggregated text features in the positive first feature combination can be narrowed, while the distance between the aggregated visual features and aggregated text features in the negative first feature combination can be widened. Thus, determining the target loss value based on the fifth loss enables the intermediate network to learn to associate visual information with semantic descriptions, enhancing the semantic understanding ability of visual representations. Simultaneously, determining the target loss value based on the fifth loss value and the sixth loss value from the first training process can prevent the intermediate network from forgetting the knowledge learned during previous training.

[0096] In the case where the training phase is the third training phase, such as Figure 5 As shown, the third training phase includes the first training process, the second training process, and the third training process (denoted as q3) executed sequentially. The first training process is the same as the first training phase, the second training process is the same as the second training process in the second training phase, and the third training process is a newly added process used to introduce essential analysis text for visual-attribute space alignment. In the third training process, the predicted segmentation result includes the second predicted mask of the abnormal object and the third predicted mask of the reference object. The auxiliary network includes a second post-processing module and a second text processing model. Accordingly, in the third training process, the entity description information corresponding to the training phase is input into the auxiliary network corresponding to the training phase to obtain auxiliary features, which may include: Using a second post-processing module, the second object features of the abnormal object and the third object features of the reference object are fused to obtain fused features. The second object features are extracted from the first encoded features according to the second prediction mask, and the third object features are extracted from the first encoded features according to the third prediction mask. Using the second post-processing module, attention calculations are performed on the fused features and preset attribute features of multiple attributes to obtain multiple target attribute features. Each target attribute feature represents the visual representation of the abnormal object on the corresponding attribute. Using a second text processing model, feature extraction is performed on the entity description information corresponding to the third training stage to obtain multiple second semantic features corresponding to the abnormal object. The multiple second semantic features correspond one-to-one with multiple attributes. Among them, auxiliary features include multiple target attribute features and multiple second semantic features corresponding to the abnormal object.

[0097] In this context, the entity description information corresponding to the third training process can be the second description text of the abnormal object. This second description text describes the target entity's internal state, properties, level, and other deeper information. For example, in the field of medical diagnostics, the entity description information corresponding to the third training process can be the TNM staging description of the lesion in a pathology report. In the field of industrial inspection, the entity description information corresponding to the third training process can be the determination of the defect type and level in a defect analysis report.

[0098] like Figure 5As shown, before entering the third training stage, the parameters of the first two-thirds of the image encoder layers in the intermediate network corresponding to the third training stage can be configured as non-updatable, i.e., frozen, to train the other network layers of the image encoder. Then, the training operation of the third training stage is executed, i.e., the first and second training processes are executed sequentially, and after the second training process is completed, the third training process is executed. This involves inputting the first sample image into the image encoder of the current intermediate network for feature extraction to obtain the first encoded features, which include feature maps at multiple scales. The image decoder of the intermediate network is then used to decode the first encoded features to obtain a second prediction mask for anomalies and a third prediction mask for reference objects. Both the second and third prediction masks are single-channel probability maps. The value of each pixel in the second prediction mask represents the probability that the pixel belongs to an anomaly, and the value of each pixel in the third prediction mask represents the probability that the pixel belongs to a reference object. Through the second post-processing module, for any scale among the multiple scales, the second prediction mask is scaled to the spatial resolution corresponding to that scale to obtain a new second prediction mask. For any spatial location in the feature map of the first encoded feature at that scale, multiply the feature vector of that spatial location by the probability of the corresponding location in the new second prediction mask to obtain a new fourth feature map. Add the feature vectors of each spatial location in this new fourth feature map to obtain the fourth cumulative feature vector. Add the probabilities in the new second prediction mask to obtain the fourth cumulative probability. Divide the fourth cumulative feature vector by the fourth cumulative probability to obtain the sub-object features of the anomalous object at that scale. Concatenate or fuse the sub-object features of the anomalous object at each scale to obtain the second object features of the anomalous object. Similarly, perform the same operation on the feature maps in the third prediction mask and the first encoded feature to obtain the third object features of the reference object. Concatenate or weightedly sum the second and third object features to obtain the fused features. Use the preset attribute features of multiple attributes as learnable query tokens, and perform attention calculations on the query tokens and the fused features to obtain multiple target attribute features. Simultaneously, the second report containing the second descriptive text is input into the second text processing model. The second text processing model extracts features from the essential analysis text in the second report to obtain multiple second semantic features of the abnormal object. These multiple second semantic features correspond one-to-one with multiple attributes. The second text processing model can be the same model as the aforementioned first text processing model, or it can be a different model.

[0099] For example, in a medical diagnostic scenario, multiple attributes representing the lesion structure, such as T-staging, N-staging, and M-staging, are used to characterize the visual representation of the abnormal object at T-staging, N-staging, and M-staging, respectively. Multiple (i.e., three) second semantic features are used to characterize the diagnostic description of the abnormal object at T-staging, N-staging, and M-staging, respectively.

[0100] Therefore, by fusing the features of the extracted anomalous objects and reference objects, and using attention computation to extract the visual representation of the anomalous objects on multiple attributes from the fused features, as well as extracting multiple second semantic features of the anomalous objects from the essence analysis text, the visual representation of the anomalous objects can be aligned with the second semantic features based on subsequent loss calculations. This enables the intermediate network to learn to understand the essential information in the essence analysis text and enhances the essential relevance of the visual representation.

[0101] Furthermore, the third training phase has at least one sample pair that is similar to the second training phase. Each sample pair includes a first sample image and its corresponding second report. Each sample pair corresponds to multiple target attribute features and multiple second semantic features. Accordingly, the method further includes: for any target attribute feature of any sample pair, combining the target attribute feature with the multiple second semantic features corresponding to the sample pair to obtain multiple second feature combinations. For any second feature combination, determining a seventh loss value for the target attribute feature and the second semantic features in the second feature combination; and determining the target loss value for the third training phase based on each seventh loss value and an eighth loss value obtained from executing the first and second training processes.

[0102] When the target attribute feature in any second feature combination corresponds to the same attribute as the second semantic feature, the second feature combination can be called a positive second feature combination; otherwise, it is called a negative second feature combination. For any target attribute feature of any sample pair, determine the third alignment probability from visual to text (also called the third alignment probability of the sample pair, representing the probability that the model considers the first sample image in the sample pair to match the second report) and the third one-hot encoding corresponding to that target attribute feature. Determine the third cross-entropy loss between the third alignment probability and the third unique encoding. For any second semantic feature corresponding to any sample pair, determine the fourth alignment probability from text to visual (also called the fourth alignment probability of the sample pair) and the fourth one-hot encoding corresponding to that second semantic feature. Determine the fourth cross-entropy loss between the fourth alignment probability and the fourth unique encoding. For any sample pair, add the third cross-entropy loss and the fourth cross-entropy loss corresponding to that sample pair to obtain the second target cross-entropy loss. Multiply the average of the second target cross-entropy losses of each sample pair by a preset parameter to obtain the second semantic contrast loss. The average value of the second semantic contrast loss for each sample pair is determined, and this average value is added to the eighth loss value obtained from the first and second training processes to obtain the target loss value for the third training stage.

[0103] The process for determining the seventh loss value is the same as that for determining the fifth loss value, the process for determining the third alignment probability is the same as that for determining the first alignment probability, the process for determining the third one-hot encoding is the same as that for determining the first one-hot encoding, the process for determining the fourth alignment probability is the same as that for determining the second alignment probability, the process for determining the fourth one-hot encoding is the same as that for determining the second one-hot encoding, and the process for determining the second semantic contrast loss is the same as that for determining the first semantic contrast loss. All of these can be found in the relevant descriptions above, and the repetitions will not be repeated here.

[0104] Since the eighth loss value obtained from executing the first and second training processes is calculated using the same method as the target loss value in the aforementioned second training phase, the eighth loss value is denoted here as... ,So = ,in, This represents the average of the second semantic contrast loss for each sample pair. This represents the target loss value for the third training phase.

[0105] Therefore, by determining the seventh loss value for each combination of second features, the distance between visual features and text features of the same attribute can be narrowed, while the distance between visual features and text features of different attributes can be widened, thus enabling the intermediate network to learn to associate visual information with textual conclusions. By retaining the eighth loss value, the knowledge learned in previous training stages can be prevented from being forgotten, achieving gradual accumulation of knowledge.

[0106] Let's take a medical diagnostic scenario as an example to illustrate, such as... Figure 6As shown, in stage Q0, the image encoder (Enc) in the initial network is self-supervised using NCCT images, enabling Enc to learn the spatial structure of NCCT images and the location information of target entities, establishing basic visual perception capabilities. In stage Q1, CECT images and annotations registered with NCCT images are introduced. Through distillation training, the morphological knowledge of target entities in CECT images is transferred to Enc, enabling Enc to accurately identify target entities and segment and locate lesion structures and organ structures. In stage Q2, compared to stage Q1, radiology reports corresponding to NCCT images are added, bridging the visual and textual modalities, enabling Enc to understand radiology reports and accurately map textual semantics to specific lesion structures in NCCT images. In stage Q3, compared to stage Q2, pathology reports corresponding to NCCT images are added, improving Enc's clinical reasoning ability, enabling it to combine multi-source textual evidence (radiology + pathology) to output more accurate and clinically significant final diagnostic results. Furthermore, the model weights trained in each stage are transferred to the next adjacent stage. The Enc obtained from Q3 training is determined as the image processing model. During the inference process using the image processing model, only NCCT images are input to obtain visual representations with rich content, thereby improving the quality and richness of visual representations.

[0107] Figure 7 A flowchart illustrating an embodiment of the image processing method 700 provided in this application is shown. This method 700 can be applied to a server, for example, by... Figure 2 The server-side 202 execution is shown. (As shown...) Figure 7 As shown, the method 700 may include: Step S701: Obtain the target image containing the target entity; Step S702: Input the target image into the image processing model and obtain the visual representation of the target entity output by the image processing model. The image processing model is trained based on method 400.

[0108] During the inference phase, the image processing model can output a visual representation of the target entity in the input target image, providing intermediate data for subsequent image analysis, medical diagnosis and other processing.

[0109] Figure 8 A flowchart illustrating a computer-aided diagnostic method 800 provided in an embodiment of this application is shown. This method 800 can be applied to a server, for example, by... Figure 3 The server-side 302 execution is shown. (As shown...) Figure 8 As shown, the method 800 may include: Step S801: Acquire a non-enhanced computed tomography image containing the entity to be screened; Step S802: Input the non-enhanced computed tomography image into the image processing model to obtain the visual representation of the entity to be screened output by the image processing model. The image processing model is trained based on method 400. Step S803: Detect and screen entities for abnormalities based on visual representation.

[0110] Corresponding to the application scenario and method 400 provided in the embodiments of this application, the embodiments of this application also provide a training device for an image processing model, including: an acquisition module, used to acquire a first sample image and entity description information associated with the first sample image, wherein the entity description information is used to describe a target entity in the first sample image; a first training module, used to perform self-supervised training on an initial network based on the first sample image to obtain an intermediate network, wherein the intermediate network is at least used to extract structural features of the target entity; and a second training module, used to perform target training on the intermediate network based on the first sample image and the entity description information, and obtain an image processing model based on the trained intermediate network.

[0111] In some implementations, the first sample image includes a first image patch and a second image patch. The first training module is specifically used to: mask the first image patch using a preset image patch to obtain a masked first sample image; input the masked first sample image into the initial network to obtain a reconstructed image, the reconstructed image including the second image patch and the reconstruction result of the first image patch; and train the initial network using a first loss value to obtain the intermediate network, the first loss value including the loss value between the reconstructed image and the first sample image.

[0112] In some implementations, the target training includes multiple training stages, and the second training module is specifically used for: for any training stage: inputting the first sample image into the intermediate network corresponding to the training stage to obtain a first prediction result of the first sample image, wherein the first prediction result is used to characterize the content of the first sample image; inputting entity description information corresponding to the training stage into the auxiliary network corresponding to the training stage to obtain auxiliary features corresponding to the entity description information; training the intermediate network corresponding to the training stage using the first prediction result and the auxiliary features; and, if the training stage is the last training stage, obtaining the image processing model based on the trained intermediate network.

[0113] In some implementations, the intermediate network includes an image encoder and an image decoder, the first prediction result includes a first encoded feature and a predicted segmentation result for the target entity, and the second training module is further specifically used to: input the first sample image into the image encoder of the intermediate network corresponding to the training stage, and use the image encoder to extract features from the first sample image to obtain the first encoded feature; and use the image decoder of the intermediate network corresponding to the training stage to decode the first encoded feature to obtain the predicted segmentation result.

[0114] In some implementations, when the training phase is a first training phase, the first training phase includes a first training process, the predicted segmentation result includes a first predicted segmentation map for the target entity, the entity description information includes a second sample image and a labeled segmentation map for the target entity, the contrast of the target entity in the second sample image is higher than the contrast in the first sample image, the auxiliary network includes a teacher model, and the auxiliary features include a second encoded feature and a second predicted segmentation map for the target entity; correspondingly, the second training module is specifically used to: input the second training sample into the image encoder of the teacher model, and use the image encoder of the teacher model to extract features from the second sample image to obtain the second encoded feature; and use the image decoder of the teacher model to decode the second encoded feature to obtain the second predicted segmentation map.

[0115] In some implementations, the second training module is further configured to: determine a second loss value based on the first predicted segmentation map and the labeled segmentation map; determine a third loss value based on the first predicted segmentation map and the second predicted segmentation map; determine a fourth loss value based on the first encoding feature, the first predicted segmentation map, the second encoding feature, and the second predicted segmentation map; and train an intermediate network corresponding to the first training phase based on the second loss value, the third loss value, and the fourth loss value.

[0116] In some implementations, the target entity includes an anomalous object and a reference object, and the second training module is specifically used to: determine a first offset feature of the anomalous object relative to the reference object based on the first encoded feature and the first predicted segmentation map; determine a second offset feature of the anomalous object relative to the reference object based on the second encoded feature and the second predicted segmentation map; and determine the fourth loss value based on the first offset feature and the second offset feature.

[0117] In some implementations, when the training phase is a second training phase, the second training phase includes a first training process and a second training process executed sequentially. In the second training process, the target entity includes a reference object, the predicted segmentation result includes a first prediction mask of the reference object, and the auxiliary network includes a first post-processing module and a first text processing model. Accordingly, the second training module is used in the second training process to obtain a first object feature of the reference object from the first encoded features according to the first prediction mask using the first post-processing module; to perform attention calculation on the first object feature and multiple preset category features of multiple categories using the first post-processing module to obtain multiple target category features, where each target category feature represents the visual representation of the reference object in the corresponding category; and to perform feature extraction on multiple descriptive texts using the first text processing model to obtain first semantic features corresponding to multiple objects, where the multiple objects include the reference object and its associated objects, and the multiple descriptive texts include entity description information and descriptive text used to describe the associated objects. The auxiliary features include the multiple target category features and the first semantic features corresponding to the multiple objects.

[0118] In some implementations, the second training module is further configured to: for any first sample image, aggregate multiple target category features corresponding to the first sample image to obtain aggregated category features, and aggregate multiple first semantic features corresponding to the first sample image to obtain aggregated semantic features; for any aggregated category feature, combine the aggregated category feature with each aggregated semantic feature to obtain multiple first feature combinations; for any first feature combination, determine a fifth loss value between the aggregated category feature and the aggregated semantic feature in the first feature combination; and train the intermediate network corresponding to the second training stage based on each fifth loss value and a sixth loss value obtained by executing the first training process.

[0119] In some implementations, when the training phase is a third training phase, the third training phase includes a first training process, a second training process, and a third training process executed sequentially. In the third training process, the target entity includes an anomaly object and a reference object, the predicted segmentation result includes a second predicted mask of the anomaly object and a third predicted mask of the reference object, and the auxiliary network includes a second post-processing module and a second text processing model. Accordingly, the second training module is used in the third training process to fuse the second object features of the anomaly object and the third object features of the reference object using the second post-processing module to obtain fused features. The second object features are obtained according to the second... The prediction mask is extracted from the first encoded feature, and the third object feature is extracted from the first encoded feature according to the third prediction mask; using the second post-processing module, the fused feature is subjected to attention calculation with the preset attribute features of multiple attributes respectively to obtain multiple target attribute features, each of which represents the visual representation of the abnormal object on the corresponding attribute; using the second text processing model, feature extraction is performed on the entity description information corresponding to the third training process to obtain multiple second semantic features corresponding to the abnormal object, and the multiple second semantic features correspond one-to-one with the multiple attributes; wherein, the auxiliary features include the multiple target attribute features and the multiple second semantic features corresponding to the abnormal object.

[0120] Corresponding to the application scenario and method 700 provided in the embodiments of this application, the embodiments of this application also provide an image processing apparatus, including an acquisition module for acquiring a target image containing a target entity; and a processing module for inputting the target image into an image processing model to acquire a visual representation of the target entity output by the image processing model, wherein the image processing model is trained based on method 400.

[0121] Corresponding to the application scenario and method 800 provided in the embodiments of this application, the embodiments of this application also provide a computer-aided diagnostic device, including: an acquisition module, used to acquire a non-enhanced computed tomography (CT) image containing a screening entity; a processing module, used to input the non-enhanced CT image into an image processing model to obtain a visual representation of the screening entity output by the image processing model, wherein the image processing model is trained based on method 400; and a detection module, used to detect whether there is an abnormality in the screening entity based on the visual representation.

[0122] The functions of each module in the devices of this application embodiment can be found in the corresponding descriptions of the methods described above, and they have corresponding beneficial effects, which will not be repeated here. Furthermore, the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application solution according to actual needs.

[0123] Figure 9 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 9 As shown, the electronic device includes a memory 901 and a processor 902. The memory 901 stores a computer program that can run on the processor 902. When the processor 902 executes the computer program, it implements the method described in the above embodiments. The number of memories 901 and processors 902 can be one or more. In a specific implementation, the electronic device may also include a communication interface 903 for communicating with external devices and exchanging data.

[0124] In practical implementation, if the memory 901, processor 902, and communication interface 903 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0125] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0126] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0127] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0128] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0129] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in this application.

[0130] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0131] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0133] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0134] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0135] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0136] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0137] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0139] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training an image processing model, comprising: Obtain a first sample image and entity description information associated with the first sample image, wherein the entity description information is used to describe the target entity in the first sample image; The initial network is trained under self-supervised guidance based on the first sample image to obtain an intermediate network, which is used at least to extract the structural features of the target entity. Based on the first sample image and the entity description information, the intermediate network is trained to target the image, and an image processing model is obtained based on the trained intermediate network.

2. The method according to claim 1, wherein, The first sample image includes a first image patch and a second image patch. The step of performing self-supervised training on the initial network based on the first sample image to obtain an intermediate network includes: The first image block is masked using a preset image block to obtain the masked first sample image. The masked first sample image is input into the initial network to obtain a reconstructed image, which includes the second image patch and the reconstruction result of the first image patch; The initial network is trained using a first loss value to obtain the intermediate network, wherein the first loss value includes the loss value between the reconstructed image and the first sample image.

3. The method according to claim 1, wherein, The target training includes multiple training stages. The step of training the intermediate network based on the first sample image and the entity description information to obtain an image processing model includes: For any training stage: the first sample image is input into the intermediate network corresponding to the training stage to obtain a first prediction result of the first sample image, the first prediction result being used to characterize the content of the first sample image; the entity description information corresponding to the training stage is input into the auxiliary network corresponding to the training stage to obtain auxiliary features corresponding to the entity description information; the intermediate network corresponding to the training stage is trained using the first prediction result and the auxiliary features. If the training phase is the last training phase, the image processing model is obtained based on the intermediate network that has been trained.

4. The method according to claim 3, wherein, The intermediate network includes an image encoder and an image decoder. The first prediction result includes a first encoded feature and a predicted segmentation result for the target entity. The step of inputting the first sample image into the intermediate network corresponding to the training phase to obtain the first prediction result of the first sample image includes: The first sample image is input into the image encoder of the intermediate network corresponding to the training stage, and the image encoder is used to extract features from the first sample image to obtain the first encoded features. The first encoded feature is decoded using the image decoder of the intermediate network corresponding to the training phase to obtain the predicted segmentation result.

5. The method according to claim 4, wherein, When the training phase is the first training phase, the first training phase includes a first training process, the predicted segmentation result includes a first predicted segmentation map for the target entity, the entity description information includes a second sample image and a labeled segmentation map for the target entity, the contrast of the target entity in the second sample image is higher than the contrast in the first sample image, the auxiliary network includes a teacher model, and the auxiliary features include a second encoding feature and a second predicted segmentation map for the target entity. The step of inputting the entity description information corresponding to the training phase into the auxiliary network corresponding to the training phase to obtain the auxiliary features corresponding to the entity description information includes: The second sample image is input into the image encoder of the teacher model, and the image encoder of the teacher model is used to extract features from the second sample image to obtain the second encoded features; The second encoded feature is decoded using the image decoder of the teacher model to obtain the second predicted segmentation map.

6. The method according to claim 5, wherein, The step of training the intermediate network corresponding to the training phase using the first prediction result and the auxiliary features includes: The second loss value is determined based on the first predicted segmentation map and the labeled segmentation map; Based on the first predicted segmentation map and the second predicted segmentation map, determine the third loss value; The fourth loss value is determined based on the first coding feature, the first predicted segmentation map, the second coding feature, and the second predicted segmentation map; Based on the second loss value, the third loss value, and the fourth loss value, the intermediate network corresponding to the first training phase is trained.

7. The method according to claim 6, wherein, The target entity includes anomaly objects and reference objects. Determining the fourth loss value based on the first encoded feature, the first predicted segmentation map, the second encoded feature, and the second predicted segmentation map includes: Based on the first encoded feature and the first predicted segmentation map, a first offset feature of the abnormal object relative to the reference object is determined; Based on the second encoded feature and the second predicted segmentation map, a second offset feature of the abnormal object relative to the reference object is determined; The fourth loss value is determined based on the first offset feature and the second offset feature.

8. The method according to claim 4, wherein, When the training phase is the second training phase, the second training phase includes a first training process and a second training process executed sequentially. In the second training process, the target entity includes a reference object, the predicted segmentation result includes a first predicted mask of the reference object, and the auxiliary network includes a first post-processing module and a first text processing model. In the second training process, the step of inputting the entity description information corresponding to the training stage into the auxiliary network corresponding to the training stage to obtain auxiliary features includes: Using the first post-processing module, the first object feature of the reference object is obtained from the first encoded feature according to the first prediction mask; Using the first post-processing module, attention calculations are performed on the first object features and multiple preset category features to obtain multiple target category features. Each target category feature represents the visual representation of the reference object in the corresponding category. Using the first text processing model, feature extraction is performed on multiple descriptive texts to obtain first semantic features corresponding to multiple objects. The multiple objects include the reference object and the associated object of the reference object. The multiple descriptive texts include the entity description information and descriptive text used to describe the associated object. The auxiliary features include the plurality of target category features and the first semantic features corresponding to the plurality of objects respectively.

9. The method according to claim 8, wherein, The step of training the intermediate network corresponding to the training phase using the first prediction result and the auxiliary features includes: For any first sample image, multiple target category features corresponding to the first sample image are aggregated to obtain aggregated category features, and multiple first semantic features corresponding to the first sample image are aggregated to obtain aggregated semantic features; For any aggregation category feature, the aggregation category feature is combined with each aggregation semantic feature to obtain multiple first feature combinations; For any combination of first features, determine the fifth loss value of the aggregated category feature and the aggregated semantic feature in the first feature combination; Based on each fifth loss value and the sixth loss value obtained from executing the first training process, train the intermediate network corresponding to the second training stage.

10. The method according to claim 4, wherein when the training phase is a third training phase, the third training phase includes a first training process, a second training process, and a third training process executed sequentially, wherein in the third training process, the target entity includes an abnormal object and a reference object, the predicted segmentation result includes a second predicted mask of the abnormal object and a third predicted mask of the reference object, and the auxiliary network includes a second post-processing module and a second text processing model: In the third training process, the step of inputting the entity description information corresponding to the training stage into the auxiliary network corresponding to the training stage to obtain auxiliary features includes: Using the second post-processing module, the second object feature of the abnormal object and the third object feature of the reference object are fused to obtain fused features. The second object feature is extracted from the first coding feature according to the second prediction mask, and the third object feature is extracted from the first coding feature according to the third prediction mask. Using the second post-processing module, attention calculations are performed on the fused features and preset attribute features of multiple attributes to obtain multiple target attribute features. Each target attribute feature represents the visual representation of the abnormal object on the corresponding attribute. Using the second text processing model, feature extraction is performed on the entity description information corresponding to the third training process to obtain multiple second semantic features corresponding to the abnormal object, and the multiple second semantic features correspond one-to-one with the multiple attributes. The auxiliary features include the multiple target attribute features and the multiple second semantic features corresponding to the abnormal object.

11. An image processing method, comprising: Obtain the target image containing the target entity; The target image is input into an image processing model to obtain a visual representation of the target entity output by the image processing model, wherein the image processing model is trained based on the method of any one of claims 1 to 10.

12. A computer-aided diagnostic method, comprising: Acquire non-enhanced computed tomography images containing the entities being screened; The unenhanced computed tomography image is input into an image processing model to obtain a visual representation of the screened entity output by the image processing model, wherein the image processing model is trained based on the method of any one of claims 1 to 10; The presence of anomalies in the screened entity is detected based on the visual representation.

13. A computer-aided diagnostic system, comprising: A client is used to send an anomaly screening request, the anomaly screening request carrying a non-enhanced computed tomography image containing the screening entity, the anomaly screening request being used to detect whether the screening entity has an anomaly; The server is configured to receive the anomaly screening request from the client, input the non-enhanced computed tomography image into an image processing model, obtain a visual representation of the screening entity output by the image processing model, detect whether the screening entity has an anomaly based on the visual representation, and send the detection result to the client, wherein the image processing model is trained based on the method of any one of claims 1 to 10, and the detection result is used to characterize whether the screening entity has an anomaly. The client is also used to output the detection results.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

Images