Image processing method, computer-assisted cancer prognosis method and system

Through the image processing model combining pathological images and prompt information, using preset risk levels and pathological fusion characteristics, the problem of insufficient efficiency and accuracy of pathological image analysis in the prior art is solved, and automated disease prognosis analysis is realized.

CN118674725BActive Publication Date: 2024-11-05ALIBABA DAMO (HANGZHOU) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411166804.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-05
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

Existing disease prognosis analysis based on pathological images requires manual analysis by doctors, making it difficult to find fine-grained features and difficult to give quantitative results, resulting in insufficient analysis efficiency and accuracy.

Method used

By acquiring target pathological images and related prompt information, input an image processing model to generate prognostic results, and use preset risk level features and pathological fusion image features to improve the accuracy and efficiency of prognostic analysis.

Benefits of technology

Accurate disease prognosis results can be obtained without the need for a doctor to manually analyze pathological images, improving analysis efficiency and accuracy and simplifying the disease prognosis process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118674725B_ABST
    Figure CN118674725B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide an image processing method, a computer-assisted cancer prognosis method and system, wherein the image processing method comprises: obtaining a target pathology image of a target disease and at least one target prompt information of the target pathology image; inputting the target pathology image and each target prompt information into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained according to a preset risk level feature and a pathology fusion image feature, and the pathology fusion image feature is obtained according to the target pathology image feature of the target pathology image and the prompt coding feature corresponding to each target prompt information. By processing the target pathology image feature of the target pathology image, adding and combining the prompt coding feature of the target prompt information, and performing feature analysis on the target pathology image, the accuracy and efficiency of determining the prognosis result of the target disease using the image processing model are further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular to an image processing method. Background Art

[0002] Pathological diagnosis is the standard for diagnosing tumors, cancer and other related diseases. Pathological diagnosis using pathological tissue sections requires experienced doctors to make the diagnosis, and the entire pathological diagnosis process is relatively long. With the development of digital pathological image technology, a large number of pathological images have been scanned into digital images, so that the original pathological images can be expressed and stored in a digital form, so that digital pathological images can be intelligently analyzed, and visual image features can be identified from digital pathological images to better describe the disease and predict the patient's disease prognosis, which can effectively reduce the burden on doctors.

[0003] Existing disease prognosis analysis based on pathological images usually requires doctors to analyze pathological images to complete, but the amount of information contained in pathological images is large, and it is difficult for doctors to notice more fine-grained features, and it is difficult for doctors to give quantitative analysis results through pathological images. Therefore, how to improve the accuracy and analysis efficiency of disease prognosis analysis based on pathological images is crucial. Summary of the invention

[0004] In view of this, the embodiments of this specification provide an image processing method and a computer-assisted cancer prognosis method. One or more embodiments of this specification also relate to an image processing device, a computer-assisted cancer prognosis device and a computer-assisted cancer prognosis system, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an image processing method, including:

[0006] Acquire a target pathological image of a target disease and at least one target prompt information of the target pathological image;

[0007] The target pathological image and each target prompt information are input into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the target pathological image and the prompt coding feature corresponding to each target prompt information.

[0008] According to a second aspect of the embodiments of this specification, there is provided an image processing apparatus, including:

[0009] an acquisition module, configured to acquire a target pathological image of a target disease and at least one target prompt information of the target pathological image;

[0010] The input module is configured to input the target pathological image and each target prompt information into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on a target pathological image feature of the target pathological image and a prompt coding feature corresponding to each target prompt information.

[0011] According to a third aspect of the embodiments of this specification, a computer-assisted cancer prognosis method is provided, comprising:

[0012] Acquire a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image;

[0013] The panoramic slice image and each target prompt information are input into an image processing model to obtain a target prognosis result of the target cancer, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on a target pathological image feature of the panoramic slice image and a prompt coding feature corresponding to each target prompt information.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-assisted cancer prognosis device is provided, comprising:

[0015] An image acquisition module is configured to acquire a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image;

[0016] The image input module is configured to input the panoramic slice image and each target prompt information into an image processing model to obtain a target prognosis result of the target cancer, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on a target pathological image feature of the panoramic slice image and a prompt coding feature corresponding to each target prompt information.

[0017] According to a fifth aspect of the embodiments of this specification, a computer-assisted cancer prognosis system is provided, including a client and a server;

[0018] The client is configured to send a cancer prognosis instruction to the server, wherein the cancer prognosis instruction carries a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image;

[0019] The server is configured to input the panoramic slice image and each target prompt information into an image processing model to obtain a target prognosis result of the target cancer, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on a target pathological image feature of the panoramic slice image and a prompt coding feature corresponding to each target prompt information.

[0020] According to a sixth aspect of an embodiment of this specification, a computing device is provided, including:

[0021] Memory and processor;

[0022] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned image processing method or computer-assisted cancer prognosis method are implemented.

[0023] According to the seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned image processing method or computer-assisted cancer prognosis method are implemented.

[0024] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing method or computer-assisted cancer prognosis method.

[0025] An embodiment of the present specification realizes that the target pathological image is processed using an image processing model. In this process, the target pathological image and the target prompt information of the target pathological image are used as input data of the image processing model, so that the image processing model can obtain more features from the target prompt information, so as to process the target pathological image through the target pathological image features of the target pathological image and the prompt coding features of the target prompt information. After obtaining the pathological fusion image features, the target prognosis result of the target disease can be determined according to the preset risk level features in the image processing model. Therefore, in practical applications, it is not necessary for doctors to analyze the pathological image to determine the patient's disease prognosis, thereby improving the accuracy and efficiency of determining the disease prognosis; by processing the target pathological image features of the target pathological image, adding and combining the prompt coding features of the target prompt information, and performing feature analysis on the target pathological image, the accuracy and efficiency of determining the target disease prognosis using the image processing model are further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of an image processing method provided by an embodiment of this specification;

[0027] Figure 2 is a schematic diagram of a model structure of an image processing model provided by an embodiment of this specification;

[0028] Figure 3 It is a structural schematic diagram of an adapter provided by an embodiment of this specification;

[0029] Figure 4 is a structural schematic diagram of an adaptation unit provided by an embodiment of this specification;

[0030] Figure 5 is a structural diagram of an image encoder provided by an embodiment of this specification;

[0031] Figure 6 is a schematic diagram of the structure of a decoder provided by an embodiment of this specification;

[0032] Figure 7 is a processing flow chart of an image processing method provided by an embodiment of this specification;

[0033] Figure 8 is a structural schematic diagram of an image processing device provided by an embodiment of this specification;

[0034] Fig. 9 is a flowchart of a computer-assisted cancer prognosis method provided by one embodiment of this specification;

[0035] Fig.10 is a schematic diagram of the structure of a computer-assisted cancer prognosis device provided in one embodiment of this specification;

[0036] Fig.11 is a schematic diagram of the architecture of a computer-assisted cancer prognosis system provided in one embodiment of this specification;

[0037] Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0038] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0039] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0042] First, the terms involved in one or more embodiments of this specification are explained.

[0043] WSI: Whole Slide Image, panoramic slice image.

[0044] VPT: Visual Prompt Tuning, visual prompt optimization.

[0045] VPTSurv: Visual Prompt Tuning Survival Analysis, visual prompt optimization survival analysis model.

[0046] Prognosis: An empirical prediction of how a disease will develop.

[0047] Pathological diagnosis is the standard for diagnosing tumors, cancer and other related diseases. Pathological diagnosis using pathological tissue sections requires experienced doctors to make the diagnosis, and the entire pathological diagnosis process is relatively long. With the development of digital pathological image technology, a large number of pathological images have been scanned into digital images, so that the original pathological images can be expressed and stored in a digital form, so that digital pathological images can be intelligently analyzed, and visual image features can be identified from digital pathological images to better describe the disease and predict the patient's disease prognosis, which can effectively reduce the burden on doctors.

[0048] Existing disease prognosis analysis based on pathological images usually requires doctors to analyze the pathological images to complete, but the amount of information contained in the pathological images is large, and it is difficult for doctors to notice more fine-grained features, and it is difficult for doctors to give quantitative analysis results through pathological images.

[0049] In this specification, an image processing method and a computer-assisted cancer prognosis method are provided. This specification also relates to an image processing device, a computer-assisted cancer prognosis device and a computer-assisted cancer prognosis system, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0050] See also Figure 1 , Figure 1 A flowchart of an image processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0051] Step 102: Acquire a target pathological image of a target disease and at least one target prompt information of the target pathological image.

[0052] With the development of digital pathology image technology, in current practical applications, digital pathology images are often analyzed to obtain relevant information from them and determine the corresponding disease prognosis of patients in a certain disease. However, how to quickly and accurately obtain effective relevant information from gigapixel digital pathology images and use the obtained effective relevant information for disease prognosis analysis is still a challenge.

[0053] In one or more embodiments provided in the present specification, after obtaining the target pathology image of the target disease, target prompt information related to the target pathology image is also obtained. By combining the target pathology image with the target prompt information, more effective relevant information can be obtained, and the target pathology image and the target prompt information are subsequently processed, so that the target prognosis result of the target disease generated based on the target pathology image and the target prompt information is more accurate.

[0054] Among them, the target disease refers to any disease for which the patient needs to determine the prognosis, such as cancer, tumors and other diseases. The target pathological image refers to the pathological image of the target disease, such as the panoramic slice image (WSI) of cancer, the panoramic slice image of the tumor, etc. The target prompt information refers to the prompt information used to assist the target pathological image in determining the target prognosis result of the target disease. The target prompt information may specifically include pathological images at different scales, cell structure information, protein detection data or data of other modalities, etc. The target prompt information may be expressed in the form of images, text and other forms. The target prognostic result is the prognostic analysis result of the target disease, specifically including the corresponding survival probability of the patient at each preset risk level.

[0055] For example, the target disease is cancer, the target pathology image may be a 20x panoramic slice image under a microscope, the target prompt information may include a 10x panoramic slice image under a microscope, a 5x panoramic slice image under a microscope, a grayscale image of the cell structure of cells in the target pathology image, protein detection content data, etc., and the target prognosis result is the prognostic analysis result of cancer.

[0056] In practical applications, microscopic cell slices of the target disease can be sampled and the corresponding target pathological images can be obtained. The pathological images at different scales of the target pathological images can be determined as target prompt information. After the cell structure grayscale image and other modal information corresponding to the target pathological image are obtained, the cell structure grayscale image and other modal information can be determined together as target prompt information of the target pathological image.

[0057] Further, with respect to the cell structure information of the target pathological image, the cell structure information of the target pathological image is used as the target prompt information of the target pathological image. In the subsequent processing using the image processing model, the image processing model can obtain more structural information about the cells in the target pathological image from the cell structure information. Based on this, when the target prompt information is specifically the cell structure information of the cells in the target pathological image, it can be obtained by the following implementation method:

[0058] In a specific implementation manner provided in this specification, obtaining at least one target prompt information of the target pathological image includes:

[0059] Extracting edge structure information of a target object in the target pathological image;

[0060] The edge structure information is determined as target prompt information of the target pathological image.

[0061] The target object refers to the cells in the target pathological image, and the target object may include normal cells and abnormal cells, which are determined according to the actual application of the target pathological image. The edge structure information refers to the cell edge structure information of each cell in the target pathological image. The edge structure information may exist in the form of a cell structure grayscale image.

[0062] Specifically, after acquiring the target pathological image, edge detection is performed on the target pathological image to extract edge structure information of each target object in the target pathological image, and the edge structure information is determined as target prompt information of the target pathological image. In the case where the target object is specifically a cell in the target pathological image, after edge detection is performed on the target pathological image, cell edge structure information of each cell in the target pathological image can be extracted, specifically a cell structure grayscale image.

[0063] By performing edge detection on the target pathological image, the cell structure information of each cell in the target pathological image is obtained, so that the image processing model can better learn the cell structure information of each cell in the target pathological image from the cell structure information during the process of processing the target pathological image.

[0064] In one or more embodiments provided in the present specification, on the basis of acquiring the target pathological image, multiple target prompt information of the target pathological image is further acquired, so that in the process of performing image processing on the target pathological image using an image processing model, the target pathological image can be processed in combination with other effective information in the target pathological image to improve the accuracy of the output target prognosis result of the target disease.

[0065] Step 104: Input the target pathological image and each target prompt information into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the target pathological image and the prompt coding feature corresponding to each target prompt information.

[0066] After acquiring the target pathological image and target prompt information, the target pathological image and target prompt information are input into the image processing model to obtain the target prognosis result of the target disease. Among them, the image processing model is used to process the target pathological image and the target prompt information, and output the target prognosis result of the target disease. The preset risk level feature refers to the disease risk level feature preset in the image processing model, specifically the feature vector of the disease risk level corresponding to the target disease. The pathological fusion image feature refers to the image feature obtained by fusing the target pathological image feature and the prompt coding feature. The target pathological image feature refers to the coding feature obtained after encoding the target pathological image, and its specific expression form can be a feature vector; the prompt coding feature refers to the coding feature obtained after encoding the target prompt information, and its specific expression form can be a feature vector.

[0067] In a specific implementation manner provided in this specification, the target pathological image and each target prompt information are input into an image processing model to obtain a target prognosis result of the target disease, including:

[0068] Dividing the target pathological image into at least two pathological image blocks, performing feature extraction on each pathological image block, and obtaining pathological image block features of each pathological image block;

[0069] Input each target prompt information into the prompt encoder to obtain the prompt coding feature corresponding to each target prompt information;

[0070] Inputting each pathological image block feature and each prompt coding feature into the adapter to obtain a prompt mapping feature corresponding to each prompt coding feature;

[0071] Inputting the target pathological image and each prompt mapping feature into the image encoder to obtain pathological fusion image features;

[0072] The pathological fusion image features are input into the decoder to obtain the target prognosis result of the target disease.

[0073] Among them, the pathological image block refers to the image block obtained by dividing the target pathological image; the pathological image block feature refers to the feature obtained by extracting the feature of the pathological image block, and its specific expression form can be a feature vector. The prompt encoder is used to encode the target prompt information and output the prompt coding feature corresponding to the target prompt information; the adapter is used to perform feature mapping on the prompt coding feature, and map the prompt coding feature to the target pathological image feature space, so that the prompt coding feature has the same dimension as the target pathological image feature. The prompt mapping feature is the output data of the adapter, specifically the mapping feature of the prompt coding feature. The image encoder is used to encode the target pathological image, and fuse the target pathological image feature with the prompt mapping feature to obtain the pathological fusion image feature. The decoder is used to decode the pathological fusion image feature to obtain the target prognosis result of the target disease.

[0074] See also Figure 2 , Figure 2 FIG. 1 shows a schematic diagram of a model structure of an image processing model provided according to an embodiment of the present specification. Figure 2 As shown, the image processing model specifically includes an image encoder, an adapter, a prompt encoder and a decoder. In practical applications, the target pathological image is divided into multiple pathological image blocks, specifically divided into multiple pathological image blocks according to the area size of 16*16, and features are extracted for each pathological image block to obtain the pathological image block features of each pathological image block. Based on this, the acquired target prompt information is input into the prompt encoder to obtain the prompt coding features of each target prompt information, and the prompt coding features and the pathological image block features are input into the adapter to obtain the prompt mapping features corresponding to each prompt coding feature. The prompt mapping features and the acquired target pathological image are used together as the input data of the image encoder, and the prompt mapping features and the target pathological image are input into the image encoder to obtain the pathological fusion image features, and then the pathological fusion image features are input into the decoder to obtain the target prognosis result of the target disease.

[0075] In practical applications, since the acquired target pathological image is often a gigapixel-level pathological image, the target pathological image can be segmented before being input into the image encoder. For example, the target pathological image can be segmented into image blocks of 256*256 pixels, and the segmented pathological image blocks are used as input data of the image encoder and input into the image encoder with each prompt mapping feature, thereby obtaining the pathological fusion image features output by the image encoder. By segmenting the target pathological image, the image processing model can reduce the amount of computing resources used when the image processing model is running during the process of processing the target pathological image.

[0076] The following is a detailed explanation of the execution process of each part of the model architecture in the image processing model.

[0077] In a specific implementation provided in this specification, the prompt encoder includes at least two modality encoding layers;

[0078] Input each target prompt information into the prompt encoder to obtain the prompt coding features corresponding to each target prompt information, including:

[0079] Determine the modality type of each target prompt information;

[0080] Determine the modal coding layer corresponding to each target prompt information according to the modal type of each target prompt information;

[0081] Input each target prompt information into the corresponding modality encoding layer to obtain the prompt encoding features.

[0082] Since the acquired target prompt information belongs to different modalities, for example, images, texts, etc., in order to ensure that the acquired target prompt information can be unified into the same dimension, encoding layers of different modalities can be configured in the prompt encoder to realize encoding of target prompt information of different modalities.

[0083] The modality type refers to the specific modality type to which each target prompt information belongs, such as image, text, etc. The modality encoding layer is an encoding layer of different modalities configured in the prompt encoder, and the modality encoding layer can specifically be a visual prompt embedding vector encoding layer.

[0084] Specifically, after obtaining each target prompt information, determine the modal type corresponding to each target prompt information, and according to the modal type of each target prompt information, determine the modal coding layer required to be input for each target prompt information, input each target prompt information into the corresponding modal coding layer, and obtain the prompt coding features of each target prompt information.

[0085] For example, the target prompt information includes a 10x panoramic slice image under a microscope, a cell structure grayscale image of cells in the target pathological image, and protein detection content data. The modal types of the 10x panoramic slice image and the cell structure grayscale image are both images. Based on this, it can be determined that the modal coding layers corresponding to the 10x panoramic slice image and the cell structure grayscale image are both image coding layers. In practical applications, protein detection content data is often expressed as "protein content is...", and its modal type is text. Based on this, it can be determined that the modal coding layer corresponding to the protein detection content data is a text coding layer. The 10x panoramic slice image and the cell structure grayscale image are input into the image coding layer to obtain the corresponding prompt coding features, and the protein detection content data is input into the text coding layer to obtain the prompt coding features corresponding to the protein detection content data.

[0086] By configuring encoding layers of different modalities in the prompt encoder, when the target prompt information includes prompt information of multiple modalities, different target prompt information can be encoded separately to unify the prompt encoding features of each target prompt information into the same vector dimension.

[0087] In a specific implementation manner provided in this specification, the adapter includes n adapter units;

[0088] Inputting each pathological image block feature and each prompt coding feature into the adapter to obtain a prompt mapping feature corresponding to each prompt coding feature, including:

[0089] Inputting each pathological image block feature and each prompt coding feature into a first adaptation unit to obtain a first prompt mapping feature corresponding to each prompt coding feature;

[0090] Input the i-1th prompt mapping feature and each prompt coding feature into the i-th adaptation unit to obtain the i-th prompt mapping feature corresponding to each prompt coding feature, where 2≤i≤n;

[0091] Increment i by 1 and determine whether i is greater than n. If not, continue to execute the step of inputting the i-1th prompt mapping feature and each prompt coding feature into the i-th adaptation unit to obtain the i-th prompt mapping feature corresponding to each prompt coding feature. If so, obtain the prompt mapping feature corresponding to each prompt coding feature.

[0092] The adapter is configured with n adapter units, which are connected in sequence in the adapter. The first adapter unit is the first adapter unit in the adapter, and the first prompt mapping feature is the prompt mapping feature output by the first adapter unit.

[0093] See also Figure 3 , Figure 3 FIG. 2 shows a schematic diagram of the structure of an adapter provided according to an embodiment of the present specification. Figure 3 As shown, the adapter includes n adapting units, namely adapting unit A1, adapting unit A2, adapting unit A3, ..., adapting unit An. Adapting unit A1 is the first adapting unit.

[0094] Specifically, each pathological image block feature and each hint coding feature is input into the first adaptation unit, and the first hint mapping feature output by the first adaptation unit can be obtained. The first hint mapping feature and each hint coding feature are used as input data of the next adaptation unit and input into the next adaptation unit to obtain the hint mapping feature output by the next adaptation unit, and so on, until the adaptation unit is the nth adaptation unit, and the hint mapping feature output by the nth adaptation unit is obtained.

[0095] Further, Figure 3The structure of the adapter shown is described as an example. Each pathological image block feature and each prompt coding feature are input into the adaptation unit A1 to obtain the prompt mapping feature F1 output by the adaptation unit A1; the prompt mapping feature F1 and each prompt coding feature are used as input data of the adaptation unit A2 and input into the adaptation unit A2 to obtain the prompt mapping feature F2 output by the adaptation unit A2; the prompt mapping feature F2 and each prompt coding feature are used as input data of the adaptation unit A3 and input into the adaptation unit A3 to obtain the prompt mapping feature F3 output by the adaptation unit A3; ... until the prompt mapping feature Fn output by the adaptation unit An is obtained.

[0096] Furthermore, in a specific implementation manner provided in this specification, each pathological image block feature and each prompt coding feature are input into a first adaptation unit to obtain a first prompt mapping feature corresponding to each prompt coding feature, including:

[0097] Each pathological image block feature is determined as a query vector, each prompt coding feature is determined as a key-value pair vector, attention calculation is performed on each pathological image block feature and each prompt coding feature, and a first prompt mapping feature corresponding to each prompt coding feature is obtained.

[0098] Specifically, in the adaptation unit, the prompt mapping feature corresponding to each prompt coding feature can be generated through attention calculation. In the first adaptation unit, each input pathological image feature is determined as a query vector, each prompt coding feature is determined as a key-value pair vector, and the first prompt mapping feature corresponding to each prompt coding feature is obtained through attention calculation. In other adaptation units, the prompt mapping feature output by the previous adaptation unit can be determined as a query vector, each prompt coding feature is determined as a key-value pair vector, and the prompt mapping feature corresponding to each prompt coding feature is obtained through attention calculation.

[0099] Further, see Figure 4 , Figure 4 A schematic diagram of the structure of an adaptation unit provided according to an embodiment of the present specification is shown. Although the vector dimensions of each prompt coding feature obtained by the prompt encoder are unified to the same vector dimension, their vector dimensions are lower than the vector dimensions of each pathological image block feature. In order to reduce the consumption of computer computing resources and reduce the computing workload, before performing the attention calculation, the dimension of each pathological image block feature can be reduced, and then the subsequent attention calculation can be performed to achieve lightweight calculation.

[0100] like Figure 4As shown, the adaptation unit includes a downsampling fully connected network layer, a cross attention layer and an upsampling fully connected network layer. The downsampling fully connected network layer can be used to perform dimensionality reduction processing on each pathological image block feature, the cross attention layer can be used to perform attention calculation on each pathological image block feature and each prompt coding feature, and the upsampling fully connected network layer can be used to perform dimensionality increase processing on the calculated prompt mapping feature, that is, to restore the calculated prompt mapping feature from the vector dimension after dimensionality reduction to the vector dimension of the pathological image block feature. Specifically, the pathological image block feature is subjected to dimensionality reduction processing by the downsampling fully connected network layer to obtain the pathological image block feature after dimensionality reduction, the pathological image block feature after dimensionality reduction is input as a query vector into the cross attention layer, each prompt coding feature is input as a key-value pair vector into the cross attention layer, and attention calculation is performed to obtain the initial prompt mapping feature, and the initial prompt mapping feature and the pathological image block feature after dimensionality reduction are fused by adding to obtain the fused prompt mapping feature. The fused prompt mapping feature is input into the upsampling fully connected network layer, and the fused prompt mapping feature is subjected to dimensionality increase processing to obtain the fused prompt mapping feature after dimensionality increase, that is, the prompt mapping feature.

[0101] The process of generating prompt mapping features by other adaptation units is similar to the above process, and will not be described in detail in this specification.

[0102] Further, in a specific implementation provided in this specification, the image encoder includes m image coding layers, and the adapter includes n adaptation units;

[0103] Inputting the target pathological image and each prompt mapping feature into the image encoder to obtain pathological fusion image features, including:

[0104] Inputting the target pathological image into a first image coding layer to obtain a first pathological image feature;

[0105] fusing the first pathological image feature and the first prompt mapping feature output by the first adaptation unit to obtain a first pathological fused image feature;

[0106] Input the j-1th pathological fusion image feature into the jth image coding layer to obtain the jth pathological image feature;

[0107] Fusing the j-th pathological image feature and the j-th prompt mapping feature output by the j-th adaptation unit to obtain the j-th pathological fusion image feature, where 2≤j≤m;

[0108] Increment j by 1 and determine whether j is greater than m. If not, continue to execute the step of inputting the j-1th pathological fusion image feature into the jth image coding layer to obtain the jth pathological image feature. If so, obtain the pathological fusion image feature.

[0109] Among them, the image encoder is configured with m image coding layers, and the m image coding layers are connected sequentially in the image encoder. Each image coding layer can be configured with a corresponding adaptation unit. In practical applications, the image coding layer corresponds to the adaptation unit one by one, that is, m=n, or except for the last image coding layer in the image encoder, each image coding layer corresponds to the adaptation unit one by one, that is, m=n-1. The first image coding layer is the first image coding layer in the image encoder, and the first pathological image feature is the pathological image feature output by the first image coding layer, that is, the encoding feature after the first image coding layer encodes the target pathological image. The first pathological fusion image feature refers to the pathological fusion image feature obtained by fusing the first pathological image feature with the first prompt mapping feature.

[0110] See also Figure 5 , Figure 5 FIG. 2 shows a schematic diagram of the structure of an image encoder provided according to an embodiment of the present specification. Figure 5 As shown, the image encoder includes m image coding layers, namely, image coding layer L1, image coding layer L2, image coding layer L3, ..., image coding layer Lm. Image coding layer L1 is the first image coding layer.

[0111] Specifically, the target pathological image is input into the first image coding layer. Specifically, the target pathological image can be cut into image blocks of 256*256 pixels and input into the first image coding layer to obtain the first pathological image feature output by the first image coding layer. The first pathological image feature is camera-fused with the first prompt mapping feature output by the first adaptation unit to generate the first pathological fusion image feature. The first pathological fusion image feature is used as the input data of the next image coding layer and input into the next image coding layer to obtain the pathological image feature output by the next image coding layer, and so on, until the image coding layer is the mth image coding layer, and the pathological image feature output by the mth image coding layer is obtained. If m=n, it is necessary to add and fuse the pathological image feature output by the mth image coding layer and the prompt mapping feature output by the nth adaptation unit to generate the mth pathological fusion image feature; if m=n-1, the pathological image feature output by the mth image coding layer is determined to be the pathological fusion image feature.

[0112] Further, Figure 5The structure of the image encoder shown is used as an example for explanation. The target pathological image is input into the image coding layer L1, and the pathological image feature P1 output by the image coding layer L1 is obtained. The prompt mapping feature F1 output by the adaptation unit A1 is feature-fused with the pathological image feature P1 to obtain the pathological fused image feature R1; the pathological fused image feature R1 is used as the input data of the image coding layer L2 and input into the image coding layer L2 to obtain the pathological image feature P2 output by the image coding layer L2, and the prompt mapping feature F2 output by the adaptation unit A2 is feature-fused with the pathological image feature P2 to obtain the pathological fused image feature R2; the pathological fused image feature R2 is used as the input data of the image coding layer L3 and input into the image coding layer L3 to obtain the pathological image feature P3 output by the image coding layer L3, and the prompt mapping feature F3 output by the adaptation unit A3 is feature-fused with the pathological image feature P3 to obtain the pathological fused image feature R3; ... until the pathological image feature Pm output by the image coding layer Lm is obtained. If m=n, the pathological image feature Pm output by the image coding layer Lm is fused with the prompt mapping feature Fn output by the adaptation unit An to obtain the pathological fused image feature Rm; if m=n-1, the pathological image feature Pm output by the image coding layer Lm is determined to be the pathological fused image feature Rm.

[0113] After obtaining the pathological fusion image features output by the image encoder, the pathological fusion image features are input into the decoder of the image processing model, and the pathological fusion image features are decoded by the decoder to obtain the target prognosis result of the target disease.

[0114] In a specific implementation manner provided in this specification, the pathological fusion image features are input into the decoder to obtain the target prognosis result of the target disease, including:

[0115] Obtaining preset risk level characteristics;

[0116] The preset risk level feature is determined as a query vector, the pathology fusion image feature is determined as a key-value pair vector, attention calculation is performed on the preset risk level feature and the pathology fusion image feature, and a target prognosis result of the target disease is obtained.

[0117] Specifically, in the decoder, the target prognosis result of the target disease can be generated through attention calculation. After obtaining the pathological fusion image features output by the image decoder, the pathological fusion image features are input into the decoder, and the preset risk level features are obtained, the preset risk level features are determined as the query vector, the pathological fusion image features are determined as the key-value pair vector, and the preset risk level and the pathological fusion image features are calculated by attention to obtain the target prognosis result of the target disease.

[0118] Further, see Figure 6 , Figure 6 FIG. 2 shows a schematic diagram of a decoder according to an embodiment of the present specification. Figure 6 As shown, the decoder includes a cross attention layer, a feedforward neural network layer, a residual connection layer, and a normalization layer. The preset risk level features and pathological fusion image features are input into the cross attention layer of the decoder for attention calculation. The result of the attention calculation is forward calculated through the feedforward neural network layer, and normalized through the residual connection layer and the normalization layer, and the target prognosis result of the target disease can be output and obtained.

[0119] In one or more embodiments provided in this specification, on the basis of the image encoder, a learnable prompt encoder and adapter are added to adjust and optimize feature embedding, integrate information from multiple sources such as target pathological images and target prompt information, predict the target prognosis result of the target disease, and enhance the accuracy of the target prognosis result.

[0120] Since the image processing model is the basis for implementing the image processing method provided in this specification, the training process of the image processing model is also very important. The implementation method of the training process is as follows:

[0121] In a specific implementation manner provided in this specification, the image processing model is trained based on the following method:

[0122] Acquiring sample training data, wherein the sample training data includes a sample pathological image, sample prompt information corresponding to the sample pathological image, and a sample prognosis result corresponding to the sample pathological image;

[0123] Inputting the sample pathological image and the sample prompt information into the image processing model to obtain a predicted prognosis result;

[0124] Calculating the loss value of the image processing model according to the predicted prognosis result and the sample prognosis result;

[0125] The model parameters of the image processing model are adjusted according to the loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

[0126] The sample training data refers to the sample data used to train the image processing model, and the sample training data includes the sample pathology image, the sample prompt information corresponding to the sample pathology image, and the sample prognosis result corresponding to the sample pathology image. The sample pathology image is the image that needs the image processing model to predict the prognosis result during the training of the image processing model; the sample prompt information is the actual prompt information corresponding to the sample pathology image; and the sample prognosis result is the actual prognosis result corresponding to the sample pathology image.

[0127] The sample training data may be sample data (including sample pathological images, sample prompt information and sample prognostic results) obtained in a sample training data set. The sample prompt information and sample prognostic results may be obtained by manually annotating and classifying data. The sample training data set refers to a set of sample pathological images, sample prompt information and sample prognostic results. The predicted prognostic result refers to the prognostic result output by inputting the sample pathological image into the image processing model. The loss value refers to the difference between the sample prognostic result and the predicted prognostic result, which is used to measure the difference between the sample prognostic result and the predicted prognostic result.

[0128] Specifically, sample pathological images, sample prompt information and sample prognosis results are obtained, and the sample pathological images and sample prompt information are input into the image processing model to obtain predicted prognosis results. At this time, the image processing model is a model that has not been trained yet. There will be a deviation between the determined predicted prognosis results and the actual sample prognosis results. The model parameters of the image processing model need to be adjusted accordingly. Specifically, the loss value of the image processing model is calculated according to the output predicted prognosis results and sample prognosis results. The loss function for calculating the model loss value can be a 0-1 loss function, a square loss function, a cross entropy loss function, etc. in actual applications. In this specification, preferably, a cross entropy loss function or a square loss function is selected as the loss function for calculating the model loss value, and the model parameters of the image processing model are adjusted according to the loss value. The image processing model is continued to be trained based on the adjusted model parameters for the next batch of sample training data until the stop condition of the model training is reached.

[0129] Specifically, the model training stopping conditions include that the model loss value is less than a preset threshold and / or the training rounds reach a preset round.

[0130] In a specific implementation provided in this specification, taking the model loss value being less than a preset threshold as the training stop condition as an example, the preset threshold is 0.2. When the loss values ​​are all less than 0.2, the image processing model training is considered completed.

[0131] In another specific implementation provided in the present specification, taking the preset training rounds as the training stop condition as an example, the preset training rounds are 30 rounds. When the training rounds of the training sample training data reach 30 rounds, the image processing model training is considered to be completed.

[0132] In another specific implementation provided in the present specification, two training stop conditions, a preset threshold and a preset training round, are set, and the model loss value and the training round are determined at the same time. When either the model loss value or the training round meets the training stop condition, the image processing model training is considered to be completed.

[0133] It should be noted that in one or more embodiments provided in this specification, the image processing model is trained using a pre-trained image encoder. During the training process, the parameters of the image encoder can be frozen, that is, the image encoder does not participate in the model training, and the prompt encoder, adapter and decoder are trained.

[0134] By freezing the parameters of the image encoder and not updating them, we can preserve the capabilities of the pre-trained encoder and reduce the amount of learnable parameters.

[0135] In order to evaluate the effectiveness of the image processing model (VPTSurv) provided in this manual for survival analysis, two cancer datasets were collected: IHC-CD4 and IHC-CD8 for analysis. The IHC-CD4 dataset includes 564 patients and 655 WSI images, which are randomly divided into training sets (449), validation sets (71), and test sets (135). The IHC-CD8 dataset includes 564 patients and 754 WSI images, which are randomly divided into training sets (509), validation sets (83), and test sets (162).

[0136] For evaluation, the consistency index (CI) is used to verify the performance and compare it with other methods. The image processing model is trained on the training set and the model with better performance is selected in the validation set based on the CI, and the results are reported in the test set.

[0137] Table 1

[0138]

[0139] Table 1 above shows the main results of VPTSurrv on the IHC-CD4 and IHC-CD8 datasets. VPTSurrv was evaluated on two datasets with different immunohistochemical pathology images in Table 1. Compared with typical two-stage methods such as TransMIL, PatchGCN, DeepAttnMISL, and HVTsurv, using only the VPTSurrv decoder, we obtained the results of 55.12% and 54.32% of the survival analysis evaluation index (CI), which is slightly lower than these two-stage methods. However, under the end-to-end scheme, VPTSurrv achieved 59.93% and 61.10% in the CI index, respectively, with significant relative improvements of 8.7% and 12.5% ​​on IHC-CD4 and IHC-CD8, respectively.

[0140] The image processing method provided in this specification includes: obtaining a target pathological image of a target disease and at least one target prompt information of the target pathological image; inputting the target pathological image and each target prompt information into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the target pathological image and the prompt coding feature corresponding to each target prompt information.

[0141] One or more embodiments of the present specification realizes the use of the target pathological image and the target prompt information of the target pathological image as input data of the image processing model, so that the image processing model can obtain more features from the target prompt information, so as to perform image processing on the target pathological image through the target pathological image features of the target pathological image and the prompt coding features of the target prompt information, and after obtaining the pathological fusion image features, the target prognosis result of the target disease can be determined according to the preset risk level features in the image processing model. Therefore, in practical applications, it is not necessary for doctors to analyze the pathological image to determine the patient's disease prognosis, thereby improving the accuracy and efficiency of determining the disease prognosis; by processing the target pathological image features of the target pathological image, adding and combining the prompt coding features of the target prompt information, and performing feature analysis on the target pathological image, the accuracy and efficiency of determining the target disease prognosis using the image processing model can be further improved.

[0142] The following combination Figure 7 , taking the application of the image processing method provided in this specification in a medical scenario as an example, the image processing method is further described. Figure 7 FIG. 1 is a flowchart of a processing process of an image processing method provided by an embodiment of the present specification, and further explains the method by taking cancer as an example of a target disease. Figure 7 As shown, a 20x panoramic slice image of cancer is obtained, and the 20x panoramic slice image is used as the main input data of the image encoder, edge detection is performed on the 20x panoramic slice image to obtain a cell grayscale image, and a 10x panoramic slice image and protein detection content data of cancer are obtained, and the cell grayscale image, 10x panoramic slice image and protein detection content data are used as target prompt information of the 20x panoramic slice image.

[0143] The 10x panoramic slice image, cell grayscale image and protein detection content data are input into the prompt encoder to obtain prompt coding features. The 20x panoramic slice image is segmented, specifically into multiple 256*256 pixel image blocks. Feature extraction is performed on the 20x panoramic slice image to obtain the target pathological image features, and the target pathological image features are input into the image encoder and adapter respectively. Among them, the image encoder includes m image coding layers, and the adapter includes n adaptation units ( Figure 7 Take m=n-1 as an example). Specifically, the target pathological image features and the prompt coding features are input into the adapter, specifically, the target pathological image features and the prompt coding features are input into the adaptation unit A1, and the prompt mapping feature F1 output by the adaptation unit A1 is obtained. The adaptation unit A1 inputs the prompt mapping feature F1 into the image encoder and the adaptation unit A2 respectively. The prompt mapping feature F1 and the prompt coding feature are used as input data of the adaptation unit A2 and input into the adaptation unit A2 to obtain the prompt mapping feature F2 output by the adaptation unit A2; the prompt mapping feature F2 and the prompt coding feature are used as input data of the adaptation unit A3 and input into the adaptation unit A3 to obtain the prompt mapping feature F3 output by the adaptation unit A3; ... until the prompt mapping feature Fn output by the adaptation unit An is obtained. For the image encoder, the target pathological image feature is input into the image encoder, specifically, the target pathological image feature is input into the image coding layer L1 to obtain the pathological image feature P1 output by the image coding layer L1, the prompt mapping feature F1 output by the adaptation unit A1 is feature fused with the pathological image feature P1 to obtain the pathological fused image feature R1; the pathological fused image feature R1 is used as the input data of the image coding layer L2 and input into the image coding layer L2 to obtain the pathological image feature P2 output by the image coding layer L2, the prompt mapping feature F2 output by the adaptation unit A2 is feature fused with the pathological image feature P2 to obtain the pathological fused image feature R2; the pathological fused image feature R2 is used as the input data of the image coding layer L3 and input into the image coding layer L3 to obtain the pathological image feature P3 output by the image coding layer L3, the prompt mapping feature F3 output by the adaptation unit A3 is feature fused with the pathological image feature P3 to obtain the pathological fused image feature R3; ... until the pathological image feature Pm output by the image coding layer Lm is obtained. Since m=n-1, there is no need to fuse the prompt mapping features with the pathological image features. The pathological image features Pm output by the image coding layer Lm can be determined as the pathological fusion image features Rm. Then, the pathological fusion image features Rm are input into the decoder, the preset risk level features are used as the query vector, and the pathological fusion image features are used as the key-value pair vector to perform attention calculation to obtain the target prognosis result for cancer. The target prognosis result can specifically be the survival probability of the patient corresponding to each preset risk level.

[0144] An embodiment of the present specification realizes that a panoramic slice image, a 10-fold panoramic slice image, a cell grayscale image, and protein detection content data are used as input data of an image processing model, so that the image processing model can obtain more effective features from the target prompt information (10-fold panoramic slice image, cell grayscale image, and protein detection content data), so as to process the panoramic slice image through the target pathological image features of the panoramic slice image and the prompt coding features of the target prompt information, and after obtaining the pathological fusion image features, the target prognosis result of cancer can be determined according to the preset risk level features in the image processing model. Therefore, it is not necessary for doctors to analyze the panoramic slice image to determine the patient's disease prognosis result, thereby improving the accuracy and efficiency of determining the disease prognosis result.

[0145] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Figure 8 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0146] An acquisition module 802 is configured to acquire a target pathological image of a target disease and at least one target prompt information of the target pathological image;

[0147] The input module 804 is configured to input the target pathological image and each target prompt information into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the target pathological image and the prompt coding feature corresponding to each target prompt information.

[0148] Optionally, the image processing model includes an image encoder, a prompt encoder, an adapter and a decoder;

[0149] The input module 804 is further configured to:

[0150] Dividing the target pathological image into at least two pathological image blocks, performing feature extraction on each pathological image block, and obtaining pathological image block features of each pathological image block;

[0151] Input each target prompt information into the prompt encoder to obtain the prompt coding feature corresponding to each target prompt information;

[0152] Inputting each pathological image block feature and each prompt coding feature into the adapter to obtain a prompt mapping feature corresponding to each prompt coding feature;

[0153] Inputting the target pathological image and each prompt mapping feature into the image encoder to obtain pathological fusion image features;

[0154] The pathological fusion image features are input into the decoder to obtain the target prognosis result of the target disease.

[0155] Optionally, the prompt encoder includes at least two modality encoding layers;

[0156] The input module 804 is further configured to:

[0157] Determine the modality type of each target prompt information;

[0158] Determine the modal coding layer corresponding to each target prompt information according to the modal type of each target prompt information;

[0159] Input each target prompt information into the corresponding modality encoding layer to obtain the prompt encoding features.

[0160] Optionally, the adapter includes n adapter units;

[0161] The input module 804 is further configured to:

[0162] Inputting each pathological image block feature and each prompt coding feature into a first adaptation unit to obtain a first prompt mapping feature corresponding to each prompt coding feature;

[0163] Input the i-1th prompt mapping feature and each prompt coding feature into the i-th adaptation unit to obtain the i-th prompt mapping feature corresponding to each prompt coding feature, where 2≤i≤n;

[0164] Increment i by 1 and determine whether i is greater than n. If not, continue to execute the step of inputting the i-1th prompt mapping feature and each prompt coding feature into the i-th adaptation unit to obtain the i-th prompt mapping feature corresponding to each prompt coding feature. If so, obtain the prompt mapping feature corresponding to each prompt coding feature.

[0165] Optionally, the input module 804 is further configured to:

[0166] Each pathological image block feature is determined as a query vector, each prompt coding feature is determined as a key-value pair vector, attention calculation is performed on each pathological image block feature and each prompt coding feature, and a first prompt mapping feature corresponding to each prompt coding feature is obtained.

[0167] Optionally, the image encoder includes m image coding layers, and the adapter includes n adaptation units;

[0168] The input module 804 is further configured to:

[0169] Inputting the target pathological image into a first image coding layer to obtain a first pathological image feature;

[0170] fusing the first pathological image feature and the first prompt mapping feature output by the first adaptation unit to obtain a first pathological fused image feature;

[0171] Input the j-1th pathological fusion image feature into the jth image coding layer to obtain the jth pathological image feature;

[0172] Fusing the j-th pathological image feature and the j-th prompt mapping feature output by the j-th adaptation unit to obtain the j-th pathological fusion image feature, where 2≤j≤m;

[0173] Increment j by 1 and determine whether j is greater than m. If not, continue to execute the step of inputting the j-1th pathological fusion image feature into the jth image coding layer to obtain the jth pathological image feature. If so, obtain the pathological fusion image feature.

[0174] Optionally, the input module 804 is further configured to:

[0175] Obtaining preset risk level characteristics;

[0176] The preset risk level feature is determined as a query vector, the pathology fusion image feature is determined as a key-value pair vector, attention calculation is performed on the preset risk level feature and the pathology fusion image feature, and a target prognosis result of the target disease is obtained.

[0177] Optionally, the acquisition module 802 is further configured to:

[0178] Extracting edge structure information of a target object in the target pathological image;

[0179] The edge structure information is determined as target prompt information of the target pathological image.

[0180] Optionally, the device further comprises a training module configured to:

[0181] Acquiring sample training data, wherein the sample training data includes a sample pathological image, sample prompt information corresponding to the sample pathological image, and a sample prognosis result corresponding to the sample pathological image;

[0182] Inputting the sample pathological image and the sample prompt information into the image processing model to obtain a predicted prognosis result;

[0183] Calculating the loss value of the image processing model according to the predicted prognosis result and the sample prognosis result;

[0184] The model parameters of the image processing model are adjusted according to the loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

[0185] The image processing device provided in this specification includes: an acquisition module, configured to acquire a target pathological image of a target disease and at least one target prompt information of the target pathological image; an input module, configured to input the target pathological image and each target prompt information into an image processing model to obtain a target prognosis result of the target disease, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the target pathological image and the prompt coding feature corresponding to each target prompt information.

[0186] One or more embodiments of the present specification realizes the use of the target pathological image and the target prompt information of the target pathological image as input data of the image processing model, so that the image processing model can obtain more features from the target prompt information, so as to perform image processing on the target pathological image through the target pathological image features of the target pathological image and the prompt coding features of the target prompt information, and after obtaining the pathological fusion image features, the target prognosis result of the target disease can be determined according to the preset risk level features in the image processing model. Therefore, in practical applications, it is not necessary for doctors to analyze the pathological image to determine the patient's disease prognosis, thereby improving the accuracy and efficiency of determining the disease prognosis; by processing the target pathological image features of the target pathological image, adding and combining the prompt coding features of the target prompt information, and performing feature analysis on the target pathological image, the accuracy and efficiency of determining the target disease prognosis using the image processing model can be further improved.

[0187] The above is a schematic scheme of an image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0188] See also Fig. 9 , Fig. 9 A flowchart of a computer-assisted cancer prognosis method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0189] Step 902: Obtain a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image.

[0190] Step 904: Input the panoramic slice image and each target prompt information into an image processing model to obtain a target prognosis result of the target cancer, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the panoramic slice image and the prompt coding feature corresponding to each target prompt information.

[0191] It should be noted that the implementation method of step 902 to step 904 is the same as the implementation method of the above-mentioned step 102 to step 104, and will not be repeated here in the embodiment of this specification.

[0192] Specifically, a target disease is cancer as an example for further explanation. By obtaining a panoramic slice image of cancer and multiple target prompt information of the panoramic slice image, and inputting the panoramic slice image and the multiple target prompt information into the image processing model, the image processing model processes the panoramic slice image and the target prompt information, and the image processing model can obtain more features about cancer from the target prompt information, thereby predicting the target prognosis result of cancer through the target pathological image features of the panoramic slice image and the prompt coding features of the target prompt information, and obtaining the target prognosis result of cancer.

[0193] The computer-assisted cancer prognosis method provided in this specification processes the target pathological image features of the panoramic slice image, adds and combines the prompt coding features of the target prompt information, performs feature analysis on the panoramic slice image, and further improves the accuracy and efficiency of using the image processing model to determine the target disease prognosis result; thereby, there is no need for doctors to analyze the panoramic slice image to determine the patient's disease prognosis result, thereby improving the accuracy and efficiency of determining the disease prognosis result.

[0194] Corresponding to the above method embodiment, this specification also provides a computer-assisted cancer prognosis device embodiment, Fig.10 FIG. 2 shows a schematic diagram of a computer-assisted cancer prognosis device provided in one embodiment of the present specification. Fig.10 As shown, the device comprises:

[0195] An image acquisition module 1002 is configured to acquire a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image;

[0196] The image input module 1004 is configured to input the panoramic slice image and each target prompt information into an image processing model to obtain a target prognosis result of the target cancer, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the panoramic slice image and the prompt coding feature corresponding to each target prompt information.

[0197] The computer-assisted cancer prognosis device provided in this specification processes the target pathological image features of the panoramic slice image, adds and combines the prompt coding features of the target prompt information, performs feature analysis on the panoramic slice image, and further improves the accuracy and efficiency of using the image processing model to determine the target disease prognosis result; thereby, there is no need for doctors to analyze the panoramic slice image to determine the patient's disease prognosis result, thereby improving the accuracy and efficiency of determining the disease prognosis result.

[0198] The above is a schematic scheme of a computer-assisted cancer prognosis device of this embodiment. It should be noted that the technical scheme of the computer-assisted cancer prognosis device and the technical scheme of the above-mentioned computer-assisted cancer prognosis method belong to the same concept, and the details not described in detail in the technical scheme of the computer-assisted cancer prognosis device can be referred to the description of the technical scheme of the above-mentioned computer-assisted cancer prognosis method.

[0199] See also Fig.11 , Fig.11 A schematic diagram of the architecture of a computer-assisted cancer prognosis system provided according to an embodiment of the present specification is shown.

[0200] The embodiments of the computer-assisted cancer prognosis system provided in this specification are as follows:

[0201] The computer-assisted cancer prognosis system comprises a client 100 and a server 200;

[0202] The client 100 is configured to send a cancer prognosis instruction to the server 200, wherein the cancer prognosis instruction carries a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image;

[0203] The server 200 is configured to input the panoramic slice image and each target prompt information into an image processing model to obtain a target prognosis result of the target cancer, wherein the target prognosis result is obtained based on a preset risk level feature and a pathological fusion image feature, and the pathological fusion image feature is obtained based on the target pathological image feature of the panoramic slice image and the prompt coding feature corresponding to each target prompt information.

[0204] The cancer prognosis instruction refers to an instruction generated by the client for obtaining a target prognosis result of a target cancer. The cancer prognosis instruction carries a panoramic slice image of the target cancer and multiple target prompt information.

[0205] The computer-assisted cancer prognosis system may include multiple clients 100 and a server 200, wherein the client 100 may be referred to as a client-side device and the server 200 may be referred to as a cloud-side device. A communication connection may be established between multiple clients 100 through the server 200. In the computer-assisted cancer prognosis system, the server 200 is used to provide computer-assisted cancer prognosis services between multiple clients 100. Multiple clients 100 may serve as a sending end or a receiving end, respectively, and realize communication through the server 200.

[0206] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the computer-assisted cancer prognosis scenario, the user can send a cancer prognosis instruction to the server 200 through the client 100. After receiving the cancer prognosis instruction, the server 200 obtains the panoramic slice image and target prompt information, and processes the panoramic slice image and target prompt information through the image processing model to generate a target prognosis result of the target cancer, and pushes the target prognosis result to other clients 100 that have established communication.

[0207] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0208] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0209] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0210] It is worth noting that the computer-assisted cancer prognosis method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the computer-assisted cancer prognosis method provided in the embodiments of this specification. In other embodiments, the computer-assisted cancer prognosis method provided in the embodiments of this specification may also be jointly executed by the client and the server.

[0211] Fig.12 The block diagram of a computing device 1200 according to one embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0212] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0213] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0214] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 may also be a mobile or stationary server.

[0215] The processor 1220 is used to execute the following computer program / instruction, which implements the steps of the above-mentioned image processing method or computer-assisted cancer prognosis method when executed by the processor.

[0216] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the image processing method or computer-assisted cancer prognosis method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing method or computer-assisted cancer prognosis method embodiment.

[0217] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned image processing method or computer-assisted cancer prognosis method when executed by a processor.

[0218] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the image processing method or computer-assisted cancer prognosis method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing method or computer-assisted cancer prognosis method embodiment.

[0219] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned image processing method or computer-assisted cancer prognosis method when executed by a processor.

[0220] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned image processing method or computer-assisted cancer prognosis method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned image processing method or computer-assisted cancer prognosis method.

[0221] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0222] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0223] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0224] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0225] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Acquire a target pathological image of a target disease and at least one target prompt information of the target pathological image; Dividing the target pathological image into at least two pathological image blocks, performing feature extraction on each pathological image block, and obtaining pathological image block features of each pathological image block; Inputting each target prompt information into a prompt encoder in the image processing model to obtain a prompt coding feature corresponding to each target prompt information; Input each pathological image block feature and each prompt coding feature into the adapter in the image processing model to obtain the prompt mapping feature corresponding to each prompt coding feature; input the target pathological image and each prompt mapping feature into the image encoder in the image processing model to obtain the pathological fusion image feature; input the pathological fusion image feature into the decoder in the image processing model to obtain the target prognosis result of the target disease.

2. The method of claim 1, wherein the hint encoder comprises at least two modality encoding layers; Input each target prompt information into the prompt encoder in the image processing model to obtain the prompt encoding features corresponding to each target prompt information, including: Determine the modality type of each target prompt information; Determine the modal coding layer corresponding to each target prompt information according to the modal type of each target prompt information; Input each target prompt information into the corresponding modality encoding layer to obtain the prompt encoding features.

3. The method according to claim 1, wherein the adapter comprises n adapter units; Inputting each pathological image block feature and each prompt coding feature into the adapter in the image processing model to obtain a prompt mapping feature corresponding to each prompt coding feature, including: Inputting each pathological image block feature and each prompt coding feature into a first adaptation unit to obtain a first prompt mapping feature corresponding to each prompt coding feature; Input the i-1th prompt mapping feature and each prompt coding feature into the i-th adaptation unit to obtain the i-th prompt mapping feature corresponding to each prompt coding feature, where 2≤i≤n; Increment i by 1 and determine whether i is greater than n. If not, continue to execute the step of inputting the i-1th prompt mapping feature and each prompt coding feature into the i-th adaptation unit to obtain the i-th prompt mapping feature corresponding to each prompt coding feature. If so, obtain the prompt mapping feature corresponding to each prompt coding feature.

4. The method according to claim 3, inputting each pathological image block feature and each prompt coding feature into a first adaptation unit to obtain a first prompt mapping feature corresponding to each prompt coding feature, comprising: Each pathological image block feature is determined as a query vector, each prompt coding feature is determined as a key-value pair vector, attention calculation is performed on each pathological image block feature and each prompt coding feature, and a first prompt mapping feature corresponding to each prompt coding feature is obtained.

5. The method according to claim 1, wherein the image encoder comprises m image coding layers, and the adapter comprises n adaptation units; Inputting the target pathological image and each prompt mapping feature into the image encoder in the image processing model to obtain the pathological fusion image feature, including: Inputting the target pathological image into a first image coding layer to obtain a first pathological image feature; fusing the first pathological image feature and the first prompt mapping feature output by the first adaptation unit to obtain a first pathological fused image feature; Input the j-1th pathological fusion image feature into the jth image coding layer to obtain the jth pathological image feature; Fusing the j-th pathological image feature and the j-th prompt mapping feature output by the j-th adaptation unit to obtain the j-th pathological fusion image feature, where 2≤j≤m; Increment j by 1 and determine whether j is greater than m. If not, continue to execute the step of inputting the j-1th pathological fusion image feature into the jth image coding layer to obtain the jth pathological image feature. If so, obtain the pathological fusion image feature.

6. The method according to claim 1, inputting the pathological fusion image features into a decoder in the image processing model to obtain a target prognosis result of the target disease, comprising: Obtaining preset risk level characteristics; The preset risk level feature is determined as a query vector, the pathology fusion image feature is determined as a key-value pair vector, attention calculation is performed on the preset risk level feature and the pathology fusion image feature, and a target prognosis result of the target disease is obtained.

7. The method according to claim 1, obtaining at least one target prompt information of the target pathological image, comprising: Extracting edge structure information of a target object in the target pathological image; The edge structure information is determined as target prompt information of the target pathological image.

8. The method according to claim 1, wherein the image processing model is trained based on the following method: Get sample training data, where The sample training data includes a sample pathological image, sample prompt information corresponding to the sample pathological image, and a sample prognosis result corresponding to the sample pathological image; Inputting the sample pathological image and the sample prompt information into the image processing model to obtain a predicted prognosis result; Calculating the loss value of the image processing model according to the predicted prognosis result and the sample prognosis result; The model parameters of the image processing model are adjusted according to the loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

9. A computer-assisted cancer prognosis method, comprising: Acquire a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image; Dividing the panoramic slice image into at least two pathological image blocks, performing feature extraction on each pathological image block, and obtaining pathological image block features of each pathological image block; Inputting each target prompt information into a prompt encoder in the image processing model to obtain a prompt coding feature corresponding to each target prompt information; Inputting each pathological image block feature and each prompt coding feature into an adapter in the image processing model to obtain a prompt mapping feature corresponding to each prompt coding feature; The panoramic slice image and each prompt mapping feature are input into the image encoder in the image processing model to obtain pathological fusion image features; the pathological fusion image features are input into the decoder in the image processing model to obtain the target prognosis result of the target cancer.

10. A computer-assisted cancer prognosis system, comprising a client and a server; The client is configured to send a cancer prognosis instruction to the server, wherein: The cancer prognosis instruction carries a panoramic slice image of a target cancer and at least one target prompt information of the panoramic slice image; The server is configured to divide the panoramic slice image into at least two pathological image blocks, perform feature extraction on each pathological image block, and obtain pathological image block features of each pathological image block; Inputting each target prompt information into a prompt encoder in the image processing model to obtain a prompt coding feature corresponding to each target prompt information; Inputting each pathological image block feature and each prompt coding feature into an adapter in the image processing model to obtain a prompt mapping feature corresponding to each prompt coding feature; The panoramic slice image and each prompt mapping feature are input into the image encoder in the image processing model to obtain pathological fusion image features; the pathological fusion image features are input into the decoder in the image processing model to obtain the target prognosis result of the target cancer.

11. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.