Image processing method and device, electronic equipment and storage medium

Abstract

This application relates to the field of data processing technology and provides an image processing method. The method includes: acquiring an image to be processed and an initial text prompt corresponding to the image; performing visual encoding processing on the image to be processed based on prior knowledge of anatomical structures to obtain visual encoding features of the image; and performing text encoding processing on the initial text prompt based on a medical knowledge graph to obtain text encoding features of the initial text prompt; determining an initial segmentation mask and a correction instruction for the initial segmentation mask based on the visual encoding features and the text encoding features; and correcting the initial segmentation mask using the correction instruction to obtain a final segmentation mask. This invention solves the problem in existing processing methods that use simple stitching or cross-attention mechanisms for one-time feature fusion, which cannot effectively address fusion interference caused by inter-modal registration errors.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Medical image segmentation is a crucial step in accurate tumor diagnosis, treatment planning, and efficacy evaluation. Traditional methods rely on radiologists manually delineating lesion boundaries, a time-consuming, labor-intensive, and subjective process with significant inter-expert variability, becoming a bottleneck restricting clinical efficiency and consistency. Currently, automatic segmentation methods based on convolutional neural networks (CNNs), such as U-Net and its variants, are widely used. However, CNN models are limited by their local receptive fields, making it difficult to model long-range dependencies, resulting in a significant decrease in segmentation accuracy in complex scenarios such as blurred tumor boundaries and invasive growth. Subsequent researchers introduced the Transformer architecture to obtain global context, but it is insensitive to local details, computationally expensive, and heavily reliant on large-scale labeled data, making it impractical in the reality of scarce medical imaging data.

[0003] Furthermore, clinical diagnosis relies on multimodal information (such as MRI multi-sequence, CT, and report text). Existing processing methods typically employ simple splicing or cross-attention mechanisms for one-time feature fusion, which is merely a "one-way, one-time" reasoning process. This cannot simulate the iterative cognitive process of human doctors, which involves "observation-hypothesis-verification-correction." It lacks the ability to assess the quality of its own output and actively correct it. It also lacks a deep understanding of key medical knowledge and spatial relationships in the text and cannot effectively solve the fusion interference caused by registration errors between modalities.

[0004] Therefore, there is an urgent need for a data processing method that can deeply integrate visual and semantic information to solve the problem that existing processing methods use simple splicing or cross-attention mechanisms for one-time feature fusion, which cannot effectively solve the fusion interference caused by inter-modal registration errors. Summary of the Invention

[0005] This application provides an image processing method that addresses the problem of existing methods using simple stitching or cross-attention mechanisms for one-time feature fusion, which fails to effectively resolve fusion interference caused by inter-modal registration errors. By applying prior knowledge of anatomical structures, the method performs visual encoding on the image to be processed, obtaining visual encoded features. Furthermore, based on a medical knowledge graph, it performs text encoding on the initial text prompt, obtaining text encoded features. Using the visual and text encoded features, it determines the initial segmentation mask and its correction instructions. The correction instructions then refine the initial segmentation mask to obtain the final segmentation mask. This method solves the problem of existing methods using simple stitching or cross-attention mechanisms for one-time feature fusion, which fails to effectively resolve fusion interference caused by inter-modal registration errors.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising the following steps:

[0007] Acquire the image to be processed and the initial text prompt corresponding to the image to be processed;

[0008] Based on prior knowledge of anatomical structure, the image to be processed is subjected to visual coding to obtain the visual coding features of the image to be processed.

[0009] Furthermore, based on a knowledge graph of medical knowledge, the initial text prompt is processed by text encoding to obtain the text encoding features of the initial text prompt;

[0010] Based on the visual encoding features and the text encoding features, the initial segmentation mask of the image to be processed and the correction instructions of the initial segmentation mask are determined.

[0011] The initial segmentation mask is modified using the correction instruction to obtain the final segmentation mask.

[0012] Optionally, the step of acquiring the image to be processed and the initial text prompt corresponding to the image to be processed includes:

[0013] Acquire 3D images and unstructured diagnostic reports that match the 3D images;

[0014] The 3D image is preprocessed to obtain the image to be processed;

[0015] Furthermore, the unstructured diagnostic report is parsed using natural language to generate structured initial text prompts.

[0016] Optionally, the prior knowledge based on anatomical structure is used to perform visual coding processing on the image to be processed to obtain the visual coding features of the image to be processed, including:

[0017] Based on prior knowledge of anatomical structures, determine anatomical structure-guided masking strategies;

[0018] The anatomically guided masking strategy is used to mask the image to be processed, resulting in a masked image.

[0019] The anatomical structures that were covered in the masked image to be processed are reconstructed using a 3D visual encoder to obtain the visual coding features of the image to be processed.

[0020] Optionally, the knowledge graph based on medical knowledge performs text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt, including:

[0021] A graph embedding layer is constructed based on a knowledge graph of medical knowledge, and a word embedding layer is constructed based on word embedding.

[0022] A text encoder is constructed based on the graph embedding layer and the word embedding layer;

[0023] The text encoder is trained by using preset positive and negative sample pairs to obtain a trained text encoder.

[0024] The trained text encoder is used to perform text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt.

[0025] Optionally, determining the initial segmentation mask of the image to be processed and the correction instructions for the initial segmentation mask based on the visual encoding features and the text encoding features includes:

[0026] Using the text encoding features as keys and values, and the visual encoding features as queries, cross-attention computation is used to fuse the initial text prompts into the visual encoding features to obtain fused visual encoding features;

[0027] The fused visual coding features are used for image segmentation to obtain the initial segmentation mask of the image to be processed;

[0028] Based on the difference between the initial segmentation mask and the gold standard segmentation mask, the visual encoding features and the text encoding features are used as evaluation dependencies. A correction instruction for the initial segmentation mask is generated by an instruction generator. The gold standard segmentation mask is a segmentation mask selected by multiple experts.

[0029] Optionally, the step of modifying the initial segmentation mask using the modification instruction to obtain the final segmentation mask includes:

[0030] The correction instruction is processed by text encoding to obtain correction encoding features;

[0031] Based on the modified encoding features, the visual encoding features, and the text encoding features, the initial segmentation mask is modified by a segmentation decoder with a cross-attention mechanism to obtain the final segmentation mask.

[0032] Optionally, the method further includes:

[0033] Calculate the initial segmentation loss between the initial segmentation mask and the gold standard segmentation mask;

[0034] And, calculate the final segmentation loss between the final segmentation mask and the gold standard segmentation mask;

[0035] And, calculate the correction instruction loss between the correction instruction and the target correction instruction, wherein the target correction instruction is the correction instruction generated by the teacher model;

[0036] Calculate the composite loss based on the initial segmentation loss, the final segmentation loss, and the correction instruction loss;

[0037] With the goal of minimizing the composite loss, the instruction generator and the segmentation decoder with the cross-attention mechanism are iteratively optimized.

[0038] Secondly, embodiments of this application provide an image processing apparatus, the image processing apparatus comprising:

[0039] The acquisition module is used to acquire the image to be processed and the initial text prompt corresponding to the image to be processed.

[0040] The first processing module is used to perform visual encoding processing on the image to be processed based on prior knowledge of anatomical structure to obtain the visual encoding features of the image to be processed.

[0041] The second processing module is used to perform text encoding processing on the initial text prompt based on a knowledge graph of medical knowledge to obtain the text encoding features of the initial text prompt.

[0042] The determination module is used to determine the initial segmentation mask of the image to be processed and the correction instructions of the initial segmentation mask based on the visual encoding features and the text encoding features.

[0043] The third processing module is used to correct the initial segmentation mask using the correction instruction to obtain the final segmentation mask.

[0044] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the image processing method provided in embodiments of the present invention.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the image processing method provided in the embodiments of the invention.

[0046] The above-mentioned solution of this application has the following beneficial effects: acquiring the image to be processed and the initial text prompt corresponding to the image to be processed; performing visual encoding processing on the image to be processed based on prior knowledge of anatomical structure to obtain the visual encoding features of the image to be processed; and performing text encoding processing on the initial text prompt based on a knowledge graph of medical knowledge to obtain the text encoding features of the initial text prompt; determining the initial segmentation mask of the image to be processed and the correction instructions for the initial segmentation mask based on the visual encoding features and the text encoding features; and correcting the initial segmentation mask through the correction instructions to obtain the final segmentation mask. This invention applies prior knowledge of anatomical structures to perform visual encoding on the image to be processed, obtaining visual encoding features of the image. Furthermore, based on a medical knowledge graph, it performs text encoding on the initial text prompt, obtaining text encoding features of the initial text prompt. Through the visual and text encoding features, the initial segmentation mask and its correction instructions are determined for the image to be processed. The initial segmentation mask is then corrected using the correction instructions to obtain the final segmentation mask. This invention solves the problem of existing processing methods that rely on simple stitching or cross-attention mechanisms for one-time feature fusion, which cannot effectively address fusion interference caused by intermodal registration errors.

[0047] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating an image processing method provided in one embodiment of this application;

[0050] Figure 2This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0053] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0054] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0055] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0056] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0057] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0058] like Figure 1 As shown, Figure 1 This is a flowchart of an image processing method provided in an embodiment of the present invention, which includes the following steps:

[0059] 101. Obtain the image to be processed and the corresponding initial text prompt.

[0060] In this embodiment of the invention, the image processing method described above can be applied to an image processing platform, which can be built on or distributed servers. The image processing platform includes a data interface (for sensors or users to upload data), a knowledge database, and a knowledge database construction program. The data interface can be used to obtain the image to be processed and its corresponding initial text prompts. The knowledge database construction program can be used to construct the knowledge database, which is specifically used to provide additional relational information for the identified data entities, thereby improving the depth of the data recognition system's understanding of the content.

[0061] The images to be processed can be three-dimensional image data, or multimodal images containing a single modality or multiple registered modalities. The images to be processed can be acquired by medical imaging equipment (such as magnetic resonance imaging (MRI) equipment, computed tomography (CT) equipment, etc.), or they can be medical images retrieved from a hospital's picture archiving and communication system (PACS).

[0062] The initial text prompts mentioned above can be text descriptions corresponding to the image to be processed, such as a list containing key information items or a coherent descriptive statement.

[0063] It should be noted that the initial text prompts mentioned above can be structured diagnostic reports that are paired with the images to be processed and retrieved from the EMR system (Electronic Medical Record system).

[0064] 102. Based on prior knowledge of anatomical structure, visual encoding processing is performed on the image to be processed to obtain the visual encoding features of the image to be processed.

[0065] In this embodiment of the invention, the prior knowledge of the aforementioned anatomical structures can be prior knowledge of "what constitutes a normal anatomical structure." The prior knowledge of anatomical structures can be a standardized set of information regarding the standard morphology, spatial location, relative adjacency, and typical imaging features (such as grayscale and texture) of normal human organs and tissues in three-dimensional space. This prior knowledge of anatomical structures can originate from standard digital anatomical atlases, such as brain atlases and whole-body organ segmentation atlases. Standard digital anatomical atlases provide precise and universally accepted three-dimensional mask labels for different anatomical structures (such as white matter, gray matter, lateral ventricles, and different lobes of the liver) in standard space.

[0066] The aforementioned visual encoding process can be described as a process of mapping the input image to be processed into low-dimensional, high-semantic anatomical perceptual visual features based on prior knowledge of anatomical structures. Visual encoding processing not only extracts general low-level features such as image texture and edges, but more importantly, it extracts high-level semantic features that contain prior knowledge of anatomical structures.

[0067] The aforementioned visual coding features can be feature representations with clear anatomical semantics output after visual coding processing of medical images to be processed, based on prior knowledge of anatomical structures. Visual coding features can clearly distinguish different organs / tissues and have an inherent understanding of their normal spatial layout and boundaries.

[0068] It should be noted that an anatomical-perceptual visual encoder can be used to perform visual encoding processing on the image to be processed. The anatomical-perceptual visual encoder can convert the input image to be processed into visual encoded features containing rich anatomical semantics. The aforementioned anatomical-perceptual visual encoder can be based on a 3D visual Transformer architecture. The 3D visual Transformer is a technology that applies the Transformer architecture to three-dimensional visual data processing, aiming to capture spatial dependencies in 3D data through a self-attention mechanism.

[0069] 103. Furthermore, based on a knowledge graph of medical knowledge, the initial text prompt is processed by text encoding to obtain the text encoding features of the initial text prompt.

[0070] In this embodiment of the invention, the aforementioned medical knowledge graph can be a structured, graphical medical domain knowledge database used to map discrete medical text terms to a continuous vector space rich in clinical semantics and logical relationships. The medical knowledge graph includes nodes and relation edges. Nodes represent medical entities, such as: disease ("glioblastoma"), anatomical location ("frontal lobe"), imaging features ("enhancement"), descriptive terms ("invasive", "micro", "satellite lesions"), and spatial location terms ("left side", "proximity"). Relation edges define multiple semantic relationships between nodes, including medical semantic relationships and spatial location relationships. Medical semantic relationships can be "is the cause" (smoking -> lung cancer), "has symptoms" (brain tumor -> headache), "common location" (glioblastoma -> cerebral hemisphere), "manifests on imaging" (malignant tumor -> blurred boundaries), etc. Spatial location relationships can be "located" (lesion -> left upper lobe of lung), "proximity" (tumor -> ventricle), "left / right side", etc.

[0071] The above text encoding process can be a process of mapping the initial text prompts into text encoding features with deep medical semantics and accurate spatial relationship discrimination based on a knowledge graph of medical knowledge.

[0072] The aforementioned text encoding features can be feature representations rich in clinical semantics and relational information, output after text encoding processing of the initial text based on a knowledge graph of medical knowledge. Each vector (or set of vectors) in the text encoding features corresponds to a semantic unit in the initial text prompt, encoding not only the lexical information of that semantic unit, but also, more profoundly, the associated concepts and spatial relational attributes of that unit's semantics in the knowledge graph of medical knowledge.

[0073] It should be noted that a text encoder can be used to encode the initial text prompts, transforming them into knowledge-relational text features rich in deep clinical semantics and precise spatial relational information. This text encoder can be based on the Transformer architecture, aiming to leverage its powerful context modeling capabilities to understand the complex grammatical and semantic dependencies between words in medical reports.

[0074] 104. Based on visual coding features and text coding features, determine the initial segmentation mask of the image to be processed and the correction instructions for the initial segmentation mask.

[0075] In this embodiment of the invention, the segmentation mask described above is a technique used in computer vision for image segmentation. The segmentation mask identifies the location and shape of a target object by assigning a label to each pixel in the image.

[0076] The initial segmentation mask described above can be a preliminary three-dimensional probability distribution map or binary segmentation label map for lesion regions in the image to be processed, generated based on visual and textual encoding features. This initial segmentation mask can be a three-dimensional data array whose size corresponds to the spatial dimension of the input image to be processed. Each voxel value in the array represents the confidence probability (between 0 and 1) of that location being classified as a lesion, or a direct binary classification label (0 for background, 1 for lesion). This initial segmentation mask can be a coarse segmentation result.

[0077] The aforementioned initial segmentation mask correction instructions can be used to guide how to modify the initial segmentation mask to obtain more accurate segmentation results.

[0078] Specifically, an initial segmentation mask can be generated by weighted fusion of visual encoding features and text encoding features. This weighted fusion can be a process of assigning corresponding weights to the visual encoding features and text visual encoding features based on their importance, and then comprehensively calculating the weighted visual encoding features and text visual encoding features.

[0079] 105. By using correction instructions, the initial segmentation mask is corrected to obtain the final segmentation mask.

[0080] In this embodiment of the invention, the above-mentioned correction instruction may be an instruction used to guide how to modify the initial segmentation mask in order to obtain a more accurate segmentation result.

[0081] The aforementioned correction can be a process of modifying the initial segmentation mask through correction instructions. It is understood that the correction is not a post-processing of the image to be recognized, but rather uses correction instructions to guide a new, more focused semantic understanding and segmentation decision.

[0082] The final segmentation mask mentioned above can be the final segmentation result obtained by modifying the initial segmentation mask through modification instructions.

[0083] In this embodiment of the invention, the invention performs visual encoding processing on the image to be processed based on prior knowledge of anatomical structures to obtain the visual encoding features of the image to be processed. Furthermore, based on a knowledge graph of medical knowledge, the invention performs text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt. Through the visual encoding features and text encoding features, the initial segmentation mask and the correction instructions for the initial segmentation mask are determined. The initial segmentation mask is then corrected using the correction instructions to obtain the final segmentation mask. This invention possesses "spatial-relationship" understanding and "self-correction" capabilities, significantly improving the detection sensitivity for small and rare lesions and enhancing the segmentation accuracy for complex and ambiguous boundaries.

[0084] In this embodiment of the invention, an image to be processed and an initial text prompt corresponding to the image to be processed are acquired; based on prior knowledge of anatomical structures, visual encoding processing is performed on the image to be processed to obtain visual encoding features of the image to be processed; and based on a knowledge graph of medical knowledge, text encoding processing is performed on the initial text prompt to obtain text encoding features of the initial text prompt; based on the visual encoding features and text encoding features, an initial segmentation mask of the image to be processed and a correction instruction for the initial segmentation mask are determined; and the initial segmentation mask is corrected using the correction instruction to obtain the final segmentation mask. This invention applies prior knowledge of anatomical structures to perform visual encoding on the image to be processed, obtaining visual encoding features of the image. Furthermore, based on a medical knowledge graph, it performs text encoding on the initial text prompt, obtaining text encoding features of the initial text prompt. Through the visual and text encoding features, the initial segmentation mask and its correction instructions are determined for the image to be processed. The initial segmentation mask is then corrected using the correction instructions to obtain the final segmentation mask. This invention solves the problem of existing processing methods that rely on simple stitching or cross-attention mechanisms for one-time feature fusion, which cannot effectively address fusion interference caused by intermodal registration errors.

[0085] It is understood that in the specific implementation of this application, data such as image data, knowledge data, and text data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0086] Optionally, in the step of obtaining the image to be processed and the corresponding initial text prompt, a 3D image and an unstructured diagnostic report matching the 3D image can be obtained; the 3D image can be preprocessed to obtain the image to be processed; and the unstructured diagnostic report can be parsed using natural language to generate a structured initial text prompt.

[0087] In this embodiment of the invention, the aforementioned 3D image can be an image with three-dimensional spatial information, that is, an image containing depth or height information. The 3D image can be image data acquired by medical imaging equipment that contains complete three-dimensional spatial voxel information. The 3D image can be a medical image retrieved from a hospital image archiving and communication system (PACS) and stored in the form of three-dimensional volumetric data.

[0088] The aforementioned unstructured diagnostic reports can be radiological diagnostic reports paired with 3D images, obtained from a hospital's electronic medical record (EMR) system. An unstructured diagnostic report is a diagnostic report whose content is freely written in natural language paragraphs and sentences.

[0089] The preprocessing described above can be a series of standardization, normalization, and alignment operations performed on 3D images. The purpose of preprocessing is to transform 3D images into high-quality, consistent, and computable standardized data. Preprocessing can include intermodal registration, N4 bias field correction, intensity normalization, and isotropic resampling. Intermodal registration, when the 3D images are multi-sequence MRI or images of different modalities (such as CT and MRI), uses non-rigid or rigid image registration algorithms to precisely align all 3D images to the same three-dimensional coordinate system, resolving modal mismatches (e.g., slight user movement at different scan times or between different scan sequences can cause coordinate mismatches for the same anatomical location in different images). N4 bias field correction, for MRI images, applies algorithms (such as N4ITK) to correct low-frequency intensity variations caused by magnetic field inhomogeneities, making the signal intensity of the same tissue at different locations more consistent, eliminating systematic intensity errors introduced by the device, and ensuring that image intensity primarily reflects tissue characteristics. The aforementioned intensity normalization can scale the voxel intensity values ​​of a 3D image to a standard range or distribution, overcoming the problem of inconsistent intensity scales caused by different scanning devices, protocols, and individual user differences. The aforementioned isotropic resampling can resample a 3D image to isotropic resolution, meaning that the voxel sizes are the same in the X, Y, and Z directions.

[0090] The image to be processed can be an image obtained after preprocessing a 3D image.

[0091] The aforementioned natural language parsing can be a process of automatically identifying, extracting, and structuring the key clinical semantic entities and their relationships contained in unstructured diagnostic reports using natural language processing technology.

[0092] The aforementioned initial text prompt can be a structured initial text prompt rich in clinical semantics, output after natural language parsing of the unstructured diagnostic report.

[0093] It should be noted that the aforementioned natural language processing technologies aim to enable machines to understand, interpret, and generate human language, thereby achieving effective communication between humans and machines and enabling computers to perform tasks such as language translation, sentiment analysis, and text summarization.

[0094] Optionally, in the step of visually encoding the image to be processed based on prior knowledge of anatomical structures to obtain the visual encoding features of the image to be processed, an anatomical structure-guided masking strategy can be determined based on prior knowledge of anatomical structures; the anatomical structure-guided masking strategy is used to mask the image to be processed to obtain the masked image to be processed; and the masked anatomical structures in the masked image to be processed are reconstructed through a 3D visual encoder to obtain the visual encoding features of the image to be processed.

[0095] In this embodiment of the invention, the prior knowledge of the above-mentioned anatomical structure can be a standardized set of information about the standard morphology, spatial position, relative adjacency, and typical imaging features (such as grayscale and texture) of normal human organs and tissues in three-dimensional space.

[0096] The masking strategy described above is primarily used in self-supervised learning. By randomly masking portions of words (or tags) in the input text, the model is forced to predict the original form of the masked content based on the context, thereby improving its ability to understand language structure.

[0097] The aforementioned anatomically guided masking strategy can be based on prior knowledge of anatomical structures to formulate a masking scheme for the image to be processed, aiming to highlight or protect certain important structures. The anatomically guided masking strategy aims to reduce interference with non-target areas, improving operational accuracy and safety. Specifically, for each 3D medical image, a non-rigid registration algorithm can be used to align prior knowledge of anatomical structures to the 3D medical image, generating a specific anatomical pseudo-label map (the anatomical pseudo-label map marks which anatomical structure each voxel in the image belongs to, such as "left upper lobe of the lung" or "right lobe of the liver"). Based on the anatomical pseudo-label map, the anatomically guided masking strategy is determined. The anatomically guided masking strategy randomly selects one or more complete anatomical structures (e.g., "entire spleen" or "left ventricle") rather than randomly selecting image patches. The aforementioned anatomical pseudo-label map can be an anatomical structure annotation map specific to an individual medical image.

[0098] The masked image to be processed can be an image generated after masking the original image using an anatomical structure-guided masking strategy, resulting in an image where specific anatomical structures are masked. This masked image can be an "incomplete" image, where the voxel values ​​of certain complete anatomical structures (such as the entire liver or spleen) are specially masked, while other parts remain unchanged. For example, in an abdominal CT image containing organs such as the liver, spleen, kidneys, and stomach, after using an anatomical structure-guided masking strategy, the spleen is randomly selected as the masking target. In the resulting masked image, the area containing the spleen becomes entirely black (value 0), while all other structures such as the liver, kidneys, and stomach remain unchanged.

[0099] The aforementioned 3D visual encoder, based on a 3D visual Transformer architecture, incorporates an anatomically guided occlusion strategy to mask entire anatomical structures (e.g., the entire "left ventricle" or "liver"), forcing the Transformer model to utilize its global contextual capabilities to reconstruct the masked anatomical structures. The 3D visual encoder is forced to learn the 3D anatomical structure, spatial adjacency, and normal boundaries of the human body. Furthermore, when used for downstream segmentation tasks, the visually encoded features provided by the 3D visual encoder already contain strong prior knowledge of "where the normal anatomical boundaries are."

[0100] The aforementioned visual coding features can be feature representations with clear anatomical semantics obtained by reconstructing the masked anatomical structures in the masked image to be processed through a 3D visual encoder.

[0101] In one possible embodiment, when processing image A, an anatomically guided masking strategy can be used to mask image A, extracting important organs and tissues such as the heart and lungs, while masking other unnecessary parts, resulting in a masked image A. A 3D visual encoder is then used to process the masked image A to obtain the visual coding features of image A. By masking unnecessary parts, the visual coding features of important organs and tissues can be extracted more quickly and accurately.

[0102] Optionally, in the step of text encoding the initial text prompt based on a knowledge graph of medical knowledge to obtain the text encoding features of the initial text prompt, a graph embedding layer can be constructed based on the knowledge graph of medical knowledge, and a word embedding layer can be constructed based on word embedding; a text encoder can be constructed based on the graph embedding layer and the word embedding layer; the text encoder can be trained using preset positive and negative sample pairs to obtain a trained text encoder; and the initial text prompt can be text encoded using the trained text encoder to obtain the text encoding features of the initial text prompt.

[0103] In this embodiment of the invention, the aforementioned medical knowledge graph can be a structured, graphical medical domain knowledge database used to map discrete medical text terms to a continuous vector space rich in clinical semantics and logical relationships. The medical knowledge graph includes nodes and relation edges. Nodes represent medical entities, such as: disease ("glioblastoma"), anatomical location ("frontal lobe"), imaging features ("enhancement"), descriptive terms ("invasive", "micro", "satellite lesions"), and spatial location terms ("left side", "proximity"). Relation edges define multiple semantic relationships between nodes, including medical semantic relationships and spatial location relationships. Medical semantic relationships can be "is the cause" (smoking -> lung cancer), "has symptoms" (brain tumor -> headache), "common location" (glioblastoma -> cerebral hemisphere), "manifests on imaging" (malignant tumor -> blurred boundaries), etc. Spatial location relationships can be "located" (lesion -> left upper lobe of lung), "proximity" (tumor -> ventricle), "left / right side", etc.

[0104] The graph embedding layer mentioned above can be a core component in graph neural networks (GNNs) or graph representation learning. Its main function is to map graph structure data (such as nodes, edges, or the entire graph) into a low-dimensional dense vector space while preserving the topological structure and semantic information of the original graph as much as possible.

[0105] The aforementioned graph embedding layer can map each medical entity to a graph embedding based on a knowledge graph of medical knowledge. This mapping can be achieved by querying the knowledge graph and utilizing graph neural networks (GNNs) or graph attention networks (GATs) to capture the structural relationships and semantics of entities within the knowledge graph.

[0106] The function of the graph embedding layer described above is to map each token in the input text to a vector representation rich in its association information within the medical knowledge graph.

[0107] The word embedding layer described above can map each word to a word embedding based on the word embedding method.

[0108] The text encoder described above can be constructed based on graph embedding layers and word embedding layers.

[0109] The aforementioned text encoder can be based on the Transformer architecture, aiming to leverage the Transformer architecture's powerful context modeling capabilities to understand the complex grammatical and semantic dependencies between words in medical reports.

[0110] The aforementioned pre-defined positive sample pairs can be texts with similar semantics, such as texts describing the same anatomical location but expressing different ideas.

[0111] The negative sample pairs mentioned above can be semantically dissimilar texts, such as texts describing different anatomical locations.

[0112] The training described above can be unsupervised training, which is a method of discovering the inherent structure and patterns in data through unlabeled data. Unsupervised training does not require predefined labels or expected outputs; instead, it allows the algorithm to autonomously explore the data and identify hidden patterns, groupings, or relationships.

[0113] The trained text encoder can be obtained by training the text encoder with preset positive and negative sample pairs. The trained text encoder can recognize the encoded text features of the initial text prompt.

[0114] The above text encoding process can be a process of mapping the initial text prompt into text encoding features with deep medical semantics and accurate spatial relationship discrimination by using a trained text encoder.

[0115] The aforementioned text encoding features can be feature representations rich in clinical semantics and relational information obtained by text encoding the initial text prompt through a trained text encoder. Each vector (or set of vectors) in the text encoding features corresponds to a semantic unit in the initial text prompt, encoding not only the lexical information of that semantic unit, but also, more profoundly, the associated concepts and spatial relational attributes of that unit's semantics in the knowledge graph of medical knowledge.

[0116] Specifically, spatial relational contrastive learning can be used to make the encoder more sensitive to spatial and relational terms that are crucial in clinical reports, with the goal of minimizing:

[0117]

[0118] in, Indicates a text encoder. Indicates a spatial anchor point. Indicates a positive sample. Indicates a negative sample; This represents a hyperparameter used to ensure that the distance between negative sample pairs (A,N) must be at least greater than the distance between positive sample pairs (A,P). .

[0119] It should be noted that, through training, the text encoder is forced to push the vectors of key spatial descriptors such as "left" and "right" to far apart positions in the feature space, thus solving the fundamental deficiency of standard BERT's insensitivity to spatial relationships. Furthermore, through medical knowledge graph fusion, the text encoder can "understand" that "glioblastoma" and "invasive" are highly correlated in medicine, even if these two terms do not appear simultaneously in the report.

[0120] Optionally, in the step of determining the initial segmentation mask and the correction instructions for the initial segmentation mask of the image to be processed based on visual coding features and text coding features, the text coding features are used as keys and values, and the visual coding features are used as queries. Cross-attention computation is used to fuse the initial text prompts into the visual coding features to obtain fused visual coding features. Image segmentation processing is performed on the fused visual coding features to obtain the initial segmentation mask of the image to be processed. Based on the difference between the initial segmentation mask and the gold standard segmentation mask, the visual coding features and text coding features are used as evaluation dependencies, and the correction instructions for the initial segmentation mask are generated by the instruction generator.

[0121] In this embodiment of the invention, the above key can be understood as an index directory of text semantics, and the above value can be understood as the specific content of text semantics.

[0122] The aforementioned cross-attention computation can be a cross-attention mechanism, which is a technique used in deep learning to process multimodal data. The cross-attention mechanism allows the model to weight and fuse features in one input sequence (such as an image) based on information from another input sequence (such as text), thereby enhancing the correlation between different modalities.

[0123] The aforementioned fused visual coding features can be obtained by using text coding features as keys and values, visual coding features as queries, and fusion of the initial text prompts into the visual coding features using cross-attention computation.

[0124] Specifically, the fusion of visual encoding features can be represented as:

[0125]

[0126] Where Attention(Q,K,V) represents the fusion of visual encoding features; Q represents the query; K represents the key; and V represents the value. Indicates the scaling factor; Represents the similarity matrix, calculating the dot product similarity between the visual location Q and each concept K in the text.

[0127] Understandably, by calculating the similarity between the query (Q) and the key (K), a set of weights is obtained, and then the values ​​(V) are weighted and summed using this set of weights to obtain the fused visual encoding features.

[0128] The image segmentation process described above can be a process of segmenting an image by fusing visual coding features to generate an initial segmentation mask.

[0129] The initial segmentation mask mentioned above can be obtained after image segmentation processing using fused visual coding features. It can be a preliminary three-dimensional probability distribution map or a binary segmentation label map for lesion regions in the image to be processed. The initial segmentation mask can be a three-dimensional data array whose size corresponds to the spatial dimension of the input image to be processed. Each voxel value in the array represents the confidence probability (between 0 and 1) of that location being classified as a lesion or a direct binary classification label (0 represents background, 1 represents lesion). The initial segmentation mask mentioned above can be a coarse segmentation result.

[0130] The gold standard segmentation mask mentioned above is a segmentation mask selected by multiple experts. Alternatively, the gold standard segmentation mask can be a consensus mask statistically estimated from the differing opinions of multiple experts using a specific algorithm.

[0131] The instruction generator described above can be a neural network model based on a sequence-to-sequence generation architecture. This architecture is used to transform one sequence into another, such as translating a sentence from one language into another, or generating text summaries. The core idea of ​​sequence-to-sequence generation is to achieve an end-to-end mapping from the input sequence to the output sequence through an encoder-decoder structure. The core task of the instruction generator is to receive the initial segmentation result and its generation basis as input, analyze the contradictions and deficiencies, and output a structured natural language text describing how to correct the initial segmentation result.

[0132] The aforementioned correction instructions can be used to guide how to modify the initial segmentation mask to obtain more accurate segmentation results.

[0133] Specifically, visual encoding features and text encoding features can be used as the objects to be evaluated. The difference between the objects to be evaluated and the gold standard segmentation mask can be compared through the instruction generator. The visual encoding features and text encoding features are used as the "visual basis" and "semantic basis" on which the evaluation is based, that is, the evaluation dependency, to generate the correction instructions for the initial segmentation mask.

[0134] Optionally, in the step of modifying the initial segmentation mask through the modification instruction to obtain the final segmentation mask, the modification instruction can be processed by text encoding to obtain modified encoded features; based on the modified encoded features, visual encoded features and text encoded features, the initial segmentation mask is modified by a segmentation decoder with a cross-attention mechanism to obtain the final segmentation mask.

[0135] In this embodiment of the invention, the above-mentioned correction instruction may be an instruction used to guide how to modify the initial segmentation mask in order to obtain a more accurate segmentation result.

[0136] The aforementioned text encoding process can be a process of text encoding correction instructions, whereby the correction instructions are input into a text encoder, and the text encoder outputs the corresponding correction encoding features. The text encoder can be obtained by training it with pre-defined positive and negative sample pairs. This text encoder can be based on the Transformer architecture, aiming to leverage the powerful context modeling capabilities of the Transformer architecture to understand the complex grammatical and semantic dependencies between words in medical reports.

[0137] The aforementioned correction encoding features can be obtained by text encoding the correction instructions.

[0138] The segmentation decoder described above using the cross-attention mechanism can be a segmentation decoder based on the cross-attention mechanism. The cross-attention mechanism is a technique used in deep learning to process multimodal data. It allows the model to weight and fuse features in one input sequence (such as an image) based on information from another input sequence (such as text), thereby enhancing the correlation between different modalities.

[0139] The initial segmentation mask described above can be a preliminary three-dimensional probability distribution map or a binary segmentation label map for lesion regions in the image to be processed. The initial segmentation mask can be a three-dimensional data array whose size corresponds to the spatial dimension of the input image to be processed. Each voxel value in the array represents the confidence probability (between 0 and 1) of that location being classified as a lesion, or a direct binary classification label (0 represents background, 1 represents lesion). The initial segmentation mask can be a coarse segmentation result.

[0140] The above correction can be a process of performing more refined segmentation on the initial segmentation mask through a segmentation decoder with a cross-attention mechanism, based on the corrected encoding features, visual encoding features, and text encoding features.

[0141] The final segmentation mask mentioned above can be the final segmentation result obtained by modifying the initial segmentation mask through a segmentation decoder with a cross-attention mechanism based on the modified encoding features, visual encoding features, and text encoding features.

[0142] Specifically, correction instructions can be input into a text encoder to generate corrected semantic features. The modified semantic features, visual encoding features, and text encoding features are then input into the segmentation decoder using a cross-attention mechanism to implement the modification instructions for the modified semantic features (e.g., "exclude edema"). Through the cross-attention mechanism, inhibitory semantic guidance is applied to the regions in the initial segmentation mask that are incorrectly labeled as "tumor" but are actually "edema". Finally, the decoder outputs the final segmentation mask.

[0143] Optionally, the image processing method can also calculate the initial segmentation loss between the initial segmentation mask and the gold standard segmentation mask; and calculate the final segmentation loss between the final segmentation mask and the gold standard segmentation mask; and calculate the correction instruction loss between the correction instruction and the target correction instruction, where the target correction instruction is the correction instruction generated by the teacher model; calculate the composite loss based on the initial segmentation loss, the final segmentation loss, and the correction instruction loss; and iteratively optimize the instruction generator and the segmentation decoder across the attention mechanism with the goal of minimizing the composite loss.

[0144] In this embodiment of the invention, the initial segmentation mask can be a preliminary three-dimensional probability distribution map or a binary segmentation label map for the lesion region in the image to be processed.

[0145] The gold standard segmentation mask mentioned above is a segmentation mask selected by multiple experts. Alternatively, the gold standard segmentation mask can be a consensus mask statistically estimated from the differing opinions of multiple experts using a specific algorithm.

[0146] The initial segmentation loss mentioned above can be the segmentation loss between the initial segmentation mask and the gold standard segmentation mask.

[0147] The final segmentation mask mentioned above can be the final classification result obtained after correction.

[0148] The aforementioned final segmentation loss can be the segmentation loss between the final segmentation mask and the gold standard segmentation mask.

[0149] The aforementioned correction instructions can be used to guide how to modify the initial segmentation mask to obtain more accurate segmentation results.

[0150] The aforementioned target correction instructions are correction instructions generated by the teacher model. The aforementioned teacher model...

[0151] The aforementioned correction instruction loss can be the correction instruction loss between the correction instruction and the target correction instruction.

[0152] The aforementioned composite loss can be the overall optimization objective generated by weighted summation of the initial segmentation loss, the final segmentation loss, and the correction instruction loss.

[0153] The iterative optimization described above is a method that gradually approaches the optimal solution by repeatedly improving the process. The core of iterative optimization lies in breaking down problem-solving into multiple iterative steps, with each step adjusting and improving based on the results of the previous step.

[0154] Furthermore, iterative optimization is used to reduce the loss of the instruction generator and the segmentation decoder across the attention mechanism. Specifically, the model's performance can be measured by calculating a composite loss function, and the model parameters can be updated using optimization algorithms such as gradient descent to minimize the loss.

[0155] Specifically, the composite loss function The formula is:

[0156]

[0157] in, Represents the initial segmentation loss, initial segmentation mask. Gold Standard Segmentation Mask Initial segmentation loss between; Represents the final segmentation loss, and the final segmentation mask. and gold standard segmentation mask The segmentation loss between classes. It should be noted that Tversky loss can be used to address class imbalance and penalties. :

[0158]

[0159] in, This represents the segmentation loss value; This represents the Tversky index; Indicates the final segmentation mask; This indicates a true positive result, which is the number of voxels predicted as lesions by both the model and the gold standard. This indicates a false positive, which is the number of voxels that the model predicts as lesions but are labeled as background by the gold standard. This indicates a false negative, which is predicted by the model as background, but is labeled as the number of voxels of the lesion in the gold standard. , Indicates hyperparameters;

[0160] This indicates the loss of the correction instruction. and target correction instructions The cross-entropy loss is calculated between the sequences. This can be a sequence-to-sequence text generation task, and the corrected instruction loss can be expressed as:

[0161]

[0162] in, Indicates the loss of the correction instruction; Indicates target correction text The total length; Represents the first in the text sequence One location index; Indicates the location The correct token; The Softmax layer of the instruction generator is in the 1st... The position is for the correct token. The assigned prediction probabilities. By minimizing the composite loss, the instruction generator and the segmentation decoder across the attention mechanism are forced to learn to generate text exactly the same as the "teacher's" correction instructions.

[0163] In one possible implementation, target correction instructions can be generated by using a large language model as a teacher model. Specifically, the difference between the object to be evaluated and the gold standard segmentation mask, along with the evaluation dependencies, can be input into a large language model. The prompt from the large language model could be something like, "Based on image coding features and text coding features, compared to the gold standard segmentation mask, the initial segmentation mask has more data in region A and less in region B. Please describe how to correct this using professional radiological terminology." The large language model can then generate target correction instructions based on these prompts.

[0164] Furthermore, the instruction generator learns as a student model. Specifically, the instruction generator receives the same input (the difference between the object to be evaluated and the gold standard segmentation mask, along with the evaluation dependency) and attempts to generate student-side correction instructions. The difference between the student-side correction instructions and the target correction instructions is calculated (e.g., using cross-entropy loss), with minimizing this difference as the optimization objective. The parameters of the instruction generator are updated through backpropagation. This iterative update of the instruction generator's parameters continues until the instruction generator learns to mimic the teacher model's "diagnostic thinking" and "expression style," enabling it to generate high-quality correction instruction bits based on the input.

[0165] Furthermore, once the instruction generator is trained, it no longer needs a gold standard segmentation mask and a large language model as a teacher model. The instruction generator directly receives the difference between the object to be evaluated and the gold standard segmentation mask, along with the evaluation dependency, as input to generate correction instructions.

[0166] like Figure 2 As shown, an embodiment of the present invention provides an image processing apparatus, which includes:

[0167] The acquisition module 201 is used to acquire the image to be processed and the initial text prompt corresponding to the image to be processed;

[0168] The first processing module 202 is used to perform visual encoding processing on the image to be processed based on prior knowledge of anatomical structure to obtain the visual encoding features of the image to be processed.

[0169] The second processing module 203 is used to perform text encoding processing on the initial text prompt based on a knowledge graph of medical knowledge to obtain the text encoding features of the initial text prompt;

[0170] The determining module 204 is used to determine the initial segmentation mask of the image to be processed and the correction instructions of the initial segmentation mask based on the visual encoding features and the text encoding features.

[0171] The third processing module 205 is used to correct the initial segmentation mask through the correction instruction to obtain the final segmentation mask.

[0172] Optionally, the acquisition module 201 is further configured to acquire a 3D image and an unstructured diagnostic report matching the 3D image; preprocess the 3D image to obtain an image to be processed; and perform natural language parsing on the unstructured diagnostic report to generate a structured initial text prompt.

[0173] Optionally, the first processing module 202 is further configured to determine an anatomically guided masking strategy based on prior knowledge of the anatomical structure; perform masking on the image to be processed using the anatomically guided masking strategy to obtain a masked image to be processed; and reconstruct the masked anatomical structure in the masked image to be processed using a 3D visual encoder to obtain the visual coding features of the image to be processed.

[0174] Optionally, the second processing module 203 is further configured to construct a graph embedding layer based on a knowledge graph of medical knowledge, and to construct a word embedding layer based on word embedding; to construct a text encoder based on the graph embedding layer and the word embedding layer; to train the text encoder using preset positive and negative sample pairs to obtain a trained text encoder; and to perform text encoding processing on the initial text prompt using the trained text encoder to obtain the text encoding features of the initial text prompt.

[0175] Optionally, the determining module 204 is further configured to use the text encoding features as keys and values, and the visual encoding features as queries, and employ cross-attention computation to fuse the initial text prompts into the visual encoding features to obtain fused visual encoding features; perform image segmentation processing on the fused visual encoding features to obtain an initial segmentation mask for the image to be processed; and, based on the difference between the initial segmentation mask and the gold standard segmentation mask, use the visual encoding features and the text encoding features as evaluation dependencies, and generate correction instructions for the initial segmentation mask through an instruction generator, wherein the gold standard segmentation mask is a segmentation mask selected by multiple experts.

[0176] Optionally, the third processing module 205 is further configured to perform text encoding processing on the correction instruction to obtain correction encoding features; based on the correction encoding features, the visual encoding features and the text encoding features, the initial segmentation mask is corrected through a cross-attention mechanism segmentation decoder to obtain the final segmentation mask.

[0177] Optionally, the apparatus is further configured to calculate the initial segmentation loss between the initial segmentation mask and the gold standard segmentation mask; and to calculate the final segmentation loss between the final segmentation mask and the gold standard segmentation mask; and to calculate the correction instruction loss between the correction instruction and the target correction instruction, wherein the target correction instruction is a correction instruction generated by the teacher model; calculate a composite loss based on the initial segmentation loss, the final segmentation loss, and the correction instruction loss; and iteratively optimize the instruction generator and the cross-attention mechanism segmentation decoder with the minimum of the composite loss as the optimization objective.

[0178] like Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described image processing methods.

[0179] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to execute image processing methods, wherein:

[0180] The processor 301 executes the calculator program for the image processing method stored in the memory 302, and performs the following steps:

[0181] Acquire the image to be processed and the initial text prompt corresponding to the image to be processed;

[0182] Based on prior knowledge of anatomical structure, the image to be processed is subjected to visual coding to obtain the visual coding features of the image to be processed.

[0183] Furthermore, based on a knowledge graph of medical knowledge, the initial text prompt is processed by text encoding to obtain the text encoding features of the initial text prompt;

[0184] Based on the visual encoding features and the text encoding features, the initial segmentation mask of the image to be processed and the correction instructions of the initial segmentation mask are determined.

[0185] The initial segmentation mask is modified using the correction instruction to obtain the final segmentation mask.

[0186] Optionally, the acquisition of the image to be processed and the initial text prompt corresponding to the image to be processed, executed by the processor 301, includes:

[0187] Acquire 3D images and unstructured diagnostic reports that match the 3D images;

[0188] The 3D image is preprocessed to obtain the image to be processed;

[0189] Furthermore, the unstructured diagnostic report is parsed using natural language to generate structured initial text prompts.

[0190] Optionally, the processor 301 executes the prior knowledge based on anatomical structure to perform visual encoding processing on the image to be processed, obtaining the visual encoded features of the image to be processed, including:

[0191] Based on prior knowledge of anatomical structures, determine anatomical structure-guided masking strategies;

[0192] The anatomically guided masking strategy is used to mask the image to be processed, resulting in a masked image.

[0193] The anatomical structures that were covered in the masked image to be processed are reconstructed using a 3D visual encoder to obtain the visual coding features of the image to be processed.

[0194] Optionally, the knowledge graph based on medical knowledge executed by processor 301 performs text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt, including:

[0195] A graph embedding layer is constructed based on a knowledge graph of medical knowledge, and a word embedding layer is constructed based on word embedding.

[0196] A text encoder is constructed based on the graph embedding layer and the word embedding layer;

[0197] The text encoder is trained by using preset positive and negative sample pairs to obtain a trained text encoder.

[0198] The trained text encoder is used to perform text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt.

[0199] Optionally, the processor 301 executes the instruction to determine the initial segmentation mask of the image to be processed based on the visual encoding features and the text encoding features, and the instruction to correct the initial segmentation mask, including:

[0200] Using the text encoding features as keys and values, and the visual encoding features as queries, cross-attention computation is used to fuse the initial text prompts into the visual encoding features to obtain fused visual encoding features;

[0201] The fused visual coding features are used for image segmentation to obtain the initial segmentation mask of the image to be processed;

[0202] Based on the difference between the initial segmentation mask and the gold standard segmentation mask, the visual encoding features and the text encoding features are used as evaluation dependencies. A correction instruction for the initial segmentation mask is generated by an instruction generator. The gold standard segmentation mask is a segmentation mask selected by multiple experts.

[0203] Optionally, the processor 301 executes the correction instruction to correct the initial segmentation mask to obtain the final segmentation mask, including:

[0204] The correction instruction is processed by text encoding to obtain correction encoding features;

[0205] Based on the modified encoding features, the visual encoding features, and the text encoding features, the initial segmentation mask is modified by a segmentation decoder with a cross-attention mechanism to obtain the final segmentation mask.

[0206] Optionally, the method executed by processor 301 further includes:

[0207] Calculate the initial segmentation loss between the initial segmentation mask and the gold standard segmentation mask;

[0208] And, calculate the final segmentation loss between the final segmentation mask and the gold standard segmentation mask;

[0209] And, calculate the correction instruction loss between the correction instruction and the target correction instruction, wherein the target correction instruction is the correction instruction generated by the teacher model;

[0210] Calculate the composite loss based on the initial segmentation loss, the final segmentation loss, and the correction instruction loss;

[0211] With the goal of minimizing the composite loss, the instruction generator and the segmentation decoder with the cross-attention mechanism are iteratively optimized.

[0212] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the image processing method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0213] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, The method includes the following steps: Acquire the image to be processed and the initial text prompt corresponding to the image to be processed; Based on prior knowledge of anatomical structure, the image to be processed is subjected to visual coding to obtain the visual coding features of the image to be processed. Furthermore, based on a knowledge graph of medical knowledge, the initial text prompt is processed by text encoding to obtain the text encoding features of the initial text prompt; Based on the visual encoding features and the text encoding features, the initial segmentation mask of the image to be processed and the correction instructions of the initial segmentation mask are determined. The initial segmentation mask is modified using the correction instruction to obtain the final segmentation mask.

2. The image processing method as described in claim 1, characterized in that, The acquisition of the image to be processed and the corresponding initial text prompt includes: Acquire 3D images and unstructured diagnostic reports that match the 3D images; The 3D image is preprocessed to obtain the image to be processed; Furthermore, the unstructured diagnostic report is parsed using natural language to generate structured initial text prompts.

3. The image processing method as described in claim 2, characterized in that, The prior knowledge based on anatomical structure is used to perform visual encoding processing on the image to be processed, obtaining the visual encoding features of the image to be processed, including: Based on prior knowledge of anatomical structures, determine anatomical structure-guided masking strategies; The anatomically guided masking strategy is used to mask the image to be processed, resulting in a masked image. The anatomical structures that were covered in the masked image to be processed are reconstructed using a 3D visual encoder to obtain the visual coding features of the image to be processed.

4. The image processing method as described in claim 2, characterized in that, The knowledge graph based on medical knowledge performs text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt, including: A graph embedding layer is constructed based on a knowledge graph of medical knowledge, and a word embedding layer is constructed based on word embedding. A text encoder is constructed based on the graph embedding layer and the word embedding layer; The text encoder is trained by using preset positive and negative sample pairs to obtain a trained text encoder. The trained text encoder is used to perform text encoding processing on the initial text prompt to obtain the text encoding features of the initial text prompt.

5. The image processing method according to any one of claims 1 to 4, characterized in that, The step of determining the initial segmentation mask of the image to be processed and the correction instructions for the initial segmentation mask based on the visual encoding features and the text encoding features includes: Using the text encoding features as keys and values, and the visual encoding features as queries, cross-attention computation is used to fuse the initial text prompts into the visual encoding features to obtain fused visual encoding features; The fused visual coding features are used for image segmentation to obtain the initial segmentation mask of the image to be processed; Based on the difference between the initial segmentation mask and the gold standard segmentation mask, the visual encoding features and the text encoding features are used as evaluation dependencies. A correction instruction for the initial segmentation mask is generated by an instruction generator. The gold standard segmentation mask is a segmentation mask selected by multiple experts.

6. The image processing method as described in claim 5, characterized in that, The step of correcting the initial segmentation mask using the correction instruction to obtain the final segmentation mask includes: The correction instruction is processed by text encoding to obtain correction encoding features; Based on the modified encoding features, the visual encoding features, and the text encoding features, the initial segmentation mask is modified by a segmentation decoder with a cross-attention mechanism to obtain the final segmentation mask.

7. The image processing method as described in claim 6, characterized in that, The method further includes: Calculate the initial segmentation loss between the initial segmentation mask and the gold standard segmentation mask; And, calculate the final segmentation loss between the final segmentation mask and the gold standard segmentation mask; And, calculate the correction instruction loss between the correction instruction and the target correction instruction, wherein the target correction instruction is the correction instruction generated by the teacher model; Calculate the composite loss based on the initial segmentation loss, the final segmentation loss, and the correction instruction loss; With the goal of minimizing the composite loss, the instruction generator and the segmentation decoder with the cross-attention mechanism are iteratively optimized.

8. An image processing apparatus, characterized in that, The image processing device includes: The acquisition module is used to acquire the image to be processed and the initial text prompt corresponding to the image to be processed. The first processing module is used to perform visual encoding processing on the image to be processed based on prior knowledge of anatomical structure to obtain the visual encoding features of the image to be processed. The second processing module is used to perform text encoding processing on the initial text prompt based on a knowledge graph of medical knowledge to obtain the text encoding features of the initial text prompt. The determination module is used to determine the initial segmentation mask of the image to be processed and the correction instructions of the initial segmentation mask based on the visual encoding features and the text encoding features. The third processing module is used to correct the initial segmentation mask using the correction instruction to obtain the final segmentation mask.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 7.

Images