Multi-condition generation method and device for medical vision task

Abstract

The invention discloses a multi-condition generation method and device for a medical vision task, and relates to the technical field of image processing. The method comprises the following steps: constructing a multi-modal pathology priori data set; encoding data in the multi-modal pathology priori data set into unified feature representation to obtain encoded multi-modal conditions; constructing a generation model based on a diffusion Transform, wherein the generation model comprises a multi-path condition injection module; wherein the multi-path condition injection module is used for injecting coded multi-mode conditions into different levels of a diffusion Transform; multi-level pathology priori is introduced as a dynamic constraint in the process of generating a synthetic pathology image by the generative model; and performing multi-dimensional quantitative evaluation on the generated synthetic pathological image, and screening out an image for a medical vision task. The problem that a current medical pathological image generation technology is insufficient in condition expression capability and cannot support controllable generation of fine-grained and high-fidelity images is solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multi-condition generation method and apparatus for medical vision tasks. Background Technology

[0002] Medical image synthesis is a key technology driving the evolution of digital pathology and other fields, and high-quality pathological image data is the foundation for training diagnostic models.

[0003] Currently, researchers are exploring medical image generation methods based on diffusion models, with conditionally controllable generation considered a core approach. This allows users to guide generation with structured priors, ensuring the synthetic images possess plausibility and realism. The GLIGEN model, by introducing a gated attention mechanism, achieves controllable generation of open sets based on bounding boxes and text, demonstrating significant success in the field of natural images. However, its conditional input only supports coarse-grained rectangular boxes, failing to depict the microscopic structural details of pathological images and ignoring the multimodal semantic alignment requirements of medical images, easily leading to false lesions. The LD-CVAE model attempts to fuse histological images and genomic data to generate multimodal consistent pathological images, preliminarily exploring cross-modal association modeling. However, its generation capabilities are limited, making it difficult to reproduce high-resolution cellular details and complex textures.

[0004] Therefore, current medical pathology image generation technology suffers from insufficient conditional expression capabilities, making it unable to support the controllable generation of fine-grained, high-fidelity images. Summary of the Invention

[0005] In this embodiment of the application, a multi-condition image generation method for medical vision tasks is provided, which solves the problem that the current medical pathology image generation technology has insufficient conditional expression ability and cannot support the controllable generation of fine-grained, high-fidelity images.

[0006] In a first aspect, embodiments of this application provide a multi-condition image generation method for medical vision tasks. The method includes: constructing a multimodal pathological prior dataset; wherein the data in the multimodal pathological prior dataset includes medical text descriptions, pathological images, and cell nucleus instance segmentation masks; encoding the data in the multimodal pathological prior dataset into unified feature representations to obtain encoded multimodal conditions; constructing a generative model based on a diffusion Transformer, the generative model including a multi-path condition injection module; wherein the multi-path condition injection module is used to inject the encoded multimodal conditions into different levels of the diffusion Transformer; designing a pathological perception diffusion scheduling mechanism to introduce multi-level pathological priors as dynamic constraints during the generation of synthetic pathological images by the generative model; and performing multi-dimensional quantitative evaluation on the generated synthetic pathological images to select images suitable for medical vision tasks.

[0007] In one possible implementation, encoding the data in the multimodal pathological prior dataset into a unified feature representation to obtain the encoded multimodal conditions includes: encoding medical text descriptions into context-aware semantic embedding vectors using a pre-trained medical language model; encoding pathological images into tissue self-embedding feature vectors using a pre-trained pathological visual encoder; and encoding cell nucleus instance segmentation masks into multi-scale spatial feature maps using a convolutional encoder.

[0008] In one possible implementation, the construction of a generative model based on a diffusion Transformer includes a multi-path conditional injection module, comprising: global text conditional injection, meso-level tissue feature injection, and micro-level kernel layout injection; global text conditional injection includes: using an adaptive layer normalization mechanism to inject context-aware semantic embedding vectors into each DiT block of the diffusion Transformer; meso-level tissue feature injection includes: using a global cross-attention mechanism to take the tissue self-embedded feature vector as the key and value, the current feature map as the query, obtaining attention weights, and then weighting and fusing the features in the self-embedded feature vectors using the attention weights, and injecting the fused features into the bottleneck layer and higher levels of the decoder of the diffusion Transformer; micro-level kernel layout injection includes: extracting the cell kernel instance segmentation mask into a multi-scale spatial feature map through a convolutional encoder, and then injecting it into the shallow decoder module of the diffusion Transformer through a local spatial alignment cross-attention mechanism; wherein, the local spatial alignment cross-attention mechanism restricts the matching range of the query and the key to within a local window, and the attention weights are aligned with the spatial position of the cell kernel mask through coordinate mapping.

[0009] In one possible implementation, the pathological perception diffusion scheduling mechanism introduces multi-level pathological priors as dynamic constraints during the generation of synthetic pathological images by the generative model. This includes: the pathological perception diffusion scheduling mechanism uses tissue self-embedded feature vectors and multi-scale spatial feature maps as multi-level pathological priors in each step of the reverse denoising diffusion; wherein the tissue self-embedded feature vectors are obtained by encoding the pathological image using a pre-trained pathological visual encoder, and the multi-scale spatial feature maps are obtained by encoding the cell nucleus instance segmentation mask using a convolutional encoder; the multi-level pathological priors are used as dynamic constraints to construct a multi-task joint optimization objective.

[0010] In one possible implementation, the expression for the joint multi-task optimization objective is: ;in, The overall training objective function is... The training objective for the standard diffusion model. For kernel layout consistency regularization, To organize semantically consistent regular expressions, The weight hyperparameter for the layout consistency regularization term. The weight hyperparameters for organizing semantic consistency regularization terms.

[0011] In one possible implementation, the step of performing multi-dimensional quantitative evaluation on the generated synthetic pathological images to select images for medical vision tasks includes: using a pre-trained cell nucleus instance segmentation model to perform end-to-end inference on the synthetic pathological images and outputting predicted cell nucleus segmentation maps; evaluating the accuracy of the position, density, and morphology of cell nuclei in the predicted cell nucleus segmentation maps using Dice coefficients and F1-scores; for images whose accuracy meets a preset standard, extracting global features of the synthetic pathological images using a pre-trained tissue classification encoder and calculating the cosine similarity between the global features and the input tissue self-embedded features, and selecting samples with consistent tissue semantics according to a preset first threshold; introducing a pre-trained image-text matching model to calculate the embedding similarity between the samples and the input medical text description, and selecting images for medical vision tasks according to a preset second threshold.

[0012] Secondly, embodiments of this application provide a multi-condition image generation device for medical vision tasks. The device includes: a dataset construction module for constructing a multimodal pathological prior dataset; wherein the data in the multimodal pathological prior dataset includes medical text descriptions, pathological images, and cell nucleus instance segmentation masks; an acquisition module for encoding the data in the multimodal pathological prior dataset into unified feature representations to obtain encoded multimodal conditions; a model construction module for constructing a generative model based on a diffusion Transformer, the generative model including a multi-path condition injection module; wherein the multi-path condition injection module injects the encoded multimodal conditions into different levels of the diffusion Transformer; a constraint module for designing a pathological perception diffusion scheduling mechanism, introducing multi-level pathological priors as dynamic constraints during the generation of synthetic pathological images by the generative model; and an evaluation module for performing multi-dimensional quantitative evaluation of the generated synthetic pathological images and selecting images suitable for medical vision tasks.

[0013] Thirdly, embodiments of this application provide a multi-condition image generation server for medical vision tasks, including a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation of the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions, which, when executed by a computer, enable the method described in the first aspect or any possible implementation thereof.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects: This application provides a multi-condition image generation method for medical vision tasks, which constructs a multimodal pathological prior dataset. The data in the multimodal pathological prior dataset includes medical text descriptions, pathological images, and cell nucleus instance segmentation masks. The data in the multimodal pathological prior dataset are encoded into unified feature representations to obtain the encoded multimodal conditions. A generative model based on a diffusion Transformer is constructed, including a multi-path condition injection module. This module injects the encoded multimodal conditions into different levels of the diffusion Transformer. A pathological perception diffusion scheduling mechanism is designed to introduce multi-level pathological priors as dynamic constraints during the generation of synthetic pathological images. The generated synthetic pathological images are quantitatively evaluated in multiple dimensions to select images suitable for medical vision tasks. This method achieves end-to-end, high-fidelity, and editable pathological image generation, from macroscopic diagnostic statements to microscopic cell arrangements, providing key technical support for constructing safe and reliable medical synthetic data. It solves the problem of insufficient conditional expression capabilities in current medical pathological image generation technologies, which cannot support the controllable generation of fine-grained, high-fidelity images. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a multi-condition generation method for medical vision tasks provided in this application embodiment; Figure 2 A schematic diagram of a multi-condition generation device for medical vision tasks provided in an embodiment of this application; Figure 3 This is a schematic diagram of a multi-condition generation server for medical vision tasks provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0020] This application provides a multi-condition image generation method for medical vision tasks, such as... Figure 1 As shown, the method includes steps S101 to S105. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order for a multi-condition image generation method for medical vision tasks. The execution order can be adjusted to achieve the desired final result. Figure 1 The steps shown can be performed in parallel or in reverse order.

[0021] S101: Construct a multimodal pathology prior dataset. The data in the multimodal pathology prior dataset includes medical text descriptions, pathological images, and cell nucleus instance segmentation masks.

[0022] Specifically, this application prioritizes high-resolution (≥0.5 μm / pixel) whole-slide images (WSIs) to ensure clear image details, good staining consistency, and the absence of severe artifacts or section defects. The dataset must provide pixel-level or instance-level annotations, with accurate spatial registration between the annotations and the image, precisely reflecting pathological features (such as tumor regions and cell nucleus locations). It should cover multiple cancer types (such as lung cancer, breast cancer, and colorectal cancer), different differentiation grades, and typical histological subtypes to ensure data diversity and model generalization ability.

[0023] Based on the above criteria, the following representative data sources were integrated. Tissue classification task: Integrating the TCGA (The Cancer Genome Atlas) and CPTAC (Clinical Proteomic Tumor Analysis Consortium) datasets to provide structured diagnostic reports covering various histological subtypes of malignant tumors.

[0024] Employing high-precision cell nucleus annotation datasets such as CoNSeP and NuCLS, this dataset provides pixel-level instance segmentation annotations of cell nuclei and distinguishes different cell types (e.g., epithelial cells, lymphocytes). For tumor region detection and segmentation, the datasets Camelyon16, Camelyon17, and PAIP are integrated to provide binary masks of tumor invasive regions annotated by multiple pathology experts, covering the complete lineage from carcinoma in situ to invasive carcinoma. A multimodal alignment dataset is also included, incorporating the PathMNIST-v2 and BACH datasets. Each image is accompanied by detailed histological descriptions and expert annotations, supporting joint modeling of text, image, and annotation.

[0025] S102: Encode the data in the multimodal pathological prior dataset into a unified feature representation to obtain the encoded multimodal conditions.

[0026] The data in the multimodal pathological prior dataset are encoded into a unified feature representation to obtain the encoded multimodal conditions, including the following.

[0027] A pre-trained medical language model is used to encode medical text descriptions into context-aware semantic embedding vectors.

[0028] Specifically, standardized pathological diagnosis text can be input into the BioClinicalBERT pre-trained language model. This model performs word-by-word analysis of the text, combining contextual information to convert each word into a high-dimensional vector representation. Through complex computation and semantic aggregation mechanisms within the model, it ultimately outputs a context-aware semantic embedding vector. This semantic embedding vector can comprehensively and accurately capture the pathological semantic information in the text, such as key information like tumor type, grade, and histological features.

[0029] A pre-trained pathological visual encoder is used to encode pathological images into tissue self-embedding feature vectors.

[0030] Specifically, pathological images contain macroscopic structural information about tissues, such as glandular arrangement patterns, stroma ratios, and tumor region distribution. This information is crucial for generating images with realistic pathological structures. For example, a pre-trained HIPT model can be used to encode WSI (pathological image) image patches (512×512) and extract tissue self-embedding vectors.

[0031] A convolutional encoder is used to encode the cell nucleus instance segmentation mask into a multi-scale spatial feature map.

[0032] Specifically, instance segmentation results from datasets such as CoNSeP and NuCLS can be converted into binary masks and kernel center point heatmaps, and their size can be normalized (0.5μm / pixel).

[0033] Specifically, a convolutional neural network (CNN) can be used to encode the cell nucleus instance segmentation mask as... Multi-scale spatial feature maps. CNN was chosen because it can effectively extract hierarchical spatial features from two-dimensional mask images, such as cell nucleus density and orientation field.

[0034] All WSIs were standardized using the H&E method to eliminate batch effects. This application also performed format conversion on all labeled data to adapt to the input requirements of the model in this application.

[0035] Following the above procedures, this application constructed a multimodal pathology prior dataset, PathoCond. This dataset contains 42,816 high-quality image-conditional triples. Each triple includes a 512×512 resolution H&E-stained image patch, a medical text description (Ttext) covering histological classification, grading, and key pathological features, a 1024-dimensional tissue self-embedding feature (Ftissue) encoding mesoscopic tissue structure information, and a cell nucleus instance segmentation mask (Mnuclei) characterizing microscopic spatial arrangement. The dataset covers 8 major cancer types and 12 histological subtypes, encompassing a complete pathological lineage from low to high grade and from in situ to invasive. All annotations have a spatial registration error of less than 2 pixels with the images.

[0036] S103: Construct a generative model based on diffusion Transformer, which includes a multi-path condition injection module. The multi-path condition injection module is used to inject encoded multimodal conditions into different levels of the diffusion Transformer.

[0037] Construct a generative model based on diffusion Transformer. The generative model includes a multi-path condition injection module, which includes the following:

[0038] The multi-path conditional injection module includes global text conditional injection, meso-level organizational feature injection, and micro-level kernel layout injection.

[0039] Global text conditional injection involves injecting context-aware semantic embedding vectors into each DiT block of the diffusion Transformer using an adaptive layer normalization mechanism.

[0040] Specifically, the expression for global text conditional injection is: .in, For adaptive layer normalization mechanism, The input features in the current DiT block, For context-aware semantic embedding vectors, Input features The mean calculated along the channel. Input features Standard deviation calculated along the channel, For the reason Predicted scaling parameters For the reason Predicted offset parameters.

[0041] This adaptive layer normalization mechanism dynamically adjusts the normalization parameters of each layer, ensuring that the overall generated style of the network (such as differentiation degree and staining intensity) remains consistent with the input text description, thus achieving macro-control of "visualization guided by semantics". For example, when the input text description is "high nucleoplasm ratio, crowded nuclei", the model can accurately generate dense, atypical nuclei in the specified region.

[0042] The meso-level organizational feature injection includes: using the organization's self-embedded feature vector as the key and value and the current feature map as the query through a global cross-attention mechanism to obtain attention weights; after weighted fusion of the features in the self-embedded feature vector through the attention weights, the fused features are injected into the bottleneck layer of the diffusion Transformer and the higher layers of the decoder.

[0043] Specifically, the expression for mesoscopic tissue feature injection is: .in, This is the fusion result of the global cross-attention mechanism. For activation function, The query vector is obtained by transforming the input feature map. The key matrix is ​​obtained by linear transformation of the organization's self-embedded eigenvectors. To and The value matrix obtained by transforming the self-embedded eigenvectors of homologous organizations through another linear transformation. This is the scaling factor.

[0044] The above operations ensure that the overall tissue structure (such as glandular arrangement pattern and degree of stromal fibrosis) of the generated synthetic pathological image is highly consistent with the actual tissue distribution, avoiding pathological errors such as "tissue misalignment" or "structural disorder". For example, when generating pathological images involving glandular tissue, it can be guaranteed that the arrangement of glands conforms to the actual pathological situation.

[0045] The micro-kernel layout injection process includes: after the cell kernel instance segmentation mask is extracted into a multi-scale spatial feature map by a convolutional encoder, it is injected into the shallow decoder module of the diffusion Transformer through a local spatial alignment cross-attention mechanism. Specifically, the local spatial alignment cross-attention mechanism limits the matching range of the query and key to within a local window, and the attention weights are aligned with the spatial position of the cell kernel mask through coordinate mapping.

[0046] Specifically, the expression for micro-kernel layout injection is: .in, Align the output feature map of the cross-attention mechanism with the entire local space. For activation function, For local query features, Characterized by nuclear bonds. Characteristic of cell nuclear values, Scaling factor This is an attention mask generated based on a cell nucleus instance segmentation mask. This is element-wise multiplication.

[0047] Microscopic nuclear layout injection enables pixel-level anchoring control of cellular structures, ensuring that the generated nuclei strictly adhere to the input pathological layout in terms of position, density, and polarity, fundamentally guaranteeing microscopic realism. For example, it can accurately generate nuclei with positions and morphologies that conform to specific pathological layouts.

[0048] S104: Design a pathological perception diffusion scheduling mechanism, and introduce multi-level pathological priors as dynamic constraints in the process of generating synthetic pathological images by the generative model.

[0049] A pathological perception diffusion scheduling mechanism is designed, which introduces multi-level pathological priors as dynamic constraints in the process of generating synthetic pathological images by the generative model, including the following:

[0050] The pathological perception diffusion scheduling mechanism uses tissue self-embedded feature vectors and multi-scale spatial feature maps as multi-level pathological priors in each step of the reverse denoising diffusion. Specifically, the tissue self-embedded feature vectors are obtained by encoding the pathological image using a pre-trained pathological visual encoder, and the multi-scale spatial feature maps are obtained by encoding the cell nucleus instance segmentation mask using a convolutional encoder.

[0051] Specifically, the pathological perception diffusion scheduling mechanism breaks through the limitations of traditional diffusion models that rely solely on latent variable autoregressive evolution, and achieves stage-by-stage assurance of the controllability of the generation path and pathological consistency.

[0052] By using multi-level pathological priors as dynamic constraints, a multi-task joint optimization objective is constructed.

[0053] The expression for the multi-task joint optimization objective is: .in, The overall training objective function is... The training objective for the standard diffusion model. For kernel layout consistency regularization, To organize semantically consistent regular expressions, The weight hyperparameter for the layout consistency regularization term. The weight hyperparameters for organizing semantic consistency regularization terms.

[0054] and It can be dynamically adjusted according to task requirements. For example, in tasks that emphasize cellular-level details (such as nuclear atypia grading), it can be increased. In organizational segmentation tasks, it enhances... The weight.

[0055] .in, The training objective of the standard diffusion model, i.e., the loss function of the standard diffusion model, measures the noise predicted by the model. With real noise The differences between them For the desired operation, These are real pathological images. For the time step of the diffusion process, For parameterized noise prediction networks, In time step Noisy data at that time.

[0056] Since the training objective of the standard diffusion model only focuses on pixel-level reconstruction accuracy, it cannot guarantee the medical correctness of the generated synthetic pathological images in terms of the rationality of tissue structure and the realism of cell spatial arrangement. To address this, this application introduces two key regularization terms to construct a multi-task joint optimization objective.

[0057] .in, For kernel layout consistency regularization, , indicating at time step At that time, for the currently generated synthetic pathological image Pre-trained kernel segmentation network using frozen weights By performing forward inference, a predicted map of cell nucleus locations is obtained. The input is a mask for the actual cell nucleus layout.

[0058] The kernel layout consistency regularization term directly affects the generator during backpropagation. The gradient update forces the model to actively "find" cell configurations that match the input kernel layout during the denoising process, which effectively prevents random aggregation or misalignment of cell kernels, especially in the early diffusion stage (high noise level).

[0059] .in, To organize semantically consistent regular expressions, , indicating a pre-trained tissue classification encoder For the currently generated synthetic pathological images The tissue feature distribution obtained after feature extraction This represents the self-embedded features of the target organization in the input. for Divergence.

[0060] The tissue semantic consistency regularization term ensures that the generated synthetic pathological images continuously approximate the visual fingerprint of the target tissue (such as macroscopic patterns like gland density, stroma ratio, and necrotic area proportion) throughout the entire evolution process. Even in the intermediate stage where local textures are blurred, the model can still make stable inferences based on global semantic anchors.

[0061] S105: Perform multi-dimensional quantitative evaluation on the generated synthetic pathological images and select images for medical vision tasks.

[0062] The generated synthetic pathological images were subjected to multi-dimensional quantitative evaluation to select images suitable for medical vision tasks, including the following: A pre-trained cell nucleus instance segmentation model is used to perform end-to-end inference on synthetic pathological images and output the predicted cell nucleus segmentation map.

[0063] The accuracy of the predicted nucleus location, density, and morphology in the nucleus segmentation map was evaluated using the Dice coefficient and F1-score.

[0064] .in, This is the Dice coefficient. The input is a mask for the actual cell nucleus layout. This is the predicted cell nucleus segmentation map. The Dice coefficient measures the similarity between the predicted cell nucleus segmentation result and the ground truth annotation (the input ground truth cell nucleus layout mask), with a value ranging from 0 to 1. A value closer to 1 indicates a higher degree of overlap between the predicted result and the ground truth annotation, and a better segmentation effect; a value closer to 0 indicates a worse segmentation effect. In this evaluation, if... If the microstructure of the sample is distorted, it will be rejected.

[0065] .in, For F1-score, A true instance is considered a correctly predicted nucleus instance when the IoU between a predicted nucleus instance and a true nucleus instance (determined based on the input true nucleus layout mask) is greater than or equal to 0.5. A false positive is a case where a region is incorrectly predicted as a nucleus but has an IoU of less than 0.5 with any real nucleus instance. A false negative is a real instance of a cell nucleus that is not detected because there is no corresponding predicted instance with an IoU greater than or equal to 0.5 in the prediction results.

[0066] The F1-score combines precision (the proportion of true positives out of predicted positives) and recall (the proportion of true positives out of actual positives), providing a comprehensive assessment of the accuracy and completeness of predictions. Its value ranges from 0 to 1. A value closer to 1 indicates better prediction performance; a value closer to 0 indicates worse prediction performance. If the F1-score is less than 0.65, the sample is considered to have distorted microstructure and is discarded.

[0067] For images whose accuracy meets the preset standard, a pre-trained tissue classification encoder is used to extract global features of the synthetic pathological image, and the cosine similarity between the global features and the input tissue self-embedded features is calculated. Based on a preset first threshold, samples with consistent tissue semantics are selected.

[0068] Specifically, the preset standards are a Dice coefficient greater than or equal to 0.7 and an F1-score greater than or equal to 0.65. The preset first threshold is 0.8, i.e. , A first threshold is set in advance. Samples below the first threshold are considered to have tissue semantic drift and cannot reflect the target tissue type. Images that are greater than or equal to the first threshold are considered to be tissue semantically consistent samples.

[0069] The formula for calculating cosine similarity is: .in, This is the result of the cosine similarity calculation. , indicating a pre-trained tissue classification encoder For the currently generated synthetic pathological images The tissue feature distribution obtained after feature extraction , representing the self-embedded features of the target tissue input. The tissue classification encoder in this application can be Phikon or HIPT.

[0070] A pre-trained image-text matching model is introduced to calculate the embedding similarity between the sample and the input medical text description. Based on a pre-set second threshold, images for medical vision tasks are selected.

[0071] Specifically, the image-text matching model can be CLIP-Hist or PathCLIP. The formula for calculating embedding similarity is: .in, To embed the results of similarity calculations, As a sample, For the input medical text description, This refers to the image encoder part in a pre-trained image-text matching model. This refers to the text encoder part in a pre-trained image-text matching model. This is a transpose operation.

[0072] If the calculated embedding similarity is lower than the 25th percentile of the same type of real sample (i.e., the pre-set second threshold), it is considered to have "semantic illusion" (such as the text description is "low-grade tumor", but the image shows a high nuclear-cytoplasmic ratio and dense nuclear division), and is judged as unqualified and not used as an image for medical vision tasks; otherwise, it is used as an image for medical vision tasks.

[0073] This application also provides a multi-condition image generation device 200 for medical vision tasks, such as... Figure 2 As shown, the device includes: a dataset construction module 201, an acquisition module 202, a model construction module 203, a constraint module 204, and an evaluation module 205.

[0074] The dataset construction module 201 is used to construct a multimodal pathology prior dataset. The data in the multimodal pathology prior dataset includes medical text descriptions, pathological images, and cell nucleus instance segmentation masks.

[0075] The acquisition module 202 is used to encode the data in the multimodal pathological prior dataset into a unified feature representation to obtain the encoded multimodal conditions.

[0076] The model building module 203 is used to build a generative model based on diffusion Transformer. The generative model includes a multi-path condition injection module. The multi-path condition injection module is used to inject encoded multimodal conditions into different levels of diffusion Transformer.

[0077] The constraint module 204 is used to design a pathological perception diffusion scheduling mechanism, which introduces multi-level pathological priors as dynamic constraints during the generation of synthetic pathological images by the generative model.

[0078] The evaluation module 205 is used to perform multi-dimensional quantitative evaluation of the generated synthetic pathological images and select images for medical vision tasks.

[0079] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0081] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, for example, as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0082] like Figure 3 As shown in the figure, this application embodiment also provides a multi-condition image generation server for medical vision tasks, including a memory 301 and a processor 302; the memory 301 is used to store computer-executable instructions; the processor 302 is used to execute the computer-executable instructions to implement the multi-condition image generation method for medical vision tasks described above in this application embodiment.

[0083] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a computer, can implement the multi-condition image generation method for medical vision tasks described above in this application.

[0084] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in the embodiments of this application.

[0085] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations.

[0086] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A multi-condition image generation method for medical vision tasks, characterized in that, The method comprises the following steps: Construct a multi-modal pathological prior dataset; wherein the data in the multi-modal pathological prior dataset includes medical text description, pathological image and cell nucleus instance segmentation mask; Encode the data in the multi-modal pathological prior dataset into unified feature representation respectively to obtain the encoded multi-modal condition; Construct a diffusion Transformer-based generation model, which includes a multi-path condition injection module; wherein the multi-path condition injection module is used to inject the encoded multi-modal condition into different levels of the diffusion Transformer; Design a pathology-aware diffusion scheduling mechanism to introduce multi-level pathological prior as a dynamic constraint during the generation of synthetic pathological images by the generation model; Perform multi-dimensional quantitative evaluation on the generated synthetic pathological images to screen out images for medical visual tasks.

2. The multi-conditional image generation method for medical vision tasks according to claim 1, characterized in that, The encoding of the data in the multi-modal pathological prior dataset into unified feature representation respectively to obtain the encoded multi-modal condition comprises: Using a pre-trained medical language model to encode the medical text description into context-aware semantic embedding vectors; Using a pre-trained pathological visual encoder to encode the pathological image into tissue self-embedding feature vectors; Using a convolutional encoder to encode the cell nucleus instance segmentation mask into multi-scale spatial feature maps.

3. The multi-conditional image generation method for medical vision tasks according to claim 2, characterized in that, The diffusion Transformer-based generation model includes a multi-path condition injection module, which comprises: The multi-path condition injection module includes global text condition injection, mesoscopic tissue feature injection and microscopic nuclear layout injection; The global text condition injection comprises: using an adaptive layer normalization mechanism to inject the context-aware semantic embedding vectors into each DiT block of the diffusion Transformer; The mesoscopic tissue feature injection comprises: using the tissue self-embedding feature vectors as keys and values and the current feature map as a query to obtain attention weights through a global cross-attention mechanism, weighting and fusing the features in the self-embedding feature vectors through the attention weights, and then injecting the fused features into the bottleneck layer and the high-level decoder of the diffusion Transformer; The microscopic nuclear layout injection comprises: after the cell nucleus instance segmentation mask is extracted into multi-scale spatial feature maps by the convolutional encoder, the local spatial alignment cross-attention mechanism is used to inject the shallow decoder module of the diffusion Transformer; wherein the local spatial alignment cross-attention mechanism limits the matching range of the query and the key within the local window, and the attention weights are aligned with the spatial position of the cell nucleus mask through coordinate mapping.

4. The multi-conditional image generation method for medical vision tasks according to claim 1, wherein, The pathology-aware diffusion scheduling mechanism introduces multi-level pathological prior as a dynamic constraint during the generation of synthetic pathological images by the generation model, which comprises: The pathology-aware diffusion scheduling mechanism uses the tissue self-embedding feature vectors and the multi-scale spatial feature maps as multi-level pathological prior at each step of the reverse denoising diffusion; wherein the tissue self-embedding feature vectors are obtained by encoding the pathological image using the pre-trained pathological visual encoder, and the multi-scale spatial feature maps are obtained by encoding the cell nucleus instance segmentation mask using the convolutional encoder; The multi-level pathological prior is used as a dynamic constraint to construct a multi-task joint optimization target.

5. The multi-conditional image generation method for medical vision tasks according to claim 4, characterized in that, The expression of the multi-task joint optimization target is: ; wherein, is an overall training target function, is a training target of a standard diffusion model, is a kernel layout consistency regularization term, is an organizational semantic consistency regularization term, is a weight hyperparameter of the layout consistency regularization term, is a weight hyperparameter of the organizational semantic consistency regularization term.

6. The multi-conditional image generation method for medical vision tasks according to claim 1, wherein, The generated synthetic pathology image is subjected to multi-dimensional quantitative evaluation, and images for medical visual tasks are screened out, including: A pre-trained nucleus instance segmentation model is used to perform end-to-end inference on the synthetic pathology image, and a predicted nucleus segmentation map is output; The accuracy of the location, density and morphology of the nucleus in the predicted nucleus segmentation map is evaluated through the Dice coefficient and F1-score indicators; For images with accuracy reaching a preset standard, a pre-trained tissue classification encoder is used to extract global features of the synthetic pathology image, and the cosine similarity between the global features and the input tissue self-embedding features is calculated, and samples with consistent tissue semantics are screened out according to a pre-set first threshold value; A pre-trained image-text matching model is introduced to calculate the embedding similarity between the samples and the input medical text description, and images for medical visual tasks are screened out according to a pre-set second threshold value.

7. A multi-condition image generation apparatus for a medical vision task, characterized by, It includes: The dataset construction module is used to construct a multi-modal pathology prior dataset; wherein the data in the multi-modal pathology prior dataset includes medical text description, pathology image and nucleus instance segmentation mask; The acquisition module is used to encode the data in the multi-modal pathology prior dataset into a unified feature representation to obtain the encoded multi-modal condition; The model construction module is used to construct a generation model based on diffusion Transformer, and the generation model includes a multi-path condition injection module; wherein the multi-path condition injection module is used to inject the encoded multi-modal condition into different levels of diffusion Transformer; The constraint module is used to design a pathology perception diffusion scheduling mechanism, and introduce multi-level pathology prior as a dynamic constraint in the process of generating synthetic pathology image by the generation model; The evaluation module is used to perform multi-dimensional quantitative evaluation on the generated synthetic pathology image, and images for medical visual tasks are screened out.

8. A multi-condition image generation server for medical vision tasks, characterized in that, It includes a memory and a processor; The memory is used to store computer executable instructions; The processor is used to execute the computer executable instructions to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores executable instructions, and the computer executes the executable instructions to implement the method of any one of claims 1-6.

Images