Doctor-like tip generation method, and medical image synthesis method and system

Abstract

This invention provides a method and system for generating doctor-style prompts. It extracts features from each medical report in a medical report set; provides non-expert prompts and extracts features from these prompts; based on the features of the medical reports and the non-expert prompts, it retrieves the medical report in the medical report set that is most similar to the non-expert prompt; it abstracts the medical knowledge in the retrieved medical reports into a knowledge graph; it searches for meta-knowledge related to the non-expert prompts in the knowledge graph and converts the meta-knowledge into doctor-style prompts. Simultaneously, it provides a method and system for medical image synthesis based on doctor-style prompts. This invention, based on a knowledge graph constructed from medical reports, limits the scope of knowledge to the medical report domain, helping to avoid irrelevant expressive information in the prompts. The generated doctor-style prompts exclude knowledge unrelated to medical reports, bringing higher fidelity and alignment improvements to medical image synthesis tasks compared to large language models.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method for generating doctor-style prompts and a method and system for synthesizing medical images, and also to a corresponding computer terminal and computer-readable storage medium. Background Technology

[0002] Text-to-image synthesis has achieved great success in generating natural-looking images. A key reason is that ordinary users can easily write detailed prompts for the synthesis. However, for medical image synthesis, due to a lack of expertise, prompts written by ordinary users often miss or misinterpret key information from the target image.

[0003] To address the aforementioned issues, large-scale model application techniques have emerged. Among them:

[0004] Large-Scale Language Image Modeling (LSLIM) has achieved great success in synthesizing natural images. However, some medical images synthesized by LSLIM differ from real images, lacking clinical plausibility. Because ordinary users cannot write prompts like experts, some essential information may be missed or misunderstood, potentially misleading LSLIM's text-to-image generation.

[0005] The use of templates or guidelines is widely accepted when creating prompts for natural image compositing. Templates or guidelines can guide users to clarify the names and classifications of objects, while also encouraging them to describe the intent, motivation, and consequences of their actions, thereby improving the fidelity and alignment of the generated images. However, when creating prompts for medical image compositing, these templates or guidelines are often incomprehensible to the average user. Therefore, this knowledge barrier hinders developers from creating sufficiently detailed prompts for medical image compositing. Thus, templates or guidelines are not a viable solution for medical image compositing.

[0006] Another approach to cue translation is to use large language models (LLMs), which can improve the quality of cues for text-to-image tasks. However, medical LLMs are trained on biomedical databases that contain knowledge far beyond the medical image domain. Therefore, they may generate irrelevant cues outside the medical image domain, which could confuse the LLM's ability to generate text-to-image cues. Summary of the Invention

[0007] To address the aforementioned shortcomings in the prior art, this invention provides a method for generating doctor-style prompts and a method and system for synthesizing medical images, along with a corresponding computer terminal and computer-readable storage medium.

[0008] According to one aspect of the present invention, a method for generating doctor-style prompts is provided, comprising:

[0009] Extract the features of each medical report in the medical report collection;

[0010] Provide non-expert suggestions and extract features from the non-expert suggestions;

[0011] Based on the characteristics of the medical reports and the characteristics of the non-expert tips, retrieve the medical reports most similar to the non-expert tips from the medical report set;

[0012] The medical knowledge in the retrieved medical reports is abstracted into a knowledge graph;

[0013] The meta-knowledge related to the non-expert hints is searched in the knowledge graph, and the meta-knowledge is converted into doctor-like hints.

[0014] Preferably, the extraction of features from each medical report in the medical report set includes:

[0015] Each medical report r in the medical report set R i Labeled with the corresponding diagnosis, a one-hot sequence containing n diagnoses is obtained, represented as:

[0016] Provide a feature extraction model that combines the medical reports with each medical report r in R. i These are respectively used as inputs to the feature extraction model, and the corresponding feature embeddings are output.

[0017] Preferably, providing non-expert hints and extracting features from the non-expert hints includes:

[0018] Provide a non-expert hint; wherein the information in the non-expert hint is represented as: { <V s / V d In the formula, s and d represent physical signs and diseases, respectively;

[0019] A feature extraction model is provided, which takes the non-expert prompts as input and outputs the corresponding feature embeddings.

[0020] Preferably, the step of retrieving the medical report most similar to the non-expert suggestion from the medical report set based on the characteristics of the medical report and the characteristics of the non-expert suggestion includes:

[0021] Based on the characteristics of the medical reports and the characteristics of the non-expert tips, a search is performed in the medical report set to obtain the similarity (sim) between the non-expert tips and the medical reports. i for:

[0022]

[0023] In the formula, Indicates the characteristics of a medical report, v nxp Features indicating non-expert suggestions;

[0024] Select the medical report with the highest similarity. As a search result:

[0025]

[0026] In the formula, i * This indicates the sequence number of the most similar medical report found. This indicates the most similar medical report.

[0027] Preferably, the step of abstracting medical knowledge from retrieved medical reports into a knowledge graph includes:

[0028] The knowledge graph is represented as follows: Each meta-knowledge k i Represented as a triple: k i ={V source ,E,V target}, where V source V represents the subject of relations in meta-knowledge. target V represents the object of a relation in meta-knowledge, and E represents the entity (vertice) and relation (edge), respectively.

[0029] Preferably, the step of searching for meta-knowledge related to the non-expert hint in the knowledge graph includes:

[0030] Provide a pre-trained BERT model;

[0031] The knowledge graph Each meta-knowledge k in j As input to the BERT model, the corresponding embedding tensor is output. Obtain the embedding tensor set of the knowledge graph K.

[0032] The non-expert hint nxp is used as input to the BER model, and the embedding tensor e of the non-expert hint nxp is output. nxp =BERT(nxp);

[0033] Calculate the embedding tensor e nxp With each embedded tensor in the set of embedded tensors ε The cross product of the non-expert hints and each meta-knowledge in the knowledge graph is used to obtain the similarity between the hints and each meta-knowledge in the knowledge graph. j for:

[0034]

[0035] The frequency of the meta-knowledge is represented as:

[0036]

[0037] Calculate the similarity sim j The meta-knowledge k is obtained by summing the weights of the frequencies. j Score:

[0038] score j =sim j +λ·f j

[0039] In the formula, λ is a hyperparameter used to utilize similarity and frequency;

[0040] Using the aforementioned meta-knowledge k j The score for the meta-knowledge k j The meta-knowledge pieces are ranked, and the highest-scoring meta-knowledge is selected as the meta-knowledge relevant to the non-expert hints.

[0041]

[0042] In the formula, j * The index indicating the highest score. This represents the meta-knowledge that achieved the highest score.

[0043] Preferably, the step of converting the meta-knowledge into doctor-like prompts includes:

[0044] Provide a cross-focus model to find meta-knowledge As input to the cross-attention model, the corresponding doctor-like prompting embedding tensor is output:

[0045]

[0046]

[0047] In the formula, LN(·) represents layer normalization, CA(·) represents cross-interest, and FFN(·) represents feedforward network; e attn Indicates attention e attn The embedding tensor, the attention e attn The embedding tensor is based on the retrieved medical reports. and the meta-knowledge found Calculations show that Embedding tensors representing meta-knowledge The embedding tensor representing the most similar report; The embedding tensor representing the doctor's prompt;

[0048] For embedding tensors Decode and de-label to generate doctor-like prompts

[0049] Among them, the doctor-style prompt The information in is represented as: { <V s E so V d >, <V d E la V as >, <V adj E m V s / V d >}, where E so E la E m These represent the three types of relationships between entities: implied, located, and modified. V as V adj These represent two entities: an anatomical location and an adjective; where sign s implies disease d, which is located at a specific anatomical site; the severity of sign s is described using an adjective.

[0050] According to a second aspect of the present invention, a doctor-style prompt generation system is provided, comprising:

[0051] The data provision module provides a collection of medical reports and non-expert tips.

[0052] The feature extraction module is used to extract features from each medical report in the medical report collection, as well as features from non-expert tips.

[0053] The knowledge graph construction module is used to abstract medical knowledge from retrieved medical reports into a knowledge graph.

[0054] The data comparison module, based on the characteristics of the medical reports and the characteristics of the non-expert tips, retrieves the medical reports most similar to the non-expert tips in the medical report set, and searches for meta-knowledge related to the non-expert tips in the knowledge graph;

[0055] A doctor-style suggestion generation module that converts the found meta-knowledge into doctor-style suggestions.

[0056] According to a third aspect of the present invention, a method for medical image synthesis is provided, comprising:

[0057] Get doctor-like tips;

[0058] Based on the doctor-like prompting, a prompting condition diffusion model is constructed following a forward process:

[0059]

[0060] In the formula, x t+1 and x t These represent the data points at time steps t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. This represents the gradient of the logarithmic probability distribution with respect to x when the model parameters are θ; ∈ t ∈N(0,σ 2 () represents the noise at time t;

[0061] The medical image synthesized using the aforementioned cue-conditional diffusion model is as follows:

[0062]

[0063]

[0064] In the formula, This represents the product of the step sizes from 0 to t; Data point x at time step t-1 t-1 The estimated average value; z t This indicates the inverted distribution of Gaussian noise predicted by the conditional diffusion model.

[0065] According to a fourth aspect of the present invention, a medical image synthesis system is provided, comprising:

[0066] Doctor-like prompt acquisition module, which is used to acquire doctor-like prompts;

[0067] The diffusion model construction module, based on the doctor-like prompts, constructs a prompt condition diffusion model following a forward process:

[0068]

[0069] In the formula, x t+1 and x t These represent the data points at time steps t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. This represents the gradient of the logarithmic probability distribution with respect to x when the model parameters are θ; ∈ t ∈N(0,σ 2 () represents the noise at time t;

[0070] The medical image synthesis module uses the cue-conditional diffusion model to synthesize medical images as follows:

[0071]

[0072]

[0073] In the formula, This represents the product of the step sizes from 0 to t; Data point x at time step t-1 t-1 The estimated average value; z t This indicates the inverted distribution of Gaussian noise predicted by the conditional diffusion model.

[0074] According to a fifth aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform the method described in any one of the above inventions, or to run the system described in the above inventions.

[0075] According to a sixth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the above-described inventions, or to run the system described in the above-described inventions.

[0076] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0077] The present invention provides a method for generating doctor-like prompts and a method and system for synthesizing medical images, which is a medical image synthesis technology based on professional doctor prompts. This technology improves medical image synthesis by converting non-expert prompts into professional doctor-like prompts. It constructs a medical knowledge graph by retrieving the most similar medical reports based on the non-expert prompts, then searches for relevant meta-knowledge from the medical knowledge graph, and combines the subject and meta-knowledge through cross-relevance to generate doctor-like prompts similar to those of an expert. Compared with prompts from Large Language Models (LLMs), doctor-like prompts are closer to medical reports, and medical images synthesized based on the converted doctor-like prompts have better alignment and fidelity than those from LLMs.

[0078] The doctor-style prompt generation method and its medical image synthesis method and system provided by this invention can limit the scope of knowledge to the field of medical reports by applying a knowledge graph based on medical reports, which helps to avoid irrelevant expressive information in the generated prompts.

[0079] This invention provides a method and system for generating doctor-like prompts and synthesizing medical images. It establishes a doctor-like prompt generation model for medical image synthesis. This model employs a prompt converter to transform non-expert prompts (from ordinary users) into doctor-like prompts that resemble those from medical experts. By retrieving the medical report most similar to the non-expert prompt, it uses this report as the subject of the doctor-like prompt. Then, it enriches the information by merging the meta-knowledge related to the non-expert prompt with the subject. The generated doctor-like prompts exclude knowledge irrelevant to medical reports, resulting in higher fidelity and alignment improvements compared to large language models for medical image synthesis tasks. Attached Figure Description

[0080] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0081] Figure 1 This is a flowchart illustrating the workflow of a doctor-style prompt generation method in one embodiment of the present invention.

[0082] Figure 2 This is a schematic diagram of the components of a doctor-style prompt generation system in one embodiment of the present invention.

[0083] Figure 3 This is a diagram illustrating the working architecture of a doctor-style prompt generation method in a specific application example of the present invention.

[0084] Figure 4 This is a flowchart illustrating the workflow of a medical image synthesis method based on doctor-like prompts in one embodiment of the present invention.

[0085] Figure 5 This is a schematic diagram of the constituent modules of a medical image synthesis system based on doctor-like prompts in one embodiment of the present invention. Detailed Implementation

[0086] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0087] In existing technologies, for medical image synthesis, prompts written by ordinary users often miss key information in the target image due to a lack of expertise. Although large models can be applied to generate more detailed prompts, these may contain information outside the specific domain, misleading the model in text-to-image synthesis.

[0088] To address the aforementioned problems, one embodiment of the present invention provides a method for generating doctor-like prompts, designed to generate medical images with high alignment and fidelity. This method proposes a prompt transformation model that can convert non-expert prompts into expert-like doctor-like prompts by retrieving medical reports and meta-knowledge, thereby improving medical image synthesis. This method excludes knowledge irrelevant to medical reports, outperforms state-of-the-art large-scale language models in terms of prompt generation metrics, and doctor-like prompts can significantly improve the alignment and fidelity of synthesized medical images.

[0089] Specifically, such as Figure 1 As shown, the doctor-style prompt generation method provided in this embodiment may include the following operations:

[0090] S1, extract the features of each medical report in the medical report collection;

[0091] S2 provides non-expert suggestions and extracts features from the non-expert suggestions;

[0092] S3, based on the characteristics of medical reports and non-expert tips, retrieve the medical reports in the medical report set that are most similar to non-expert tips;

[0093] S4, abstracts the medical knowledge in the retrieved medical reports into a knowledge graph;

[0094] S5 searches for meta-knowledge related to non-expert tips in the knowledge graph and converts the meta-knowledge into doctor-like tips.

[0095] In some preferred embodiments, the above-mentioned S1, which extracts the features of each medical report in the medical report set, may further include the following operations:

[0096] S11, retrieve each medical report r from the medical report set R. i Labeled with the corresponding diagnosis, a one-hot sequence containing n diagnoses is obtained, represented as:

[0097] S12 provides a feature extraction model that takes each medical report r in R as input and outputs the corresponding feature embedding.

[0098] In some preferred embodiments, S2 above, which provides non-expert hints and extracts features from the non-expert hints, may further include the following operations:

[0099] S21, provide a non-expert hint; wherein the information in the non-expert hint is represented as: { <V s / V d In the formula, s and d represent physical signs and diseases, respectively;

[0100] S22 provides a feature extraction model that takes non-expert prompts as input and outputs corresponding feature embeddings.

[0101] In S21 above, non-expert tips are tips provided by non-experts.

[0102] In some preferred embodiments, S3 above, which involves retrieving medical reports most similar to non-expert suggestions from the medical report set based on the characteristics of medical reports and non-expert suggestions, may further include the following operations:

[0103] S31, Based on the characteristics of medical reports and non-expert suggestions, a search is performed in the medical report set to obtain the similarity sim between non-expert suggestions and medical reports. i for:

[0104]

[0105] In the formula, Indicates the characteristics of a medical report, v nxp Features indicating non-expert suggestions;

[0106] S32, Select the medical report with the highest similarity. As a search result:

[0107]

[0108] In the formula, i * This indicates the sequence number of the most similar medical report found. This indicates the most similar medical report.

[0109] In some preferred embodiments, S4 above, which abstracts the medical knowledge in the retrieved medical reports into a knowledge graph, may further include the following operations:

[0110] Knowledge graphs are represented as: Each meta-knowledge k i Represented as a triple: k i ={V source ,E,V target}, where V source V represents the subject of relations in meta-knowledge. target V represents the object of a relation in meta-knowledge, and E represents the entity (vertice) and relation (edge), respectively.

[0111] In some preferred embodiments, the above-mentioned step S5, which involves searching for meta-knowledge related to non-expert hints in the knowledge graph, may further include the following operations:

[0112] S51 provides a pre-trained BERT model;

[0113] S52, knowledge graph Each meta-knowledge k in j As input to the BERT model, the corresponding embedding tensor is output. Obtain the set of embedding tensors of knowledge graph K

[0114] S53, taking the non-expert suggestion nxp as input to the BER model, outputs the embedding tensor e of the non-expert suggestion nxp. nxp =BERT(nxp);

[0115] S54, Calculate the embedding tensor e nxp With each embedded tensor in the set of embedded tensors ε The cross product of the non-expert hints and each meta-knowledge in the knowledge graph is used to obtain the similarity between the hints and each meta-knowledge in the knowledge graph. j for:

[0116]

[0117] S55, the frequency of meta-knowledge is expressed as:

[0118]

[0119] S56, Calculate similarity sim j The meta-knowledge k is obtained by summing the weights of the frequencies. j Score:

[0120] score j =sim j +λ·f j

[0121] In the formula, λ is a hyperparameter used to utilize similarity and frequency;

[0122] S57, utilizing meta-knowledge k j The score for meta-knowledge k j The meta-knowledge with the highest score is ranked, and the meta-knowledge with the highest score is selected as the meta-knowledge relevant to non-expert hints.

[0123]

[0124] In the formula, j * The index indicating the highest score. This represents the meta-knowledge that achieved the highest score.

[0125] In some preferred embodiments, the above-described S5, which converts meta-knowledge into doctor-like prompts, may further include the following operations:

[0126] S58 provides a cross-focus model that integrates the found meta-knowledge. As input to the cross-attention model, the output is the embedding tensor of the corresponding doctor-like prompts:

[0127]

[0128]

[0129] In the formula, LN(·) represents layer normalization, CA(·) represents cross-interest, and FFN(·) represents feedforward network; e attn Indicates attention e attn The embedding tensor, the attention e attn The embedding tensor is based on the retrieved medical reports. and the meta-knowledge found Calculations show that This represents the embedding tensor of the retrieved meta-knowledge. The embedding tensor representing the retrieved report is calculated as follows: and After cross attention and Add them together, then perform layer regularization; The embedding tensor representing the doctor's prompt;

[0130] S59, for embedding tensors Decode and de-label to generate doctor-like prompts

[0131] Among them, doctor-style prompts The information in is represented as: { <V s E so V d >, <V d E la V as >, <V adj E m V s / V d >}, where E so E la E m These represent the three types of relationships between entities: implied, located, and modified. V as V adj These represent two entities: an anatomical location and an adjective; where sign s implies disease d, which is located at a specific anatomical site; the severity of sign s is described using an adjective.

[0132] Based on the same inventive concept, an embodiment of the present invention also provides a doctor-style prompt generation system.

[0133] Specifically, such as Figure 2 As shown, the doctor-style prompt generation system provided in this embodiment may include:

[0134] The data provision module provides a collection of medical reports and non-expert tips.

[0135] The feature extraction module is used to extract features from each medical report in the medical report collection, as well as features from non-expert tips.

[0136] The knowledge graph construction module is used to abstract medical knowledge from retrieved medical reports into a knowledge graph.

[0137] The data comparison module, based on the characteristics of medical reports and non-expert tips, retrieves the medical reports most similar to non-expert tips from the medical report set, and searches for meta-knowledge related to non-expert tips in the knowledge graph;

[0138] A doctor-style suggestion generation module that converts the found meta-knowledge into doctor-style suggestions.

[0139] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0140] The doctor-style prompt generation method and system provided in the above embodiments of the present invention will be further described in detail below.

[0141] like Figure 3 As shown, the doctor-style prompt generation method and system provided in the above embodiments of the present invention mainly include the following components in their working architecture:

[0142] I. Establishing a Medical Knowledge Graph

[0143] The medical knowledge in medical reports can be abstracted into a knowledge graph: Each of these knowledge κ i Represented as a triple: κ i ={V source ,E,V target}, where V source It means, V target Let V and E represent entities (vertices) and relations (edges), respectively. Setting non-expert hints can be expressed as: { <V s / V dIn the formula, s and d represent physical signs and diseases, respectively. Triples are retrieved from K to find meta-knowledge related to non-expert tips, and this meta-knowledge is then converted into doctor-style tips accordingly. The information in the doctor-style tips can be expressed as: { <V s E so V d >, <V d E la V as >, <V adj E m V s / V d >}. This doctor-style prompt introduces three relationships between entities: <hint> (so), <located> (la), and <modify> (m), and also includes two entities:

[0144] <anatomical location>(as), <adjective>(adj) are used to add more details to the prompt. The sign s implies the disease d, which is located at a specific anatomical location; the severity of the sign s is described using <adjective>.

[0145] II. Medical Report Retrieval

[0146] Search for medical reports most similar to non-expert suggestions. These medical reports will be the core of the translated physician-style suggestions. First, each medical report in the collection is labeled as a single-hot sequence containing n types of diagnoses. Each diagnosis can be labeled as positive, negative, mentioned, or not mentioned. A feature extraction model is provided, taking the single-hot sequence as input, to extract the medical report r. i This is converted into the corresponding feature embedding. In this way, a set of medical reports can be obtained. Every medical report in the journal i Feature embeddings and, based on similar methods, feature embeddings for non-expert hints NXP: and v nxp =O(nxp), where O(·) represents the process of obtaining feature embeddings. Let be the features of medical report r, where q represents the length of the feature vector. The set containing the feature embeddings of all medical reports can be formulated as: Perform a search to find medical knowledge most similar to non-expert tips. The similarity between non-expert tips and medical reports is calculated using the following formula:

[0147]

[0148] Select the medical report with the highest similarity. As a search result:

[0149]

[0150] III. Meta-knowledge Retrieval

[0151] To bridge the knowledge gap between non-expert suggestions and actual medical reports, meta-knowledge relevant to non-expert suggestions is retrieved from the knowledge graph. A pre-trained BERT model is provided. For the knowledge graph... Each meta-knowledge κ j Embedded as: A set of embedding tensors of knowledge graph K Non-expert hint: The embedding tensor of NXP is e nxp =BERT(nxp), e nxp The cross product of each tensor in ε represents the similarity between the non-expert hint and each meta-knowledge in the knowledge graph:

[0152]

[0153] Since different meta-knowledge appears at different frequencies in medical reports, this embodiment of the invention also utilizes similarity and frequency to obtain scores and rank the retrieved triples:

[0154] The frequency of meta-knowledge can be expressed as:

[0155]

[0156] Meta-knowledge κ j The score is the similarity score. j and frequency f j Weights sum:

[0157] scorej=simj+λ·fj

[0158] Here, λ is a hyperparameter used to leverage similarity and frequency. The meta-knowledge with the highest score is ultimately selected. as follows:

[0159]

[0160] In the above embodiments of the present invention, three triplets containing suggestive, localizing, and modifying relationships are found through retrieval, and these triplets are used for cross-attention. Suggestion indicates the relationship between s and disease, while localizing indicates the relationship between anatomical location as and disease d. Modification is the connection between the adjective adj and s or disease.

[0161] IV. Meta-knowledge-based attention

[0162] Based on the search results, the embedding tensor of the retrieved meta-knowledge is e. The embedding tensor for generating doctor-style prompts is obtained, with the formula:

[0163]

[0164]

[0165] Where LN stands for layer normalization and CA stands for cross-attention. The feedforward network is represented as FFN. Attention e attn The embedding tensor is based on the retrieved medical reports. and the meta-knowledge found Calculated The embedding tensor represents the physician-like cues; cross-attention can incorporate the retrieved meta-knowledge into the body of the radiology report. After decoding and de-labeling, based on the embedding tensor... Generate doctor-style prompts

[0166] V. Comparative Learning

[0167] Evaluating medical text is difficult for ordinary users. Loss functions based on natural language metrics may fail to capture the semantics within the text. On the other hand, reward models based on human feedback require labels from professionals, which is extremely costly. Contrastive learning enables neural networks to learn representations of radiology reports, and embodiments of the present invention can further incorporate contrastive loss to improve the performance of rapid translation. It rewards neural networks for generating generative physician tips. Similar to real data reference r. The similarity matrix between r and r is:

[0168]

[0169]

[0170] In the formula, express The transpose of the matrix and e r The product of; e r The embedding tensor representing the real data reference r; e r The transpose of the matrix and The product;

[0171] The similarity between r and r can be calculated using Softmax activation:

[0172]

[0173] In the formula, express The similarity score between r and τ; τ represents the temperature parameter;

[0174] The comparative loss can be expressed as follows:

[0175]

[0176] Among them, L C L represents the sum of the comparative losses; ce represents the comparison loss value between two similarity matrices; g represents the diagonal matrix of basic facts of similarity.

[0177] An embodiment of the present invention also provides a medical image synthesis method based on doctor-like prompts, wherein the doctor-like prompts are obtained using the doctor-like prompt generation method or system provided in any of the above embodiments of the present invention.

[0178] Specifically, such as Figure 4 As shown, the medical image synthesis method provided in this embodiment may further include the following operations:

[0179] M1, get doctor-like prompts;

[0180] M2, based on doctor-like cues, constructs a cue condition diffusion model following a forward process:

[0181]

[0182] In the formula, x t+1 and x t These represent the data points at time steps t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. This represents the gradient of the logarithmic probability distribution with respect to x when the model parameters are θ; ∈ t ∈N(0,σ 2 () represents the noise at time t;

[0183] M3, using a cue-conditional diffusion model to synthesize a medical image:

[0184]

[0185]

[0186] In the formula, This represents the product of the step sizes from 0 to t; Data point x at time step t-1 t-1 The estimated average value; z t This indicates the inverted distribution of Gaussian noise predicted by the conditional diffusion model.

[0187] Based on the same inventive concept, an embodiment of the present invention also provides a medical image synthesis system.

[0188] Specifically, such as Figure 5As shown, the medical image synthesis system provided in this embodiment may further include the following modules:

[0189] Doctor-like prompt acquisition module, which is used to acquire doctor-like prompts;

[0190] The diffusion model construction module, based on doctor-like cues, constructs a cue-conditional diffusion model following a forward process.

[0191]

[0192] In the formula, x t+1 and x t These represent the data points at time steps t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. This represents the gradient of the logarithmic probability distribution with respect to x when the model parameters are θ; ∈ t ∈N(0,σ 2 () represents the noise at time t;

[0193] The medical image synthesis module uses a cue-based conditional diffusion model to synthesize medical images as follows:

[0194]

[0195]

[0196] In the formula, This represents the product of the step sizes from 0 to t; Data point x at time step t-1 t-1 The estimated average value; z t This indicates the inverted distribution of Gaussian noise predicted by the conditional diffusion model.

[0197] The medical image synthesis method and system provided in the above embodiments of the present invention will be further described in detail below.

[0198] The medical image synthesis method and system provided in the above embodiments of the present invention perform text-to-image synthesis based on doctor-like prompts.

[0199] The conditional diffusion model follows a forward process:

[0200]

[0201] In the formula, x t+1 and x t These are the data points at time step t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. It is the gradient of the log probability distribution with respect to x when the model parameters are θ; ∈ t ∈N(0,σ 2 () represents the noise at time t; the image is synthesized using a diffusion model through the following process:

[0202]

[0203]

[0204] In the formula, It is x t-1 The estimated average value; z t It is the inverted distribution of Gaussian noise predicted by the diffusion model.

[0205] The above embodiments of the present invention also used the DALLE-2 to test the effectiveness of physician-based suggestions. Using chest X-ray image synthesis as an example, the experimental results showed that the SSIM was 0.780 and the clinical rationality was 0.803, which were 0.049 and 0.063 higher than the SOTA, respectively.

[0206] An embodiment of the present invention also provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to execute the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0207] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0208] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0209] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0210] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0211] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0212] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0213] The doctor-style prompt generation method and medical image synthesis method and system provided in the above embodiments of the present invention can limit the scope of knowledge to the field of medical reports by applying a knowledge graph constructed based on medical reports, which helps to avoid irrelevant expressive information in the prompts.

[0214] The doctor-like prompt generation method and medical image synthesis method and system provided in the above embodiments of the present invention establish a model for creating doctor-like prompts in medical image synthesis. This model employs a prompt converter to transform non-expert prompts (from ordinary users) into doctor-like prompts similar to those from medical experts. By retrieving the medical report most similar to the non-expert prompt as the subject of the doctor-like prompt, and then merging the meta-knowledge related to the subject and the non-expert prompt to enrich its information, the generated doctor-like prompt excludes knowledge irrelevant to the medical report, bringing higher fidelity and alignment improvements than large language models to the medical image synthesis task.

[0215] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0216] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0217] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for generating doctor-style prompts, characterized in that, include: Extract the features of each medical report in the medical report collection; Provide non-expert suggestions and extract features from the non-expert suggestions; Based on the characteristics of the medical reports and the characteristics of the non-expert tips, retrieve the medical reports most similar to the non-expert tips from the medical report set; The medical knowledge in the retrieved medical reports is abstracted into a knowledge graph; Search the knowledge graph for meta-knowledge related to the non-expert tips, and convert the meta-knowledge into doctor-like tips; The process of abstracting medical knowledge from retrieved medical reports into a knowledge graph includes: The knowledge graph is represented as follows: Each meta-knowledge k i Represented as a triple: k i ={V source ,E,V target }, where V source V represents the subject of relations in meta-knowledge. target The objects representing relations in meta-knowledge, V and E represent entities and relations, respectively; The process of converting the meta-knowledge into doctor-like prompts includes: Provide a cross-focus model to find meta-knowledge As input to the cross-attention model, the corresponding doctor-like prompting embedding tensor is output: In the formula, LN(·) represents layer normalization, CA(·) represents cross-interest, and FFN(·) represents feedforward network; e attn Indicates attention e attn The embedded tensor, the attention e attn The embedding tensor is based on the retrieved medical reports. and the meta-knowledge found Calculations show that Embedding tensors representing meta-knowledge The embedding tensor representing the most similar report; The embedding tensor representing the doctor's prompt; For embedding tensors Decode and de-label to generate doctor-like prompts Among them, the doctor-style prompt The information in is represented as: { <V s E so V d >, <V d E la V as >, <V adj E m V s / V d >}, where E so E la E m These represent the three types of relationships between entities: implied, located, and modified. V as V adj These represent two entities: an anatomical location and an adjective; where sign s implies disease d, which is located at a specific anatomical site; the severity of sign s is described using an adjective.

2. The doctor-style prompt generation method according to claim 1, characterized in that, The features extracted from each medical report in the medical report collection include: Each medical report r in the medical report set R i Labeled with the corresponding diagnosis, a one-hot sequence containing n diagnoses is obtained, represented as: Provide a feature extraction model that combines the medical reports with each medical report r in R. i These are respectively used as inputs to the feature extraction model, and the corresponding feature embeddings are output.

3. The doctor-style prompt generation method according to claim 1, characterized in that, The provision of non-expert hints and the extraction of features from the non-expert hints include: Provide a non-expert hint, wherein the information in the non-expert hint is represented as: { <V s / V d In the formula, s and d represent physical signs and diseases, respectively; A feature extraction model is provided, which takes the non-expert prompts as input and outputs the corresponding feature embeddings.

4. The doctor-style prompt generation method according to claim 1, characterized in that, The step of retrieving the medical report most similar to the non-expert suggestion from the medical report set based on the characteristics of the medical report and the characteristics of the non-expert suggestion includes: Based on the characteristics of the medical reports and the characteristics of the non-expert tips, a search is performed in the medical report set to obtain the similarity (sim) between the non-expert tips and the medical reports. i for: In the formula, Indicates the characteristics of a medical report, v nxp Features indicating non-expert suggestions; Select the medical report with the highest similarity. As a search result: In the formula, i * R represents the sequence number of the most similar medical report found. i * indicates the most similar medical report.

5. The doctor-style prompt generation method according to claim 1, characterized in that, The step of searching for meta-knowledge related to the non-expert hint in the knowledge graph includes: Provide a pre-trained BERT model; The knowledge graph Each meta-knowledge k in j As input to the BERT model, the corresponding embedding tensor is output. Obtain the embedding tensor set of the knowledge graph K. The non-expert hint nxp is used as input to the BERT model, and the embedding tensor e of the non-expert hint nxp is output. nxp =BERT(nxp); Calculate the embedding tensor e nxp With each embedded tensor in the set of embedded tensors ε The cross product of the non-expert hints and each meta-knowledge in the knowledge graph is used to obtain the similarity between the hints and each meta-knowledge in the knowledge graph. j for: The frequency of the meta-knowledge is represented as: Calculate the similarity sim j The meta-knowledge k is obtained by summing the weights of the frequencies. j Score: score j =sim j +λ·f j In the formula, λ is a hyperparameter used to utilize similarity and frequency; Using the aforementioned meta-knowledge k j The score for the meta-knowledge k j The meta-knowledge pieces are ranked, and the highest-scoring meta-knowledge is selected as the meta-knowledge relevant to the non-expert hints. In the formula, j * K represents the index of the highest score. j * indicates the meta-knowledge with the highest score.

6. A doctor-style prompt generation system, characterized in that, include: The data provision module provides a collection of medical reports and non-expert tips. The feature extraction module is used to extract features from each medical report in the medical report collection, as well as features from non-expert tips. The knowledge graph construction module is used to abstract medical knowledge from retrieved medical reports into a knowledge graph. The data comparison module, based on the characteristics of the medical reports and the characteristics of the non-expert tips, retrieves the medical reports most similar to the non-expert tips in the medical report set, and searches for meta-knowledge related to the non-expert tips in the knowledge graph; A doctor-style suggestion generation module that converts the found meta-knowledge into doctor-style suggestions; The process of abstracting medical knowledge from retrieved medical reports into a knowledge graph includes: The knowledge graph is represented as follows: Each meta-knowledge k i Represented as a triple: k i ={V source ,E,V target }, where V source V represents the subject of relations in meta-knowledge. target The objects representing relations in meta-knowledge, V and E represent entities and relations, respectively; The process of converting the meta-knowledge into doctor-like prompts includes: Provide a cross-focus model to find meta-knowledge As input to the cross-attention model, the corresponding doctor-like prompting embedding tensor is output: In the formula, LN(·) represents layer normalization, CA(·) represents cross-interest, and FFN(·) represents feedforward network; e attn Indicates attention e attn The embedded tensor, the attention e attn The embedding tensor is based on the retrieved medical reports. and the meta-knowledge found Calculations show that Embedding tensors representing meta-knowledge The embedding tensor representing the most similar report; The embedding tensor representing the doctor's prompt; For embedding tensors Decode and de-label to generate doctor-like prompts Among them, the doctor-style prompt The information in is represented as: { <V s E so V d >, <V d E la V as >, <V adj E m V s / V d >}, where E so E la E m These represent the three types of relationships between entities: implied, located, and modified. V as V adj These represent two entities: an anatomical location and an adjective; where sign s implies disease d, which is located at a specific anatomical site; the severity of sign s is described using an adjective.

7. A method for synthesizing medical images, characterized in that, include: Get doctor-like tips; The doctor-style prompt is generated by a doctor-style prompt generation method according to any one of claims 1-5; Based on the doctor-like prompting, a prompting condition diffusion model is constructed following a forward process: In the formula, x t+1 and x t These represent the data points at time steps t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. This represents the gradient of the logarithmic probability distribution with respect to x when the model parameters are θ; ∈ t ∈N(0,σ 2 () represents the noise at time t; The medical image synthesized using the aforementioned cue-conditional diffusion model is as follows: In the formula, This represents the product of the step sizes from 0 to t; Data point x at time step t-1 t-1 The estimated average value; z t This indicates the inverted distribution of Gaussian noise predicted by the conditional diffusion model.

8. A medical image synthesis system, characterized in that, include: Doctor-like prompt acquisition module, which is used to acquire doctor-like prompts; The doctor-style prompt is generated by a doctor-style prompt generation method according to any one of claims 1-5; The diffusion model construction module, based on the doctor-like prompts, constructs a prompt condition diffusion model following a forward process: In the formula, x t+1 and x t These represent the data points at time steps t+1 and t, respectively; α t Indicates the step size at point t; x and These represent images and doctor-style prompts, respectively. This represents the gradient of the logarithmic probability distribution with respect to x when the model parameter is θ; ∈ t ∈N(0,σ 2 () represents the noise at time t; The medical image synthesis module uses the cue-conditional diffusion model to synthesize medical images as follows: In the formula, This represents the product of the step sizes from 0 to t; Data point x at time step t-1 t-1 The estimated average value; z t This indicates the inverted distribution of Gaussian noise predicted by the conditional diffusion model.

9. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-5 and 7, or to run the system of claim 6 or 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-5 and 7, or to run the system of claim 6 or 8.

Images