Doctor type prompt generation method and medical image synthesis method and system thereof

Abstract

The invention provides a doctor-type prompt generation method and system. The method comprises the following steps: extracting features of each medical report in a medical report set; providing a non-expert prompt, and extracting features in the non-expert prompt; based on the features of the medical reports and the features of the non-expert prompts, retrieving the medical reports most similar to the non-expert prompts in the medical report set; abstracting medical knowledge in the retrieved medical report into a knowledge graph; meta-knowledge related to non-expert prompts is searched in the knowledge graph, and the meta-knowledge is converted into doctor-type prompts. Meanwhile, the invention provides a medical image synthesis method and system based on the doctor-type prompt. According to the knowledge graph constructed based on the medical reports, the knowledge range is limited in the field of the medical reports, and irrelevant expression information in prompts can be avoided. The generated doctor-type prompt excludes knowledge irrelevant to a medical report, and higher fidelity and alignment improvement than a large language model are brought to a medical image synthesis task.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for generating doctor-style prompts and a medical image synthesis method and system thereof, as well as a corresponding computer terminal and a computer-readable storage medium. Background Art

[0002] Text-to-image synthesis has achieved great success in generating natural images. One important reason is that general users can easily write detailed prompts for synthesis. However, for medical image synthesis, due to the lack of professional knowledge, the prompts written by general users often miss or misunderstand the key information of the target image.

[0003] In order to solve the above problems, large model application technology came into being. Among them:

[0004] Large-Scale Language Image Model (LSLIM) has achieved great success in synthesizing natural images. However, some medical images synthesized by LSLIM are different from real images, i.e., they are not clinically plausible. Since ordinary users cannot write prompts like experts, some necessary information may be lost or misunderstood by ordinary users, which may mislead LSLIM's text-to-image generation.

[0005] The use of templates or guidelines is widely accepted when writing prompts for natural image synthesis. Templates or guidelines can guide users to clarify the names and classifications of objects, while also encouraging users to describe the intentions, motivations, and consequences of actions, thereby improving the fidelity and alignment of generated images. However, when writing prompts for medical image synthesis, these templates or guidelines are usually incomprehensible to ordinary users, and thus, the knowledge barrier prevents developers from writing sufficiently detailed prompts for medical image synthesis. Therefore, patterns or guidelines are not a feasible solution for medical image synthesis.

[0006] Another approach to prompt transfer is to use Large Language Models (LLMs), which are able to improve prompt quality for text-to-image tasks. However, medical LLMs are trained on biomedical databases that contain much more knowledge than the medical image domain and, therefore, may generate irrelevant prompts outside the medical image domain, which may confuse the LLM for text-to-image generation. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for generating doctor-style prompts and a medical image synthesis method and system thereof, and also provides a corresponding computer terminal and computer-readable storage medium.

[0008] According to one aspect of the present invention, a method for generating a doctor-style prompt is provided, comprising:

[0009] Extracting features of each medical report in the medical report collection;

[0010] providing non-expert hints and extracting features from the non-expert hints;

[0011] Based on the characteristics of the medical report and the characteristics of the non-expert suggestion, searching the medical report most similar to the non-expert suggestion in the medical report collection;

[0012] Abstract the medical knowledge in the retrieved medical reports into a knowledge graph;

[0013] Find meta-knowledge related to the non-expert prompt in the knowledge graph, and convert the meta-knowledge into doctor-style prompts.

[0014] Preferably, extracting the features of each medical report in the medical report set includes:

[0015] Each medical report r in the medical report set R i Marked as the corresponding diagnosis, the onehot sequence containing n types of diagnosis is obtained, which is expressed as:

[0016] Provide a feature extraction model to combine the medical report with each medical report r in R i They are respectively used as inputs of the feature extraction model and output corresponding feature embeddings.

[0017] Preferably, providing non-expert prompts and extracting features from the non-expert prompts includes:

[0018] Provide a non-expert tip; wherein the information in the non-expert tip is represented as: { <V s / V d >}, where s and d represent signs and diseases, respectively;

[0019] A feature extraction model is provided, the non-expert prompt is used as an input of the feature extraction model, and a corresponding feature embedding is output.

[0020] Preferably, the retrieving a medical report most similar to the non-expert suggestion from the medical report set based on the features of the medical report and the features of the non-expert suggestion comprises:

[0021] Based on the characteristics of the medical report and the characteristics of the non-expert prompt, a search is performed in the medical report collection to obtain the similarity sim between the non-expert prompt and the medical report. i for:

[0022]

[0023] In the formula, Represents the characteristics of medical reports, v nxp Characteristic of non-expert prompting;

[0024] Select the medical report with the highest similarity As search results:

[0025]

[0026] In the formula, i * Indicates the serial number of the most similar medical report retrieved. Indicates the most similar medical reports.

[0027] Preferably, the step of abstracting the medical knowledge in the retrieved medical reports into a knowledge graph includes:

[0028] The knowledge graph is represented as: Each meta-knowledge k i Expressed as a triple: k i = {V source ,E,V target}, where V source Represents the subject of the relation in meta-knowledge, V target It represents the object of the relationship in meta-knowledge, V and E represent entities (vertices) and relationships (edges), respectively.

[0029] Preferably, searching the knowledge graph for meta-knowledge related to the non-expert prompt includes:

[0030] Provide a pre-trained BERT model;

[0031] The knowledge graph Each meta-knowledge k in j As input to the BERT model, the corresponding embedding tensor is output Get the embedded tensor set of the knowledge graph K

[0032] The non-expert prompt nxp is used as the input of the BER model, and the embedding tensor e of the non-expert prompt nxp is output nxp =BERT(nxp);

[0033] Compute the embedding tensor e nxp With each embedded tensor in the embedded tensor set ε The cross product of , we get the similarity between non-expert tips and each meta-knowledge in the knowledge graph sim j for:

[0034]

[0035] The frequency of the meta-knowledge is expressed as:

[0036]

[0037] Calculate the similarity sim j The weighted sum of the frequencies gives the meta-knowledge k j Score:

[0038] score j =sim j +λ·f j

[0039] Where λ is a hyperparameter used to exploit similarity and frequency;

[0040] Using the meta-knowledge k j The score of the meta-knowledge k j Ranking is performed, and finally the meta-knowledge with the highest score is selected as the meta-knowledge related to the non-expert prompt

[0041]

[0042] In the formula, j * represents the subscript of the highest score, Represents the highest scoring meta-knowledge.

[0043] Preferably, converting the meta-knowledge into doctor-style prompts comprises:

[0044] Provide a cross-attention model to find meta-knowledge As input to the cross-attention model, the corresponding doctor-style prompt embedding tensor is output:

[0045]

[0046]

[0047] Where LN(·) represents layer normalization, CA(·) represents cross attention, and FFN(·) represents feed-forward network; e attn Indicates attention attn The embedding tensor of the attention attn The embedding tensor of is based on the retrieved medical reports and the meta-knowledge found Calculated, An embedding tensor representing meta-knowledge, The embedding tensor representing the most similar report; Embedding tensor representing the doctor's prompt;

[0048] Embedding tensor Decode and de-tag to generate doctor-style prompts

[0049] Among them, the doctor-style prompt The information in is represented as: <V s ,E so ,V d >, <V d ,E la ,V as >, <V adj ,E m ,V s / V d >}, where E so ,E la ,E m Represents three types of relationships between entities: implies, located in, and modifies, V as ,V adj They represent two entities: anatomical location and adjective; among them, sign s implies disease d, and disease d is located in a specific anatomical part; the severity of sign s is described by adjective.

[0050] According to a second aspect of the present invention, there is provided a doctor-style prompt generation system, comprising:

[0051] A data providing module for providing a collection of medical reports and non-expert prompts;

[0052] A feature extraction module, which is used to extract features of each medical report in the medical report set and features in the non-expert prompts respectively;

[0053] A knowledge graph construction module, which is used to abstract the medical knowledge in the retrieved medical reports into a knowledge graph;

[0054] a data comparison module, which, based on the characteristics of the medical report and the characteristics of the non-expert suggestion, retrieves the medical report most similar to the non-expert suggestion in the medical report set, and searches for meta-knowledge related to the non-expert suggestion in the knowledge graph;

[0055] A doctor-style prompt generation module, which converts the found meta-knowledge into doctor-style prompts.

[0056] According to a third aspect of the present invention, there is provided a medical image synthesis method, comprising:

[0057] Get doctor-style tips;

[0058] Based on the doctor-style prompts, the prompt conditional diffusion model is constructed following the forward process:

[0059]

[0060] In the formula, x t+1 and x t Respectively represent the data points at time steps t+1 and t; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; It means the gradient of log probability distribution to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) represents the noise at time t;

[0061] The medical image synthesized using the prompt conditional diffusion model is:

[0062]

[0063]

[0064] In the formula, represents the product of the step lengths from 0 to t; represents the data point x at time step t-1 t-1 The estimated mean value of z t Represents the inverse distribution of Gaussian noise predicted by the cue-conditioned diffusion model.

[0065] According to a fourth aspect of the present invention, there is provided a medical image synthesis system, comprising:

[0066] A doctor-style prompt acquisition module, which is used to acquire doctor-style prompts;

[0067] A diffusion model building module, which builds a prompt condition diffusion model based on the doctor-style prompts following the forward process:

[0068]

[0069] In the formula, x t+1 and x t Respectively represent the data points at time steps t+1 and t; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; It means the gradient of log probability distribution to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) represents the noise at time t;

[0070] A medical image synthesis module, which uses the prompt conditional diffusion model to synthesize a medical image as follows:

[0071]

[0072]

[0073] In the formula, represents the product of the step lengths from 0 to t; represents the data point x at time step t-1 t-1 The estimated mean value of z t Represents the inverse distribution of Gaussian noise predicted by the cue-conditioned diffusion model.

[0074] According to a fifth aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0075] According to a sixth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above in the present invention, or to run the system described above in the present invention.

[0076] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0077] The doctor-style prompt generation method and medical image synthesis method and system provided by the present invention are a medical image synthesis technology based on professional doctor prompts. The technology improves medical image synthesis by converting non-expert prompts into professional doctor-style prompts, retrieves the most similar medical reports based on non-expert prompts to build a medical knowledge graph, and then searches for relevant meta-knowledge from the medical knowledge graph, and combines the subject and meta-knowledge through cross-attention to generate expert-like doctor-style prompts. Compared with the prompts of the large language model (LLM), the doctor-style prompts are closer to medical reports, and the medical images synthesized based on the converted doctor-style prompts have better alignment and fidelity than the large language model (LLM).

[0078] The doctor-style prompt generation method and medical image synthesis method and system provided by the present invention can limit the scope of knowledge to the field of medical reports by applying a knowledge graph constructed based on medical reports, which helps to avoid irrelevant expression information in the generated prompts.

[0079] The present invention provides a method for generating doctor-style prompts and a method and system for synthesizing medical images. A doctor-style prompt generation model for medical image synthesis is established. The model uses a prompt converter to convert non-expert prompts (ordinary users) into doctor-style prompts similar to those of medical experts. The medical report that is most similar to the non-expert prompt is retrieved as the subject of the doctor-style prompt, and then the subject is merged with the meta-knowledge related to the non-expert prompt to enrich its information. The generated doctor-style prompt excludes knowledge that is not related to the medical report, bringing higher fidelity and alignment improvements to the medical image synthesis task than large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0081] Figure 1 The figure is a workflow diagram of a method for generating a doctor-style prompt in one embodiment of the present invention.

[0082] Figure 2 Schematic diagram of the components of a system for generating doctor-style prompts in one embodiment of the present invention.

[0083] Figure 3 The figure is a working framework diagram of a method for generating doctor-style prompts in a specific application example of the present invention.

[0084] Figure 4 The figure is a workflow diagram of a medical image synthesis method based on doctor-style prompting in one embodiment of the present invention.

[0085] Figure 5 Schematic diagram of the components of a medical image synthesis system based on doctor-style prompts in one embodiment of the present invention. DETAILED DESCRIPTION

[0086] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0087] In the existing technology, for medical image synthesis, the prompts written by ordinary users usually miss the key information of the target image due to lack of expertise. Although large models can be applied to generate more detailed prompts, they may contain information outside the specific field and mislead the model for text-to-image synthesis.

[0088] In view of the above problems, an embodiment of the present invention provides a method for generating doctor-style prompts, which is intended to be used to generate medical images with high alignment and fidelity. The method proposes a prompt conversion model that can convert non-expert prompts into expert-like doctor-style prompts by retrieving medical reports and meta-knowledge, which can then be used to improve medical image synthesis. The method excludes knowledge that is not related to medical reports and outperforms the most advanced large language models in generating prompt indicators. Doctor-style prompts can also be used to significantly improve the alignment and fidelity of synthesized medical images.

[0089] Specifically, Figure 1 As shown, the doctor-style prompt generation method provided in this embodiment may include the following operations:

[0090] S1, extract the features of each medical report in the medical report collection;

[0091] S2, provides non-expert prompts and extracts features from non-expert prompts;

[0092] S3, based on the characteristics of the medical report and the characteristics of the non-expert suggestion, retrieve the medical report that is most similar to the non-expert suggestion in the medical report collection;

[0093] S4, abstracts the medical knowledge in the retrieved medical reports into a knowledge graph;

[0094] S5, finds meta-knowledge related to non-expert tips in the knowledge graph and converts the meta-knowledge into doctor-style tips.

[0095] In some preferred implementations, the above S1, extracting the features of each medical report in the medical report set, may further include the following operations:

[0096] S11, each medical report r in the medical report set R i Marked as the corresponding diagnosis, the onehot sequence containing n types of diagnosis is obtained, which is expressed as:

[0097] S12, providing a feature extraction model, taking the medical report combined with each medical report r in R as input of the feature extraction model, and outputting corresponding feature embedding.

[0098] In some preferred implementations, the above S2, providing non-expert prompts and extracting features from the non-expert prompts, may further include the following operations:

[0099] S21, providing a non-expert tip; wherein the information in the non-expert tip is represented as: { <V s / V d >}, where s and d represent signs and diseases, respectively;

[0100] S22, providing a feature extraction model, taking the non-expert prompt as an input of the feature extraction model, and outputting corresponding feature embedding.

[0101] In the above S21, the non-expert prompt is prompt information provided by non-experts.

[0102] In some preferred embodiments, the above S3, based on the features of the medical report and the features of the non-expert suggestion, searching the medical report set for the medical report that is most similar to the non-expert suggestion, may further include the following operations:

[0103] S31, based on the characteristics of the medical report and the characteristics of the non-expert prompt, search in the medical report collection to obtain the similarity between the non-expert prompt and the medical report i for:

[0104]

[0105] In the formula, Represents the characteristics of medical reports, v nxp Characteristic of non-expert prompting;

[0106] S32, select the medical report with the highest similarity As search results:

[0107]

[0108] In the formula, i * Indicates the serial number of the most similar medical report retrieved. Indicates the most similar medical reports.

[0109] In some preferred implementations, the above S4, abstracting the medical knowledge in the retrieved medical report into a knowledge graph, may further include the following operations:

[0110] The knowledge graph is represented as: Each meta-knowledge k i Expressed as a triple: k i = {V source ,E,V target}, where V source Represents the subject of the relation in meta-knowledge, V target It represents the object of the relationship in meta-knowledge, V and E represent entities (vertices) and relationships (edges), respectively.

[0111] In some preferred implementations, the above S5, searching the knowledge graph for meta-knowledge related to the non-expert prompt, may further include the following operations:

[0112] S51, providing a pre-trained BERT model;

[0113] S52, knowledge graph Each meta-knowledge k in j As input to the BERT model, output the corresponding embedding tensor Get the embedded tensor set of the knowledge graph K

[0114] S53, taking the non-expert prompt nxp as the input of the BER model, and outputting the embedding tensor e of the non-expert prompt nxp nxp =BERT(nxp);

[0115] S54, calculate the embedding tensor e nxp With each embedded tensor in the embedding tensor set ε The cross product of , we get the similarity between non-expert tips and each meta-knowledge in the knowledge graph sim j for:

[0116]

[0117] S55, the frequency of meta-knowledge is expressed as:

[0118]

[0119] S56, calculate similarity sim j The weighted sum of the frequencies gives us meta-knowledge k j Score:

[0120] score j =sim j +λ·f j

[0121] Where λ is a hyperparameter used to exploit similarity and frequency;

[0122] S57, using meta-knowledge k j The score of meta-knowledge k j Ranking is performed and the meta-knowledge with the highest score is finally selected as the meta-knowledge related to the non-expert prompts

[0123]

[0124] In the formula, j * represents the subscript of the highest score, Represents the highest scoring meta-knowledge.

[0125] In some preferred implementations, the above S5, converting meta-knowledge into doctor-style prompts, may further include the following operations:

[0126] S58, providing a cross-attention model to find the meta-knowledge As input to the cross-attention model, the output is the embedding tensor of the corresponding doctor-style prompts:

[0127]

[0128]

[0129] Where LN(·) represents layer normalization, CA(·) represents cross attention, and FFN(·) represents feed-forward network; e attn Indicates attention attn The embedding tensor of the attention attn The embedding tensor of is based on the retrieved medical reports and the meta-knowledge found Calculated, An embedding tensor representing the retrieved meta-knowledge, Represents the embedding tensor of the retrieved report, which is calculated as follows: and After cross attention Add and then perform layer regularization; Embedding tensor representing the doctor's prompt;

[0130] S59, for the embedding tensor Decode and de-tag to generate doctor-style prompts

[0131] Among them, doctor-style tips The information in is represented as: <V s ,E so ,V d >, <V d ,E la ,V as >, <V adj ,E m ,V s / V d >}, where E so ,E la ,E m Represents three types of relationships between entities: implies, located in, and modifies, V as ,V adj They represent two entities: anatomical location and adjective; among them, sign s implies disease d, and disease d is located in a specific anatomical part; the severity of sign s is described by adjective.

[0132] Based on the same inventive concept, an embodiment of the present invention further provides a doctor-style prompt generation system.

[0133] Specifically, Figure 2 As shown, the doctor-style prompt generation system provided by this embodiment may include:

[0134] A data providing module for providing a collection of medical reports and non-expert prompts;

[0135] A feature extraction module, which is used to extract features of each medical report in the medical report set and features in the non-expert prompts respectively;

[0136] A knowledge graph construction module, which is used to abstract the medical knowledge in the retrieved medical reports into a knowledge graph;

[0137] A data comparison module, which searches for medical reports that are most similar to non-expert tips in a medical report collection based on the characteristics of medical reports and non-expert tips, and searches for meta-knowledge related to non-expert tips in a knowledge graph;

[0138] A doctor-style prompt generation module, which converts the found meta-knowledge into doctor-style prompts.

[0139] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0140] The doctor-style prompt generation method and system provided by the above-mentioned embodiment of the present invention are further described in detail below.

[0141] like Figure 3 As shown, the doctor-style prompt generation method and system provided by the above embodiment of the present invention, its working architecture mainly includes the following parts:

[0142] 1. Establishing a Medical Knowledge Graph

[0143] The medical knowledge in medical reports can be abstracted into a knowledge graph: Each of these knowledge i is expressed as a triple: i = {V source ,E,V target}, where V source Indicates that V target Indicates that V and E represent entities (vertices) and relationships (edges) respectively. Setting non-expert prompts can be expressed as: <V s / V d>}, where s and d represent signs and diseases respectively. Retrieve triples in K to find meta-knowledge related to non-expert prompts, and convert the meta-knowledge into doctor-style prompts accordingly. The information in the doctor-style prompts can be expressed as: { <V s ,E so ,V d >, <V d ,E la ,V as >, <V adj ,E m ,V s / V d >}. This doctor-style prompt introduces three relationships between entities: <implies> (so), <is located> (la), <modifies> (m), and also includes two entities:

[0144] <anatomical part> (as), <adjective> (adj) to add more details to the prompt. Sign s indicates disease d, which is located in a specific anatomical part; the severity of sign s is described using <adjective>.

[0145] 2. Medical Report Retrieval

[0146] Search for medical reports that are most similar to non-expert tips. Medical reports will be the main body of the converted doctor-style tips. First, label each medical report in the medical report collection as a one-hot sequence containing n types of diagnoses. Each diagnosis can be labeled as positive, negative, mentioned, or not mentioned. Provide a feature extraction model and use the one-hot sequence as the input of the model to convert the medical report r i Convert to corresponding feature embedding. In this way, a collection of medical reports can be obtained Every medical report in i The feature embedding of non-expert tips nxp is obtained based on a similar method: and v nxp =O(nxp), where O(·) represents the process of obtaining feature embedding. is the feature of medical report r, where q represents the length of the feature vector. The set of feature embeddings containing all medical reports can be formulated as: A search was performed to find the medical knowledge that was most similar to the non-expert prompts. The similarity between the non-expert prompts and the medical reports was calculated using the following formula:

[0147]

[0148] Select the medical report with the highest similarity As search results:

[0149]

[0150] 3. Meta-knowledge Retrieval

[0151] In order to fill the knowledge gap between non-expert tips and real medical reports, meta-knowledge related to non-expert tips in the knowledge graph is retrieved. A pre-trained BERT model is provided. Each meta-knowledge κ j is embedded as: A collection of embedding tensors of the knowledge graph K Non-expert tip: The embedding tensor of NXP is E nxp =BERT(nxp),e nxp The cross product with each tensor in ε represents the similarity between the non-expert hint and each meta-knowledge in the knowledge graph:

[0152]

[0153] Since different meta-knowledge appears at different frequencies in medical reports, the embodiment of the present invention also uses similarity and frequency to obtain scores and rank the retrieved triples:

[0154] The frequency of meta-knowledge can be expressed as:

[0155]

[0156] Meta-knowledge j The score is the similarity sim j and frequency f j The weight and:

[0157] scorej=simj+λ·fj

[0158] Among them, λ is a hyperparameter used to take advantage of similarity and frequency. Finally, the meta-knowledge with the highest score is selected as follows:

[0159]

[0160] In the above embodiment of the present invention, three triplets containing the relationship of suggestion, location and modification are found through retrieval, and they are used for cross attention. Suggestion indicates the relationship between s and disease, while location indicates the relationship between anatomical part as and disease d. Modification is the connection between adjective adj and s or disease.

[0161] 4. Meta-knowledge-based Attention

[0162] According to the search results, the embedding tensor of the retrieved meta-knowledge is e, and the embedding tensor for generating doctor-style prompts is obtained as follows:

[0163]

[0164]

[0165] Among them, LN is layer normalization and CA is cross attention. The feed-forward network is denoted as FFN. Attention attn The embedding tensor is based on the retrieved medical reports and the meta-knowledge found Calculated, Embedding tensor representing physician tips; cross-attention can incorporate the retrieved meta-knowledge into the body of the radiology report. After decoding and de-labeling, the embedded tensor Generate doctor-style tips

[0166] 5. Contrastive Learning

[0167] Evaluating medical text is difficult for ordinary users. Loss functions based on natural language metrics may not capture the semantics in the text. On the other hand, using a reward model based on human feedback requires professionals to provide labels, which is very costly. Contrastive learning enables neural networks to learn representations of radiology reports. Embodiments of the present invention can also introduce contrastive loss to improve the performance of fast conversion. It rewards neural networks for generating generative doctor prompts. Similar to the real data reference r. The similarity matrix with r is:

[0168]

[0169]

[0170] In the formula, express The transposed matrix of e r The product of r The embedding tensor representing the real data reference r; Indicates e r The transposed matrix of The product of

[0171] The similarity between and r can be calculated by Softmax activation:

[0172]

[0173] In the formula, express The similarity score between and r; τ represents the temperature parameter;

[0174] The contrast loss can be expressed as follows:

[0175]

[0176] Among them, L C represents the sum of contrast losses; L ce represents the contrast loss value between two similarity matrices; g represents the diagonal matrix of the basic facts of similarity.

[0177] An embodiment of the present invention further provides a medical image synthesis method based on doctor-style prompts, wherein the doctor-style prompts are obtained using the doctor-style prompt generation method or system provided in any one of the above embodiments of the present invention.

[0178] Specifically, Figure 4 As shown, the medical image synthesis method provided in this embodiment may further include the following operations:

[0179] M1, get doctor-style tips;

[0180] M2, based on the doctor-style prompt, follows the forward process to build a prompt conditional diffusion model:

[0181]

[0182] In the formula, x t+1 and x t Respectively represent the data points at time steps t+1 and t; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; It means the gradient of log probability distribution to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) represents the noise at time t;

[0183] M3, using the prompt conditional diffusion model to synthesize medical images:

[0184]

[0185]

[0186] In the formula, represents the product of the step lengths from 0 to t; represents the data point x at time step t-1 t-1 The estimated mean value of z t Represents the inverse distribution of Gaussian noise predicted by the cue-conditioned diffusion model.

[0187] Based on the same inventive concept, an embodiment of the present invention further provides a medical image synthesis system.

[0188] Specifically, Figure 5As shown, the medical image synthesis system provided in this embodiment may further include the following modules:

[0189] A doctor-style prompt acquisition module, which is used to acquire doctor-style prompts;

[0190] The diffusion model building module, based on the doctor-style prompts, follows the forward process to build a prompt-conditioned diffusion model:

[0191]

[0192] In the formula, x t+1 and x t Respectively represent the data points at time steps t+1 and t; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; It means the gradient of log probability distribution to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) represents the noise at time t;

[0193] Medical image synthesis module, which uses the prompt conditional diffusion model to synthesize medical images as:

[0194]

[0195]

[0196] In the formula, represents the product of the step lengths from 0 to t; represents the data point x at time step t-1 t-1 The estimated mean value of z t Represents the inverse distribution of Gaussian noise predicted by the cue-conditioned diffusion model.

[0197] The medical image synthesis method and system provided by the above-mentioned embodiments of the present invention are further described in detail below.

[0198] The medical image synthesis method and system provided by the above-mentioned embodiments of the present invention perform text-to-image synthesis based on doctor-style prompts.

[0199] The cue-conditioned diffusion model follows a forward process:

[0200]

[0201] In the formula, x t+1 and x t are the data points at time steps t+1 and t respectively; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; is the gradient of the log probability distribution with respect to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) is the noise at time t; the image is synthesized using the diffusion model through the following process:

[0202]

[0203]

[0204] In the formula, is x t-1 The estimated mean value of z t is the inverse distribution of the Gaussian noise predicted by the diffusion model.

[0205] The above embodiment of the present invention also uses DALLE-2 to test the effectiveness of doctor-style prompts. The test takes chest X-ray image synthesis as an example. The experimental results show that the SSIM is 0.780 and the clinical rationality is 0.803, which are 0.049 and 0.063 higher than SOTA respectively.

[0206] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0207] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0208] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0209] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0210] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0211] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0212] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0213] The doctor-style prompt generation method and medical image synthesis method and system provided by the above-mentioned embodiments of the present invention can limit the scope of knowledge to the field of medical reports by applying a knowledge graph constructed based on medical reports, which helps to avoid irrelevant expression information in the prompts.

[0214] The doctor-style prompt generation method and medical image synthesis method and system provided in the above-mentioned embodiment of the present invention establish a doctor-like prompt model for medical image synthesis, which uses a prompt converter to convert non-expert prompts (ordinary users) into doctor-style prompts similar to medical experts. The medical report most similar to the non-expert prompt is retrieved as the subject of the doctor-style prompt, and then the subject is merged with the meta-knowledge related to the non-expert prompt to enrich its information. The generated doctor-style prompt excludes knowledge irrelevant to the medical report, bringing higher fidelity and alignment improvements to the medical image synthesis task than large language models.

[0215] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0216] All matters not covered in the above embodiments of the present invention are well known in the art.

[0217] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for generating doctor-style prompts, characterized in that: include: Extracting features of each medical report in the medical report collection; providing non-expert hints and extracting features from the non-expert hints; Based on the characteristics of the medical report and the characteristics of the non-expert suggestion, searching the medical report most similar to the non-expert suggestion in the medical report collection; Abstract the medical knowledge in the retrieved medical reports into a knowledge graph; Find meta-knowledge related to the non-expert prompt in the knowledge graph, and convert the meta-knowledge into doctor-style prompts.

2. The method for generating doctor-style prompts according to claim 1, characterized in that: The extracting the features of each medical report in the medical report set includes: Each medical report r in the medical report set R i Marked as the corresponding diagnosis, the onehot sequence containing n types of diagnosis is obtained, which is expressed as: Provide a feature extraction model to combine the medical report with each medical report r in R i They are respectively used as inputs of the feature extraction model and output corresponding feature embeddings.

3. The method for generating doctor-style prompts according to claim 1, characterized in that: The providing of non-expert prompts and extracting features from the non-expert prompts includes: Provide a non-expert tip, wherein the information in the non-expert tip is represented as: <V s / V d >}, where s and d represent signs and diseases, respectively; A feature extraction model is provided, the non-expert prompt is used as an input of the feature extraction model, and a corresponding feature embedding is output.

4. The method for generating doctor-style prompts according to claim 1, characterized in that: The step of searching the medical report set for a medical report that is most similar to the non-expert suggestion based on the features of the medical report and the features of the non-expert suggestion includes: Based on the characteristics of the medical report and the characteristics of the non-expert prompt, a search is performed in the medical report collection to obtain the similarity sim between the non-expert prompt and the medical report. i for: In the formula, Represents the characteristics of medical reports, v nxp Characteristic of non-expert prompting; Select the medical report with the highest similarity As search results: In the formula, i * Indicates the serial number of the most similar medical report retrieved. Indicates the most similar medical reports.

5. The method for generating doctor-style prompts according to claim 1, characterized in that: The medical knowledge in the retrieved medical reports is abstracted into a knowledge graph, including: The knowledge graph is represented as: Each meta-knowledge k i Expressed as a triple: k i = {V sourece ,E,V target }, where V source Represents the subject of the relation in meta-knowledge, V target It represents the object of the relationship in meta-knowledge, V and E represent entities and relationships respectively.

6. The method for generating doctor-style prompts according to claim 1, characterized in that: The searching for meta-knowledge related to the non-expert prompt in the knowledge graph includes: Provide a pre-trained BERT model; The knowledge graph Each meta-knowledge k in j As input to the BERT model, the corresponding embedding tensor is output Get the embedded tensor set of the knowledge graph K The non-expert prompt nxp is used as the input of the BER model, and the embedding tensor e of the non-expert prompt nxp is output nxp =BERT(nxp); Compute the embedding tensor e nxp With each embedded tensor in the embedded tensor set ε The cross product of , we get the similarity between non-expert tips and each meta-knowledge in the knowledge graph sim j for: The frequency of the meta-knowledge is expressed as: Calculate the similarity sim j The weighted sum of the frequencies gives the meta-knowledge k j Score: score j =sim j +λ·f j Where λ is a hyperparameter used to exploit similarity and frequency; Using the meta-knowledge k j The score of the meta-knowledge k j Ranking is performed, and finally the meta-knowledge with the highest score is selected as the meta-knowledge related to the non-expert prompt In the formula, j* 表 The subscript of the highest score is shown. represents the highest scoring meta-knowledge.

7. The method for generating doctor-style prompts according to claim 1, characterized in that: The step of converting the meta-knowledge into a doctor-style prompt comprises: Provide a cross-attention model to find meta-knowledge As input to the cross-attention model, the corresponding doctor-style prompt embedding tensor is output: Where LN(·) represents layer normalization, CA(·) represents cross attention, and FFN(·) represents feed-forward network; e attn Indicates attention attn The embedding tensor of the attention attn The embedding tensor of is based on the retrieved medical reports and the meta-knowledge found Calculated, An embedding tensor representing meta-knowledge, The embedding tensor representing the most similar report; Embedding tensor representing the doctor's prompts; Embedding tensor Decode and de-tag to generate doctor-style prompts Among them, the doctor-style prompt The information in is represented as: <V s ,E so ,V d >, <V d ,E la ,V as >, <V adj ,E m ,V s / V d >}, where E so ,E la ,E m Represents three types of relationships between entities: implies, located in, and modifies, V as ,V adj They represent two entities: anatomical location and adjective; among them, sign s implies disease d, and disease d is located in a specific anatomical part; the severity of sign s is described by adjective.

8. A doctor-style prompt generation system, characterized in that: include: A data providing module for providing a collection of medical reports and non-expert prompts; A feature extraction module, which is used to extract features of each medical report in the medical report set and features in the non-expert prompts respectively; A knowledge graph construction module, which is used to abstract the medical knowledge in the retrieved medical reports into a knowledge graph; a data comparison module, which, based on the characteristics of the medical report and the characteristics of the non-expert suggestion, retrieves the medical report most similar to the non-expert suggestion in the medical report set, and searches for meta-knowledge related to the non-expert suggestion in the knowledge graph; A doctor-style prompt generation module, which converts the found meta-knowledge into doctor-style prompts.

9. A medical image synthesis method, characterized in that: include: Get doctor-style tips; Based on the doctor-style prompts, the prompt conditional diffusion model is constructed following the forward process: In the formula, x t+1 and x t Respectively represent the data points at time steps t+1 and t; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; It means the gradient of log probability distribution to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) represents the noise at time t; The medical image synthesized using the prompt conditional diffusion model is: In the formula, represents the product of the step lengths from 0 to t; represents the data point x at time step t-1 t-1 The estimated mean value of z t Represents the inverse distribution of Gaussian noise predicted by the cue-conditioned diffusion model.

10. A medical image synthesis system, characterized in that: include: A doctor-style prompt acquisition module, which is used to acquire doctor-style prompts; A diffusion model building module, which builds a prompt condition diffusion model based on the doctor-style prompts following the forward process: In the formula, x t+1 and x t Respectively represent the data points at time steps t+1 and t; α t represents the step size at t; x and Respectively represent images and doctor-style prompts; It means the gradient of log probability distribution to x when the model parameter is θ; ∈ t ∈N(0,σ 2 ) represents the noise at time t; A medical image synthesis module, which uses the prompt conditional diffusion model to synthesize a medical image as follows: In the formula, represents the product of the step lengths from 0 to t; represents the data point x at time step t-1 t-1 The estimated mean value of z t Represents the inverse distribution of Gaussian noise predicted by the cue-conditioned diffusion model.

11. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 7 and 9, or to run the system described in claim 8 or 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 7 and 9, or to run the system described in claim 8 or 10.

Images