Method, system and terminal for synthesizing ultrasound images based on medical reports

Abstract

The present application provides a method, system, and terminal for synthesizing ultrasound images based on medical reports, including encoding the medical reports to form text vectors, generating medical images based on the text vectors, and identifying the medical images to generate ultrasound images. The above-mentioned method, system, and terminal for synthesizing ultrasound images synthesize medical images directly from medical reports. Medical students and researchers can compile samples of interest based on sample reports, reducing dependence on actual image data, lowering the threshold for image synthesis, more conveniently solving research problems, improving the efficiency of image synthesis, reducing costs, and providing more learning and research opportunities for medical students and researchers. At the same time, it can also serve as an excellent data expansion tool to provide more training data for deep learning models.

Description

Technical Field

[0001] The present application relates to the field of medical imaging technology, and in particular to a method, system, and terminal for synthesizing ultrasound images based on medical reports. Background Art

[0002] Ultrasound imaging is currently the most commonly used imaging modality in clinical practice and is widely used in medical diagnosis. Radiologists review numerous ultrasound images daily to assist outpatient physicians in their diagnosis. Training a radiologist requires a large number of case images, but due to data privacy concerns, accessing these patient ultrasound images is difficult. Therefore, using deep learning technology to synthesize ultrasound images would effectively address this data privacy issue. Furthermore, synthesized ultrasound images can assist in training machine learning or deep learning algorithms, improving network performance.

[0003] Currently, most mainstream medical image synthesis methods are based on deep learning methods, which use deep learning networks to synthesize medical images of the target domain from medical images of other modalities. The main methods include U-shaped deep convolutional neural networks and generative adversarial networks (GANs).

[0004] Although the image-to-image cross-modal generation method can effectively achieve the synthesis of the target image, the cost of collecting medical image data is very high, and different medical images must be collected for the synthesis of different parts, such as lung CT, heart MRI, etc. As a result, researchers need to spend a lot of time and money to collect and label data for future training. In addition, since medical image data usually contains personal sensitive information, special data privacy measures need to be taken. These measures can prevent data leakage and improper use, but they also increase the complexity of data processing. Therefore, although the image-to-image cross-modal generation method is very effective in achieving the synthesis of the target image, due to the high cost and difficulty in collecting medical image data, this method still faces some challenges in practical application.

[0005] Compared to collecting patient image data, medical report data is easier to obtain because a large amount of medical report data can be obtained from medical institutions or research institutions, and most reports are publicly available. If a method can be developed to synthesize medical images directly from medical reports, medical students and researchers can compile samples of interest based on sample reports, reducing their reliance on actual image data. This will greatly lower the threshold for image synthesis and make it easier to solve research problems. Summary of the Invention

[0006] In view of this, it is necessary to provide a method, system and terminal for synthesizing ultrasound images based on medical reports, which can reduce the dependence on actual image data and improve the efficiency of image synthesis, in order to address the defects in the existing technology.

[0007] To solve the above problems, this application adopts the following technical solutions:

[0008] One of the purposes of this application is to provide a method for synthesizing an ultrasound image based on a medical report, comprising the following steps:

[0009] performing encoding processing on the medical report to form a text vector;

[0010] generating a medical image based on the text vector;

[0011] The medical image is authenticated to generate an ultrasound image.

[0012] In some embodiments, the step of encoding the medical report to form a text vector specifically includes the following steps:

[0013] Identifying word boundaries of the medical report, dividing each report into a plurality of Chinese words according to the word boundaries, and reading the Chinese words one by one into a dictionary to construct a dictionary of all characters;

[0014] Establishing a mapping relationship between the constructed dictionary and the index, and using the index as a digital representation of a word, converting the medical report into a digital code consisting of the index;

[0015] A word sequence in a sentence of the medical report is taken as input according to the digital encoding, and a latent embedding of each word and sentence is learned as input to form a text vector or an embedding of a series of sentences of the medical report is taken as input to form a text vector.

[0016] In some embodiments, the step of generating a medical image according to the text vector specifically includes the following steps:

[0017] generating a low-resolution ultrasound image using the text vector as input;

[0018] Taking the low-resolution ultrasound image as input, a series of images with increasing resolutions are generated.

[0019] In some embodiments, the low-resolution ultrasound image is an ultrasound image with a resolution of 32*32.

[0020] In some embodiments, a subsequent image in the series of images with increasing resolution has a resolution twice that of an adjacent previous image.

[0021] In some embodiments, the step of identifying the medical image to generate an ultrasound image specifically includes the following steps:

[0022] RestNet is used as the backbone network to determine whether the medical image is generated or real.

[0023] A second object of the present application is to provide a system for synthesizing ultrasound images based on medical reports, comprising:

[0024] Hierarchical text encoder: configured to encode the medical report to form a text vector;

[0025] Image generator: used for generating a medical image according to the text vector;

[0026] Discriminator: used for discriminating the medical image to generate an ultrasound image.

[0027] In some embodiments, the hierarchical text encoder includes a word segmentation module, a word embedding module, and an encoding module, wherein:

[0028] Word segmentation module: used to identify word boundaries in the medical report, segment each report into several Chinese words according to the word boundaries, and read the Chinese words one by one into the dictionary to build a dictionary of all words;

[0029] A word embedding module is used to establish a mapping relationship between the constructed dictionary and the index, and use the index as the digital representation of the word to convert the medical report into a digital code composed of the index;

[0030] An encoding module: takes a word sequence in a sentence of the medical report as input according to the digital encoding, and learns a latent embedding of each word and sentence as input to form a text vector or takes the embedding of a series of sentences of the medical report as input to form a text vector.

[0031] In some embodiments, the encoding module includes a word-level LSTM layer and a sentence-level LSTM layer, wherein the word-level LSTM layer takes a word sequence in a sentence of the medical report as input according to the digital encoding, and learns the potential embedding of each word and sentence as input to form a text vector; the sentence-level LSTM layer takes the embedding of a series of sentences of the medical report as input to form a text vector.

[0032] In some embodiments, the image generator includes a basic generator and a progressive generator, wherein:

[0033] The basic generator takes the text vector as input and generates a low-resolution ultrasound image;

[0034] The progressive generator takes the low-resolution ultrasound image as input and generates a series of images with increasing resolutions.

[0035] In some embodiments, the discriminator uses RestNet as a backbone network to determine whether the medical image is generated or real.

[0036] In some embodiments, the discriminator is further configured to determine whether the image generated by the hierarchical text encoder stage is real or synthetic.

[0037] The third object of the present application is to provide a terminal, the terminal comprising a processor and a memory coupled to the processor, wherein:

[0038] The memory stores program instructions for implementing the method for synthesizing an ultrasound image based on a medical report;

[0039] The processor is configured to execute the program instructions stored in the memory to control synthesis of ultrasound images.

[0040] This application adopts the above technical solution, and its beneficial effects are as follows:

[0041] The present application provides a method, system, and terminal for synthesizing ultrasound images based on medical reports, including encoding the medical reports to form text vectors, generating medical images based on the text vectors, and identifying the medical images to generate ultrasound images. The above-mentioned method, system, and terminal for synthesizing ultrasound images synthesize medical images directly from medical reports. Medical students and researchers can compile samples of interest based on sample reports, reducing dependence on actual image data, lowering the threshold for image synthesis, more conveniently solving research problems, improving the efficiency of image synthesis, reducing costs, and providing more learning and research opportunities for medical students and researchers. At the same time, it can also serve as an excellent data expansion tool to provide more training data for deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 Schematic diagram of the principle of the method for synthesizing ultrasound images based on medical reports provided in Example 1 of the present application.

[0044] Figure 2A flowchart of the method for synthesizing ultrasound images based on medical reports provided in Example 1 of the present application.

[0045] Figure 3 A schematic diagram of synthesizing an ultrasound image based on a medical report provided in Example 1 of the present application.

[0046] Figure 4 A schematic diagram of the structure of the system for synthesizing ultrasound images based on medical reports provided in Example 2 of the present application.

[0047] Figure 5 A schematic diagram of the structure of the hierarchical text encoder provided in Example 2 of the present application.

[0048] Figure 6 This is a schematic diagram of the structure of the terminal provided in Example 2 of the present application. DETAILED DESCRIPTION

[0049] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0050] In the description of this application, it should be understood that the terms "upper", "lower", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on this application.

[0051] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0053] Example 1

[0054] Please combine Figure 1 and Figure 2 , which is a flowchart of a method for synthesizing an ultrasound image based on a medical report provided in this embodiment 1, including the following steps S110 to S130. The implementation of each step is described in detail below.

[0055] Step S110: Encoding the medical report to form a text vector.

[0056] In this embodiment, the step of encoding the medical report to form a text vector specifically includes the following steps S111 to S113, and the implementation method of each step is described in detail below.

[0057] Step S111: Identify word boundaries of the medical report, segment each report into a number of Chinese words according to the word boundaries, and read the Chinese words into the dictionary one by one to construct a dictionary of all characters.

[0058] In this embodiment, the jieba tool is used to automatically identify word boundaries of the medical report.

[0059] Step S112: Establish a mapping relationship between the constructed dictionary and the index, use the index as the digital representation of the word, and convert the medical report into a digital code composed of the index.

[0060] It can be understood that in this embodiment, the process of mapping words in the text into the digital space is word embedding.

[0061] Step S113: taking a word sequence in a sentence of the medical report as input according to the digital encoding, and learning a potential embedding of each word and sentence as input to form a text vector or taking the embedding of a series of sentences of the medical report as input to form a text vector.

[0062] It is understandable that in a sentence, some words (e.g., medical terms) are more important than other words (e.g., function words). To account for the different importance of different words, a word-level attention module is used to calculate the attention score for each word. These attention scores are used to reweight the word embeddings when generating the embedding of the entire sentence. Similarly, at the sentence level, some sentences are more important than others, and the sentence-level attention mechanism is used to account for this fact.

[0063] Step S120: Generate a medical image according to the text vector.

[0064] In this embodiment, the step of generating a medical image according to the text vector specifically includes the following steps S121 to S122 , and the implementation method of each step is described in detail below.

[0065] Step S121: using the text vector as input, generating a low-resolution ultrasound image.

[0066] In this example, we use the embedded medical report as input to generate an ultrasound image with a resolution of 32*32. Due to the low resolution of the ultrasound image at this point, generation is relatively easy. By capturing coarse-grained information in the image, we determine the rough appearance of the ultrasound image. Fine-grained details are then filled in by the progressive generator.

[0067] Step S122: using the low-resolution ultrasound image as input, generating a series of images with increasing resolutions.

[0068] In this embodiment, the low-resolution ultrasound image is used as input, and a series of images with increasing resolutions are generated in a cascaded progressive manner. At each stage of the cascade, the resolution of the subsequent image in the series of images with increasing resolutions is twice that of the adjacent previous image. The output of the progressive generator at the n-1 stage is the input of the progressive generator at the n stage. In addition, the progressive generators at all stages also attach the embedded report as a condition. In the present invention, two progressive generators are attached after the basic generator, so the image resolution is first expanded from 32*32 to 64*64, and then from 64*64 to 128*128.

[0069] Step S130: authenticating the medical image to generate an ultrasound image.

[0070] In this embodiment, the step of identifying the medical image to generate an ultrasound image specifically includes the following steps: using RestNet as the backbone network to determine whether the medical image is generated or real, see Figure 3 , which is the ultrasound image obtained in the above-mentioned embodiment 1 of the present application.

[0071] As you can understand, RestNet is used as the discriminator backbone network. Similar to GAN networks, this application adds a discriminator at each stage to determine whether an image is generated or real. The discriminator's inputs include two: 1) images generated at different stages; 2) the embedding vector of the medical report. Based on these two inputs, the discriminator determines whether the predicted synthesized image is real or synthetic. The discriminator's role is to supervise the generator to learn how to generate more realistic medical images that are consistent with the textual knowledge.

[0072] The method for synthesizing ultrasound images in the above-mentioned embodiment 1 of the present application synthesizes medical images directly from medical reports. Medical students and researchers can compile samples of interest based on sample reports, reduce dependence on actual image data, lower the threshold for image synthesis, more conveniently solve research problems, improve the efficiency of image synthesis, reduce costs, and provide medical students and researchers with more learning and research opportunities. It can also serve as a good data expansion tool to provide more training data for deep learning models.

[0073] Example 2

[0074] Please combine Figure 2 and Figure 4 , is a structural diagram of a system for synthesizing ultrasound images based on medical reports provided in this embodiment 2. The implementation method of each component is described in detail below.

[0075] Embodiment 2 of the present application provides a system for synthesizing ultrasound images based on medical reports, including: a hierarchical text encoder 110 for encoding the medical report to form a text vector; an image generator 120 for generating a medical image based on the text vector; and a discriminator 130 for discriminating the medical image to generate an ultrasound image.

[0076] See also Figure 5 The hierarchical text encoder 110 includes a word segmentation module 111, a word embedding module 112 and an encoding module 113, wherein:

[0077] The word segmentation module 111 is used to identify word boundaries of the medical report, segment each report into a number of Chinese words according to the word boundaries, and read the Chinese words into the dictionary one by one to construct a dictionary of all characters.

[0078] In this embodiment, the jieba tool is used to automatically identify word boundaries of the medical report.

[0079] The word embedding module 112 is used to establish a mapping relationship between the constructed dictionary and the index, and use the index as a digital representation of the word to convert the medical report into a digital code composed of the index.

[0080] The encoding module 113 takes a word sequence in a sentence of the medical report as input according to the digital encoding, and learns the potential embedding of each word and sentence as input to form a text vector or takes the embedding of a series of sentences of the medical report as input to form a text vector.

[0081] In some embodiments, the encoding module 113 includes a word-level LSTM layer and a sentence-level LSTM layer, wherein the word-level LSTM layer takes a word sequence in a sentence of the medical report as input according to the digital encoding, and learns the potential embedding of each word and sentence as input to form a text vector; the sentence-level LSTM layer takes the embedding of a series of sentences of the medical report as input to form a text vector.

[0082] It is understandable that in a sentence, some words (e.g., medical terms) are more important than other words (e.g., function words). To account for the different importance of different words, a word-level attention module is used to calculate the attention score for each word. These attention scores are used to reweight the word embeddings when generating the embedding of the entire sentence. Similarly, at the sentence level, some sentences are more important than others, and the sentence-level attention mechanism is used to account for this fact.

[0083] It is understandable that in a sentence, some words (e.g., medical terms) are more important than other words (e.g., function words). To account for the different importance of different words, a word-level attention module is used to calculate the attention score for each word. These attention scores are used to reweight the word embeddings when generating the embedding of the entire sentence. Similarly, at the sentence level, some sentences are more important than others, and the sentence-level attention mechanism is used to account for this fact.

[0084] It can be understood that using hierarchical LSTM as a text encoder can extract word-based and sentence-based fine-grained text features in medical reports.

[0085] In some embodiments, the image generator 114 includes a basic generator and a progressive generator.

[0086] As can be understood, in order to generate high-resolution images, the image generator 114 adopts a multi-scale progressive generation method: first, a high-quality but low-resolution image is generated, and then a high-resolution image is gradually generated from the low-resolution image in multiple stages. The generator consists of two types of generators: (1) a basic generator that generates the lowest-resolution ultrasound image; and (2) a progressive generator that takes the generated low-resolution image as input and generates a high-resolution image. These generators are conditional generators, where the generation of the image is conditioned on the embedding of the ultrasound report.

[0087] Wherein: the basic generator takes the text vector as input to generate a low-resolution ultrasound image; the progressive generator takes the low-resolution ultrasound image as input to generate a series of images with increasing resolution.

[0088] As can be understood, the basic generator takes the embedding of the medical report as input and generates an ultrasound image with a resolution of 32*32. Since the resolution of the ultrasound images at this time is relatively low, it is relatively easy to generate them. The basic generator captures the coarse-grained information in the image and determines the rough appearance of the ultrasound image. The fine-grained details will be filled in by the progressive generator; the progressive generator takes the image generated by the basic generator as input, and the cascaded progressive generators will generate a series of images with increasing resolution. At each stage of the cascade, the image resolution will increase by 2 times. The output of the progressive generator in the n-1 stage is the input of the progressive generator in the nth stage. In addition, the progressive generators in all stages will also attach the embedding of the report as a condition. In the present invention, two progressive generators are attached after the basic generator, so the image resolution is first expanded from 32*32 to 64*64, and then from 64*64 to 128*128.

[0089] It can be understood that in the generator part, a cascade approach is used to generate high-resolution medical images, making the generation task easier to train and converge.

[0090] In some embodiments, the discriminator 130 uses RestNet as a backbone network to determine whether the medical image is generated or real.

[0091] In some embodiments, the discriminator 130 is further configured to determine whether the image generated by the hierarchical text encoder stage is real or synthetic.

[0092] It is understood that in this embodiment, RestNet is used as the backbone network of the discriminator. Similar to the GAN network, this application can add a discriminator at each stage to determine whether the image is generated or real. The input of the discriminator 130 includes two: 1) images generated at different stages; 2) the embedding vector of the medical report. Based on these two inputs, it determines whether the predicted synthetic image is real or synthetic. The role of the discriminator 130 is to supervise the generator to learn how to generate more realistic medical images that are consistent with textual knowledge.

[0093] During training, the objective function of our invention consists of two terms: an adversarial loss and an image reconstruction loss. For the adversarial loss, we use the loss function from WGAN-GP, which measures the difference between real and generated images. The goal of the generator is to minimize this loss, while the goal of the discriminator is to maximize it. The reconstruction loss is defined as the pixel-wise L2 distance between the generated and real images. The generator aims to minimize this loss. The final objective function during the entire training process is the weighted sum of these two terms.

[0094] The system for synthesizing ultrasound images in the above-mentioned embodiment 2 of the present application synthesizes medical images directly from medical reports. Medical students and researchers can compile samples of interest based on sample reports, reduce dependence on actual image data, lower the threshold for image synthesis, more conveniently solve research problems, improve the efficiency of image synthesis, reduce costs, and provide medical students and researchers with more learning and research opportunities. It can also serve as a good data expansion tool to provide more training data for deep learning models.

[0095] Example 3

[0096] See also Figure 6 , is a schematic diagram of the terminal structure of an embodiment of the present application. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0097] The memory 52 stores program instructions for implementing the method for synthesizing ultrasound images based on medical reports;

[0098] The processor 51 is configured to execute the program instructions stored in the memory to control the synthesis of ultrasound images.

[0099] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0100] It can be understood that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The above are merely preferred embodiments of the present application and only specifically describe the technical principles of the present application. These descriptions are intended only to explain the principles of the present application and should not be construed in any way as limiting the scope of protection of the present application. Based on the explanations herein, any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present application, as well as other specific implementations of the present application that can be conceived by those skilled in the art without inventive effort, shall be included within the scope of protection of the present application.

Claims

1. A method for synthesizing an ultrasound image based on a medical report, characterized in that: The steps include: performing encoding processing on the medical report to form a text vector; generating a medical image based on the text vector; performing identification on the medical image to generate an ultrasound image; The step of encoding the medical report to form a text vector specifically includes the following steps: Identifying word boundaries of the medical report, dividing each report into a plurality of Chinese words according to the word boundaries, and reading the Chinese words one by one into a dictionary to construct a dictionary of all characters; Establishing a mapping relationship between the constructed dictionary and the index, and using the index as a digital representation of a word, converting the medical report into a digital code consisting of the index; Taking a word sequence in a sentence of the medical report as input according to the digital encoding and learning a latent embedding of each word and sentence as input to form a text vector or taking an embedding of a series of sentences of the medical report as input to form a text vector; The step of generating a medical image according to the text vector specifically includes the following steps: generating a low-resolution ultrasound image using the text vector as input; Taking the low-resolution ultrasound image as input, generating a series of images with increasing resolutions; In the series of images with increasing resolution, the resolution of each subsequent image is twice that of the adjacent previous image; The step of identifying the medical image to generate an ultrasound image specifically includes the following steps: RestNet is used as the backbone network to determine whether the medical image is generated or real.

2. The method for synthesizing ultrasound images based on medical reports according to claim 1, wherein: The low-resolution ultrasound image is an ultrasound image with a resolution of 32*32.

3. A system for implementing the method for synthesizing ultrasound images based on medical reports according to claim 1, characterized in that: include: Hierarchical text encoder: configured to encode the medical report to form a text vector; Image generator: used for generating a medical image according to the text vector; Discriminator: used for discriminating the medical image to generate an ultrasound image.

4. The system according to claim 3, wherein: The hierarchical text encoder includes a word segmentation module, a word embedding module and an encoding module, wherein: Word segmentation module: used to identify word boundaries in the medical report, segment each report into several Chinese words according to the word boundaries, and read the Chinese words one by one into the dictionary to build a dictionary of all words; A word embedding module is used to establish a mapping relationship between the constructed dictionary and the index, and use the index as the digital representation of the word to convert the medical report into a digital code composed of the index; An encoding module: takes a word sequence in a sentence of the medical report as input according to the digital encoding, and learns a latent embedding of each word and sentence as input to form a text vector or takes the embedding of a series of sentences of the medical report as input to form a text vector.

5. The system according to claim 4, wherein: The encoding module includes a word-level LSTM layer and a sentence-level LSTM layer, wherein the word-level LSTM layer takes a word sequence in a sentence of the medical report as input according to the digital encoding, and learns the potential embedding of each word and sentence as input to form a text vector; the sentence-level LSTM layer takes the embedding of a series of sentences of the medical report as input to form a text vector.

6. The system according to claim 3, wherein: The image generator includes a basic generator and a progressive generator, wherein: The basic generator takes the text vector as input and generates a low-resolution ultrasound image; The progressive generator takes the low-resolution ultrasound image as input and generates a series of images with increasing resolutions.

7. The system according to claim 3, wherein: The discriminator uses RestNet as the backbone network to determine whether the medical image is generated or real.

8. The system according to claim 3 or 7, characterized in that The discriminator is also used to determine whether the image formed by the hierarchical text encoder stage is real or synthetic.

9. A terminal, characterized in that: The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the method for synthesizing an ultrasound image based on a medical report according to any one of claims 1 to 2; The processor is configured to execute the program instructions stored in the memory to control synthesis of ultrasound images.

Images