An ultrasound report generation method integrating image recognition and diffusion generation model
By fusing image recognition and diffusion generation models in intelligent ultrasound image analysis, the problems of low accuracy of ultrasound image report generation and high computing resources are solved, and high accuracy of ultrasound report generation is achieved, reducing doctor copywriting work and improving Check efficiency.
Patent Information
- Application Number
- CN202411190635.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The reports generated by the prior art in ultrasound image intelligent analysis have low accuracy, relying on a large number of computing resources and high-quality labeled data, making it difficult to effectively apply in grassroots hospitals, and there are technical problems such as large-model hallucinations in image semantics generation.
Using an ultrasonic report generation method that integrates image recognition and diffusion generation model, the spatial information adapter algorithm improves tumor segmentation and classification accuracy in image big models, and uses cross entropy loss function and Adam optimizer to train the diffusion generation model to generate Medical diagnostic report.
It greatly improves the accuracy of tumor area segmentation and classification of ultrasound images, reduces the time for doctors to write reports, improves examination efficiency, and partially replaces expert doctors, alleviates the shortage of talents in grassroots hospitals.
Smart Images

Figure CN118711744B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of medical image analysis and natural language, and in particular to an ultrasound report generation method integrating image recognition and a diffusion generation model. Background Art
[0002] The automatic generation technology of scanning reports based on intelligent analysis of ultrasound images can be used as an auxiliary tool for ultrasound scanning examinations, reducing the time of doctors' paperwork, improving examination efficiency, and increasing the number of patients undergoing ultrasound examinations per unit time, thereby alleviating the shortage of ultrasound doctors in primary hospitals. In order to meet the needs of digital medicine and electronic medical records, doctors generate ultrasound examination reports through keyboard or voice input. This method saves a certain amount of time, but still relies entirely on the doctor's professional knowledge and personal experience. Furthermore, image description processing models represented by LSTM are mainly concentrated in the field of natural images, and the ultrasound image reports they generate lack accuracy.
[0003] The accuracy of medical image description based on commercial language big models is low. In theory, the language big model should be deployed locally, and the ultrasound image text report database should be used as corpus for enhanced training through retrieval enhancement generation to generate ultrasound reports with medical value. However, the deployment and training of language big models require a lot of computing resources, and the image semantic generation through retrieval enhancement has technical problems such as big model hallucination that are difficult to completely solve. It is not suitable for the generation of ultrasound medical reports without a large amount of high-quality annotated data and computing power. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides an ultrasound report generation method that integrates image recognition and a diffusion generation model. The method studies and develops a spatial information adapter algorithm model structure, which is embedded in a large image model framework to provide high-accuracy segmentation and classification results; the tumor segmentation area is used as the region of interest when simulating a doctor's diagnosis, replacing the Faster-RCNN method to determine the key image description area for the diffusion generation model; finally, the diffusion generation algorithm model is trained by the cross entropy loss function and the Adam optimizer training method.
[0005] The present invention is achieved through the following technical solutions:
[0006] A method for generating an ultrasound report by integrating image recognition and a diffusion generation model comprises the following steps:
[0007] (1) Establish an ultrasound image and text report database and perform data enhancement on the two different modalities;
[0008] (2) Design a spatial adapter and feature fusion module for ultrasound images and embed them into the image encoder of the image segmentation model to accurately segment and classify tumor images;
[0009] (3) Perform data enhancement on the training data of ultrasound images, including rotation, flipping, displacement, zooming in and out operations, to enhance the generalization of the image training data. At the same time, process the text report corresponding to each image, align the diagnostic keywords semantically, and list them in the form of a table;
[0010] (4) Establish a diffusion text generation algorithm model, which includes a text semantic encoder, a tumor region image encoder, and a hybrid decoder. The output result is a text description of the input ultrasound tumor image.
[0011] (5) Use the Adam optimizer to complete network training of the diffusion text generation algorithm model. It is necessary to independently select the initial learning rate and continuously change the learning rate according to the gradient changes during training. Finally, a text description and auxiliary diagnosis report based on the ultrasound image of the tumor area is generated.
[0012] Specifically, the spatial adapter in the step (2) is composed of a convolution layer and a maximum pooling layer with different convolution kernel sizes, and the feature fusion module is composed of a cross attention layer, a linear transformation layer and an activation function; the output of each transformer module and the spatial adapter in the image encoder is fused by the feature fusion module and used as the input of the next transformer module, and the number of embedded spatial adapters and feature fusion modules can be adjusted according to different data.
[0013] Specifically, the training data of the ultrasound image and the text report in step (3) are in a one-to-one correspondence.
[0014] Specifically, the input of the text semantic encoder in step (4) is extracted from the text report database in step (1) and has a one-to-one correspondence with the image input. The sentence text is converted into a vector by embedding and input into the semantic encoder, and its encoder architecture is the same as that of the tumor image encoder.
[0015] Specifically, the tumor region image encoder in step (4) uses the segmentation result in step (2) as the region of interest input. According to the ultrasound image, it can be known that the region of interest is one or more. Its structure is a transformer architecture encoder, which consists of a multi-head self-attention mechanism layer and a forward propagation layer.
[0016] Specifically, the structure of the hybrid decoder in step (4) consists of a multi-head self-attention mechanism layer, a multi-head cross-attention mechanism layer and a forward propagation layer; the output features of the tumor image encoder need to pass through the self-attention and cross-attention layers in sequence, and the output features of the semantic encoder are directly input into the cross-attention layer, and calculated with the image features in the cross-attention layer, and finally the text result is output through the forward propagation layer and the linear layer.
[0017] Specifically, the diffusion text generation algorithm model in step (4) uses the cross-entropy loss function CrossEntropy for calculation to measure the difference between the real text and the generated text in the training data; at the same time, when the computing power is sufficient, the model can improve the accuracy of text generation by stacking multiple diffusion modules.
[0018] Specifically, the diffusion text generation algorithm model supports the generation of ultrasound tumor image reports in different parts, including the pelvis, breast and thyroid.
[0019] Specifically, the tumor area of the ultrasound image is regarded as the area of interest for the doctor's diagnosis, and the extraction is completed through the segmentation algorithm, which better replaces the role of Faster-RCNN in the original model. The same ultrasound image can have one or more tumor areas.
[0020] Specifically, the ultrasound report generation belongs to the text generation within 20 words in the text generation task. It is necessary to use a single autoregressive training method to characterize the tumor image and the corresponding text, as well as the conditional probability between each text and other text, and then train the diffusion text generation algorithm model based on the maximum likelihood estimation MLE.
[0021] The beneficial effects of the present invention are as follows: the present invention greatly improves the accuracy of ultrasound image tumor area segmentation and classification tasks by changing the encoder structure in the large image segmentation model, and at the same time simulates the doctor's diagnostic thinking, uses the tumor area as the region of interest for ultrasound report generation, and further uses the diffusion text generation model to perform medical diagnosis description of the tumor area, and automatically completes the generation of ultrasound examination reports. The invention can reduce the copywriting time of doctors writing reports and improve the efficiency of examinations. At the same time, it can play a role in partially replacing the examination of expert doctors in grassroots hospitals, alleviating the problem of the lack of grassroots ultrasound doctors. Among them, the spatial adapter, based on the characteristics of ultrasound tumor images, retains more image spatial structure features of the tumor area through the image feature extraction method of variable receptive field convolution operation, and further uses the feature fusion module to fuse general image features and spatial features to achieve the role of information feature completion; using tumor areas and type information with diagnostic value to replace the traditional text generation model, using multiple regions of interest output by Faster-RCNN as training input, and using transformer as the basic architecture to design the autoregressive model of the encoder and decoder structure, the model reduces the generation time while ensuring the accuracy of text generation. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 It is the overall flow chart of the present invention;
[0024] Figure 2 It is a neural network architecture diagram for the pre-tumor image segmentation and classification tasks in the present invention;
[0025] Figure 3 It is a specific neural network architecture diagram of the spatial adapter and feature fusion module of the spatial adaptive encoder of the present invention;
[0026] Figure 4 It is a specific architecture diagram of the diffusion generation model of the present invention. DETAILED DESCRIPTION
[0027] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0028] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0029] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0030] The present invention provides an ultrasound report generation method integrating image recognition and diffusion generation model, comprising the following steps: Figure 1 As shown, the method comprises the following steps:
[0031] S1, establish a database of ultrasound tumor scanning images and corresponding text reports;
[0032] In this database, as training data, scanned images and text reports need to have a one-to-one correspondence; the image and text data in the database need to undergo data enhancement preprocessing, and the processing method involved is the same as the general data enhancement processing method for deep learning.
[0033] S2, simulates the doctor’s prior knowledge of the region of interest and text report of diagnosis through the tumor segmentation and classification task of ultrasound images;
[0034] The tumor segmentation task algorithm is an adaptive improvement based on the SAM image segmentation model. It mainly adds a spatial adapter and a feature fusion module to the image encoder to enhance the performance in ultrasound image tumor segmentation. Figure 2 In the paper, a single Transformer is taken as an example (which contains the specific architecture of the main innovative spatial adaptive image encoder); the algorithm model framework of the segmentation and classification tasks and the specific structure of the spatial image encoder we designed are demonstrated. In the specific implementation case, the image encoder is composed of 12 Transformer modules in series, and the spatial adapter and feature fusion modules are embedded in the 4th and 8th Transformer modules respectively;
[0035] like Figure 3 As shown in the figure, the spatial adapter is composed of a 7×7 convolution layer, a maximum pooling layer, a 5×5 convolution layer, a 3×3 convolution layer, and a 1×1 convolution layer in sequence, and each layer performs a downsampling with a stride of 2; the feature fusion module is composed of 2 cross attention layers and 1 feedforward network, which is used to cross-fuse the output features of the Transformer module and the spatial adapter;
[0036] The tumor classification algorithm uses the image encoder in the segmentation algorithm to extract image features, and then performs classification calculations through a classifier, which consists of 2 fully connected layers and a SoftMax activation function;
[0037] In the training of the tumor image segmentation model, the torch AdamW optimizer is used, the weight decay is 0.0005, the initial learning rate is 0.0005, the batch size is 8, and the loss function uses a combination of Dice loss and focal loss. During the training process, the model only updates the parameters of the spatial adapter, feature fusion module and decoder, and freezes the remaining network parameters to save computing resources.
[0038] S3, data integration of tumor diagnosis regions, tumor type labels, and corresponding text reports;
[0039] Create an index table and correspond the tumor area, tumor classification, and text report by annotation to ensure that the subsequent image-to-text algorithm can accurately locate the input data through the table.
[0040] S4, build a diffusion generation model based on the Transformer encoder-decoder architecture, such as Figure 4 As shown, the network mainly includes three parts: image encoder, semantic encoder and sentence decoder;
[0041] The Transformer image encoder consists of two visual encoders, each of which includes a multi-head self-attention layer (MultiHead), two stacking and normalization layers and a feed-forward network (FFN); the output expression of the nth visual encoder is: ;
[0042] ;
[0043] The specific structure of the Transformer semantic encoder is the same as that of the image encoder. Its input is the corresponding text and words obtained by retrieving image tags, and the pre-processing of the encoder input is further completed through the text embedding layer.
[0044] The Transformer sentence decoder includes 2 decoders and 1 linear fully connected layer. Each decoder consists of 1 multi-head self-attention layer, 1 cross self-attention layer and 1 feedforward network. The output expression of the n+1th decoder is:
[0045] ;
[0046] ;
[0047] in, It is the semantic feature output by the Transformer semantic encoder. Furthermore, the output of the last decoder calculates the probability distribution of the semantic vector through the SoftMax activation function, and finally selects the text word with the highest probability.
[0048] The diffusion generation model uses the cross entropy loss function, selects the Adam optimizer, the batch size is 5, the initial learning rate is 0.00001, and the label smoothing method is applied for training.
[0049] The diffusion generative model can be composed of multiple Figure 4 The structures in the figure are connected in series and the accuracy of ultrasound generation is further improved by superimposing computing power. Only a single structure is used in this implementation case.
[0050] In one embodiment, after training and verification converge simultaneously, the network model structure and its hyperparameters are saved. When the neural network algorithm is called, the network model can be read and then input data, and classification can be performed directly according to the saved hyperparameters.
[0051] In one embodiment, a portable ultrasound scanning device with a network transmission function is proposed, which is miniaturized, portable, and has a built-in or external wireless network data transmission function. At the same time, a computer device is proposed as a remote server, including a memory and a processor, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps in the method and system for rapid classification of acute abdomen in portable ultrasound scanning images in the above-mentioned embodiments.
[0052] In one embodiment, a storage medium storing computer-readable instructions is provided, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the neural network method and system for rapid classification of acute abdomen in portable ultrasound scanning images in the above-mentioned embodiments. The storage medium may be a non-volatile storage medium.
[0053] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0054] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.
Claims
1. A method for generating ultrasound reports by integrating image recognition and diffusion generation model, characterized in that: The following steps are involved: (1) Establish an ultrasound image and text report database and perform data enhancement on the two different modalities; (2) Designing a spatial adapter and a feature fusion module for ultrasound images, and embedding them into the image encoder of the large image segmentation model to accurately segment and classify tumor images; the spatial adapter is composed of a convolution layer with different convolution kernel sizes and a maximum pooling layer, and the feature fusion module is composed of a cross attention layer, a linear transformation layer, and an activation function; the output of each transformer module and the spatial adapter in the image encoder is fused by the feature fusion module and used as the input of the next transformer module, and the number of embedded spatial adapters and feature fusion modules can be adjusted according to different data; (3) Perform data enhancement on the training data of ultrasound images, including rotation, flipping, displacement, zooming in and out operations, to enhance the generalization of the image training data. At the same time, process the text report corresponding to each image, align the diagnostic keywords semantically, and list them in the form of a table; (4) Establishing a diffusion text generation algorithm model, which includes a text semantic encoder, a tumor region image encoder and a hybrid decoder, and the output result is a text description of the input ultrasound tumor image; the diffusion text generation algorithm model uses a cross entropy loss function Cross Entropy is calculated to measure the difference between the real text and the generated text in the training data; at the same time, when the computing power is sufficient, the model can improve the accuracy of text generation by superimposing multiple diffusion modules; the input of the text semantic encoder is extracted from the text report database, and there is a one-to-one correspondence with the image input. The sentence text is converted into a vector by embedding and input into the semantic encoder. Its encoder architecture is the same as that of the tumor image encoder; the tumor area image encoder uses the segmentation result as the input of the region of interest. According to the ultrasound image, it can be known that the region of interest is one or more. Its structure is a transformer architecture encoder, which consists of a multi-head self-attention mechanism layer and a forward propagation layer; the structure of the hybrid decoder consists of a multi-head self-attention mechanism layer, a multi-head cross-attention mechanism layer and a forward propagation layer; the output features of the tumor image encoder need to pass through the self-attention and cross-attention layers in turn, and the output features of the semantic encoder are directly input into the cross-attention layer, calculated with the image features in the cross-attention layer, and finally the text result is output through the forward propagation layer and the linear layer; (5) Use the Adam optimizer to complete network training of the diffusion text generation algorithm model. It is necessary to independently select the initial learning rate and continuously change the learning rate according to the gradient changes during training. Finally, a text description and auxiliary diagnosis report based on the ultrasound image of the tumor area is generated.
2. The ultrasound report generation method integrating image recognition and diffusion generation model according to claim 1, characterized in that: The training data of the ultrasound image and the text report in step (3) are in a one-to-one correspondence.
3. The ultrasound report generation method integrating image recognition and diffusion generation model according to claim 1, characterized in that: The diffusion text generation algorithm model supports the generation of ultrasound tumor image reports in different parts, including the pelvis, breast and thyroid.
4. The ultrasound report generation method integrating image recognition and diffusion generation model according to claim 1, characterized in that: The tumor area of the ultrasound image is regarded as the area of interest for the doctor's diagnosis, and is extracted through a segmentation algorithm to replace Faster-RCNN in the original model. The same ultrasound image can have one or more tumor areas.
5. The ultrasound report generation method integrating image recognition and diffusion generation model according to claim 1, characterized in that: The ultrasound report generation belongs to the text generation within 20 words in the text generation task. It is necessary to use a single autoregressive training method to characterize the tumor image and the corresponding text, as well as the conditional probability between each text and other text, and then train the diffusion text generation algorithm model based on the maximum likelihood estimation MLE.
Citation Information
Cited By
Detection sheet generation method and system based on big data
CN121215153A
