Method and device for automatically generating image description based on ultrasonic image

Abstract

The invention relates to a method and device for automatically generating image description based on an ultrasonic image, and the method comprises the steps: obtaining the ultrasonic image and a corresponding image report, and carrying out the preprocessing of the image report, which comprises data cleaning and report standardization; splitting a text in the preprocessed image report into a plurality of word segmentation units; constructing a Chinese list based on the plurality of word segmentation units and preset medical professional words; constructing a data set based on the Chinese list; training an image text generation model through the data set; and inputting the target ultrasonic image into the trained image text generation model to obtain an image report of the target ultrasonic image. Through standardization of the image text generation model and the image report, the problem of inaccurate report value is solved, and report generation quality and accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and in particular relates to a method and device for automatically generating image descriptions based on ultrasonic images. Background Art

[0002] Image captioning is an important task at the intersection of computer vision and natural language processing (NLP), which aims to generate a natural language description for a given image. This task involves extracting visual features from images and converting these features into coherent text descriptions. Medical Image Report Generation is an important task in medical image analysis, which aims to automatically generate detailed diagnostic reports from medical images (such as X-rays, CT scans, MRI, etc.). This technology can help radiologists and other medical professionals improve their work efficiency and reduce human errors. At present, ultrasound reports are mainly written manually by physicians based on ultrasound images, but manual report writing is very time-consuming and labor-intensive. In recent years, the intersection of computer vision and natural language processing has developed rapidly, extracting visual features from images and converting these features into coherent text descriptions. Inspired by the image captioning task, the image report generation task has received increasing attention.

[0003] Traditional image captioning tasks use convolutional neural networks (CNNs) to extract image features and recurrent neural networks or transformers to generate natural language descriptions. Image report generation draws on the technical methods of image caption generation and also divides the task into two aspects: image feature extraction and text generation. Although the existing technology has made some progress in image report generation, there are still some significant defects: it is impossible to generate numerical information. The existing image report generation model does not study the generation of numerical values ​​in the report and cannot capture and generate numerical values ​​in the report, which are crucial in ultrasound reports. Therefore, the current image report generation model is difficult to apply to ultrasound report generation tasks. Summary of the invention

[0004] In order to solve the problem of inaccurate numerical values ​​in ultrasound image descriptions, a first aspect of the present invention provides a method for automatically generating image descriptions based on ultrasound images, comprising: acquiring an ultrasound image and its corresponding image report, and preprocessing the image report, wherein the preprocessing includes data cleaning and report standardization; splitting the text in the preprocessed image report into multiple word segmentation units; constructing a Chinese list based on the multiple word segmentation units and preset medical professional terms; constructing a data set based on the Chinese list; training an image-text generation model through the data set; and inputting a target ultrasound image into the trained image-text generation model to obtain an image report of the target ultrasound image.

[0005] In some embodiments of the present invention, constructing a Chinese list based on multiple word segmentation units and preset medical professional terms includes: based on word frequency statistics, merging long word segmentation units or long phrases whose word frequency is higher than a preset value; based on the merged long word segmentation units or long phrases, constructing a Chinese list with preset medical professional terms.

[0006] In some embodiments of the present invention, the image-text generation model includes: a visual feature extractor, used to extract visual features from an input image; an encoder, used to encode the visual features to obtain a hidden state of the visual features; a decoder, used to decode the hidden state of the visual features and a preset text sequence to obtain a placeholder output report; and a digital filling module, used to extract size information from the input image and fill the size information into the placeholder output report.

[0007] Furthermore, the decoder obtains an output text probability distribution through the input hidden state and text sequence, and generates a placeholder output report based on the probability distribution.

[0008] Furthermore, the digital filling module includes: a target detection module for extracting category information and size information from an input image; and a contrast learning module for matching the size information with the highest similarity based on the category information, size information and the placeholder report.

[0009] In some embodiments of the present invention, the preprocessing of the imaging report includes: filtering personal information in the imaging report and information beyond the image expression; replacing numerical information in the report with placeholders; and correcting text errors in the imaging report.

[0010] According to a second aspect of the present invention, there is provided a device for automatically generating image descriptions based on ultrasound images, comprising: an acquisition module for acquiring an ultrasound image and its corresponding image report, and preprocessing the image report, wherein the preprocessing includes data cleaning and report standardization; a construction module for splitting the text in the preprocessed image report into a plurality of word segmentation units; constructing a Chinese list based on the plurality of word segmentation units and preset medical professional terms; a training module for constructing a data set based on the Chinese list; training an image-text generation model using the data set; and a generation module for inputting a target ultrasound image into the trained image-text generation model to obtain an image report of the target ultrasound image.

[0011] Furthermore, the acquisition module includes: a filtering unit for filtering personal information in the imaging report and information beyond the image expression; a replacement unit for replacing numerical information in the report with placeholders; and a correction unit for correcting text errors in the imaging report.

[0012] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for automatically generating image descriptions based on ultrasound images provided in the first aspect of the present invention.

[0013] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for automatically generating image descriptions based on ultrasound images provided in the first aspect of the present invention.

[0014] The beneficial effects of the present invention are: Traditional models still have huge defects in the generation of numerical information. They cannot generate or cannot generate numerical values ​​accurately, and the numerical values ​​are limited by the vocabulary. However, the present invention effectively avoids the disadvantages of the image subtitle task model in the generation of numerical information, completes the report generation task in steps, and makes the text description and numerical information better than the traditional model. It effectively solves the shortcomings of the existing technology and improves the professionalism and accuracy of ultrasound report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a basic flow chart of a method for automatically generating image descriptions based on ultrasound images in some embodiments of the present invention; Figure 2 A schematic diagram of the structure of an image text generation model in some embodiments of the present invention; Figure 3 A schematic diagram of an encoder structure in an image text generation model in some embodiments of the present invention; Figure 4 A schematic diagram of a decoder structure in an image-text generation model in some embodiments of the present invention; Figure 5 A schematic diagram of the structure of a digital filling module in an image text generation model in some embodiments of the present invention; Figure 6 is a schematic structural diagram of an apparatus for automatically generating image descriptions based on ultrasound images in some embodiments of the present invention; Figure 7 It is a schematic diagram of the structure of an electronic device in some embodiments of the present invention. DETAILED DESCRIPTION

[0016] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0017] refer to Figure 1 and Figure 2 In a first aspect of the present invention, a method for automatically generating image descriptions based on ultrasound images is provided, comprising: S100. Acquire an ultrasound image and its corresponding image report, and preprocess the image report, wherein the preprocessing includes data cleaning and report standardization; S200. Split the text in the preprocessed image report into multiple word segmentation units; construct a Chinese list based on the multiple word segmentation units and preset medical professional terms; S300. Construct a data set based on the Chinese list; train an image-text generation model using the data set; S100. Input a target ultrasound image into the trained image-text generation model to obtain an image report of the target ultrasound image.

[0018] It should be noted that token: In natural language processing (NLP), it can refer to a word segmentation unit in a text, such as a word, phrase, or symbol.

[0019] Since ultrasound reports usually consist of two parts, the part automatically generated by the ultrasound equipment and the part made by the doctor based on his or her own professional experience, the image report needs to be preprocessed. The purpose of preprocessing is to improve the standardization and readability of ultrasound reports, as well as to reduce harmful samples and the processing complexity of the model.

[0020] In step S100 of some embodiments of the present invention, the preprocessing of the image report includes: S101. Filter personal information in the image report and information beyond the image expression; Specifically, after obtaining ultrasound images and imaging reports from a hospital, all content containing patient personal information in the report is deleted. The data set is an ultrasound image-imaging report pair: the patient's ultrasound image and the doctor's imaging findings. Information in the report that exceeds the image is removed so that the information expressed in the report can correspond to the information expressed in the image.

[0021] S102. Use placeholders to replace numerical information in the report; Specifically, using placeholders <num>Replace the numerical information in the report; placeholders or other identifiers are not limited to <num>It can be set according to actual situation.

[0022] S103.Correct text errors in imaging reports.

[0023] Specifically, deal with common errors in the text, such as spelling errors, grammatical errors, punctuation errors, and unit errors. Unify the format of the report to make it more concise and the expression more reasonable.

[0024] It is understandable that some word segmenters, especially those based on statistics or rules, may incorrectly segment these professional terms into single words or subwords, which may result in the loss of important medical information. Therefore, in step S200 of some embodiments of the present invention, the Chinese list is constructed based on multiple word segmentation units and preset medical professional terms, including: S201. Based on word frequency statistics, long word units or long phrases with a word frequency higher than a preset value are merged; S202. Based on the merged long word segmentation units or long phrases, a Chinese list is constructed together with preset medical professional terms.

[0025] Specifically, based on the report processed in the previous step, the text data is divided into tokens using a tokenizer, and the word frequency is counted. The tokens with high frequency that are actually the entire long sentence are counted and merged to solve the problem of the tokenizer splitting up medical professional terms. All tokens and the merged medical professional terms are combined to form a Chinese list. For example, "acute lymphocytic leukemia" may be incorrectly segmented into "acute", "lymphocyte", "sexual", and "leukemia". Through the above steps, the tokens: "acute", "lymphocyte", "sexual", and "leukemia" will be merged into "acute lymphocytic leukemia".

[0026] refer to Figures 3 to 5 In step S300 or S400 of some embodiments of the present invention, the image text generation model includes: A visual feature extractor, used to extract visual features from an input image; an encoder, used to encode the visual features to obtain a hidden state of the visual features; Specifically, Resnet101 is used as a visual feature extractor to extract features from the input image to obtain visual features Xi. The encoder uses the standard encoder of Transformer to obtain the hidden state hi of the input Xi. In the encoding stage, the visual features Xi are upsampled three times, and features of different scales are fused into the encoder from bottom to top to provide richer visual features.

[0027] A decoder, configured to decode the hidden state of the visual feature and a preset text sequence to obtain a placeholder output report; Specifically, the decoder uses the Transformer decoder to obtain the output text probability distribution through the input hi and the text sequence yt, and generates a placeholder output report based on the probability selection. The placeholder output report is sent to the digital filling module to complete the numerical information and generate the final report.

[0028] A digital fill module (DFI) is used to extract size information from an input image and fill the size information into the placeholder output report.

[0029] Furthermore, the digital filling module includes: a target detection module for extracting category information and size information from an input image; and a contrast learning module for matching the size information with the highest similarity based on the category information, size information and the placeholder report.

[0030] Specifically, the digit filling module consists of object detection and contrastive learning.

[0031] The object detection module is a Faster-RCNN using Resnet101, which is trained using ultrasound images and labeled ultrasound images to obtain category information and size information based on region boxes. The contrastive learning module is trained using real reports with placeholders and category size information of object detection to enable the placeholders to match category size information with high similarity, thereby replacing and generating the final report.

[0032] It should be noted that the training process of the image text generation model includes: dividing the data set into a training set, a validation set, and a test set, with a division ratio of 7:1:2. In the data loading stage, the image is preprocessed, scaled, cropped, and normalized. The text is segmented and padded and truncated. The vector representation of the image and text is input into the model training to generate a predicted text vector. Using the Adam optimizer with the characteristics of adaptive learning rate, fast convergence and strong adaptability, the self-constructed loss function preprocesses the input image before the ultrasound image is input to unify the data format, improve the generalization ability of the model, reduce overfitting, accelerate the training process, and improve the robustness and data balance of the model. We crop the ultrasound image to remove irrelevant information on the image, so that the model focuses more on the target features and reduces the model from overfitting to irrelevant features in the training data. The cropped image is scaled to unify the image size for model processing. Finally, the image is normalized to avoid the input data value being too large or too small, making the model training more stable.

[0033] It can be understood that the existing image text generation task or image report generation task has reached a very high level in text processing, but is very general in digital representation and cannot capture and generate the size numerical information in the image. Compared with the traditional encoder-decoder structure model, the present invention discards the difficult-to-learn digital information in the text generation stage, so that the model focuses more on the generation of plain text and obtains a high-level plain text report. Then, a digital generation module is added to fill in the missing numerical information in the plain text report.

[0034] Exemplarily, refer to the following table, which shows the comparison text between the real report and the method for automatically generating image description based on ultrasound images provided by the present application:

[0035] It can be seen that the traditional model still has huge defects in the generation of numerical information. It cannot generate or cannot accurately generate numerical values, and the numerical values ​​are limited by the vocabulary. However, the present invention effectively avoids the disadvantages of the image subtitle task model in the generation of numerical information, completes the report generation task in steps, and makes the text description and numerical information better than the traditional model. It effectively solves the shortcomings of the existing technology and improves the professionalism and accuracy of ultrasound report generation.

[0036] In a specific embodiment of the method for automatically generating image descriptions based on ultrasound images of the present invention, the method includes: step one: obtaining and preprocessing ultrasound reports; step two: constructing a professional vocabulary for ultrasound reports; step three: model network training; step four: model network inference to obtain image reports.

[0037] Example 2 refer to Figure 6 According to a second aspect of the present invention, there is provided an apparatus 1 for automatically generating image descriptions based on ultrasound images, comprising: an acquisition module 11, for acquiring ultrasound images and their corresponding image reports, and preprocessing the image reports, wherein the preprocessing includes data cleaning and report standardization; a construction module 12, for splitting the text in the preprocessed image report into a plurality of word segmentation units; constructing a Chinese list based on the plurality of word segmentation units and preset medical professional terms; a training module 13, for constructing a data set based on the Chinese list; training an image-text generation model using the data set; and a generation module, for inputting a target ultrasound image into the trained image-text generation model to obtain an image report of the target ultrasound image.

[0038] Furthermore, the acquisition module 11 includes: a filtering unit for filtering personal information and information beyond the image expression in the imaging report; a replacement unit for replacing numerical information in the report with placeholders; and a correction unit for correcting text errors in the imaging report.

[0039] Example 3 refer to Figure 7 According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for automatically generating image descriptions based on ultrasound images according to the first aspect of the present invention.

[0040] The electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0041] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0042] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0043] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more computer programs. When the one or more programs are executed by the electronic device, the electronic device: Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, Python, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0044] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0045] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / num> < / num>

Claims

1. A method for automatically generating image description based on ultrasound images, characterized in that: include: Acquire ultrasound images and their corresponding imaging reports, and preprocess the imaging reports, wherein the preprocessing includes data cleaning and report standardization; Split the text in the preprocessed imaging report into multiple word segmentation units; construct a Chinese list based on multiple word segmentation units and preset medical professional terms; Constructing a data set based on the Chinese list; training an image-text generation model using the data set; The target ultrasound image is input into the trained image-text generation model to obtain an image report of the target ultrasound image.

2. The method for automatically generating image description based on ultrasound images according to claim 1, characterized in that: The Chinese list constructed based on multiple word segmentation units and preset medical professional terms includes: Based on word frequency statistics, long word units or long phrases with word frequencies higher than a preset value are merged; Based on the merged long word units or long phrases, a Chinese list is constructed with preset medical professional terms.

3. The method for automatically generating image description based on ultrasound images according to claim 1, characterized in that: The image text generation model includes: A visual feature extractor for extracting visual features from an input image; An encoder, used for encoding the visual feature to obtain a hidden state of the visual feature; A decoder, configured to decode the hidden state of the visual feature and a preset text sequence to obtain a placeholder output report; A digital filling module is used to extract size information from an input image and fill the size information into the placeholder output report.

4. The method for automatically generating image description based on ultrasound images according to claim 3, characterized in that: The decoder obtains the output text probability distribution through the input hidden state and text sequence, and generates a placeholder output report based on the probability distribution.

5. The method for automatically generating image description based on ultrasound images according to claim 3, characterized in that: The digital filling module includes: Object detection module, used to extract category information and size information from the input image; The comparative learning module is used to match the size information with the highest similarity based on category information, size information and placeholder reports.

6. The method for automatically generating image description based on ultrasound images according to claim 1, characterized in that: The preprocessing of the image report comprises: Filter personal information and information beyond the image expression in the imaging report; Use placeholders to replace numerical information in reports; Fixed text errors in imaging reports.

7. A device for automatically generating image description based on ultrasound images according to claim 1, characterized in that: include: An acquisition module, used for acquiring ultrasound images and their corresponding image reports, and preprocessing the image reports, wherein the preprocessing includes data cleaning and report standardization; A construction module is used to split the text in the pre-processed imaging report into multiple word segmentation units; and construct a Chinese list based on multiple word segmentation units and preset medical professional terms; A training module, used for constructing a data set based on the Chinese list; and training an image-text generation model using the data set; The generation module is used to input the target ultrasound image into the trained image-text generation model to obtain an image report of the target ultrasound image.

8. The device for automatically generating image description based on ultrasound images according to claim 7, characterized in that: The acquisition module comprises: A filtering unit, used for filtering personal information in the image report and information beyond the image expression; Replacement unit, used to replace the numerical information in the report with placeholders; Correction unit, used to correct text errors in image reports.

9. An electronic device, comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for automatically generating image descriptions based on ultrasound images as described in any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method for automatically generating image descriptions based on ultrasound images as described in any one of claims 1 to 6 is implemented.

Images