Image report generation method and device, computer equipment and storage medium

Abstract

The invention relates to an image report generation method and device, computer equipment and a storage medium, and is applied to the field of image processing, and the method comprises the steps: obtaining a medical image and prompt information; the prompt information is information set for a region of interest in the medical image; and according to the prompt information, generating a report statement for the medical image. According to the method, analysis is carried out in combination with the actual medical image, the image report of the region of interest in the medical image is directly generated according to the prompt information, manual selection of feature description of the features of interest is not needed, generation of the image report is more intelligent and automatic, the report generation efficiency can be greatly improved, and the user experience is improved. And the report generation is more objective and accurate, and has higher logicality and readability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image report generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In medical imaging diagnosis, traditional text reports lack accuracy and comprehensibility. Moreover, since doctors need to manually input a large amount of information when writing reports, report generation efficiency is low and prone to errors. Therefore, in order to improve the quality and generation efficiency of medical imaging reports, structured reports came into being.

[0003] However, the existing structured report generation method mainly relies on the doctor's experience and knowledge. In the structured report system for a specific disease, the doctor needs to manually select the selectable feature descriptions based on the preset fixed descriptions. Therefore, this method still requires doctors to perform long-term manual operations, and the efficiency of generating reports is still low. Summary of the invention

[0004] Based on this, it is necessary to provide an image report generation method, apparatus, computer device, computer readable storage medium and computer program product to address the technical problem that the report generated by the above-mentioned report generation method is inefficient.

[0005] In a first aspect, the present application provides a method for generating an image report. The method comprises:

[0006] Acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image;

[0007] A report statement for the medical image is generated according to the prompt information.

[0008] In one embodiment, the prompt information is determined by:

[0009] The prompt information is obtained in response to a selection operation on a region of interest in the medical image and / or a prompt information input operation.

[0010] In one embodiment, generating a report statement for the medical image according to the prompt information includes:

[0011] According to the prompt information, the medical image is segmented to obtain a corresponding segmented image;

[0012] A report statement for the medical image is generated according to the segmented image and the prompt information.

[0013] In one embodiment, the medical image includes images of the same medical object from multiple perspectives, and the prompt information includes perspective information of the specified image;

[0014] The step of segmenting the medical image according to the prompt information to obtain a corresponding segmented image includes:

[0015] According to the viewing angle information of the designated image in the prompt information, determining a target image from the images of multiple viewing angles included in the medical image;

[0016] In the target image, the region of interest corresponding to the prompt information is segmented to obtain the segmented image.

[0017] In one embodiment, the prompt information includes location prompt information and text prompt information of the region of interest in the medical image;

[0018] The step of segmenting the medical image according to the prompt information to obtain a corresponding segmented image includes:

[0019] Inputting the medical image, the text prompt information and the position prompt information in the prompt information into a trained segmentation model to obtain the segmented image;

[0020] The step of generating a report statement for the medical image according to the segmented image and the prompt information includes:

[0021] The segmented image and the text prompt information in the prompt information are input into the trained visual language model to obtain a report sentence for the medical image.

[0022] In one embodiment, the trained visual language model is obtained by:

[0023] Fine-tuning a pre-trained image encoder using a first sample data set to obtain a trained image encoder; the first sample data set includes a first sample medical image;

[0024] Pre-training the initial query conversion model using the trained image encoder and a second sample data set to obtain a pre-trained query conversion model; the second sample data set includes a second sample medical image and text description information of the second sample medical image;

[0025] Retraining the pre-trained query conversion model through a pre-trained large language model and a third sample data set to obtain a trained query conversion model; the third sample data set includes a third sample medical image and a real report sentence of the third sample medical image;

[0026] The trained visual language model is composed based on the trained image encoder, the pre-trained large language model and the trained query conversion model.

[0027] In one embodiment, the trained segmentation model is obtained by:

[0028] Obtain a pre-trained basic large model; the basic large model includes an image encoder, a prompt encoder and a mask decoder;

[0029] The image encoder in the basic large model is fine-tuned using the first sample medical image to obtain a new image encoder, and the image encoder in the basic large model is replaced with the new image encoder to obtain the trained segmentation model.

[0030] In a second aspect, the present application also provides an image report generating device. The device comprises:

[0031] An acquisition module, used to acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image;

[0032] A generating module is used to generate a report statement for the medical image according to the prompt information.

[0033] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0034] Acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image;

[0035] A report statement for the medical image is generated according to the prompt information.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0037] Acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image;

[0038] A report statement for the medical image is generated according to the prompt information.

[0039] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image;

[0041] A report statement for the medical image is generated according to the prompt information.

[0042] The above-mentioned image report generation method, apparatus, computer equipment, storage medium and computer program product, when generating an image report, analyze the actual medical image and directly generate an image report of the area of ​​interest in the medical image according to the prompt information, without the need for manual selection of feature descriptions of the features of interest, making the generation of image reports more intelligent and automated, which can greatly improve the efficiency of report generation and make the report generation more objective, accurate, logical and readable. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is a schematic diagram of a flow chart of a method for generating an image report in one embodiment;

[0044] FIG2 is a schematic diagram of an image of a medical image before and after masking in one embodiment;

[0045] FIG3 is a flow chart of a visual language model training step in one embodiment;

[0046] FIG4 is a schematic flow chart of an image report generating method in another embodiment;

[0047] FIG5 is a structural block diagram of an image report generating device in one embodiment;

[0048] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0051] In one embodiment, as shown in FIG1 , a method for generating an image report is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0052] Step S110, obtaining a medical image and prompt information; the prompt information is information set for a region of interest in the medical image.

[0053] The medical image is an image containing a medical object, which may be a two-dimensional image or a three-dimensional image. For example, the medical image may be an X-ray of the cervical spine, a computed tomography (CT) image of the cervical spine, etc.

[0054] The region of interest may represent a region of particular concern in a medical image. The region of interest may be a specific part of the patient's body, such as a damaged area, abnormal tissue, etc., or may be a specific structure or organ, such as the abdomen.

[0055] The prompt information may be in the form of location prompt information, such as clicking or selecting a box for the region of interest; or it may be in the form of text prompt information, such as "the cervical curvature can be seen from the lateral view", indicating that the designated region of interest is a region that can characterize the cervical curvature.

[0056] In a specific implementation, a medical image of a medical object to be detected can be collected by a medical imaging device and then obtained from the medical imaging device. A pre-stored medical image can also be obtained from a storage device such as a database or a server. After the medical image is obtained, prompt information set by the user for the area of ​​interest in the medical image can be received.

[0057] In this step, prompt information for the medical image is set so as to accurately locate the region of interest according to the prompt information, thereby identifying and analyzing the region of interest.

[0058] Step S120: Generate a report statement for the medical image according to the prompt information.

[0059] The report statement may represent the diagnosis result of the medical image. For example, if the medical image is a cervical spine X-ray, and the prompt information is "the cervical spine curvature can be seen from the lateral view", the region of interest representing the cervical spine curvature is analyzed, and the resulting report statement may be "the cervical spine curvature is normal", or "the cervical spine curvature is arched", or "the cervical spine curvature becomes straight", etc.

[0060] In a specific implementation, the features of the region of interest in the medical image can be extracted, and a report statement of the region of interest can be generated according to the features of the region of interest and the prompt information, as a report statement for the medical image. In one application mode, a report statement for the medical image can be generated by a neural network model. Specifically, a report generation model can be trained in advance with sample images and prompt information as input variables and real report statements of the sample images as supervisory information. After the training is completed, the medical image and prompt information are input into the trained report generation model for processing, and a report statement for the medical image is output.

[0061] In the above-mentioned image report generation method, when generating the image report, the actual medical image is analyzed, and an image report of the area of ​​interest in the medical image is directly generated according to the prompt information. There is no need to manually select the feature description of the feature of interest, which makes the generation of the image report more intelligent and automated, can greatly improve the report generation efficiency, and also makes the report generation more objective, accurate, logical and readable.

[0062] In an exemplary embodiment, the prompt information is determined in the following manner: in response to a selection operation on a region of interest in a medical image, and / or a prompt information input operation, the prompt information is obtained.

[0063] Specifically, the user can select a region of interest in the medical image by drawing a bounding box, clicking, etc., or can enter text information through an input box, drop-down menu or other interactive methods. The terminal integrates the region of interest selected by the user and the text information entered by the user to obtain prompt information.

[0064] Among them, the difference in medical objects and the difference in image types of medical images will affect the setting of prompt information. For example, for the same image type, if the medical objects are different, the prompt information will also be different. For example, the image type is X-ray, but the prompt information when the medical object in the medical image is the cervical spine and the prompt information when the medical object is the abdomen will be different. For example, the prompt information of the cervical spine is generally information related to the cervical curvature, cervical alignment, morphology of the cervical vertebrae, gaps, etc., while the prompt information of the abdomen is generally information related to the abdominal organs. Similarly, for the same medical object, but different image types, the representation of the medical object will also be different. For example, the medical objects are all cervical vertebrae. When the image type is an X-ray film, the anteroposterior and lateral films of the cervical vertebrae are generally collected, and the cervical vertebrae bone structure is displayed. Then the prompt information is information related to the anteroposterior film, lateral film, and cervical vertebrae bone structure; when the image type is a CT image, a three-dimensional image of the cervical vertebrae is generated, and the detailed anatomical information or soft tissue structure information of the cervical vertebrae bone structure is displayed. Therefore, the prompt information is generally information related to the detailed anatomical information or soft tissue structure information of the cervical vertebrae bone structure. In summary, the prompt information is determined by the medical object in the medical image and the image type of the medical image, thereby improving the rationality of the determined prompt information, so that the region of interest can be accurately located according to the prompt information, and an accurate report can be generated. Among them, the medical object can be a part of the patient's body, a tissue, an organ, etc., such as the cervical vertebra, abdomen, arm, etc. Among them, the image type is determined according to the acquisition method of the medical image, for example, the image type can be an X-ray film, an ultrasound image, and a computed tomography (CT) image.

[0065] Furthermore, in addition to the medical object and image type, actual needs can also be considered when setting the prompt information. For example, for X-rays of the cervical spine, when you want to know the curvature of the cervical spine, the prompt information set is "The curvature of the cervical spine can be seen from the lateral view"; when you want to know the morphology of the C2 vertebra of the cervical spine, the prompt information set is "The morphology of the C2 vertebra of the cervical spine can be seen from this image"; when you want to know the gap between the C2-C3 vertebrae of the cervical spine, the prompt information set is "The gap between the C2-C3 vertebrae of the cervical spine can be seen from the lateral view".

[0066] In this embodiment, the generation and application of prompt information can realize the effective analysis and processing of medical images and improve the efficiency and accuracy of report generation.

[0067] In an exemplary embodiment, the above-mentioned step S120 generates a report statement for the medical image according to the prompt information, specifically including: performing segmentation processing on the medical image according to the prompt information to obtain a corresponding segmented image; and generating a report statement for the medical image according to the segmented image and the prompt information.

[0068] The segmented image is used to represent the mask of the region of interest corresponding to the prompt information.

[0069] In a specific implementation, when segmenting a medical image according to the prompt information set for the region of interest in the medical image, the segmentation can be performed through a neural network model, and the medical image and the prompt information are input into the trained segmentation model to obtain a segmented image. Specifically, the segmentation model may include an image encoder, a prompt encoder, and a mask decoder, and the input of the mask decoder is the output of the image encoder and the prompt encoder. The image encoder is used to encode the input medical image into an embedding, that is, to convert the medical image into a vector representation; the prompt encoder is used to embed the input prompt information encoder, that is, to convert the prompt information into a vector representation; the mask decoder is used to predict the mask in real time, integrate the two embeddings output by the image encoder and the prompt encoder, and decode the segmentation mask from the feature map of the integrated embedding.

[0070] Correspondingly, the medical image and the prompt information are input into the trained segmentation model, and the medical image and the prompt information are processed by the trained segmentation model to obtain the specific process of the segmented image: the prompt information is input into the prompt encoder in the trained segmentation model, and the encoded prompt information is output, and the medical image is input into the image encoder in the trained segmentation model, and the encoded image information is output; the encoded prompt information and the encoded image information are further input into the mask decoder, and the encoded prompt information and the encoded image information are mapped to the mask by the mask decoder, and the segmented image is output. In this way, according to the prompt information, the segmentation model can segment the structure or area related to the prompt information in the medical image in a targeted manner, more accurately identify the specific structure or tissue in the medical image, and realize automated analysis and recognition, so that the part of interest can be located and identified more quickly.

[0071] Furthermore, after the segmentation is completed, a report statement for the medical image can be generated according to the segmented image and the prompt information. The specific implementation method can be to input the segmented image and the prompt information into a trained visual language model to obtain a report statement for the medical image. Among them, the trained visual language model may include an image encoder, a large language model and a query conversion model (Querying-Former), the input of the large language model is the output of the query conversion model, and the input of the query conversion model is the output of the image encoder and the prompt information. The specific process of inputting the segmented image and the prompt information into the trained visual language model, processing the segmented image and the prompt information through the trained visual language model, and obtaining the report statement for the medical image is: inputting the segmented image into the image encoder in the trained visual language model, extracting the visual features of the segmented image through the image encoder; the extracted visual features and the prompt information are input into the query conversion model together, and after being processed by the query conversion model, the processing result will be input into the large language model, and the report statement of the segmented image is obtained through the large language model as the report statement for the medical image. In this method, the segmented image and prompt information are processed by a trained visual language model to obtain report sentences for medical images, which can improve the efficiency of report generation. When generating report sentences, the visual (segmented image) and language information (prompt information) are combined to better understand the relationship between medical image content and prompt information, thereby generating more semantically accurate report sentences.

[0072] In this embodiment, the corresponding segmented image is first segmented from the medical image according to the prompt information to achieve accurate positioning of the region of interest, and then an image report is generated based on the segmented image and the prompt information, thereby reducing the impact of interference noise in the medical image and further improving the efficiency of image report generation.

[0073] In an exemplary embodiment, the medical image includes images of multiple perspectives of the same medical object, and the prompt information includes perspective information of the specified image; the above-mentioned segmentation processing of the medical image according to the prompt information to obtain the corresponding segmented image includes: determining a target image from the images of the multiple perspectives included in the medical image according to the perspective information of the specified image in the prompt information; in the target image, segmenting the region of interest corresponding to the prompt information to obtain a segmented image.

[0074] It can be understood that the medical image collected for the medical object may include not only one but multiple perspective images, for example, the medical image includes the anteroposterior and lateral views of the cervical spine. In this case, the information obtained from images of different perspectives will be different, for example, the cervical curvature and cervical alignment are generally obtained from the lateral view. Therefore, when setting the prompt information, the image to be observed can be specified, so the prompt information will include the perspective information of the specified image, so that when performing image segmentation, targeted segmentation can be performed.

[0075] Specifically, when a medical image includes multiple images, when performing image segmentation, the target image can be first determined from the images of multiple perspectives included in the medical image according to the perspective information of the image specified in the prompt information. In the target image, the region of interest corresponding to the prompt information is segmented to obtain a segmented image. In this process, the unspecified image can be used as reference information for segmentation.

[0076] For example, as shown in FIG2 , the medical image includes an AP and lateral view of the cervical spine, and the prompt information is “the cervical curvature can be seen from the lateral view”. Then, based on the prompt information, the lateral view of the cervical spine can be used as the target image, and in the lateral view of the cervical spine, the area representing the cervical curvature can be segmented to obtain a segmented image.

[0077] In this embodiment, based on the prompt information, the target image is determined from the images of multiple perspectives included in the medical image, and then the region of interest corresponding to the prompt information is segmented in the target image. This can achieve precise positioning and segmentation of the region of interest, thereby ensuring the accuracy of the segmented image. The segmentation process can remove irrelevant areas, reduce interference information, and improve the accuracy and efficiency of subsequent analysis and report generation.

[0078] It should be noted that the prompt information set for the region of interest in the medical image includes position prompt information and text prompt information. The position prompt information refers to a click or box selection for the region of interest; the text prompt information refers to the text edited for the region of interest.

[0079] Correspondingly, in an exemplary embodiment, the above-mentioned step of segmenting the medical image according to the prompt information to obtain the corresponding segmented image includes: inputting the medical image, the position prompt information and the text prompt information in the prompt information into a trained segmentation model to obtain a segmented image.

[0080] Specifically, the segmentation model can be a model of SAM (Segment-Anything Model) architecture. SAM is an interactive promptable model, including an image encoder, a prompt encoder and a mask decoder. SAM can return a valid segmentation mask given any segmentation prompt. Therefore, when performing mask segmentation through the segmentation model of SAM architecture, different forms of segmentation prompts can be applied, so the medical image, the position prompt information in the prompt information and the text prompt information can be input into the trained segmentation model to obtain the segmented image. In this step, the medical image is masked and segmented in combination with the position prompt information and the text prompt information. The position prompt information can guide the model to focus on the important features of a specific area in the medical image, thereby improving the accuracy and precision of the segmentation; the text prompt information can provide additional information about the content of the medical image, help the model better understand the image, and enable the model to more effectively capture the features related to the text prompt information, thereby improving the segmentation results.

[0081] The above step of generating a report sentence for the medical image based on the segmented image and the prompt information includes: inputting the segmented image and the text prompt information in the prompt information into a trained visual language model to obtain a report sentence for the medical image.

[0082] Specifically, the visual language model may be a model of the BLIP-2 (Bootstrapping Language-Image Pre-training, a pre-training model that freezes existing visual models and large language models) architecture. When the visual image is converted into a report sentence through the visual language model of the BLIP-2 architecture, the prompt information input into the visual language model only includes the text prompt information in the prompt information, and does not include the location prompt information in the prompt information. That is, when generating the report sentence, the segmented image and the text prompt information in the prompt information are input into the trained visual language model for processing to obtain the report sentence for the medical image.

[0083] In this embodiment, by combining the generation capability of the large language model in the medical image and visual language model, structured medical image report sentences can be automatically and efficiently generated, thereby improving the doctor's work efficiency and diagnostic accuracy.

[0084] In an exemplary embodiment, as shown in FIG3 , the trained visual language model is obtained by:

[0085] Step S310, fine-tuning the pre-trained image encoder using a first sample data set to obtain a trained image encoder; the first sample data set includes a first sample medical image;

[0086] Step S320, pre-training the initial query conversion model by using the trained image encoder and the second sample data set to obtain a pre-trained query conversion model; the second sample data set includes the second sample medical image and the text description information of the second sample medical image;

[0087] Step S330, retraining the pre-trained query conversion model by using the pre-trained large language model and the third sample data set to obtain a trained query conversion model; the third sample data set includes a third sample medical image and a real report sentence of the third sample medical image;

[0088] Step S340, composing a trained visual language model based on the trained image encoder, the pre-trained large language model and the trained query conversion model.

[0089] Specifically, the visual language model can be a BLIP2 model. By applying the BLIP2 architecture to medical image analysis, the data of two different modalities, medical images and natural language, are organically integrated, and the information of medical images and natural language is processed and understood at the same time to generate report statements for medical images.

[0090] As shown in the schematic diagram of the BLIP-2 model in Figure 4, BLIP-2 consists of a pre-trained, frozen-parameter visual model (image encoder) and text model (large language model), as well as a trainable query transformation model (Q-Former). Among them, Q-Former consists of two Transformer submodules (Transformer is a model based on the attention mechanism), namely image transformer and text transformer. The self-attention layer of these two submodules is shared, which can be understood as having two inputs to the self-attention layer, namely: Queries and Text. The input of the image transformer is learnable query embeddings (queries), which first interact with the text through the self-attention layer, and then interact with the frozen image features through the cross-attention layer. The parameters of the cross-attention layer of Q-Former are randomly initialized, and the other parameters are initialized with the parameters of the pre-trained BERT_{base}.

[0091] It can be understood that since the image encoder and the large language model in the BLIP-2 model are both pre-trained models, and the pre-trained image encoder is generally obtained based on natural image training, and the support for medical images is relatively limited, therefore, before using the BLIP-2 model, this embodiment first uses the collected medical image data set to train a specific medical image image encoder method to improve the processing effect of the BLIP-2 model on medical images. That is, first perform the above step S310, fine-tune the pre-trained image encoder through the first sample data set, and obtain a trained image encoder. Afterwards, the query conversion model (Q-Former) is trained based on the trained image encoder and the pre-trained large language model.

[0092] The training of the query conversion model (Q-Former) is divided into two steps. First, learn visual text encoding from a trained image encoder with fixed parameters, that is, execute step S320, pre-train the initial query conversion model through the pre-trained image encoder and the second sample data set to obtain a pre-trained query conversion model. This step includes three training objectives: Image-Text Contrastive Learning (ITC, image-text contrast learning), Image-grounded Text Generation (ITG, image text generation), and Image-Text Matching (ITM, image text matching). Secondly, learn to understand image to text generation from a fixed parameter language large model, that is, execute step S330, re-train the pre-trained query conversion model through a pre-trained large language model and a third sample data set. Specifically, the third sample medical image can be used as an input variable, and the real report sentence of the third sample medical image can be used as the supervision information for training to obtain a trained query conversion model. And connect the query conversion model (Q-Former) through a fully connected layer, map the queries to the same dimension as the text vector of the large language model, and insert them in front of the text vector. Since Q-Former has been trained to extract linguistic information from visual representations, it acts as an information bottleneck, effectively providing the most useful information to the large language model while removing irrelevant visual information. This reduces the burden of learning visual-linguistic alignment for the large language model, thereby alleviating the catastrophic forgetting problem.

[0093] Finally, the trained image encoder, the pre-trained large language model and the trained query conversion model are combined into a trained visual language model according to the association between input and output.

[0094] In this embodiment, by using the first sample medical image to fine-tune the pre-trained image encoder in the visual language model, the visual language model can be better adapted to the characteristics of the medical image dataset, thereby improving the accuracy of the generated medical image report. At the same time, by fine-tuning the image encoder based on the pre-trained visual language model, this transfer learning method can accelerate the training process and improve the training efficiency of the visual language model.

[0095] In an exemplary embodiment, the trained segmentation model is obtained in the following manner: obtaining a pre-trained basic large model; the basic large model includes an image encoder, a prompt encoder and a mask decoder; fine-tuning the image encoder in the basic large model through a first sample medical image to obtain a new image encoder, and replacing the image encoder in the basic large model with the new image encoder to obtain a trained segmentation model.

[0096] In a specific implementation, a pre-trained SAM model can be used as a basic large model. At the same time, considering that the pre-trained basic large model is generally obtained by training based on natural images, and its support for medical images is relatively limited, this embodiment proposes a method of using a collected medical image dataset to train an image encoder for a specific medical image to improve the segmentation effect of SAM on medical images.

[0097] Specifically, a batch of first sample medical images can be obtained, and the first sample medical images can be used as input variables. The model parameters can be continuously adjusted through the back propagation algorithm to fine-tune the pre-trained image encoder, and the trained image encoder can be used as a new image encoder. The new image encoder replaces the image encoder in the basic large model, and the model including the new image encoder, the prompt encoder and the mask decoder is used as the trained segmentation model.

[0098] In this embodiment, by using the first sample medical image to train a new image encoder and replacing it with the image encoder in the basic large model, the segmentation model can be better adapted to the characteristics of the medical image dataset, thereby improving the accuracy of medical image segmentation. At the same time, by fine-tuning the image encoder based on the pre-trained basic large model, this transfer learning method can accelerate the training process and improve the training efficiency of the segmentation model.

[0099] In one embodiment, in order to facilitate those skilled in the art to understand the embodiments of the present application, the following will be explained in conjunction with specific examples of the accompanying drawings. Referring to Figure 4, a medical image structured report framework based on the prompt (prompt information) engineering of the BLIP-2 visual language model and the SAM framework is shown. This application is based on the BLIP-2 visual language multimodal large language model framework, and processes both medical image and natural language data forms, and introduces the prompt mechanism of the SAM model. It can preset different prompts for different medical images and diagnostic needs, assist the image in understanding the diagnostic part that the current text is concerned about, and finally guide the model to generate complete and accurate report statements, effectively improving the quality and generation efficiency of medical image reports. A specific example is given below using the diagnostic report of cervical spine X-ray images as an example.

[0100] It should be noted that this embodiment regards the SAM model and the BLIP-2 model as a whole. The SAM model can be trained in advance and used as a preprocessing stage for medical images. The following model training mainly refers to the training of the BLIP-2 model.

[0101] First, in the model training stage, corresponding prompts are preset for different types of sample medical images. For example, for cervical spine X-rays, the following prompt can be preset: the cervical curvature can be seen from the lateral view, and the expected result of this prompt is straightening, normal or reverse arch. Among them, the lateral view is based on the doctor's actual diagnostic experience. When observing the cervical curvature, the judgment is mainly based on the lateral cervical spine X-ray. The training data also takes both the frontal and lateral images as input, and the model can automatically select the X-ray to be observed according to the prompt. In the model training stage, a large number of cervical spine X-ray images and corresponding prompts are input. By using a large amount of training data, a model that can generate a complete report statement according to the preset prompt is trained.

[0102] In the model application stage, the trained model is applied to actual medical imaging diagnosis. For cervical spine X-rays, the X-ray images and preset prompts (such as the cervical curvature can be seen from the lateral view) can be input into the model. After the model is processed, it can automatically generate a complete medical imaging report statement, such as "the cervical curvature is normal" or "the cervical curvature becomes straight". Similarly, for other types of medical images, other prompts can also be preset, such as "the cervical alignment can be seen from the frontal view", and the model can generate complete report statements such as "the cervical alignment is acceptable" or "the cervical alignment is poor".

[0103] The model can be optimized and adjusted based on the actual application results and feedback. For example, more prompts can be preset for different types of medical images to cover more diagnostic information. At the same time, the model parameters and structure can be adjusted to improve the efficiency and accuracy of the model.

[0104] This application proposes a medical image structured report framework based on a visual language multimodal large model, which can realize the automation and intelligence of medical image reports and improve the efficiency of report generation. Specifically, this application uses the BLIP-2 framework for image analysis, information extraction and text matching. The BLIP-2 model is a multimodal visual language deep learning framework that can automatically identify and extract key information from medical images, such as lesion location, size, shape, etc., and align it with the corresponding text information to provide doctors with detailed and accurate diagnosis basis. The SAM (point and frame interaction) architecture can help with precise positioning and interaction in medical images. Through simple point or frame operations, it is easy to circle or delineate the diagnostic area of ​​interest, providing an accurate data basis for subsequent analysis and diagnosis. At the same time, the architecture can also set prompt words to guide the model. Simply enter simple keywords or phrases to guide the model to automatically identify and extract specific diagnostic site information. Finally, this application utilizes the capabilities of the large language model so that the framework can generate complete medical image report statements based on the prompt information preset by the doctor. This not only makes report generation more intelligent and automated, greatly improving the efficiency of report generation, but also makes report generation more objective, accurate, logical and readable.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides an image report generation device for implementing the image report generation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image report generation device embodiments provided below can refer to the limitations of the image report generation method above, and will not be repeated here.

[0107] In one embodiment, as shown in FIG5 , an image report generating device is provided, including: an acquisition module 510 and a generation module 520, wherein:

[0108] The acquisition module 510 is used to acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image;

[0109] The generating module 520 is used to generate a report statement for the medical image according to the prompt information.

[0110] In one embodiment, the acquisition module 510 is further configured to obtain prompt information in response to a selection operation on a region of interest in a medical image and / or a prompt information input operation.

[0111] In one embodiment, the generating module 520 is used to perform segmentation processing on the medical image according to the prompt information to obtain a corresponding segmented image; and generate a report statement for the medical image according to the segmented image and the prompt information.

[0112] In one embodiment, the medical image includes images of multiple perspectives of the same medical object, and the prompt information includes perspective information of the specified image; the generation module 520 is further used to determine a target image from the images of the multiple perspectives included in the medical image based on the perspective information of the specified image in the prompt information; in the target image, the region of interest corresponding to the prompt information is segmented to obtain a segmented image.

[0113] In one embodiment, the prompt information includes location prompt information and text prompt information of the area of ​​interest in the medical image; the generation module 520 is also used to input the medical image, the text prompt information in the prompt information and the location prompt information into the trained segmentation model to obtain a segmented image; the segmented image and the text prompt information in the prompt information are input into the trained visual language model to obtain a report sentence for the medical image.

[0114] In one of the embodiments, the device also includes a visual language model training module, which is used to fine-tune a pre-trained image encoder through a first sample data set to obtain a trained image encoder; the first sample data set includes a first sample medical image; the initial query conversion model is pre-trained through the trained image encoder and the second sample data set to obtain a pre-trained query conversion model; the second sample data set includes the second sample medical image and the text description information of the second sample medical image; the pre-trained query conversion model is re-trained through a pre-trained large language model and a third sample data set to obtain a trained query conversion model; the third sample data set includes a third sample medical image and a real report sentence of the third sample medical image; based on the trained image encoder, the pre-trained large language model and the trained query conversion model, a trained visual language model is composed.

[0115] In one embodiment, the device also includes a segmentation model training module for obtaining a pre-trained basic large model; the basic large model includes an image encoder, a prompt encoder and a mask decoder; the image encoder in the basic large model is fine-tuned through a first sample medical image to obtain a new image encoder, and the image encoder in the basic large model is replaced with the new image encoder to obtain a trained segmentation model.

[0116] Each module in the above-mentioned image report generating device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0117] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image report generation method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, a trackball or a touch pad provided on the housing of the computer device, or an external keyboard, touch pad or mouse, etc.

[0118] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0120] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0121] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0124] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating an image report, characterized in that: The method comprises: Acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image; A report statement for the medical image is generated according to the prompt information.

2. The method according to claim 1, characterized in that The prompt information is determined in the following manner: The prompt information is obtained in response to a selection operation on a region of interest in the medical image and / or a prompt information input operation.

3. The method according to claim 1, characterized in that Generating a report statement for the medical image according to the prompt information includes: According to the prompt information, the medical image is segmented to obtain a corresponding segmented image; A report statement for the medical image is generated according to the segmented image and the prompt information.

4. The method according to claim 3, characterized in that: The medical image includes images of a same medical object from multiple perspectives, and the prompt information includes perspective information of a specified image; The step of segmenting the medical image according to the prompt information to obtain a corresponding segmented image includes: According to the viewing angle information of the designated image in the prompt information, determining a target image from the images of multiple viewing angles included in the medical image; In the target image, the region of interest corresponding to the prompt information is segmented to obtain the segmented image.

5. The method according to claim 3, characterized in that: The prompt information includes position prompt information and text prompt information of the region of interest in the medical image; The step of segmenting the medical image according to the prompt information to obtain a corresponding segmented image includes: Inputting the medical image, the text prompt information and the position prompt information in the prompt information into a trained segmentation model to obtain the segmented image; The step of generating a report statement for the medical image according to the segmented image and the prompt information includes: The segmented image and the text prompt information in the prompt information are input into the trained visual language model to obtain a report sentence for the medical image.

6. The method according to claim 5, characterized in that The trained visual language model is obtained in the following way: Fine-tuning a pre-trained image encoder using a first sample data set to obtain a trained image encoder; the first sample data set includes a first sample medical image; Pre-training the initial query conversion model using the trained image encoder and a second sample data set to obtain a pre-trained query conversion model; the second sample data set includes a second sample medical image and text description information of the second sample medical image; Retraining the pre-trained query conversion model through a pre-trained large language model and a third sample data set to obtain a trained query conversion model; the third sample data set includes a third sample medical image and a real report sentence of the third sample medical image; The trained visual language model is composed based on the trained image encoder, the pre-trained large language model and the trained query conversion model.

7. The method according to claim 5, characterized in that The trained segmentation model is obtained in the following way: Obtain a pre-trained basic large model; the basic large model includes an image encoder, a prompt encoder and a mask decoder; The image encoder in the basic large model is fine-tuned using the first sample medical image to obtain a new image encoder, and the image encoder in the basic large model is replaced with the new image encoder to obtain the trained segmentation model.

8. An image report generating device, characterized in that: The device comprises: An acquisition module, used to acquire a medical image and prompt information; the prompt information is information set for a region of interest in the medical image; A generating module is used to generate a report statement for the medical image according to the prompt information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the image report generation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image report generating method according to any one of claims 1 to 7 are implemented.

Images