Method and program product for generating medical image report

Abstract

The invention relates to a medical image report generation method and a program product. The method comprises the following steps: acquiring a medical image of a target object and a text cue word corresponding to the medical image; inputting the medical image and the text cue word into a medical report generation model for processing, and generating an initial medical report; inputting the medical image into a target disease classification model for target disease detection to obtain a detection result; and fusing the detection result into the initial medical report to form a medical image report of the target object. By adopting the method, the medical image report can be automatically generated, and the medical image report does not need to be written manually in the process, so that the generation efficiency of the medical image report can be improved. In addition, the accurate detection result and the initial medical report are fused in the medical image report, so that the accuracy of the medical image report can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the medical technology field, and in particular to a medical image report generation method and program product. BACKGROUND

[0002] With the development of medical technology, an image department doctor can write a medical image report on an interface of a RIS system (Radiology Information System).

[0003] Currently, a structured report template is usually pre-configured, and a doctor manually fills in options in the structured report template to generate a medical image report.

[0004] However, the current medical image report generation method has the problem of low efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a medical image report generation method and program product to improve the efficiency of generating a medical image report.

[0006] In a first aspect, the present application provides a medical image report generation method, comprising:

[0007] obtaining a medical image of a target object and a text prompt word corresponding to the medical image;

[0008] inputting the medical image and the text prompt word into a medical report generation model for processing to generate an initial medical report;

[0009] inputting the medical image into a target disease classification model for target disease detection to obtain a detection result;

[0010] fusing the detection result into the initial medical report to form a medical image report of the target object.

[0011] In one embodiment, the step of fusing the detection result into the initial medical report to form the medical image report of the target object comprises:

[0012] arranging the detection result according to a preset structured rule to generate a structured report of the target disease;

[0013] fusing the structured report and the initial medical report to form the medical image report of the target object.

[0014] In one embodiment, the step of fusing the structured report and the initial medical report to form the medical image report of the target object comprises:

[0015] If the initial medical report contains first content corresponding to the target disease, the second content in the structured report is used to replace the first content to form a medical image report of the target object;

[0016] Alternatively, if the initial medical report does not contain the first content corresponding to the target disease, the second content is added to the initial medical report to form a medical image report of the target object.

[0017] In one of the embodiments, the method further comprises:

[0018] The initial medical report is input into a disease-specific placeholder model to identify the description information of the target disease, and the description content of the target disease is obtained;

[0019] If the initial medical report includes the description content, a placeholder is used to replace the description content of the target disease in the initial medical report to generate an intermediate medical report;

[0020] Alternatively, if the initial medical report does not include the description content, the placeholder is added to any position in the initial medical report to generate an intermediate medical report;

[0021] The structured report and the initial medical report are fused to form a medical image report of the target object, comprising:

[0022] The structured report and the intermediate medical report are fused to form a medical image report of the target object.

[0023] In one of the embodiments, the structured report and the intermediate medical report are fused to form a medical image report of the target object, comprising:

[0024] The second content in the structured report is used to replace the placeholder to form a medical image report of the target object.

[0025] In one of the embodiments, the method further comprises:

[0026] In response to a triggering operation on any field of the second content in the structured report, the area image corresponding to the any field is located from the medical image, and the area image is displayed on a display interface; the any field is used to describe the target disease.

[0027] In one of the embodiments, the method further comprises:

[0028] In response to the update operation on the detection results, the detection results, the structured report, and the medical imaging report are updated according to the update operation.

[0029] In one embodiment, the medical report generation model includes an image encoder, a text encoder, a mapper, and a text decoder. The step of inputting the medical image and the text prompts into the medical report generation model for processing to generate an initial medical report includes:

[0030] The medical image is input into the image encoder for conversion to obtain an image embedding vector;

[0031] The image embedding vector is input into the mapper to generate a processed image embedding vector; the mapper is used to perform feature alignment processing on the image and text based on the mapping relationship between the image and text learned during training.

[0032] The text prompts are input into the text encoder for conversion to obtain text embedding vectors;

[0033] The processed image embedding vector and the text embedding vector are input into the text decoder for processing to obtain the initial medical report.

[0034] In one embodiment, obtaining the text prompts corresponding to the medical image includes:

[0035] The medical image is input into an image encoder for conversion to obtain an image embedding vector;

[0036] The image embedding vector is input into a disease classifier to detect disease categories and obtain abnormal category labels;

[0037] The text prompt word is constructed based on the anomaly category label.

[0038] Secondly, this application also provides a medical image report generation device, comprising:

[0039] The acquisition module is used to acquire the medical images of the target object and the corresponding text prompts for the medical images;

[0040] The processing module is used to input the medical images and the text prompts into the medical report generation model for processing, and generate an initial medical report;

[0041] The detection module is used to input the medical images into the target disease classification model to detect the target disease and obtain the detection results;

[0042] The fusion module is used to fuse the detection results into the initial medical report to form a medical image report of the target object.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for generating medical image reports in the first aspect described above.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for generating a medical image report as described in the first aspect.

[0045] Fifthly, this application also provides a computer program product comprising a computer program, which is processed by a processor to perform the steps of the method for generating medical image reports in the first aspect described above.

[0046] The aforementioned method and program for generating medical imaging reports involve: acquiring medical images of the target object and corresponding text prompts; inputting the medical images and text prompts into a medical report generation model for processing to generate an initial medical report; inputting the medical images into a target disease classification model for target disease detection to obtain detection results; and integrating the detection results into the initial medical report to form the target object's medical imaging report. This application embodiment can utilize a target disease classification model specifically for target disease detection, accurately obtaining detection results and automatically generating the medical imaging report by integrating these accurate results into the initial medical report. This process eliminates the need for manual writing of medical imaging reports, thus improving the efficiency of medical imaging report generation. Furthermore, since the medical imaging report integrates accurate detection results and the initial medical report, this application embodiment also improves the accuracy of the medical imaging report. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is an application environment diagram of a method for generating medical image reports in one embodiment;

[0049] Figure 2 This is a flowchart illustrating a method for generating a medical image report in one embodiment;

[0050] Figure 3 This is a flowchart illustrating the initial medical report generation steps in one embodiment;

[0051] Figure 4 This is a schematic diagram illustrating the generation of an initial medical report based on a medical report generation model in one exemplary embodiment.

[0052] Figure 5 This is a flowchart illustrating the fusion steps in one embodiment;

[0053] Figure 6 This is a schematic diagram of structured rules in an exemplary embodiment;

[0054] Figure 7 This is a flowchart illustrating the fusion step in another embodiment;

[0055] Figure 8 This is a flowchart illustrating the steps for generating an intermediate medical report in one embodiment;

[0056] Figure 9 This is a schematic diagram of the display interface in one embodiment;

[0057] Figure 10 This is a flowchart illustrating a method for generating medical image reports in another embodiment;

[0058] Figure 11 This is a structural block diagram of a medical image report generation device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0061] With the development of medical technology, radiologists can write medical imaging reports on the interface of the RIS (Radiology Information System).

[0062] Currently, medical imaging reports are usually written manually, or pre-configured structured report templates are prepared, and doctors manually fill in / check the options in the structured report templates to generate medical imaging reports.

[0063] However, the current method of manually writing medical imaging reports is time-consuming and labor-intensive, and the volume of radiology imaging examinations is huge, resulting in low efficiency in the current method of generating medical imaging reports.

[0064] Having described the background technology of the medical image report generation method provided in the embodiments of this application, the implementation environment involved in the medical image report generation method provided in the embodiments of this application will be briefly described below. The medical image report generation method provided in the embodiments of this application can be applied to, for example... Figure 1 The computer device shown can be a terminal or a server. It includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for generating medical image reports. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0065] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0066] In one embodiment, such as Figure 2As shown, a method for generating medical image reports is provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0067] S201, Obtain the medical image of the target object and the corresponding text prompts for the medical image.

[0068] The target object can include, but is not limited to, any object such as a patient, an object, or a human model. Medical images can include, but are not limited to, CT (Computed Tomography) images, PET (Positron Emission Computed Tomography) images, and MR (Magnetic Resonance) images. Text prompts are used to instruct the medical report generation model to convert the images into medical reports.

[0069] In this embodiment of the application, optionally, the computer device can obtain the medical images of the target object from a database or a public platform in advance, or the computer device can scan the target object in real time to obtain the medical images of the target object.

[0070] In this embodiment of the application, optionally, the computer device can respond to a text instruction entered by the user on the display interface and obtain a text prompt corresponding to the medical image based on the text instruction. For example, the text prompt could be "Please convert the input image into an image report". Alternatively, the computer device can first determine the image embedding vector corresponding to the medical image, and then construct the text prompt based on the image embedding vector corresponding to the medical image.

[0071] S202, input the medical images and text prompts into the medical report generation model for processing to generate an initial medical report.

[0072] The medical report generation model is used to generate medical image reports from medical images. For example, the medical report generation model can be any of the pre-trained large models such as the Vision-Language Model (VLM). The initial medical report refers to the initial text report output by the medical report generation model. The initial medical report includes at least one of the disease / site described in text, such as pulmonary nodules, enlarged heart, pleural effusion, etc.

[0073] In this embodiment of the application, optionally, the computer device can also acquire multiple historical medical images and corresponding historical text reports, and train a model based on the multiple historical medical images and corresponding historical text reports to obtain a medical report generation model. Alternatively, the computer device can directly obtain a pre-trained medical report generation model from commercial sources or public platforms.

[0074] Optionally, the computer device can input the medical image of the target object into the medical report generation model to generate a report, thereby outputting an initial medical report for the target object. Alternatively, the computer device can first perform image processing on the medical image of the target object, and then input the processed medical image into the medical report generation model to generate a report, thereby outputting an initial medical report for the target object. The image processing methods may include, but are not limited to, at least one of the following: filtering, denoising, and image enhancement.

[0075] S203: Input medical images into the target disease classification model to detect the target disease and obtain the detection results.

[0076] The target disease classification model is used for the detection and segmentation of target diseases. That is, the target disease classification model can be a detection and segmentation model, also known as a disease-specific model. Disease-specific models can detect subtle lesions that are difficult to observe in medical report generation models, such as lung nodules and fractures. Furthermore, disease-specific models are typically sensitive to measured values, such as the diameter of aortic dilation; therefore, diseases based on measured values ​​can also be detected using disease-specific models. For example, disease-specific models can include, but are not limited to, lung nodule detection models, fracture detection models, and aortic segmentation models. The target disease refers to lesions that may be incorrectly detected in the initial medical report (such as subtle lesions like lung nodules or fractures), and / or lesions with diverse descriptive styles (such as lung nodules), and / or lesions that require linkage with the lesion area in medical images. The detection result refers to the list of lesions of the target disease output by the target disease classification model.

[0077] It should be noted that the initial medical report generated based on the aforementioned medical report generation model may contain errors in the detection and measurement results of the target disease, or the description style of the target disease in the initial medical report may be inconsistent with the hospital's usual practices. Therefore, in this embodiment, for the aforementioned target disease, the computer device can input medical images into the target disease classification model to detect the target disease and obtain the detection results.

[0078] Optionally, the computer device can directly input the medical image of the target object into the target disease classification model to detect the target disease and obtain the detection result. Alternatively, the computer device can first perform image processing on the medical image of the target object, and then input the processed medical image into the target disease classification model to detect the target disease and obtain the detection result. The image processing method may include, but is not limited to, at least one processing method such as filtering, denoising, and image enhancement.

[0079] S204 integrates the test results into the initial medical report to form a medical imaging report for the target subject.

[0080] In this embodiment, optionally, the computer device can directly integrate the detection results into the corresponding position of the target disease in the initial medical report to form a medical imaging report for the target object. Alternatively, the computer device can first organize the detection results according to preset structured rules to generate a structured report for the target disease, and then integrate the structured report for the target disease into the corresponding position of the target disease in the initial medical report to form a medical imaging report for the target object. Of course, this embodiment does not limit the specific implementation method of integrating the detection results into the initial medical report.

[0081] In addition, the computer device can automatically display the medical image report of the target object on the computer device's display interface; or, in response to the user's operation of opening the display interface in the RIS system, the computer device can display the medical image report of the target object on the display interface so that the user can view, edit or change the medical image report.

[0082] In the aforementioned method for generating medical imaging reports, the medical images of the target object and corresponding text prompts are obtained; the medical images and text prompts are input into a medical report generation model for processing to generate an initial medical report; the medical images are input into a target disease classification model for target disease detection to obtain detection results; and the detection results are integrated into the initial medical report to form a medical imaging report for the target object. This embodiment of the application can use a target disease classification model specifically for target disease detection, accurately obtaining the detection results and integrating the accurate detection results into the initial medical report, thus automatically generating the medical imaging report. This process eliminates the need for manual writing of medical imaging reports, thereby improving the efficiency of medical imaging report generation. Furthermore, since the medical imaging report integrates accurate detection results and the initial medical report, this embodiment of the application can also improve the accuracy of the medical imaging report.

[0083] In one embodiment, such as Figure 3As shown, the aforementioned medical report generation model includes an image encoder, a text encoder, a mapper, and a text decoder. Based on this, an implementation method for generating an initial medical report is provided, namely, "inputting medical images and text prompts into the medical report generation model for processing to generate an initial medical report" in S202 above, including:

[0084] S301, the medical image is input into the image encoder for conversion to obtain the image embedding vector.

[0085] In this embodiment of the application, a computer device can input medical images into an image encoder for encoding and conversion to obtain image embedding vectors. For example, the image encoder can be a frozen pre-trained visual encoder.

[0086] S302, the image embedding vector is input into the mapper to generate the processed image embedding vector.

[0087] The mapper is used to align the features of images and text based on the learned mapping relationship between images and text. For example, the mapper can be a multimodal projector. Through training, the mapper can map image features to a space with the same dimension as the text features; that is, it has pre-trained and learned the cross-modal mapping relationship between images and text. Essentially, the cross-modal mapping relationship is stored in the model beforehand. Therefore, in practical applications, only the image embedding vector needs to be input into the mapper, and the mapper can automatically align the image embedding vector and the text embedding vector.

[0088] In this embodiment, to align the image embedding vector and the text embedding vector, the computer device can input the image embedding vector into a multimodal projector for feature alignment processing to obtain the processed image embedding vector. For example, the projector can be an MLP (Multilayer Perceptron), or it can be a linear layer. The processed image embedding vector and the text embedding vector can reside in the same feature space, meaning the lengths of the two vectors are the same.

[0089] S303, input the text prompt words into the text encoder for conversion to obtain the text embedding vector.

[0090] In this embodiment of the application, after obtaining the text prompts corresponding to the medical images, the computer device can input the text prompts (prompt / question) into a text encoder for encoding and conversion to obtain text embedding vectors (Text Embedding).

[0091] S304, the processed image embedding vector and text embedding vector are input into the text decoder for processing to obtain the initial medical report.

[0092] In this embodiment, the computer device can input the processed image embedding vector and text embedding vector into a text decoder for processing, thereby outputting an initial medical report. The text decoder can also be replaced by an LLM (Large Language Model).

[0093] For example, such as Figure 4 As shown, Figure 4 As an exemplary embodiment, this diagram illustrates the generation of an initial medical report based on a medical report generation model. The computer device can input a medical image Xv into a frozen, pre-trained vision encoder for encoding and transformation, obtaining an image embedding vector, i.e., visual features Zv, where Zv = g(Xv). Then, the computer device can input the visual features Zv into a linear projection layer W for feature alignment, obtaining aligned visual features Hv (i.e., the processed image embedding vector), where Hv = W·Zv. Thus, the aligned visual features Hv and the text features Hq reside in the same text space.

[0094] In addition, combined Figure 4 As shown, the computer device can acquire the text prompts Xq corresponding to the medical image and input them into a text encoder for encoding and transformation to obtain text features Hq (i.e., text embedding vectors). Therefore, the computer device can input the aligned visual features Hv and text features Hq together into an LLM model (i.e.,... Figure 4 The initial medical report Xa is generated by processing the Language Model (LLM) through multiple Transformer Decoder layers of attention mechanism.

[0095] In this embodiment, accurate image embedding vectors and accurate text embedding vectors can be obtained through image encoders and text encoders, respectively. The image embedding vectors and text embedding vectors can be feature aligned through a mapper. Thus, the aligned image embedding vectors and text embedding vectors can be input into the text decoder for processing, thereby automatically and accurately outputting the initial medical report for one-scan-multiple-checks based on the medical report generation model.

[0096] In one optional implementation, "obtaining the text prompt words corresponding to the medical image" in S201 includes:

[0097] The medical images are input into an image encoder for conversion to obtain image embedding vectors.

[0098] Image embedding vectors are input into a disease classifier to detect disease categories and obtain abnormal category labels.

[0099] Construct text prompts based on anomaly category labels.

[0100] The classification head module is a module in the deep learning model used to map features to category prediction, and it is usually composed of fully connected layers. Optionally, the classification head module can be included in the VLM model, i.e., the medical report generation model mentioned above can also include a classification head module; or, the classification head module can be two separate models set up separately from the medical report generation model mentioned above. For example, abnormal category labels can include, but are not limited to, lung shadows, atelectasis, etc. Text prompts are used to prompt the text decoder to generate a medical report based on the input abnormal category labels. For example, text prompts can include, but are not limited to: prompts related to the target object, such as age, gender, etc.; prompts about the style of the generated report, for example, some hospitals' report style is more detailed; some hospitals' report style is more comprehensive and general, etc.

[0101] In this embodiment, the computer device can input medical images into an image encoder for encoding and conversion to obtain image embedding vectors. For example, the image encoder can be a frozen pre-trained visual encoder. Then, the computer device can input the image embedding vectors into a classification head module (disease classification head) for disease category detection and output abnormal category labels. Subsequently, the computer device can use the abnormal category labels to construct text prompts. For example, the text prompts could be: "The abnormal category label is XX. Please output an image report based on the image and the abnormal category label." Thus, the text prompts contain the abnormal category labels, enabling the model to generate medical reports using the abnormal category labels.

[0102] In one embodiment, such as Figure 5 As shown, a method for generating a medical image report is provided, namely, "integrating the detection results into the initial medical report to form a medical image report of the target object" in S204 above, including:

[0103] S401 uses preset structured rules to arrange the test results and generate a structured report of the target disease.

[0104] In this embodiment, the computer device can pre-set structured rules. Different structured rules can be set for different hospitals to generate reports with different styles, meeting the needs of different hospitals. Of course, different structured rules can also be set for the same hospital to suit the writing styles of different doctors. Furthermore, optionally, structured rules can be simple and general, or they can be specific and detailed, or they can be described by lesion type, or they can be custom-defined. For example, as shown... Figure 6 As shown, Figure 6 This is a schematic diagram of structured rules in an exemplary embodiment, wherein the structured rules can be set in any of the following ways: enumeration, classification, maximum, placeholder, etc.

[0105] Therefore, computer equipment can extract explanatory text for each target disease from the test results, and use preset structured rules to arrange and organize the test results to generate a structured report for the target disease.

[0106] S402 integrates structured reports and initial medical reports to form a medical imaging report for the target subject.

[0107] In this embodiment, optionally, the computer device can add a structured report to any location in the initial medical report to generate a medical image report for the target object. Alternatively, the computer device can identify the text region corresponding to the target disease from the initial medical report and replace the text region corresponding to the target disease in the initial medical report with a structured report to generate a medical image report for the target object. Of course, this embodiment does not limit the specific implementation method of fusing the structured report and the initial medical report.

[0108] In this embodiment, the detection results can be arranged using preset structured rules, which can automatically populate the structured reports for each target disease, thus automatically and accurately generating structured reports for the target diseases. Therefore, the structured reports and initial medical reports can be merged to automatically generate medical imaging reports. This process eliminates the need for manual writing of medical imaging reports, thereby improving the efficiency of medical imaging report generation.

[0109] In one embodiment, such as Figure 7 As shown, a method for generating a medical imaging report is provided, namely, "merging the structured report and the initial medical report to form a medical imaging report for the target object" in S402 above, including:

[0110] S501, determine whether the initial medical report contains the first content corresponding to the target disease. If the initial medical report contains the first content corresponding to the target disease, proceed to S502; or, if the initial medical report does not contain the first content corresponding to the target disease, proceed to S503.

[0111] S502 replaces the first content with the second content in the structured report to form a medical imaging report for the target object.

[0112] S503, add the second content to the initial medical report to form a medical imaging report for the target object.

[0113] The first content refers to the textual information related to the target disease in the initial medical report. The second content refers to the textual information related to the target disease in the structured report. Optionally, the first and second content can be plain text or rich text, where rich text includes text and information such as links / tables / colors / bold text.

[0114] In this embodiment, the computer device can identify the first content corresponding to the target disease in the initial medical report, thereby determining whether the initial medical report contains the first content corresponding to the target disease. If the initial medical report does not contain the first content corresponding to the target disease, it means that there is no text information related to the target disease in the initial medical report. In this case, the computer device can add the second content to any position in the initial medical report to form a medical image report of the target object.

[0115] Alternatively, if the initial medical report contains first content corresponding to the target disease, indicating the presence of textual information related to the target disease, the computer device can replace the first content in the initial medical report with the second content from the structured report to generate a medical imaging report for the target object. Optionally, if the initial medical report includes one first content corresponding to the same target disease, the computer device can directly replace that first content in the initial medical report with the second content to generate a medical imaging report for the target object. Alternatively, if the initial medical report includes multiple first contents corresponding to the same target disease, the computer device can replace any one of the multiple first contents with the second content from the structured report and delete the other contents to generate a medical imaging report for the target object.

[0116] In this embodiment, the second content in the structured report can replace the first content, or the second content can be added to the initial medical report while retaining other descriptive text in the initial medical report, so that the fused medical image report can be generated automatically and accurately.

[0117] In one embodiment, such as Figure 8 As shown, a method for generating intermediate medical reports is provided, namely, the above method further includes:

[0118] S601: Input the initial medical report into the disease placeholder model to identify the description information of the target disease and obtain the description content of the target disease. If the initial medical report includes the description content, proceed to S602; or, if the initial medical report does not include the description content, proceed to S603.

[0119] S602, use placeholders to replace the description of the target disease in the initial medical report to generate an intermediate medical report.

[0120] S603, add a placeholder to any position in the initial medical report to generate an intermediate medical report.

[0121] S402 includes:

[0122] S604 integrates structured reports and intermediate medical reports to form a medical imaging report for the target subject.

[0123] For example, a placeholder is a special character used to describe a specific disease; for instance, a placeholder...<lung_nodule> Used to describe pulmonary nodule disease<bone_fracture> This is used to describe fracture diseases. The disease-specific placeholder model is a model that, upon inputting a complete initial medical report, identifies the disease-specific content within the report and replaces it with placeholders. For example, the disease-specific placeholder model can be a large language model. During reasoning, when the large language model identifies descriptions of pulmonary nodules and fractures, it can replace the corresponding disease-specific descriptions with the placeholders corresponding to those diseases (including pulmonary nodules and fractures, etc.). An intermediate medical report refers to the initial medical report including placeholders.

[0124] In this embodiment, the computer device can input an initial medical report into a disease-specific placeholder model to identify the descriptive information of the target disease and obtain the descriptive content of the target disease. If the initial medical report does not include descriptive content, it means that there is no text information related to the target disease in the initial medical report. In this case, a placeholder can be directly inserted at any position in the initial medical report through the disease-specific placeholder model to generate an intermediate medical report.

[0125] If the initial medical report is found to contain descriptive content, the disease-specific placeholder model can use placeholders to replace the descriptive content corresponding to the target disease in the initial medical report to generate an intermediate medical report.

[0126] It should be noted that if the target disease includes only one type, the computer device can use a placeholder to replace the description of that target disease. If the target disease includes multiple types, different target diseases correspond to different placeholders. For example, if the placeholder for a lung nodule is...<lung_nodule> The placeholder for the fracture is<bone_fracture> At this point, the computer equipment can use placeholders corresponding to each target disease to replace the descriptions of each target disease and generate an intermediate medical report.

[0127] If the initial medical report contains multiple discontinuous text segments describing the same target disease, the computer device can use placeholders to replace any segment of the discontinuous text and delete the other segments.

[0128] In this way, placeholders can be used to replace the description of the target disease in the initial medical report. The disease-specific placeholder model can automatically change the description of the target disease into a placeholder, so that the second content can be automatically used to replace the placeholder in the future.

[0129] When training the disease-specific placeholder model, it is first necessary to obtain the disease description statements that need to be replaced, as well as a training dataset labeled with placeholders. Then, historical medical reports can be input into the disease-specific placeholder model to be trained, which will output reports that replace the disease description statements with the specified placeholders. For example, the training dataset labeled with placeholders is shown in Table 1 below:

[0130] Table 1

[0131]

[0132] In Table 1, <|lung_nodule|> represents a lung nodule, <|bone_fracture|> represents a fracture, <|breast_mass|> represents a breast mass, and <|lymph_node|> represents an enlarged lymph node.

[0133] Furthermore, when training the disease-specific placeholder model, the model parameters can be fine-tuned using the qwen3 lora mode, or all model parameters can be fine-tuned. It should be noted that in this embodiment, placeholders can be treated as special custom tokens and added to the model's tokenizer for fine-tuning. This ensures the accuracy and completeness of placeholder predictions. For example, lung nodules can maintain the same placeholder style as qwen, such as <|lung_nodule|>, while general qwen styles include <|im_start|>.

[0134] Therefore, computer devices can merge structured reports and intermediate medical reports to form a medical imaging report for the target object. For example, the computer device can replace placeholders in the intermediate medical report with the second content from the structured report to generate the medical imaging report for the target object. Thus, by replacing placeholders in the intermediate medical report with the second content from the structured report, the merged medical imaging report can be generated automatically and accurately.

[0135] In one embodiment, a method for implementing linked image display is provided, wherein the above method further includes:

[0136] In response to a trigger operation on any field of the second content in the structured report, the corresponding regional image is located from the medical images and displayed on the display interface; any field is used to describe the target disease.

[0137] In this embodiment, the computer device can automatically display a structured report on its display interface. Thus, in response to a user's triggering operation on any field of the second content in the structured report, the computer device can locate the corresponding region image from the medical image and display the region image on the display interface. The triggering operation can include, but is not limited to, clicking, pressing a button, or checking a box. Any field describes the target disease; optionally, any field can be a lesion identifier of the target disease, such as IM37(lung_uuid); or, any field can be the specific lesion coordinates of the target disease, such as IM37(x,y,z). In this way, the structured report has the function of linking and interacting with the lesion region image.

[0138] For example, if a structured report includes: multiple solid nodules are seen in the left upper lobe (IM37), right upper lobe (IM55), and right lower lobe (IM107), with a long diameter of approximately 5mm, the largest measuring approximately 5mm × 5mm, located in the lateral basal segment of the right lower lobe. Then, when the user clicks on "IM37" in "left upper lobe (IM37)," the computer device can automatically jump to the image of the region corresponding to the first nodule in the medical imaging.

[0139] In one embodiment, an implementation method for updating the detection result is provided, namely, the above method further includes:

[0140] In response to update operations on test results, update the test results, structured reports, and medical imaging reports according to the update operations.

[0141] In this embodiment of the application, optionally, the computer device can automatically display the detection results on the computer device's display interface; or, the computer device can automatically display the detection results and the structured report on the same display interface of the computer device. For example, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of the display interface in one embodiment, where the content in the red box represents the detection result, the content in the yellow box represents the structured rules, and the content in the green box represents the structured report. Furthermore, in response to a user's click on the content in the yellow box, the computer device can pop up... Figure 6 The pop-up interface.

[0142] In this embodiment, when a user manually updates (e.g., deletes, modifies, or adds lesions) the lesion list (i.e., the detection results) output by the disease-specific model, the computer device can respond to the update operation on the detection results and update the detection results and structured report in real time according to the update operation. Furthermore, the real-time updated detection results can be integrated into the initial medical report to form a real-time updated medical imaging report. Thus, by modifying the detection results of the disease-specific model, the structured report can be automatically updated, ensuring that the user is always presented with the latest medical imaging report.

[0143] In summary, as described in the above embodiments, Figure 10 As shown, a method for generating medical image reports is provided, applied to a computer device, including:

[0144] S701, acquire medical images of the target object;

[0145] S702, the medical image input image encoder is transformed to obtain the image embedding vector;

[0146] S703: The image embedding vector is input into the disease classifier to detect the disease category and obtain the abnormal category label;

[0147] S704, Construct text prompts based on anomaly category labels;

[0148] S705, the image embedding vector is input into the mapper to generate the processed image embedding vector;

[0149] S706, Input the text prompt words into the text encoder for conversion to obtain the text embedding vector;

[0150] S707, the processed image embedding vector and text embedding vector are input into the text decoder for processing to obtain the initial medical report;

[0151] For example, the initial medical report is as follows:

[0152] Small nodules, about 3-7 mm in diameter, were observed in both lungs, and a small calcification was observed in the lower right lung.

[0153] All bronchi were found to be patent.

[0154] No enlarged lymph nodes were observed in the hilum and mediastinum;

[0155] There was no pleural thickening or pleural effusion; there was slightly more soft tissue shadow in the anterior mediastinum, suggesting incomplete thymic involution.

[0156] Fracture of the fourth anterior rib on the left side.

[0157] S708, Input the initial medical report into the disease placeholder model to identify the description information of the target disease and obtain the description content of the target disease;

[0158] If the initial medical report includes descriptive content, then execute S709; if the initial medical report does not include descriptive content, then add a placeholder to any position in the initial medical report to generate an intermediate medical report.

[0159] For example, the recognition results are as follows:

[0160] Small nodules, approximately 3-7 mm in diameter, were observed in both lungs, with a small calcification in the right lower lung; (pulmonary nodules)

[0161] All bronchi were found to be patent.

[0162] No enlarged lymph nodes were observed in the hilum and mediastinum;

[0163] There was no pleural thickening or pleural effusion; there was slightly more soft tissue shadow in the anterior mediastinum, suggesting incomplete thymic involution.

[0164] Fracture of the fourth anterior rib on the left side. (Fracture)

[0165] S709, Use placeholders to replace the description of the target disease in the initial medical report to generate an intermediate medical report;

[0166] For example, the intermediate medical report is as follows:

[0167] <lung_nodule>

[0168] All bronchi were found to be patent.

[0169] No enlarged lymph nodes were observed in the hilum and mediastinum;

[0170] There was no thickening of the pleura, and no effusion in the pleural cavity;

[0171] There is a slightly larger amount of soft tissue shadow in the anterior mediastinum, suggesting incomplete thymic involution;

[0172] <bone_fracture> .

[0173] S710: Input medical images into the target disease classification model to detect the target disease and obtain the detection results;

[0174] For example, the target disease classification model includes a lung nodule detection model and a fracture detection model. The detection results of the lung nodule detection model from the image are shown in Table 2 below:

[0175] Table 2

[0176]

[0177] The detection results of lung nodules detected by the fracture detection model from the images are shown in Table 3 below:

[0178] Table 3

[0179]

[0180] S711 uses preset structured rules to arrange the test results and generate a structured report of the target disease;

[0181] For example, structured reports include reports of lung nodules and fractures, wherein, optionally, a report of lung nodules can be described as follows:

[0182] Multiple solid, ground-glass opacities were observed in both lungs (IM205, IM300, IM311, etc.), with a long diameter ranging from approximately 3 to 7 mm. The largest opacity was located in the anterior segment of the left upper lobe, measuring approximately 5 × 7 mm. Calcifications were also observed in the right upper lobe.

[0183] Alternatively, if a structured report is generated based on location-priority structured rules, the description of lung nodules can also be described as follows:

[0184] Multiple nodules were observed in the left upper lobe (IM300), left middle lobe (IM205), right upper lobe (IM311, IM318), and right middle lobe (IM199), with a long diameter ranging from 3 to 7 mm. The largest nodule was located in the anterior segment of the left upper lobe, measuring approximately 5 × 7 mm.

[0185] A fracture report can be described as follows:

[0186] Fractures of the 4th anterior rib on the left and the 5th anterior rib on the right, and a compression fracture of the T11 thoracic vertebra.

[0187] S712, in response to a trigger operation on any field of the second content in the structured report, locates the region image corresponding to any field in the medical image and displays the region image on the display interface; any field is used to describe the target disease;

[0188] S713, replace the placeholders in the intermediate medical report with the second content in the structured report to form a medical imaging report for the target object;

[0189] For example, the target patient's medical imaging report is as follows:

[0190] Multiple solid, ground-glass opacities were observed in both lungs (IM205, IM300, IM311, etc.), with a long diameter ranging from approximately 3 to 7 mm. The largest opacity was located in the anterior segment of the left upper lobe, measuring approximately 5 × 7 mm. Calcifications were also observed in the right upper lobe.

[0191] All bronchi were found to be patent.

[0192] No enlarged lymph nodes were observed in the hilum and mediastinum;

[0193] There was no pleural thickening or pleural effusion; there was slightly more soft tissue shadow in the anterior mediastinum, suggesting incomplete thymic involution.

[0194] Fractures of the 4th anterior rib on the left and the 5th anterior rib on the right, and a compression fracture of the T11 thoracic vertebra.

[0195] S714, in response to an update operation on test results, updates the test results, structured reports, and medical imaging reports according to the update operation.

[0196] In the aforementioned method for generating medical imaging reports, a target disease classification model can be used to specifically detect the target disease, accurately obtaining the detection results. These accurate results can be integrated into the initial medical report, automatically generating the medical imaging report. This process eliminates the need for manual report writing by doctors, thus optimizing the report-writing process, reducing writing time, and improving the efficiency of medical imaging report generation. Furthermore, since the medical imaging report integrates accurate detection results and the initial medical report, it not only possesses the ability of the medical report generation model to scan and check multiple diseases, but also the ability of the target disease classification model to detect and measure lesions. Therefore, the embodiments of this application can further improve the accuracy of medical imaging reports.

[0197] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0198] Based on the same inventive concept, this application also provides a medical image report generation apparatus for implementing the above-described method for generating medical image reports. The solution provided by this apparatus is similar to the implementation described in the above-described method; therefore, the specific limitations of one or more embodiments of the medical image report generation apparatus provided below can be found in the limitations of the medical image report generation method described above, and will not be repeated here.

[0199] In one exemplary embodiment, such as Figure 11 As shown, a medical image report generation device is provided, comprising: an acquisition module 31, a processing module 32, a detection module 33, and a fusion module 34, wherein:

[0200] The acquisition module 31 is used to acquire the medical images of the target object and the corresponding text prompts for the medical images.

[0201] The processing module 32 is used to input medical images and text prompts into the medical report generation model for processing and to generate an initial medical report.

[0202] The detection module 33 is used to input medical images into the target disease classification model to detect the target disease and obtain the detection results.

[0203] The fusion module 34 is used to fuse the test results into the initial medical report to form a medical image report of the target object.

[0204] In one embodiment, the fusion module 34 includes:

[0205] The arrangement unit is used to arrange the test results using preset structured rules to generate a structured report of the target disease;

[0206] The fusion unit is used to merge the structured report and the initial medical report to form a medical image report for the target object.

[0207] In one embodiment, the fusion unit is specifically used for:

[0208] If the initial medical report contains the first content corresponding to the target disease, the second content in the structured report is used to replace the first content to form a medical imaging report for the target object.

[0209] Alternatively, if the initial medical report does not contain the first content corresponding to the target disease, the second content is added to the initial medical report to form a medical imaging report for the target object.

[0210] In one embodiment, the above-mentioned medical image report generation apparatus further includes:

[0211] The identification module is used to identify the description information of the target disease in the disease placeholder model from the initial medical report, and obtain the description content of the target disease.

[0212] The substitution module is used to replace the description of the target disease in the initial medical report with a placeholder to generate an intermediate medical report if the initial medical report includes descriptive content; or,

[0213] The module adds a placeholder to any position in the initial medical report if the initial medical report does not include descriptive content, thus generating an intermediate medical report.

[0214] The fusion unit includes:

[0215] The fusion subunit is used to merge structured reports and intermediate medical reports to form a medical image report for the target object.

[0216] In one embodiment, the fusion subunit is specifically used for:

[0217] Replace the placeholders with the second content from the structured report to generate a medical imaging report for the target object.

[0218] In one embodiment, the above-mentioned medical image report generation apparatus further includes:

[0219] The positioning module is used to respond to a trigger operation on any field of the second content in the structured report, locate the region image corresponding to any field in the medical image, and display the region image on the display interface; any field is used to describe the target disease.

[0220] In one embodiment, the above-mentioned medical image report generation apparatus further includes:

[0221] The update module is used to respond to update operations on test results, and to update the test results, structured reports, and medical imaging reports according to the update operations.

[0222] In one embodiment, the medical report generation model includes an image encoder, a text encoder, a mapper, and a text decoder, and the processing module 32 includes:

[0223] The first conversion unit is used to convert the medical image input image encoder to obtain the image embedding vector;

[0224] The alignment unit is used to input the image embedding vector into the mapper to generate the processed image embedding vector; the mapper is used to perform feature alignment processing on the image and text according to the mapping relationship between the image and text learned during training.

[0225] The second conversion unit is used to input the text prompt words into the text encoder for conversion to obtain the text embedding vector;

[0226] The processing unit is used to input the processed image embedding vector and text embedding vector into the text decoder for processing to obtain the initial medical report.

[0227] In one embodiment, the acquisition module 31 includes:

[0228] The third conversion unit is used to convert the medical image input image encoder to obtain the image embedding vector;

[0229] The label generation unit is used to input the image embedding vector into the disease classifier for disease category detection and to obtain abnormal category labels;

[0230] The building block is used to construct text prompts based on the exception category label.

[0231] The modules in the aforementioned medical image report generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0232] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 1As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for generating medical image reports. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0233] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0234] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for generating a medical image report in any of the above embodiments.

[0235] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for generating a medical image report in any of the above embodiments.

[0236] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method for generating a medical image report in any of the above embodiments.

[0237] It should be noted that the user information (including but not limited to relevant information of the target object, user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0238] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0239] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0240] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating a medical image report, characterized in that, The method includes: Obtain the medical images of the target object and the corresponding text prompts for the medical images; The medical images and text prompts are input into a medical report generation model for processing to generate an initial medical report. The medical images are input into a target disease classification model to detect the target disease, and the detection results are obtained. The test results are integrated into the initial medical report to form a medical imaging report for the target object.

2. The method according to claim 1, characterized in that, The process of integrating the detection results into the initial medical report to form a medical imaging report for the target object includes: The detection results are arranged using preset structured rules to generate a structured report of the target disease; The structured report and the initial medical report are merged to form a medical image report for the target object.

3. The method according to claim 2, characterized in that, The process of fusing the structured report and the initial medical report to form a medical imaging report for the target object includes: If the initial medical report contains the first content corresponding to the target disease, then the second content in the structured report is used to replace the first content to form a medical image report for the target object; Alternatively, if the initial medical report does not contain the first content corresponding to the target disease, the second content is added to the initial medical report to form a medical imaging report for the target object.

4. The method according to claim 2, characterized in that, The method further includes: The initial medical report is input into the disease placeholder model to identify the description information of the target disease, thereby obtaining the description content of the target disease. If the initial medical report includes the description, then placeholders are used to replace the description of the target disease in the initial medical report to generate an intermediate medical report; Alternatively, if the initial medical report does not include the description, the placeholder is added to any position in the initial medical report to generate an intermediate medical report; The process of fusing the structured report and the initial medical report to form a medical imaging report for the target object includes: The structured report and the intermediate medical report are merged to form a medical imaging report for the target object.

5. The method according to claim 4, characterized in that, The process of fusing the structured report and the intermediate medical report to form the medical imaging report for the target object includes: The placeholders are replaced with the second content from the structured report to form the medical image report for the target object.

6. The method according to any one of claims 2-5, characterized in that, The method further includes: In response to a trigger operation on any field of the second content in the structured report, the region image corresponding to the any field is located from the medical image and displayed on the display interface; the any field is used to describe the target disease.

7. The method according to any one of claims 2-5, characterized in that, The method further includes: In response to the update operation on the detection results, the detection results, the structured report, and the medical imaging report are updated according to the update operation.

8. The method according to any one of claims 1-5, characterized in that, The medical report generation model includes an image encoder, a text encoder, a mapper, and a text decoder. The process of inputting the medical image and the text prompts into the medical report generation model for processing to generate an initial medical report includes: The medical image is input into the image encoder for conversion to obtain an image embedding vector; The image embedding vector is input into the mapper to generate a processed image embedding vector; the mapper is used to perform feature alignment processing on the image and text based on the mapping relationship between the image and text learned during training. The text prompts are input into the text encoder for conversion to obtain text embedding vectors; The processed image embedding vector and the text embedding vector are input into the text decoder for processing to obtain the initial medical report.

9. The method according to any one of claims 1-5, characterized in that, Obtain the text prompts corresponding to the medical images, including: The medical image is input into an image encoder for conversion to obtain an image embedding vector; The image embedding vector is input into a disease classifier to detect disease categories and obtain abnormal category labels; The text prompt word is constructed based on the anomaly category label.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images