Medical image report generation method and system based on interactive guidance
By introducing an interactive guidance mechanism in the medical image report generation system, combining a large language model and attention mechanism, the problems of single report content and low accuracy in the existing technology are solved, and more accurate and interpretable medical image report generation is achieved.
Patent Information
- Application Number
- CN202411787985.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-03-28
AI Technical Summary
The existing automatic generation methods of medical imaging reports lack effective interaction mechanisms with doctors, resulting in a single report content, a lack of targeted analysis, and low accuracy.
It provides an interactive guidance-based medical image report generation method and system. By displaying the target user interface on the screen of an electronic device, acquiring and displaying medical images, detecting the doctor's marking operations on the interface, segmenting the lesion area, extracting subimage features, generating report content using a large language model, and generating the final image report through the fusion of attention mechanism and multi-grained model.
It improves the accuracy and interpretability of medical imaging reports, can analyze the imaging areas more deeply, is suitable for large-scale clinical applications, and assists radiologists in completing report writing, saving manpower.
Smart Images
Figure CN119724465A_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate to the field of auxiliary medical technology, and in particular to a method and system for generating a medical imaging report based on interactive guidance. Background Art
[0002] In the modern medical system, medical imaging reports play a vital bridging role. Medical images include X-rays, CT scans, MRI scans and other images, which can intuitively present the internal structure of the human body. Medical imaging reports are the results of in-depth analysis of these images by radiologists based on their professional knowledge of radiology. The descriptions and diagnostic content in the imaging reports are key auxiliary bases for clinicians to assess the health status of patients. For example, in the diagnosis of tumor diseases, medical imaging reports can accurately describe the location, size, morphology and other characteristics of the tumor, helping clinicians to formulate reasonable treatment plans.
[0003] Although the automatic generation method of medical imaging reports in related technologies can perform simple image recognition and annotation, it lacks an effective interaction mechanism with doctors. Doctors can often only passively accept the given results, and cannot flexibly guide more in-depth analysis based on their own experience and professional judgment. This leads to the fact that the content of the reports generated by existing methods is usually relatively simple, lacking targeted analysis of the image area, and the accuracy of the imaging reports is low. For example, most of them are currently only a general description of the overall situation of the image, and the detailed characteristics of specific areas, the development trend of lesions and other information are not reflected enough. At the same time, for non-professional image interpreters such as clinicians and patients, the connotation of the imaging report is difficult to directly and accurately obtain, and the interpretability is poor, which is not conducive to large-scale clinical application. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method and system for generating a medical imaging report based on interactive guidance.
[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a medical imaging report based on interactive guidance, which is applied to an electronic device, comprising:
[0006] In response to a preset operation, a target user interface is displayed on the screen of the electronic device, and a medical image to be processed is acquired and displayed on the target user interface;
[0007] Detecting a doctor's marking operation on the lesion area of the medical image on the target user interface, and segmenting the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes an image area corresponding to the marking operation and a sub-image of an area outside the image area;
[0008] Extracting image features of each of the sub-images, generating report content corresponding to each of the sub-images based on the image features of each of the sub-images and using a large language model; and generating a preliminary image report based on splicing of the report content corresponding to each of the sub-images;
[0009] The preliminary image report is input into a first model with an attention mechanism to obtain interpretability-related content of the preliminary image report; the preliminary image report is input into a second model to obtain a multi-granularity image report; and the interpretability-related content of the preliminary image report and the multi-granularity image report are integrated to obtain a target image report.
[0010] In one embodiment, the interpretability-related content includes at least the relationship between each sentence in the preliminary image report and the corresponding sub-image.
[0011] In one embodiment, the method further comprises:
[0012] receiving the doctor's modification and / or supplementary content for the target imaging report on the target user interface;
[0013] Based on the modified and / or supplemented content, the target image report is modified and / or supplemented to obtain a final image report.
[0014] In one embodiment, the method further comprises:
[0015] Inputting the image features of each sub-image into a pre-trained classifier to obtain a classification result corresponding to each sub-image;
[0016] Based on the classification result corresponding to each of the sub-images, selecting and determining at least one target sub-image; the at least one target sub-image is a partial sub-image in each of the sub-images;
[0017] Based on the image features of each of the target sub-images and using the large language model, report content corresponding to each of the target sub-images is generated; and based on the splicing of the report content corresponding to each of the target sub-images, the preliminary image report is generated.
[0018] In one embodiment, the second model is used to capture image features of different granularities from overall to local for the image features of each sub-image in the preliminary image report and the corresponding report content, and to process image features of different granularities and corresponding report contents to select image features of appropriate target granularity and corresponding report contents for combined expression to obtain a multi-granularity image report.
[0019] In one embodiment, the image features of each of the sub-images include at least shape, size, texture, and pixel density.
[0020] In one embodiment, the medical images include at least X-ray images of tumor patients.
[0021] In a second aspect, the present disclosure provides a medical imaging report generation system based on interactive guidance, comprising:
[0022] An image acquisition module, used to display a target user interface on the screen of the electronic device in response to a preset operation, acquire a medical image to be processed and display it on the target user interface;
[0023] a marking segmentation module, configured to detect a marking operation performed by a doctor on the lesion area of the medical image on the target user interface, and segment the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes an image area corresponding to the marking operation and a sub-image of an area outside the image area;
[0024] A preliminary generation module, used to extract the image features of each sub-image, generate the report content corresponding to each sub-image based on the image features of each sub-image and using a large language model; and generate a preliminary image report based on the splicing of the report content corresponding to each sub-image;
[0025] A fusion module is generated, which is used to input the preliminary image report into a first model with an attention mechanism to obtain interpretability-related content of the preliminary image report; input the preliminary image report into a second model to obtain a multi-granularity image report; and fuse the interpretability-related content of the preliminary image report and the multi-granularity image report to obtain a target image report.
[0026] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating a medical imaging report based on interactive guidance as described in any of the above embodiments.
[0027] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including:
[0028] Processor; and
[0029] Memory for storing computer programs;
[0030] The processor is configured to execute the medical imaging report generation method based on interactive guidance described in any of the above embodiments by executing the computer program.
[0031] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0032] The medical image report generation method and system based on interactive guidance provided by the embodiments of the present disclosure display a target user interface on the screen of the electronic device in response to a preset operation, obtain the medical image to be processed and display it on the target user interface; detect the doctor's marking operation on the lesion area of the medical image on the target user interface, and segment the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes the image area corresponding to the marking operation and the sub-image of the area outside the image area; extract the image features of each of the sub-images, and generate the report content corresponding to each of the sub-images based on the image features of each of the sub-images and using a large language model; generate a preliminary image report based on the report content corresponding to each of the sub-images; input the preliminary image report into a first model with an attention mechanism to obtain the interpretability-related content of the preliminary image report; input the preliminary image report into a second model to obtain a multi-granularity image report; and fuse the interpretability-related content of the preliminary image report and the multi-granularity image report to obtain a target image report. In this embodiment, based on the marking operation on the user interface, targeted key areas of the medical image lesion area can be annotated, and multiple sub-images (including sub-images of key areas) based on the segmentation of the lesion area can guide a more in-depth and accurate analysis. The image features of each sub-image extracted by the large language model are combined to accurately generate the report content corresponding to each sub-image to splice and form a preliminary image report, and then the interpretable content of the preliminary image report is determined and a multi-granularity image report is generated. Finally, the target image report is fused, which contains accurate interpretable content and in-depth and accurate analysis content, thereby improving the accuracy of automatically generated medical image reports; at the same time, for non-professional image interpreters such as clinicians and patients, their image reports contain interpretable content, which is conducive to intuitive and accurate understanding and is conducive to large-scale clinical applications. The automatic generation scheme of medical image reports in this embodiment can assist radiologists in completing the writing of reports, saving manpower to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0035] Figure 1 This is a flow chart of a method for generating a medical imaging report based on interactive guidance according to an embodiment of the present disclosure;
[0036] Figure 2 This is a flow chart of a method for generating a medical imaging report based on interactive guidance according to another embodiment of the present disclosure;
[0037] Figure 3 This is a flow chart of a method for generating a medical imaging report based on interactive guidance according to another embodiment of the present disclosure;
[0038] Figure 4 A schematic diagram of a medical imaging report generation system based on interactive guidance according to an embodiment of the present disclosure;
[0039] Figure 5 Schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0040] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0041] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0042] It should be understood that, in the following, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0043] Figure 1 The following is a flow chart of a method for generating a medical imaging report based on interactive guidance according to an embodiment of the present disclosure. The method can be applied to electronic devices such as computers or mobile terminals, and can specifically include the following steps:
[0044] Step S101: In response to a preset operation, a target user interface is displayed on the screen of the electronic device, and a medical image to be processed is acquired and displayed on the target user interface.
[0045] Exemplarily, in one embodiment, the medical image may at least include but is not limited to an X-ray image of a tumor patient. Figure 2 As shown, the method can be implemented by a software system such as an application. Doctors can log in to the main interface of the system to achieve unified authentication of the doctor's identity in the system, such as entering a user name and password, clicking the "authentication" control, which is the preset operation, and entering the target user interface after automatic authentication. This user interface can create a patient file, enter the patient's basic information, and provide the system with basic data to assist in judgment; create a medical image generation report record, and add a new image report generation record for a patient to the file, upload the patient's chest CT image data, and upload all the DCM image files in the selected chest CT scan folder to this system and display the medical image on the target user interface.
[0046] Step S102: Detecting the doctor's marking operation on the lesion area of the medical image on the target user interface, and segmenting the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes the image area corresponding to the marking operation and a sub-image of an area outside the image area.
[0047] Exemplarily, the target user interface serves as an interactive interface, and the doctor can mark the lesion area of the displayed medical image on the target user interface, such as by marking with a mouse, or by touching with a stylus, and the corresponding electronic device has a touch screen. In this embodiment, the medical image can be automatically segmented to obtain at least one sub-image related to the lesion area, that is, the lesion area of the medical image is divided into regions, and the image is divided into multiple precise regional images, i.e., sub-images, using image segmentation technology. For the sub-image of each region, its image features can be extracted later. The segmented sub-image contains the area marked by the doctor, such as the key area, i.e., the area that has a great impact on the diagnosis result of the lesion. The sub-image of the area marked by the doctor can be given priority for report generation and analysis.
[0048] In one example, the lesion region is to locate several anatomical regions in the chest cavity, and the image region corresponding to the marking operation may be a part of the anatomical region, such as an important feature region that may have abnormalities.
[0049] Step S103: extracting image features of each of the sub-images, generating report content corresponding to each of the sub-images based on the image features of each of the sub-images and using a large language model; and generating a preliminary image report based on the splicing of the report content corresponding to each of the sub-images.
[0050] Exemplarily, in one embodiment, the image features of each sub-image may at least include, but are not limited to, image features such as shape, size, texture, pixel density, etc. The selected features, i.e., the image features of each sub-image, are input into a large language model such as a medical large language model. Since doctors mark the regions of interest on medical images, descriptive sentences for each region (including the marked regions of interest), i.e., sub-images, can be generated as report content and spliced into a preliminary image report, thereby achieving accurate preliminary generation of an image-text correspondence report.
[0051] Step S104: input the preliminary image report into a first model with an attention mechanism to obtain interpretability-related content of the preliminary image report; input the preliminary image report into a second model to obtain a multi-granularity image report; and fuse the interpretability-related content of the preliminary image report and the multi-granularity image report to obtain a target image report.
[0052] Exemplarily, the first model with the attention mechanism can be obtained by pre-training a Transformer model based on sample data, and the sample data can include a sample image report and corresponding interpretability-related content. In one embodiment, the interpretability-related content can at least include but is not limited to the relationship between each sentence in the preliminary image report and the corresponding sub-image.
[0053] The preliminary imaging report is processed by the first model with an attention mechanism, which can show which areas of the image, i.e., sub-images, each sentence in the report is based on. This detailed information can be used as relevant content for the interpretability of the report, which helps non-professional imaging interpreters such as clinicians and patients to understand the report.
[0054] The second model can be a natural language generation model, which can automatically generate a high-quality natural language text such as the content of a medical imaging report through an automatic planning process based on the information provided by the preliminary imaging report. In this embodiment, the second model can output a multi-granularity imaging report, that is, the generated preliminary report is based on natural language generation technology, and according to different needs, the image features of appropriate granularity and the corresponding report content can be selected for combined expression to generate a multi-granularity imaging report containing more accurate content, and then the interpretability-related content of the preliminary imaging report is integrated to obtain the target imaging report.
[0055] In this embodiment, the doctor user creates a patient file and uploads chest CT scan image data. The system automatically identifies and processes the image file, then segments the image lesion area into multiple areas, namely sub-images, extracts the features of each sub-image and inputs them into the large language model. The system generates a preliminary report with descriptive sentences for each area. The preliminary report is then processed by the attention mechanism model and natural language generation technology, and finally fused to generate a comprehensive and hierarchical multi-granularity report, which contains accurate and detailed interpretable content.
[0056] In this embodiment, based on the marking operation on the user interface, targeted key areas of the medical image lesion area can be annotated, and multiple sub-images (including sub-images of key areas) based on the segmentation of the lesion area can guide a more in-depth and accurate analysis. The image features of the extracted sub-images are combined with the large language model to accurately generate the report content corresponding to each sub-image to splice into an accurate preliminary image report, and then the interpretable content of the preliminary image report is determined and a multi-granularity image report is generated, and finally a more accurate target image report is obtained by fusion, which contains accurate and detailed interpretable content and in-depth and accurate analysis content, thereby improving the accuracy of automatically generated medical image reports; at the same time, for non-professional image interpreters such as clinicians and patients, their image reports contain accurate and detailed interpretable content, which is conducive to intuitive and accurate understanding and is conducive to large-scale clinical applications.
[0057] In one embodiment, the second model is used to capture image features of different granularities from overall to local for the image features of each sub-image in the preliminary image report and the corresponding report content, and to process image features of different granularities and corresponding report contents to select image features of appropriate target granularity and corresponding report contents for combined expression to obtain a multi-granularity image report.
[0058] Exemplarily, the image features of each sub-image may be image features of different granularities, and image features of different granularities may be expressed as different image feature parameters such as shape, size, texture, pixel density, etc. By selecting image features of appropriate target granularity from image features of different granularities from the whole to the local and combining them with corresponding report content to obtain a multi-granularity image report, the report content may have multi-level feature fusion and semantic understanding, and the generated medical image report may be more accurate.
[0059] Based on the above embodiments, in one embodiment, the method may further include the following steps: receiving the doctor's modified and / or supplemented content for the target image report on the target user interface; based on the modified and / or supplemented content, modifying and / or supplementing the target image report to obtain a final image report.
[0060] Exemplarily, the modification and / or supplementary content may be a professional text description provided by a doctor. In this embodiment, the doctor may modify and supplement the generated report through interactive information to generate a more accurate final imaging report, thereby further improving the accuracy of automatically generated medical imaging reports.
[0061] Based on any of the above embodiments, in one embodiment, reference Figure 3 As shown in , the method may also include the following steps:
[0062] Step S301: input the image features of each sub-image into a pre-trained classifier to obtain a classification result corresponding to each sub-image.
[0063] Exemplarily, the classifier can be a binary classifier. For example, a high-precision object detector is used to locate several unique anatomical regions in the chest cavity of a chest CT image, and two binary classifiers are used to select important and possibly abnormal feature regions based on the classification results of each sub-image. For example, the classification result corresponding to each sub-image can be a lesion feature category corresponding to the image feature of the sub-image. A disease may have multiple categories of lesion features at the same time, but some of them are key lesion feature categories that can distinguish the type of disease. The key lesion feature categories can be accurately determined through the classification results of the classifier. Two binary classifiers can ultimately determine, for example, the sub-images corresponding to two key lesion feature categories.
[0064] Step S302: Based on the classification results corresponding to each of the sub-images, at least one target sub-image is selected and determined; the at least one target sub-image is a sub-image in each of the sub-images. Exemplarily, the at least one target sub-image is a sub-image corresponding to a key lesion feature category.
[0065] Step S303: Generate report content corresponding to each target sub-image based on the image features of each target sub-image and using the large language model; and generate the preliminary image report based on the splicing of the report content corresponding to each target sub-image.
[0066] Exemplarily, the image features of the sub-images corresponding to the key lesion feature categories, i.e., the target sub-images, are used as input to the large language model to generate the report content corresponding to each target sub-image; based on the splicing of the report content corresponding to each target sub-image, an accurate preliminary image report can be generated. Then, step S104 can be executed. In this way, through the classification results of the classifier, it is possible to identify the sub-images in the medical image whose importance related to the lesion area is greater than a preset value, i.e., the sub-images corresponding to the key lesion feature categories, and thus a preliminary image report and a final image report can be generated efficiently and accurately, because it does not need to use all the sub-images, but only uses a part of the important sub-images to generate the preliminary image report, which can improve the accuracy of the generated report while improving the efficiency of the report generation process.
[0067] In some embodiments, the above-mentioned classifier is obtained by training the original classifier based on multiple image features of the sample medical image. The original classifier may include a first module and a second module, and the first module and the second module may be convolutional neural networks. The training process of the original classifier may include: inputting the image feature into the first module to obtain a first classification result; inputting the first classification result and the image feature into the second module together to obtain a second classification result, and iteratively training based on the difference between the lesion feature type label of the corresponding image feature in the sample medical image and the second classification result, until the loss value of the loss function of the original classifier is less than a preset value to terminate the training.
[0068] The model architecture of the above-mentioned original classifier is combined with the training process, so that the data input to the second module includes not only the image features of the sample medical image, but also the first classification result output by the first module, so that the second classification result output by the second module is enhanced by the first classification result output by the first module, so that the second classification result, that is, the final classification result, is more accurate, so that the trained classifier can more accurately identify the corresponding lesion feature type for each sub-image, that is, more accurately identify the lesion feature type result, and then execute steps S302 to S303. Combined with the above, it can be seen that the accuracy of automatically generated image reports can be ultimately improved.
[0069] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc. In addition, it is also easy to understand that these steps may be, for example, executed synchronously or asynchronously in multiple modules / processes / threads.
[0070] like Figure 4As shown, the embodiment of the present disclosure provides a medical imaging report generation system based on interactive guidance, comprising:
[0071] The image acquisition module 401 is used to display a target user interface on the screen of the electronic device in response to a preset operation, acquire the medical image to be processed and display it on the target user interface;
[0072] The marking and segmentation module 402 is used to detect the marking operation of the doctor on the lesion area of the medical image on the target user interface, and segment the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes the image area corresponding to the marking operation and the sub-image of the area outside the image area;
[0073] The preliminary generation module 403 is used to extract the image features of each sub-image, generate the report content corresponding to each sub-image based on the image features of each sub-image and using the large language model; and generate a preliminary image report based on the splicing of the report content corresponding to each sub-image;
[0074] A fusion module 404 is generated, which is used to input the preliminary image report into a first model with an attention mechanism to obtain interpretability-related content of the preliminary image report; input the preliminary image report into a second model to obtain a multi-granularity image report; and fuse the interpretability-related content of the preliminary image report and the multi-granularity image report to obtain a target image report.
[0075] In one embodiment, the interpretability-related content includes at least the relationship between each sentence in the preliminary image report and the corresponding sub-image.
[0076] In one embodiment, the system also includes a report adjustment module, which is used to: receive the doctor's modifications and / or supplementary content for the target image report on the target user interface; modify and / or supplement the target image report based on the modifications and / or supplementary content to obtain a final image report.
[0077] In one embodiment, the system may further include:
[0078] A classification module, used for inputting the image features of each sub-image into a pre-trained classifier to obtain a classification result corresponding to each sub-image;
[0079] A selection module, configured to select and determine at least one target sub-image based on the classification result corresponding to each of the sub-images; the at least one target sub-image is a partial sub-image in each of the sub-images;
[0080] The preliminary generation module 403 can also be used to generate report content corresponding to each target sub-image based on the image features of each target sub-image and using the large language model; and to generate the preliminary image report based on the splicing of the report content corresponding to each target sub-image.
[0081] In one embodiment, the second model is used to capture image features of different granularities from overall to local for the image features of each sub-image in the preliminary image report and the corresponding report content, and to process image features of different granularities and corresponding report contents to select image features of appropriate target granularity and corresponding report contents for combined expression to obtain a multi-granularity image report.
[0082] In one embodiment, the image features of each of the sub-images include at least shape, size, texture, and pixel density.
[0083] In one embodiment, the medical images include at least X-ray images of tumor patients.
[0084] The above-mentioned classifier is obtained by training the original classifier based on multiple image features of the sample medical image. The original classifier may include a first module and a second module, and the first module and the second module may be convolutional neural networks. The training process of the original classifier may include: inputting the image feature into the first module to obtain a first classification result; inputting the first classification result and the image feature into the second module together to obtain a second classification result, and iteratively training based on the difference between the lesion feature type label corresponding to the image feature in the sample medical image and the second classification result, until the loss value of the loss function of the original classifier is less than a preset value to terminate the training.
[0085] Regarding the system in the above embodiment, the specific manner in which each module performs operations and the corresponding technical effects brought about have been described in detail in the embodiment of the method, and will not be elaborated here.
[0086] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units for concretization. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative work.
[0087] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating a medical imaging report based on interactive guidance as described in any one of the above embodiments is implemented.
[0088] Exemplarily, the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0089] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0090] The present disclosure also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, wherein the processor is configured to execute the interactive guidance-based medical imaging report generation method in any of the above embodiments by executing the computer program.
[0091] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0092] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0093] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method embodiment section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 The steps of the method shown in .
[0094] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0095] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.
[0096] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0097] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0098] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the method steps of the above-mentioned embodiments according to the implementation methods of the present disclosure.
[0099] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0100] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for generating a medical imaging report based on interactive guidance, applied to an electronic device, characterized in that: include: In response to a preset operation, a target user interface is displayed on the screen of the electronic device, and a medical image to be processed is acquired and displayed on the target user interface; Detecting a doctor's marking operation on the lesion area of the medical image on the target user interface, and segmenting the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes an image area corresponding to the marking operation and a sub-image of an area outside the image area; Extracting image features of each of the sub-images, generating report content corresponding to each of the sub-images based on the image features of each of the sub-images and using a large language model; and generating a preliminary image report based on splicing of the report content corresponding to each of the sub-images; Inputting the preliminary imaging report into a first model with an attention mechanism to obtain interpretability-related content of the preliminary imaging report; The preliminary image report is input into a second model to obtain a multi-granularity image report; and the interpretability-related content of the preliminary image report and the multi-granularity image report are integrated to obtain a target image report.
2. The method according to claim 1, characterized in that The interpretability-related content includes at least the relationship between each statement in the preliminary image report and the corresponding sub-image.
3. The method according to claim 1, characterized in that The method further includes: receiving the doctor's modification and / or supplementary content for the target imaging report on the target user interface; Based on the modified and / or supplemented content, the target image report is modified and / or supplemented to obtain a final image report.
4. The method according to any one of claims 1 to 3, characterized in that: The method further includes: Inputting the image features of each sub-image into a pre-trained classifier to obtain a classification result corresponding to each sub-image; Based on the classification result corresponding to each of the sub-images, selecting and determining at least one target sub-image; the at least one target sub-image is a partial sub-image in each of the sub-images; Based on the image features of each of the target sub-images and using the large language model, report content corresponding to each of the target sub-images is generated; and based on the splicing of the report content corresponding to each of the target sub-images, the preliminary image report is generated.
5. The method according to any one of claims 1 to 3, characterized in that: The second model is used to capture image features of different granularities from overall to local for the image features of each sub-image in the preliminary image report and the corresponding report content, and to process image features of different granularities and corresponding report contents to select image features of appropriate target granularity and corresponding report contents for combined expression to obtain a multi-granularity image report.
6. The method according to any one of claims 1 to 3, characterized in that: The image features of each of the sub-images include at least shape, size, texture, and pixel density.
7. The method according to any one of claims 1 to 3, characterized in that: The medical images at least include X-ray images of tumor patients.
8. A medical imaging report generation system based on interactive guidance, characterized in that: include: An image acquisition module, configured to display a target user interface on the screen of the electronic device in response to a preset operation, acquire a medical image to be processed and display it on the target user interface; a marking segmentation module, configured to detect a marking operation performed by a doctor on the lesion area of the medical image on the target user interface, and segment the medical image to obtain at least one sub-image related to the lesion area; the at least one sub-image includes an image area corresponding to the marking operation and a sub-image of an area outside the image area; A preliminary generation module, used to extract the image features of each sub-image, generate the report content corresponding to each sub-image based on the image features of each sub-image and using a large language model; and generate a preliminary image report based on the splicing of the report content corresponding to each sub-image; generating a fusion module, configured to input the preliminary imaging report into a first model with an attention mechanism to obtain interpretability-related content of the preliminary imaging report; The preliminary image report is input into a second model to obtain a multi-granularity image report; and the interpretability-related content of the preliminary image report and the multi-granularity image report are integrated to obtain a target image report.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a medical imaging report based on interactive guidance according to any one of claims 1 to 7 is implemented.
10. An electronic device, characterized in that: include: processor; as well as Memory for storing computer programs; Wherein, the processor is configured to execute the medical imaging report generation method based on interactive guidance according to any one of claims 1 to 7 by executing the computer program.