Medical image report automatic generation method and system

Abstract

The invention relates to a medical image report automatic generation method and system. The method comprises the following steps: extracting local image features and global image features related to a focus in a medical image; fusing the local image features and the global image features to obtain fused image features, and inputting the fused image features into an image diagnosis classifier to obtain a medical image related disease diagnosis classification result; wherein the image diagnosis classifier is obtained by pre-training an original classifier based on a multi-dimensional image fusion feature, and the multi-dimensional image fusion feature is obtained by fusing a local image feature and a global image feature which are extracted from a sample medical image and related to a focus; querying a pre-constructed medical field knowledge graph based on the disease diagnosis classification result to obtain a corresponding first medical field knowledge code; and fusing the local image features, the global image features and the first medical domain knowledge codes to obtain target text features, and inputting the target text features into a medical big language model to automatically generate an image diagnosis report.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of auxiliary medical technology, and more particularly to a method and system for automatically generating a medical imaging report. Background Art

[0002] At present, medical imaging reports are mainly for radiology medical images such as X-rays, CT scans, MRI scans and other image files. Doctors analyze the images based on their professional knowledge and issue disease descriptions and diagnosis results based on what they see in the image files to form imaging reports. However, manual processing is inefficient and wastes manpower. In order to improve efficiency, with the application of artificial intelligence deep learning technology, related technologies can use massive amounts of medical images and imaging report texts as training data sets, and use deep learning methods to learn image features and generate text to achieve automated medical imaging report generation. However, the medical imaging report generation methods in related technologies still have problems in terms of report interpretability and accuracy. Therefore, there is still a lot of room for improvement in the intelligent generation technology of medical imaging reports. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method and system for automatically generating a medical imaging report.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for automatically generating a medical imaging report, comprising:

[0005] Acquire a medical image to be processed, and extract local image features and global image features related to lesions in the medical image;

[0006] The local image features and the global image features are fused to obtain fused image features, and the fused image features are input into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image; wherein the image diagnosis classifier is obtained by pre-training the original classifier based on multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing local image features related to lesions extracted from sample medical images and global image features;

[0007] Based on the disease diagnosis classification result, a pre-constructed medical field knowledge graph is queried to obtain the corresponding first medical field knowledge code; the local image features, the global image features and the first medical field knowledge code are integrated to obtain the target text features, and the target text features are input into the medical big language model to automatically generate an imaging diagnosis report.

[0008] In one embodiment, extracting local image features and global image features related to lesions in the medical image includes:

[0009] The medical image is input into a first image encoder and a second image encoder respectively to extract local image features and global image features related to the lesion respectively; wherein the first image encoder is pre-trained based on a sample medical image to a RetinaNet model; and / or the second image encoder is pre-trained based on a sample medical image to a Vision Transformer model.

[0010] In one embodiment, the method further comprises:

[0011] Performing semantic mapping on the local image features and the global image features to obtain feature codes; inputting the feature codes into a medical imaging knowledge template model to obtain a second medical field knowledge code; and determining a target medical field knowledge code based on the second medical field knowledge code and the first medical field knowledge code;

[0012] The target text feature is obtained by fusing the local image feature, the global image feature and the target medical field knowledge encoding.

[0013] In one embodiment, the semantic mapping of the local image features and the global image features to obtain feature coding includes:

[0014] Image multi-scale features are generated based on the local image features and the global image features, and semantic mapping is performed based on the image multi-scale features to obtain feature encoding.

[0015] In one embodiment, the original classifier comprises a multi-label classifier.

[0016] In one embodiment, the method further comprises:

[0017] Obtaining descriptive questions of the medical image as prompt content;

[0018] The step of inputting the target text features into a medical large language model to automatically generate an imaging diagnosis report includes:

[0019] The target text features and the prompt content are input into the medical big language model to automatically generate an imaging diagnosis report.

[0020] In one embodiment, the medical images include at least X-ray images.

[0021] In a second aspect, an embodiment of the present disclosure provides a system for automatically generating a medical imaging report, comprising:

[0022] A feature acquisition module, used to acquire the medical image to be processed, and extract local image features and global image features related to the lesion in the medical image;

[0023] An image diagnosis module, used for fusing the local image features and the global image features to obtain fused image features, and inputting the fused image features into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image; wherein the image diagnosis classifier is obtained by pre-training the original classifier based on the multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing the local image features related to the lesions extracted from the sample medical image and the global image features;

[0024] A report generation module is used to query a pre-constructed medical field knowledge graph to obtain the corresponding first medical field knowledge code based on the disease diagnosis classification result; fuse the local image features, the global image features and the first medical field knowledge code to obtain the target text features, and input the target text features into the medical large language model to automatically generate an imaging diagnosis report.

[0025] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for automatically generating a medical imaging report according to any of the above embodiments.

[0026] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0027] Processor; and

[0028] Memory for storing computer programs;

[0029] The processor is configured to execute the method for automatically generating a medical imaging report of any of the above-mentioned embodiments by executing the computer program.

[0030] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0031] The medical image report automatic generation method and system provided by the embodiments of the present disclosure obtain a medical image to be processed, extract local image features and global image features related to lesions in the medical image; fuse the local image features and global image features to obtain fused image features, and input the fused image features into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image; wherein the image diagnosis classifier is obtained by pre-training the original classifier based on multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing local image features and global image features related to lesions extracted from sample medical images; based on the disease diagnosis classification result, a pre-constructed medical field knowledge graph is queried to obtain a corresponding first medical field knowledge code; the local image features, the global image features and the first medical field knowledge code are fused to obtain a target text feature, and the target text feature is input into a medical large language model to automatically generate an image diagnosis report. In the scheme of this embodiment, when a medical imaging report is automatically generated, the image diagnosis classifier trained based on multi-dimensional image fusion features is combined with the fused image features, i.e., the fusion features of local image features and global image features from medical images, to accurately obtain the lesion diagnosis classification results. Based on this, the medical field knowledge encoding related to the lesion can be accurately obtained from the pre-constructed medical field knowledge graph. Finally, the local image features, the global image features and the medical field knowledge encoding, i.e., the lesion-related professional knowledge encoding, are fused to obtain the target text features. The image diagnosis report is automatically generated in combination with the medical large language model. In this way, a more accurate medical imaging report can be obtained for the patient, and the report content is interpretable, i.e., the professional report content that can be easily understood by doctors other than radiologists or patients is conducive to clinical application. This scheme can realize automatic feature extraction and processing of medical images to automatically generate imaging reports, assist radiologists in completing the writing of reports, greatly reduce the workload of radiologists, improve work efficiency, and save human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is a flow chart of the method for automatically generating a medical imaging report according to an embodiment of the present disclosure;

[0035] Figure 2 This is a flow chart of a method for automatically generating a medical imaging report according to another embodiment of the present disclosure;

[0036] Figure 3 This is a flow chart of a method for automatically generating a medical imaging report according to another embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram of a system for automatically generating medical imaging reports according to an embodiment of the present disclosure;

[0038] Figure 5 Schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0041] It should be understood that, in the following, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0042] Figure 1 The following is a flow chart of a method for automatically generating a medical imaging report according to an embodiment of the present disclosure. The method can be executed by a computing device such as a computer or a mobile terminal. The method for automatically generating a medical imaging report can include the following steps:

[0043] Step S101: obtaining a medical image to be processed, and extracting local image features and global image features related to lesions in the medical image.

[0044] In one embodiment, the medical images may at least include but are not limited to X-ray images. It is understandable that in some embodiments, users such as doctors can log in to the main interface of the hospital system, and the users complete, for example, single sign-on (SSO) to achieve unified identity authentication of doctor users in the system of this platform. Afterwards, the user can create a new patient record, enter the patient's basic information such as name, ID number, etc., and provide auxiliary data for disease diagnosis. Then, for example, an X-ray medical image report generation record can be created, and a new image report generation record for a certain patient can be added to the list, and the chest X-ray image data of the uploaded patient can be detected. When the report is automatically generated, the previously uploaded X-ray image can be obtained, and then the specific image feature extraction operation in step S101 is automatically executed. Among them, the local image features and global image features extracted related to the lesion represent the image detail features of the lesion area and the overall global features, respectively.

[0045] Exemplarily, since X-ray tomography is a three-dimensional radiological medical image reconstructed by digital geometric processing, in some embodiments, three-dimensional images of the lesion area at different angles can be generated based on the patient's X-ray image, and the global image features related to the lesion can be determined based on the three-dimensional images at different angles. In this way, the global image features of the lesion area determined based on the three-dimensional images at multiple angles are more accurate, thereby making the final image diagnosis report more accurate.

[0046] Step S102: Fusing the local image features and the global image features to obtain fused image features, and inputting the fused image features into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image. The image diagnosis classifier is obtained by pre-training the original classifier based on multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing local image features related to lesions extracted from sample medical images and global image features.

[0047] Exemplarily, in this embodiment, when classifying image diagnosis, a classifier trained based on multidimensional image fusion features, namely, an image diagnosis classifier, is pre-set, and diagnostic classification is performed based on the fusion image features of the input medical image to accurately determine the disease diagnosis classification results related to the medical image. The fusion of global image features and local image features, namely, regional features, can better mine the image hierarchical information of sample medical images and medical images. When the image diagnosis classifier trained accordingly is applied and the fusion image features are input, the disease category corresponding to the medical image can be more accurately identified and confirmed. Among them, the original classifier can be a convolutional neural network, but it is not limited to this, and any one that can achieve the classification task can be used. When the original classifier is trained, the sample medical image is first processed, that is, the local image features and global image features related to the lesion are extracted, and then the multidimensional image fusion features are obtained by fusion and input into the original classifier, and the difference between the disease type label of the sample medical image and the output result corresponding to the original classifier is combined for iterative training, until the loss value of the loss function of the original classifier meets the set convergence condition, the training can be terminated.

[0048] Step S103: Based on the disease diagnosis classification result, query the pre-constructed medical field knowledge graph to obtain the corresponding first medical field knowledge code; fuse the local image features, the global image features and the first medical field knowledge code to obtain the target text features, and input the target text features into the medical big language model to automatically generate an imaging diagnosis report.

[0049] Exemplarily, after obtaining accurate disease diagnosis and classification results of medical images, knowledge nodes are retrieved from a pre-constructed medical field knowledge graph based on the diagnosis and classification results of the images to accurately obtain medical field expertise related to the lesions involved in the medical images. Among them, the medical field knowledge graph is pre-constructed based on medical expertise, and it can contain the corresponding relationship between different diagnosis and classification results and corresponding medical field knowledge codes. In this way, the enhancement of medical knowledge related to lesions can be achieved, and the diagnosis and classification results of the images are retrieved as knowledge nodes. Through the joint query of entities and relationships, the first medical field knowledge code, i.e., the feature code of the professional field knowledge related to the lesion, is obtained. Afterwards, the multi-dimensional image features, i.e., local image features, global image features, and the first field knowledge code, i.e., the professional medical knowledge corresponding to the lesion, are fused to generate multimodal fusion features, i.e., target text features, and finally, the multimodal fusion features are input into the medical large language model to intelligently generate an image diagnosis report, i.e., the input medical large language model is decoded and processed to generate an image diagnosis report. Among them, the medical big language model can be a model obtained by training a big language model based on medical field knowledge content. It can refer to the commercial medical big language models on the market, such as the MedGPT model developed and released by Yilian or the Zuoyi medical big model, but it is not limited to this, and any other medical big language model can be used. The local image features, global image features, and target text features of the first medical field knowledge encoding fusion are summarized and analyzed through the medical big language model, wherein enhanced lesion-related medical professional field knowledge is introduced by querying the medical field knowledge graph, so that the medical big language model can output accurate, professional, and highly interpretable imaging diagnosis report content.

[0050] In the above scheme of this embodiment, when a medical imaging report is automatically generated, the image diagnosis classifier trained based on the multi-dimensional image fusion feature can combine the fused image features, i.e., the fusion features of the local image features and the global image features from the medical image, to accurately obtain the lesion diagnosis classification result. Based on this, the medical field knowledge coding related to the lesion can be accurately obtained from the pre-constructed medical field knowledge graph. Finally, the local image features, the global image features and the medical field knowledge coding, i.e., the lesion-related professional knowledge coding, are fused to obtain the target text features. The image diagnosis report is automatically generated in combination with the medical large language model. In this way, a more accurate medical imaging report can be obtained for the patient, and the report content is interpretable, i.e., the professional report content that can be easily understood by doctors other than radiologists or patients. This scheme can realize automatic feature extraction and processing of medical images to automatically generate imaging reports, assist radiologists in completing the writing of reports, greatly reduce the workload of radiologists, improve work efficiency, and save human resources.

[0051] In one embodiment, the original classifier may include a multi-label classifier. In the diagnosis and identification of medical images, at least one lesion feature in the medical image may involve multiple different types of diseases. In order to more accurately identify the lesion diagnosis classification results and improve the accuracy of automatically generated image reports, when training the image diagnosis classifier, in addition to using multi-dimensional image fusion features, a multi-label classifier can also be set up for training. The image diagnosis classifier trained in this way can more accurately identify at least one type of disease corresponding to it from the medical image, thereby more accurately identifying the lesion diagnosis classification results, and ultimately improving the accuracy of automatically generated image reports.

[0052] In one example, the original classifier may include a feature acquisition network, a first module, and a second module, and the first module and the second module may be convolutional neural networks. The training process of the original classifier may include: the feature acquisition network extracts a feature map of multi-dimensional image fusion features; inputs the feature map into the first module to obtain a first classification result; and compares the first classification result with the feature map. Figure 1 The second module is input in the same manner to obtain a second classification result, and iterative training is performed based on the difference between at least one disease type label of the sample medical image and the second classification result corresponding to the original classifier, until the loss value of the loss function of the original classifier meets the set convergence condition, and the training is terminated.

[0053] The model architecture of the above-mentioned original classifier is combined with the training process, so that the data input to the second module includes the first classification result output by the first module in addition to the extracted feature map, so that the output result of the second module, i.e., the second classification result, is enhanced by the output result of the first module, i.e., the first classification result, so that the second classification result, i.e., the final classification result, is more accurate, so that the trained image diagnosis classifier can more accurately identify at least one type of disease corresponding to the medical image, that is, more accurately identify the lesion diagnosis classification result, and ultimately improve the accuracy of automatically generated image reports.

[0054] Exemplarily, the feature acquisition network may be a deep neural network with a self-attention mechanism, but is not limited thereto.

[0055] Based on any one of the above embodiments, in one embodiment, in order to further improve the accuracy of automatically generated imaging reports, extracting local image features and global image features related to lesions in the medical image in step S101 may specifically include: inputting the medical image into a first image encoder and a second image encoder respectively to extract local image features and global image features related to lesions respectively; wherein the first image encoder is pre-trained based on a RetinaNet model based on sample medical images; and / or the second image encoder is pre-trained based on a Vision Transformer model based on sample medical images.

[0056] The Transformer model uses the Self-Attention mechanism, which enables the model to have global information. Vision Transformer (ViT) is a deep learning model based on the Transformer architecture. ViT successfully extends the Transformer model from the field of natural language processing (NLP) to the field of computer vision (CV). The core idea of ​​ViT is to regard the input image as a serialized input and process the pixel relationship in the image through the self-attention mechanism. Specifically, ViT first divides the input image into multiple fixed-size image patches (Patches), and then performs linear mapping (Flattening+Linear Projection) on each image patch to convert it into a vector. These vectors, together with a special classification token, serve as the input sequence of the Transformer, so that the self-attention mechanism can be directly applied for feature extraction. For the specific content of the ViT model and the RetinaNet model, please refer to the existing technology for understanding, which will not be repeated here.

[0057] In this embodiment, in order to further improve the accuracy of automatically generating imaging reports, a single step S101 in the above steps is further improved. Specifically, in response to different requirements for medical image feature extraction, a pre-trained first image encoder and a pre-trained second image encoder are used to extract local image features and global image features related to lesions, respectively. The second image encoder is trained based on a Vision Transformer model of sample medical images, which is more suitable for extracting global features, while the first image encoder is more suitable for extracting local features. Therefore, local image features and global image features can be extracted more accurately, and then steps S102 to S103 are performed, which can ultimately further improve the accuracy of automatically generating imaging reports.

[0058] Based on any of the above embodiments, Figure 2As shown, in one embodiment, the method may further include the following steps:

[0059] Step S201: semantically map the local image features and the global image features to obtain feature codes; input the feature codes into a medical imaging knowledge template model to obtain a second medical field knowledge code; and determine a target medical field knowledge code based on the second medical field knowledge code and the first medical field knowledge code.

[0060] Exemplarily, feature coding represents semantic information of local image features and global image features, such as semantic information related to diseases, and different feature coding can represent semantic knowledge information related to different diseases. The medical imaging knowledge template model can be obtained by pre-training an original model such as a convolutional neural network based on a pair of training data of sample feature coding and corresponding medical field knowledge coding. In this way, after obtaining feature coding through semantic mapping based on the above-mentioned local image features and global image features, the corresponding second medical field knowledge coding can be accurately determined, and then fused with the first medical field knowledge coding obtained by the query to determine the target medical field knowledge coding, which can also assist in enhancing the accuracy of the first medical field knowledge coding.

[0061] Step S202: fusing the local image features, the global image features and the target medical field knowledge encoding to obtain the target text features.

[0062] Exemplarily, in this embodiment, local image features, global image features, and target medical field knowledge encoding, i.e., enhanced first medical field knowledge encoding, can be integrated to obtain more accurate target text features, and then the target text features are input into the medical large language model to automatically generate an imaging diagnosis report. In this way, the obtained feature encoding can be accurately aligned with the medical field knowledge through the medical imaging knowledge template model, and the accuracy of the first medical field knowledge encoding can be enhanced, so that the accuracy of automatically generated imaging reports can be further improved.

[0063] Based on the above embodiments, in one embodiment, the local image features and the global image features are semantically mapped to obtain feature codes in step S201, which may specifically include: generating image multi-scale features based on the local image features and the global image features, and obtaining feature codes by performing semantic mapping based on the image multi-scale features.

[0064] Exemplarily, based on the local image features and the global image features, image multi-scale features can be fused to generate the image. The fusion of global features and local features can better mine image hierarchical information, making the feature coding obtained by semantic mapping more accurate, thereby further improving the accuracy of automatically generated image reports. Different image multi-scale features can correspond to different feature codings, and the corresponding relationship can be preset.

[0065] Based on any of the above embodiments, in one embodiment, reference Figure 3 As shown, the method may also include the following steps:

[0066] Step S301: Obtain descriptive questions of the medical image as prompt content. Exemplarily, the descriptive questions of the medical image can be obtained from reports of other historical medical images with similar image features to the medical image, or determined by a doctor based on medical image diagnosis input. The descriptive questions can include image feature descriptions of lesions in the image, diagnosis results of the diseases to which the lesions belong, and the like.

[0067] Accordingly, in step S103, the target text features are input into the medical language model to automatically generate an imaging diagnosis report, which may specifically include the following steps:

[0068] Step S302: input the target text features and the prompt content into the medical big language model to automatically generate an imaging diagnosis report.

[0069] In this embodiment, a large medical language model is used to generate an image diagnosis report by combining multimodal fusion features, i.e., target text features, with image descriptive questions as prompt content. The prompt content can further assist the large medical language model in accurately understanding image features and diagnosis results, thereby generating more accurate, professional, and interpretable image diagnosis report content.

[0070] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc. In addition, it is also easy to understand that these steps may be, for example, executed synchronously or asynchronously in multiple modules / processes / threads.

[0071] like Figure 4 As shown, the embodiment of the present disclosure provides a medical imaging report automatic generation system, comprising:

[0072] The feature acquisition module 401 is used to acquire the medical image to be processed and extract the local image features and global image features related to the lesion in the medical image;

[0073] An image diagnosis module 402 is used to fuse the local image features and the global image features to obtain fused image features, and input the fused image features into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image; wherein the image diagnosis classifier is obtained by pre-training the original classifier based on the multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing the local image features related to the lesion and the global image features extracted from the sample medical image;

[0074] The report generation module 403 is used to query the pre-constructed medical field knowledge graph to obtain the corresponding first medical field knowledge code based on the disease diagnosis classification result; fuse the local image features, the global image features and the first medical field knowledge code to obtain the target text features, and input the target text features into the medical big language model to automatically generate an imaging diagnosis report.

[0075] In one example, the original classifier may include a feature acquisition network, a first module, and a second module, and the first module and the second module may be convolutional neural networks. The training process of the original classifier may include: the feature acquisition network extracts a feature map of multi-dimensional image fusion features; inputs the feature map into the first module to obtain a first classification result; and compares the first classification result with the feature map. Figure 1 The second module is input in the same manner to obtain a second classification result, and iterative training is performed based on the difference between at least one disease type label of the sample medical image and the second classification result corresponding to the original classifier, until the loss value of the loss function of the original classifier meets the set convergence condition, and the training is terminated.

[0076] In one embodiment, the feature acquisition module 401 extracts local image features and global image features related to lesions in the medical image, including:

[0077] The medical image is input into a first image encoder and a second image encoder respectively to extract local image features and global image features related to the lesion respectively; wherein the first image encoder is pre-trained based on a sample medical image to a RetinaNet model; and / or the second image encoder is pre-trained based on a sample medical image to a Vision Transformer model.

[0078] In one embodiment, the system may also include a mapping encoding module, which is used to: perform semantic mapping on the local image features and the global image features to obtain feature encoding; input the feature encoding into the medical imaging knowledge template model to obtain a second medical field knowledge encoding; based on the second medical field knowledge encoding and the first medical field knowledge encoding, determine the target medical field knowledge encoding; the report generation module 403 can also be used to fuse the local image features, the global image features and the target medical field knowledge encoding to obtain the target text features.

[0079] In one embodiment, the mapping coding module performs semantic mapping on the local image features and the global image features to obtain feature coding, including: generating image multi-scale features based on the local image features and the global image features, and performing semantic mapping based on the image multi-scale features to obtain feature coding.

[0080] In one embodiment, the original classifier comprises a multi-label classifier.

[0081] In one embodiment, the system may also include a prompt determination module, which is used to: obtain descriptive questions of the medical image as prompt content; the report generation module 403 may also be used to: input the target text features and the prompt content into the medical big language model to automatically generate an image diagnosis report.

[0082] In one embodiment, the medical images include at least X-ray images.

[0083] Regarding the system in the above embodiment, the specific manner in which each module performs operations and the corresponding technical effects brought about have been described in detail in the embodiment of the method, and will not be elaborated here.

[0084] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units for concretization. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0085] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for automatically generating a medical imaging report according to any one of the above embodiments is implemented.

[0086] Exemplarily, the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0087] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0088] The present disclosure also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, wherein the processor is configured to execute the method for automatically generating a medical imaging report of any of the above embodiments by executing the computer program.

[0089] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0090] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0091] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method embodiment section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 The steps of the method shown in .

[0092] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0093] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0094] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0095] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0096] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the method steps of the above-mentioned embodiments according to the implementation methods of the present disclosure.

[0097] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0098] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatically generating a medical imaging report, characterized in that: include: Acquire a medical image to be processed, and extract local image features and global image features related to lesions in the medical image; The local image features and the global image features are fused to obtain fused image features, and the fused image features are input into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image; wherein the image diagnosis classifier is obtained by pre-training the original classifier based on multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing local image features related to lesions extracted from sample medical images and global image features; Based on the disease diagnosis classification result, a pre-constructed medical field knowledge graph is queried to obtain the corresponding first medical field knowledge code; the local image features, the global image features and the first medical field knowledge code are integrated to obtain the target text features, and the target text features are input into the medical big language model to automatically generate an imaging diagnosis report.

2. The method according to claim 1, characterized in that: The extracting of local image features and global image features related to lesions in the medical image includes: The medical image is input into a first image encoder and a second image encoder respectively to extract local image features and global image features related to the lesion respectively; wherein the first image encoder is pre-trained based on a sample medical image to a RetinaNet model; and / or the second image encoder is pre-trained based on a sample medical image to a Vision Transformer model.

3. The method according to claim 1, characterized in that The method further includes: Performing semantic mapping on the local image features and the global image features to obtain feature codes; inputting the feature codes into a medical imaging knowledge template model to obtain a second medical field knowledge code; and determining a target medical field knowledge code based on the second medical field knowledge code and the first medical field knowledge code; The target text feature is obtained by fusing the local image feature, the global image feature and the target medical field knowledge encoding.

4. The method according to claim 3, characterized in that The semantic mapping of the local image features and the global image features to obtain feature coding includes: Image multi-scale features are generated based on the local image features and the global image features, and semantic mapping is performed based on the image multi-scale features to obtain feature encoding.

5. The method according to any one of claims 1 to 4, characterized in that: The original classifier includes a multi-label classifier.

6. The method according to any one of claims 1 to 4, characterized in that: The method further includes: Obtaining descriptive questions of the medical image as prompt content; The step of inputting the target text features into a medical large language model to automatically generate an imaging diagnosis report includes: The target text features and the prompt content are input into the medical big language model to automatically generate an imaging diagnosis report.

7. The method according to any one of claims 1 to 4, characterized in that: The medical images at least include X-ray images.

8. A medical imaging report automatic generation system, characterized in that: include: A feature acquisition module, used to acquire the medical image to be processed, and extract local image features and global image features related to the lesion in the medical image; An image diagnosis module, used for fusing the local image features and the global image features to obtain fused image features, and inputting the fused image features into an image diagnosis classifier to obtain a disease diagnosis classification result related to the medical image; wherein the image diagnosis classifier is obtained by pre-training the original classifier based on the multi-dimensional image fusion features, and the multi-dimensional image fusion features are obtained by fusing the local image features related to the lesions extracted from the sample medical image and the global image features; A report generation module is used to query a pre-constructed medical field knowledge graph to obtain the corresponding first medical field knowledge code based on the disease diagnosis classification result; fuse the local image features, the global image features and the first medical field knowledge code to obtain the target text features, and input the target text features into the medical large language model to automatically generate an imaging diagnosis report.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically generating a medical imaging report according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as Memory for storing computer programs; Wherein, the processor is configured to execute the method for automatically generating a medical imaging report according to any one of claims 1 to 7 by executing the computer program.

Images