Medical image report generation device and method, electronic equipment and storage medium

Abstract

The invention discloses a medical image report generation device and method, electronic equipment and a storage medium. The medical image report generation device comprises a feature alignment module which is used for carrying out cross-modal alignment on image features of a medical image and preset text features; the label classification module is used for determining at least one label of a patient corresponding to the medical image, and the label represents a clinical symptom of the patient; and the report output module is used for inputting the at least one label and the image features after cross-modal alignment into a large language model to obtain an image report of the medical image output by the large language model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a medical image report generation device, method, electronic device, and storage medium. Background Technology

[0002] Medical image report generation is a special type of image-to-text generation task. Related technologies utilize similarity functions and domain transfer techniques to find the most similar image in an existing database for the medical image to be generated for the report. Once the most similar image is found, domain transfer techniques can be used to transfer report content related to that image to the new image. This process involves transferring knowledge from the source domain (existing images and their reports) to the target domain (new images).

[0003] Related technologies tend to rely too much on similar cases and ignore the unique features of new images, and the generated results lack interpretability and are difficult for medical staff to understand and apply. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a medical image report generation device, method, electronic device, and storage medium.

[0005] The technical solution of this invention is implemented as follows: On one hand, embodiments of the present invention provide a medical image report generation device, the device comprising: The feature alignment module is used to perform cross-modal alignment of image features of medical images with preset text features; A tag classification module is used to determine at least one tag for a patient corresponding to the medical image, wherein the tag characterizes the patient's clinical symptoms; The report output module is used to input the at least one label and the cross-modal aligned image features into the large language model to obtain the image report of the medical image output by the large language model.

[0006] In the above scheme, the medical image report generation device further includes: The encoding module is used to encode the medical image to obtain the image features.

[0007] In the above scheme, the feature alignment module is specifically used for: Based on the image features and the preset text features, an attention score is determined; the image features are then weighted based on the attention score to obtain cross-modal aligned image features.

[0008] In the above scheme, the label classification module is specifically used for: The image features are input into a label classification model to obtain at least one label output by the label classification model.

[0009] In the above scheme, the report output module is specifically used for: The at least one label is used as a semantic anchor to guide the large language model in generating the image report.

[0010] In the above scheme, the medical image report generation device further includes: A matching module is used to determine training data in the training dataset that matches the image features. The training data includes image features of a preset medical image and text features of a medical report corresponding to the preset medical image. The determination module is used to determine the text features of the medical report corresponding to the preset medical image as preset text features.

[0011] In the above scheme, the medical image report generation device further includes: The first training module is used to train the large language model based on the training dataset; the training data includes image features of preset medical images and text features of image reports of the preset medical images; The second training module is used to train the label classification model based on the image features of the preset medical images.

[0012] On the other hand, embodiments of this application also provide a method for generating medical image reports, the method comprising: Cross-modal alignment of medical image features with preset text features; Identify at least one label for a patient corresponding to the medical image, the label representing the patient's clinical symptoms; The at least one label and the cross-modal aligned image features are input into a large language model to obtain an image report of the medical image output by the large language model.

[0013] On the other hand, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described medical image report generation method.

[0014] On the other hand, embodiments of the present invention provide an electronic device including a processor and a memory, the processor and the memory being interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the steps of the medical image report generation method provided in embodiments of the present invention.

[0015] On the other hand, embodiments of the present invention provide a computer-readable storage medium, comprising: the computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the medical image report generation method provided in embodiments of the present invention.

[0016] The medical image report generation device of this invention includes: a feature alignment module for cross-modal alignment of image features of medical images with preset text features; a label classification module for determining at least one label for a patient corresponding to the medical image, wherein the label represents the patient's clinical symptoms; and a report output module for inputting at least one label and the cross-modal aligned image features into a large language model to obtain an image report of the medical image output by the large language model. This invention aligns image features to a text feature space and inputs the patient's clinical symptoms along with the image features into the large language model, making the report generated by the large language model more interpretable and reliable, and easier for medical personnel to understand and apply. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a medical image report generation device provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a label classification model provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a report generation network model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the implementation process of a medical image report generation method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Deep learning technology has made significant progress in the field of medical image processing, enabling automated or semi-automated disease diagnosis. However, the lack of interpretability in the end-to-end operation of deep learning limits its further development in the field of intelligent healthcare. To address this issue, researchers have begun exploring the integration of Natural Language Processing (NLP) technology with medical image analysis to develop multimodal models capable of automatically generating medical diagnostic reports. This approach not only improves diagnostic efficiency and accuracy but also increases the transparency and interpretability of the diagnostic process.

[0020] The technology for automatically generating medical image reports is based on massive amounts of image diagnostic reports and medical image data. It utilizes advanced machine learning and deep learning algorithms to extract and analyze key features in the images. This technology combines image feature analysis with natural language processing (NLP) to bridge the gap between medical images and textual descriptions, providing strong support for medical diagnosis and research. Currently, the technical solutions for generating medical image reports include the following: 1. Using similarity functions and domain transfer techniques, the most similar image to the medical image for which a report is to be generated is found in the existing database. Once the most similar image is found, domain transfer techniques can be used to transfer the report content related to that image to the new image. This process involves transferring knowledge from the source domain (existing images and their reports) to the target domain (new images).

[0021] 2. Instead of aligning multimodal features or using only linear transformations for simple feature alignment, this approach utilizes neural networks to extract visual features from medical images and relevant textual features from domain knowledge. Then, the features from different modalities are simply concatenated and input into the report generation module to generate the corresponding medical report.

[0022] The drawbacks of the related technologies are: 1. Domain transfer approaches can effectively utilize existing medical knowledge and report structures, but they also face challenges, such as how to handle rare or unique medical cases, and how to ensure that the generated reports do not overly rely on similar cases while ignoring the unique features of new images. Therefore, in practical applications, this approach often needs to be combined with other technologies, such as deep learning and natural language processing, to improve its flexibility and accuracy.

[0023] 2. Using simple feature alignment schemes may lead to the loss of complex nonlinear feature information, reducing the model's sensitivity to subtle medical features. Furthermore, this alignment scheme cannot fully capture the complex relationships between visual and textual features, affecting the fusion effect.

[0024] The aforementioned technical solutions also suffer from a common problem: a lack of interpretability. The "black box" nature of the model may make the diagnostic process difficult to explain, affecting the trust of doctors and patients.

[0025] To address the shortcomings of the aforementioned related technologies, embodiments of the present invention provide a medical image report generation device that enables the generated medical reports to have better interpretability and reliability. To illustrate the technical solution described in this invention, specific embodiments are described below.

[0026] refer to Figure 1 , Figure 1 This is a schematic diagram of a medical image report generation device provided in an embodiment of the present invention. The medical image report generation device includes: The feature alignment module is used to perform cross-modal alignment of image features of medical images with preset text features; A tag classification module is used to determine at least one tag for a patient corresponding to the medical image, wherein the tag characterizes the patient's clinical symptoms; The report output module is used to input the at least one label and the cross-modal aligned image features into the large language model to obtain the image report of the medical image output by the large language model.

[0027] In order to enable large language models to better understand and process medical image information, it is necessary to align the image features of medical images to the text feature space. Since there are huge modal differences between image modalities and text modalities, it is impossible to bridge the differences between them through simple linear projection. Therefore, this embodiment performs cross-modal alignment between image features and preset text features.

[0028] The preset text features can be the text features of image reports of other known medical images that are similar to the medical images. They can be predefined or obtained through real-time matching.

[0029] For example, image features can be aligned to the text feature space using self-attention and cross-attention mechanisms.

[0030] With the continuous improvement of domestic medical standards and quality, the electronic health record (EHR) system for patients is becoming increasingly sophisticated, making it easy to obtain case metadata. Effectively utilizing this multi-omics data can help with automated medical diagnosis.

[0031] This embodiment acquires the patient's clinical symptom data and uses the clinical symptoms as labels to assist in the report generation task. The clinical symptoms, as additional semantic information, make the reports generated by the large speech model more interpretable and reliable. This method not only improves the accuracy of the reports but also makes the diagnostic process more transparent.

[0032] Here, a patient's clinical symptoms can be retrieved from the patient's electronic health record system or extracted from medical images. For example, clinically relevant semantic features can be extracted as labels from medical image features using a label classification model.

[0033] At least one label is combined with the aligned image features as additional semantic information and input into the large language model, which outputs a medical image report.

[0034] The large language model can be pre-trained using a training dataset, which includes known image features of medical images and text features of medical image reports. The image features and text features of the training data are aligned before iteratively training the large language model.

[0035] For example, for generating Chinese image reports, a pre-trained Chinese language model, GLM-4, can be used.

[0036] Because of the close correlation and correlation between clinical symptoms and imaging features, in the text modality, in addition to utilizing the report corresponding to the images, the patient's clinical symptoms are also input into the large language model along with the imaging features. In medical reports, combining specific clinical symptoms makes the report content easier for medical staff to understand and apply. For example, the consistency between the imaging features pointed out in the report and the patient's symptoms can enhance medical staff's confidence in diagnostic and treatment decisions.

[0037] The medical image report generation device of this invention includes: a feature alignment module for cross-modal alignment of image features of medical images with preset text features; a label classification module for determining at least one label for a patient corresponding to the medical image, wherein the label represents the patient's clinical symptoms; and a report output module for inputting at least one label and the cross-modal aligned image features into a large language model to obtain an image report of the medical image output by the large language model. This invention aligns image features to a text feature space and inputs the patient's clinical symptoms along with the image features into the large language model, making the report generated by the large language model more interpretable and reliable, and easier for medical personnel to understand and apply.

[0038] In one embodiment, the medical image report generation device further includes: The encoding module is used to encode the medical image to obtain the image features.

[0039] For example, image features can be extracted from medical images using an image feature encoder.

[0040] In one embodiment, the feature alignment module is specifically used for: Based on the image features and the preset text features, an attention score is determined; the image features are then weighted based on the attention score to obtain cross-modal aligned image features.

[0041] This embodiment aligns image features to the text feature space through self-attention and cross-attention mechanisms. Cross-attention, as a key alignment technique, can focus on the differences between text and image modalities and align image features with the text semantic space. The steps of cross-attention include: 1. First, map the visual feature I and the preset text feature T to the same feature dimension through a linear transformation:

[0042]

[0043] in, and It is the learned weight matrix. and It is a bias.

[0044] 2. In cross-attention, text features are used as the query, and visual features are used as the key and value.

[0045] 3. Calculate attention score using dot-product attention.

[0046]

[0047] The aligned image features are represented by the attention output calculated through the above steps, which represents the alignment of image features in the text feature space.

[0048] In one embodiment, the label classification module is specifically used for: The image features are input into a label classification model to obtain at least one label output by the label classification model.

[0049] For example, a label classification network is used to predict a patient's clinical symptoms, generating a distribution across m labels. An adaptive thresholding mechanism is then employed to select the k most representative labels (i.e., clinical symptoms) from the predicted distribution. This step ensures the high relevance and representativeness of the selected features. The selected k labels are then used as semantic anchors to guide the large language model generation process. This method significantly improves the accuracy and clinical relevance of the generated reports.

[0050] The structure of the label classification model is as follows Figure 2As shown, the input image features first pass through N fully connected layers and ReLU layers, then through a fully connected layer and a sigmoid activation function, as shown in the following formula:

[0051]

[0052] in, It is the predicted label distribution. , and These are the weight matrix and bias vector of the classifier, respectively. It is the sigmod activation function. It is a label for clinical symptoms. These are the k tags with the highest probability.

[0053] In one embodiment, the report output module is specifically used for: The at least one label is used as a semantic anchor to guide the large language model in generating the image report.

[0054] Extracting clinically relevant semantic features from image data provides strong semantic guidance for subsequent report generation. Through a TopK mechanism, the model can dynamically adjust the number of selected features to adapt to medical cases of varying complexity. Utilizing the extracted semantic features as prior knowledge significantly improves the accuracy and professionalism of report generation. Explicit clinical symptom labels enhance the interpretability of model decisions, which is particularly important in the medical field.

[0055] The results of multi-label classification are combined with the aligned image features as additional semantic information and input into the large language model. In one example, the large language model uses the Chinese pre-trained large language model GLM-4, as shown in the following formula:

[0056] in, It is the final image report generated by the large language model. These are the aligned image features. It is a label for clinical symptoms.

[0057] In one embodiment, the medical image report generation device further includes: A matching module is used to determine training data in the training dataset that matches the image features. The training data includes image features of a preset medical image and text features of a medical report corresponding to the preset medical image. The determination module is used to determine the text features of the medical report corresponding to the preset medical image as preset text features.

[0058] During training, the training data contains paired image features and text features. However, during prediction, the input only contains image features of medical images. In order to achieve cross-modal feature alignment, preset text features are used as text data to guide image modality alignment.

[0059] Training data and retrieval tools can be used to perform comparative semantic ranking using relevant instance retrieval processes to obtain preset text features input to the network, which can then be used as text data to guide image modality alignment.

[0060] For example, one can find training data from the training dataset that is most similar to the image features of the input medical image, and use the text features in that training data as the preset text features.

[0061] In one embodiment, the medical image report generation device further includes: The first training module is used to train the large language model based on the training dataset; the training data includes image features of preset medical images and text features of image reports of the preset medical images; The second training module is used to train the label classification model based on the image features of the preset medical images.

[0062] The training data can also include clinical symptoms from medical images. These clinical symptoms are used as additional semantic information to input into the large semantic model and guide the model training process.

[0063] In one embodiment, during model training, a low-rank adaptation (LoRA) approach is used to fine-tune the large language model.

[0064] The core of LoRA technology lies in preserving the weights of the pre-trained model while injecting a trainable low-rank factorization matrix into the Transformer layer. The key advantage of this strategy is that by limiting the number of parameters in downstream tasks, it significantly reduces the computational complexity and resource requirements when the model adapts to new tasks.

[0065] For example, a multi-label classification loss function and an autoregressive loss are used together during training to optimize the neural network. The corresponding overall loss function can be written as:

[0066] in, This represents the multi-label classification loss function. It is its corresponding weight coefficient; Represents the autoregressive loss function. It is its corresponding weighting coefficient.

[0067] The loss function of the label classification model uses the average binary cross-entropy loss, which is optimized for multi-label classification tasks. The formula is as follows:

[0068] in, The total number of tags. It is the actual value of the i-th label. This is the network's original prediction of the i-th label. Using this loss function, the model is trained to predict the presence or absence of each label. When the actual label... When the probability is 1, and the model's predicted probability approaches 0, or when When the model's predicted probability approaches 1 while the loss function is 0, the model is penalized. This mechanism allows the model to optimize its performance in multi-label classification tasks, reducing the gap between the prediction and the true label.

[0069] In large language models, the autoregressive loss function is used to train the model to predict the next word or character sequentially. The main objective is to maximize the probability of the next word given the conditional probabilities of the preceding parts of the sequence. This example uses cross-entropy loss:

[0070] in It is the current time step target words, It refers to all words before the current time step. Is the model in a given Predicting under the following conditions The probability of the next word is used to predict the probability of the next word through this loss function, thereby guiding the generation of medical image reports.

[0071] Before model training, the training data needs to be preprocessed. Large-scale medical imaging data often comes from multiple centers and multiple devices, and these images may vary in format, resolution, and quality. Therefore, standardization is necessary to ensure data consistency. Specific processing steps include: data cleaning, format conversion, image registration, image enhancement, and labeling.

[0072] The compiled image reports are structured to make them more standardized, making it easier for the neural network to learn their features. The medical named entity recognition model ERNIE-health is used to extract clinical symptom information from patient medical records. Each patient can have 0-n different clinical symptoms as labels for the multi-label classification module.

[0073] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a report generation network model provided in an embodiment of the present invention. The report generation network model includes an image feature encoder, a text feature encoder, an alignment module, a multi-label classification (MLC) model, and a large language model.

[0074] The training process requires simultaneous input of medical images and image reports. An image feature encoder extracts image features from the medical images, while a text feature encoder extracts text features from the image reports. These image and text features then enter an alignment module. In this module, a self-attention mechanism (Self-Attn) is first employed to achieve deep interaction within the modality. This allows each token to adaptively learn the most relevant visual features, thus accurately capturing key information from complex visual inputs. Next, cross-modal attention (Cross-Attn) is introduced to achieve dynamic alignment between visual and linguistic features. Cross-attention, as a key alignment technique, addresses the differences between the text and image modalities and aligns image features with the text semantic space. Simultaneously, the extracted image features are input into an MLC model, which predicts the patient's clinical symptoms. The results of multi-label classification are combined with the aligned image features as additional semantic information and input into a large language model to obtain the image report generated by the large language model.

[0075] In the inference process, only the medical image to be generated as input is needed. The image feature encoder extracts image features and searches the training dataset for training data that is most similar to the image features of the input medical image. The text features in this training data are used as preset text features. The image features and preset text features are then input into the alignment module for cross-modal alignment. Clinical symptoms are extracted from the image features using MLC. The results of multi-label classification are combined with the aligned image features as additional semantic information and input into the large language model to obtain the image report generated by the large language model.

[0076] This invention employs self-attention and cross-attention mechanisms to align image features to the text feature space, effectively improving the expressive power of the features and enabling large language models to better understand and process medical image information. By using a multi-label classification model to predict patients' clinical symptoms and inputting these labels as additional semantic information into the large language model, the generated reports become more clinically relevant and interpretable. This approach not only improves the accuracy of reports but also makes the diagnostic process more transparent.

[0077] This invention provides an end-to-end intelligent medical report generation process that seamlessly integrates multimodal feature alignment, multi-label classification, and a large language model to construct a complete intelligent diagnostic assistance system. It boasts a high degree of automation, enabling rapid processing of large amounts of medical image data and improving diagnostic efficiency. It is highly adaptable, capable of handling different modalities of medical images (such as X-rays, CT scans, and MRIs) to meet diverse clinical needs. The generated reports include detailed image descriptions and highlight key clinical findings, providing doctors with comprehensive diagnostic references and assisting them in making more accurate diagnoses.

[0078] Automated generation of medical imaging reports can significantly reduce physicians' workload and improve diagnostic efficiency. Furthermore, by combining imaging features with clinical symptoms, the generated reports are more accurate. Integrating information from both clinical symptoms and imaging features provides more detailed and interpretable reports, enhancing support for physicians' clinical decision-making and improving the quality of diagnosis and treatment. Automated report generation reduces the time and human resources required for manual diagnosis, lowers healthcare costs, and improves the operational efficiency of healthcare institutions.

[0079] It should be understood that, in addition to medical imaging and clinical symptoms, other types of medical data, such as electronic health records, genomic data, and pathology reports, can be integrated to further enhance the input information of the large language model and improve the comprehensiveness and accuracy of the generated reports. When generating reports, patient historical data and individual characteristics can be combined to produce personalized medical reports. This not only helps to provide targeted treatment recommendations but also improves patient satisfaction and trust.

[0080] refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the implementation flow of a medical image report generation method provided in an embodiment of the present invention. The medical image report generation method includes: S401 performs cross-modal alignment between the image features of medical images and preset text features.

[0081] S402, determine at least one label for a patient corresponding to the medical image, the label representing the patient's clinical symptoms.

[0082] S403, input the at least one label and the cross-modal aligned image features into the large language model to obtain the image report of the medical image output by the large language model.

[0083] In one embodiment, the medical image report generation method further includes: The medical image is encoded to obtain the image features.

[0084] In one embodiment, the cross-modal alignment of image features of medical images with preset text features includes: Based on the image features and the preset text features, an attention score is determined; the image features are then weighted based on the attention score to obtain cross-modal aligned image features.

[0085] In one embodiment, determining at least one tag of a patient corresponding to the medical image includes: The image features are input into a label classification model to obtain at least one label output by the label classification model.

[0086] In one embodiment, the at least one tag is used as a semantic anchor to guide the large language model in generating the image report.

[0087] In one embodiment, the medical image report generation method further includes: Determine training data in the training dataset that matches the image features, the training data including image features of preset medical images and text features of medical reports corresponding to the preset medical images; The text features of the medical report corresponding to the preset medical image are determined as preset text features.

[0088] In one embodiment, the medical image report generation method further includes: The large language model is trained based on a training dataset; the training data includes image features of preset medical images and text features of image reports of the preset medical images. The label classification model is trained based on the image features of the preset medical images.

[0089] This method corresponds to the medical image report generation device at the receiving end described above, and the steps in the above medical image report generation device embodiment are also fully applicable to this method embodiment. The medical image report generation device and the medical image report generation method embodiment provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the device embodiment, which will not be repeated here.

[0090] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0091] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0092] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0093] In addition, in the embodiments of the present invention, "first," "second," etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0094] In practical applications, the feature alignment module, label classification module, and report output module can be implemented by processors in electronic devices, such as central processing units (CPUs), digital medical image report generators (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0095] It should be noted that the medical image report generation device provided in the above embodiments is only illustrated by the division of the above modules when generating medical image reports. In actual applications, the above processing can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different modules to complete all or part of the processing described above. In addition, the medical image report generation device and the medical image report generation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0096] The aforementioned medical image report generation device can be in the form of an image file. After execution, this image file can run as a container or virtual machine to implement the medical image report generation method described in this application. However, it is not limited to the image file format; any software that can implement the medical image report generation method described in this application is within the scope of protection of this application.

[0097] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 5 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device includes: The communication interface 501 enables information exchange with other devices, such as network devices. The processor 502 is connected to the communication interface 501 to enable information interaction with other devices and to execute the methods provided by one or more of the above-described technical solutions when running a computer program. The computer program is stored in the memory 503.

[0098] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system 504. It can be understood that the bus system 504 is used to implement communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general designated all buses as Bus System 504.

[0099] The memory 503 in this embodiment is used to store various types of data to support the operation of the computer device. Examples of such data include any computer program used to operate on the electronic device.

[0100] It is understood that memory 503 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0101] The methods disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0102] Optionally, when the processor 602 executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0103] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory storing a computer program, which can be executed by a processor of a computer device to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, computer devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0105] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0107] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0108] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0109] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 502 of an electronic device to perform the steps described in the medical image report generation method of this application.

[0110] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0111] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A medical image report generation device, characterized in that, The method includes: The feature alignment module is used to perform cross-modal alignment of image features of medical images with preset text features; A tag classification module is used to determine at least one tag for a patient corresponding to the medical image, wherein the tag characterizes the patient's clinical symptoms; The report output module is used to input the at least one label and the cross-modal aligned image features into the large language model to obtain the image report of the medical image output by the large language model.

2. The medical image report generation device according to claim 1, characterized in that, The medical image report generation device also includes: The encoding module is used to encode the medical image to obtain the image features.

3. The medical image report generation device according to claim 1, characterized in that, The feature alignment module is specifically used for: Based on the image features and the preset text features, an attention score is determined; the image features are then weighted based on the attention score to obtain cross-modal aligned image features.

4. The medical image report generation device according to claim 1, characterized in that, The label classification module is specifically used for: The image features are input into a label classification model to obtain at least one label output by the label classification model.

5. The medical image report generation device according to claim 1, characterized in that, The report output module is specifically used for: The at least one label is used as a semantic anchor to guide the large language model in generating the image report.

6. The medical image report generation device according to claim 1, characterized in that, The medical image report generation device also includes: A matching module is used to determine training data in the training dataset that matches the image features. The training data includes image features of a preset medical image and text features of a medical report corresponding to the preset medical image. The determination module is used to determine the text features of the medical report corresponding to the preset medical image as preset text features.

7. The medical image report generation device according to claim 4, characterized in that, The medical image report generation device also includes: The first training module is used to train the large language model based on the training dataset; the training data includes image features of preset medical images and text features of image reports of the preset medical images; The second training module is used to train the label classification model based on the image features of the preset medical images.

8. A method for generating medical image reports, characterized in that, include: Cross-modal alignment of medical image features with preset text features; Identify at least one label for a patient corresponding to the medical image, the label representing the patient's clinical symptoms; The at least one label and the cross-modal aligned image features are input into a large language model to obtain an image report of the medical image output by the large language model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the medical image report generation method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the steps of the medical image report generation method as described in claim 8.

Images