Chinese medical report generation method and system based on pre-training large model, and terminal

Abstract

The invention provides a Chinese medical report generation method and system based on a pre-trained large model and a terminal, and relates to the technical field of artificial intelligence, and the method comprises the steps: S1, obtaining an X-ray medical image of a to-be-generated report; s2, obtaining a Chinese medical report generation requirement input by a user; s3, importing the obtained X-ray medical image and a Chinese medical report generation requirement input by a user into a pre-trained large model to obtain a Chinese medical report of the X-ray medical image; the large model comprises a medical visual encoder, a medical large language model and a similar report retrieval module. According to the method, complex contents and lesion features in the medical image can be fully understood based on the pre-trained large model, the accuracy of the generated medical report in lesion description and diagnosis suggestion aspects is improved, the illusion problem when the model generates the report is relieved, and the quality and efficiency of the medical report are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system and terminal for generating Chinese medical reports based on a pre-trained large model. Background Art

[0002] With the rapid development of medical technology, medical imaging plays an increasingly critical role in clinical diagnosis. As an important basis for doctors' diagnosis and treatment, the accuracy and detail of medical imaging reports directly affect the quality of medical care and patient safety.

[0003] At present, the automatic generation technology of medical imaging reports is mainly based on deep learning and multimodal large model methods. Multimodal large models usually consist of visual encoders and large language models (LLM), which provide an effective way to generate text from images. These models realize the automatic conversion of medical images to diagnostic reports by combining image information and language information. Multimodal large models may have hallucination problems when generating text, that is, generating descriptions that are inconsistent with the content of the input image or logically unreasonable. This problem is particularly serious in medical report generation because it may lead to misdiagnosis or missed diagnosis, bringing uncontrollable risks to patients. This is the shortcoming of the existing technology. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the present invention provides a Chinese medical report generation method, system and terminal based on a pre-trained large model. The trained large model can fully understand the complex content and lesion characteristics in medical images and alleviate the hallucination problem when generating text.

[0005] In a first aspect, the present invention provides a method for generating a Chinese medical report based on a pre-trained large model, the method comprising: S1, obtaining an X-ray medical image for which a report is to be generated; S2, obtaining the Chinese medical report generation requirements input by the user; S3, importing the acquired X-ray medical image and the Chinese medical report generation requirements into a pre-trained large model to obtain the Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder, a medical large language model and a similar report retrieval module; The medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical large language model; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model is used to generate a Chinese medical report for the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder and based on the input Chinese medical report generation requirements and the retrieved medical report.

[0006] Furthermore, the input X-ray medical image is encoded to obtain visual features of the X-ray medical image, including: The input X-ray medical image is divided into N pre-set non-overlapping image blocks of fixed size to obtain an image block sequence ; The image patch sequence is transformed into Mapped to dimension The vector space of the image block embedding sequence is obtained ; Add image classification embedding at the beginning of the image block embedding sequence , and then add image classification embedding to the first part Each image patch in the image patch embedding sequence is embedded with the image classification embedding Adding positional encoding , get the image block coding sequence of the X-ray medical image ; Encode the image block sequence Import the first layer of the Transformer encoder in the medical vision encoder, use the output of each layer as the input of the next layer of Transformer encoder, perform feature extraction layer by layer, and output the visual features of the X-ray medical image after being encoded by the last layer of Transformer encoder ; in, represents the X-ray medical image, Represents the segmentation of the acquired X-ray medical image image blocks, , Indicates The height of the image block, Indicates The width of the image block, Indicates The number of channels of an image block, , Indicates the number of image blocks obtained by segmenting the acquired X-ray medical image, represents the fully connected layer, represents the dimension of the vector space, represents image classification embedding, represents the image position encoding, Represents an image block coding sequence; represents the visual features of the X-ray medical image, represents the global features of the X-ray medical image, Representing local features of the X-ray medical image; Represents the Transformer encoder.

[0007] Furthermore, before the medical large language model generates a Chinese medical report of the X-ray medical image according to the visual features of the X-ray medical image obtained by the medical visual encoder and according to the input Chinese medical report generation requirements and the retrieved medical report, it also includes: performing visual embedding processing on the visual features of the X-ray medical image, including: Through the trained projection layer The local features of the X-ray medical image Mapped to the same dimension as the word embedding space of the medical large language model, the visual embedding of the X-ray medical image is obtained , ; The projection layer The training methods include: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a projection layer sample training set; Image-text alignment training: Use the data in the projection layer sample training set to The multimodal template performs data conversion and uses the converted data to project the layer Perform iterative training, using the converted mixed data Partial data and loss function for projection layer Calculate the loss value and then use the back propagation algorithm to calculate the projection layer The gradient of the parameters is used to update the projection layer using the gradient descent method Parameters until the preset number of iterations of training is completed or the calculated projection layer is The loss value no longer decreases, completing the projection layer Train to get the trained projection layer ; in, The visual features of the images representing the data in the training set of the projection layer samples, A text description of an image representing data from the projection layer sample training set.

[0008] Furthermore, the method for constructing a large medical language model includes: Create the Qwen-7b model as the initial model and perform model initialization; The training method of the medical large language model includes: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a model sample training set; Instruction fine-tuning training: After importing the data in the model sample training set into the initial model, use the imported data Multimodal templates are used to convert data, and then the converted data is used The loss value of the medical language model is calculated using partial data and loss function, the gradient of the medical language model is calculated using the back propagation algorithm, and then the model parameters of other linear layers except the lm_head layer of the medical language model are updated using the low-rank adaptation fine-tuning method until the preset number of iterative training times is completed or the calculated loss value of the medical language model no longer decreases, the training of the medical language model is completed, and a trained medical language model is obtained; in, Visual features of images representing the data in the model’s sample training set, A text description of an image representing the data in the model’s training set, Representing the model roles and capabilities of the medical large language model, Indicates the user tag. Indicates the report generation requirements of the user input medical large language model, Represents a helper tag, A stop mark indicating that the large medical language model has finished responding to the user's request for generating a report on the large medical language model.

[0009] Further, in step S3, according to the global features of the encoded X-ray medical image, a medical report corresponding to an X-ray medical image similar to the X-ray medical image is retrieved from a preset image memory storage and a medical report file to be queried, including: Normalizing the global features of the X-ray medical image encoded by the medical vision encoder to obtain a global feature vector of the X-ray medical image; Calculate the square of the Euclidean distance between the global feature vector of the processed X-ray medical image and each feature vector in the preset image memory storage, and sort each feature vector in the image memory storage from small to large according to the calculated square of the Euclidean distance, and select the feature vector in the image memory storage with the smallest square of the Euclidean distance calculated from the global feature vector of the X-ray medical image in the sorting, and record it as the feature vector ; The feature vector in the selected image memory The corresponding historical X-ray medical image is an X-ray medical image similar to the X-ray medical image; Find feature vectors in the pre-set medical report files to be queried The corresponding medical report, the found medical report is a medical report of an X-ray medical image similar to the X-ray medical image.

[0010] Furthermore, the method for constructing the preset image memory storage device includes: Collection of historical X-ray medical images; Use a medical visual encoder to encode the collected historical X-ray medical images to obtain the global features of each historical X-ray medical image; Normalizing the global features of each historical X-ray medical image to obtain a global feature vector of each historical X-ray medical image; Setting an index number for the global feature vector of each historical X-ray medical image obtained; The global feature vector of each historical X-ray medical image and its corresponding index number are stored in a preset memory to obtain a preset image memory; The method for constructing the preset medical report file to be queried includes: Medical reports corresponding to the historical X-ray medical images are collected, and the medical reports and the index numbers of the global feature vectors of the corresponding historical X-ray medical images are stored in a preset file to obtain a preset medical report file to be queried.

[0011] In a second aspect, the present invention provides a Chinese medical report generation system based on a pre-trained large model, the system comprising an image acquisition module, an instruction acquisition module and a report generation module; An image acquisition module, used to acquire X-ray medical images for generating reports; An instruction acquisition module is used to obtain the Chinese medical report generation requirements input by the user; A report generation module, used for importing the acquired X-ray medical image and the Chinese medical report generation requirements input by the user into a pre-trained large model to obtain a Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder, a medical large language model and a similar report retrieval module; The medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical large language model; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model is used to generate a Chinese medical report for the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder and based on the input Chinese medical report generation requirements and the retrieved medical report.

[0012] Furthermore, in the report generation module, according to the global features of the X-ray medical image obtained by encoding, a medical report corresponding to an X-ray medical image similar to the X-ray medical image is retrieved from a preset image memory storage and a medical report file to be queried, including: Normalizing the global features of the X-ray medical image encoded by the medical vision encoder to obtain a global feature vector of the X-ray medical image; Calculate the square of the Euclidean distance between the global feature vector of the processed X-ray medical image and each feature vector in the preset image memory storage, and sort each feature vector in the image memory storage from small to large according to the calculated square of the Euclidean distance, and select the feature vector in the image memory storage with the smallest square of the Euclidean distance calculated from the global feature vector of the X-ray medical image in the sorting, and record it as the feature vector ; The feature vector in the selected image memory The corresponding historical X-ray medical image is an X-ray medical image similar to the X-ray medical image; Find feature vectors in the pre-set medical report files to be queried The corresponding medical report, the found medical report is a medical report of an X-ray medical image similar to the X-ray medical image.

[0013] Furthermore, the method for constructing the preset image memory storage device includes: Collection of historical X-ray medical images; Use a medical visual encoder to encode the collected historical X-ray medical images to obtain the global features of each historical X-ray medical image; Normalizing the global features of each historical X-ray medical image to obtain a global feature vector of each historical X-ray medical image; Setting an index number for the global feature vector of each historical X-ray medical image obtained; The global feature vector of each historical X-ray medical image and its corresponding index number are stored in a preset memory to obtain a preset image memory; The method for constructing the preset medical report file to be queried includes: Medical reports corresponding to the historical X-ray medical images are collected, and the medical reports and the index numbers of the global feature vectors of the corresponding historical X-ray medical images are stored in a preset file to obtain a preset medical report file to be queried.

[0014] In a third aspect, the present invention provides a terminal, the terminal comprising a memory and a processor; A memory, used for storing a program for automatically generating Chinese medical reports; A processor is used to implement the Chinese medical report generation method based on the pre-trained large model described in any one of the first aspects when executing the Chinese medical report automatic generation program.

[0015] It can be seen from the above technical solutions that the present invention has the following advantages: The present invention constructs a large model, sets up similar report retrieval, and realizes accurate matching of medical images and their corresponding reports by combining the image feature processing technology of the medical large language model and the medical visual encoder. The medical reports obtained through retrieval optimize the performance of the medical large language model in the reasoning stage, further alleviate the hallucination problem caused by the medical large language model, and improve the accuracy of the generated medical reports in lesion description and diagnostic suggestions, effectively improving the quality and efficiency of medical reports.

[0016] The present invention extracts the global and local features of X-ray medical images through the medical visual encoder in the constructed large model, thereby improving the accuracy of lesion feature recognition. Secondly, the medical large language model in the large model constructed by the present invention is based on the Qwen-7b model, and performs image-text alignment training and instruction fine-tuning through the constructed mixed data sample training set, thereby improving the accuracy, robustness and stability of the large model's understanding of the visual features of X-ray medical images.

[0017] In addition, the present invention constructs a mixed data sample training set containing rich image-text correspondences, and performs image-text alignment training on the set projection layer. This not only enhances the projection layer's ability to understand the visual features of the X-ray image, but also promotes the deep fusion of image information and text description by the medical big language model, so that the generated medical report can more accurately reflect the lesion condition in the image.

[0018] Moreover, the present invention constructs a mixed data sample training set containing rich image-text correspondences to perform instruction fine-tuning training on the medical language model, guiding the model to learn how to generate standardized, accurate, and logical medical reports based on image features, including detailed lesion descriptions, precise diagnostic suggestions, etc. In addition, the present invention uses a rank adaptation fine-tuning method to perform efficient parameter fine-tuning on the medical language model, which not only effectively reduces the number of fine-tuning parameters and significantly reduces the consumption of computing resources of the medical language model, but also ensures that the fine-tuned medical language model can effectively adapt to downstream tasks. This process effectively alleviates the hallucination problem that may occur when the model generates reports (i.e., generating text that does not match the image content), and improves the credibility and practicality of the report.

[0019] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A schematic flow chart of an embodiment of the method for generating a Chinese medical report based on a pre-trained large model according to the present invention; Figure 2 A schematic block diagram of an embodiment of the Chinese medical report generation system based on the pre-trained large model of the present invention; Figure 3 The figure is a schematic diagram of the structure of an embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0024] like Figure 1 As shown, the present invention provides a method for generating Chinese medical reports based on a pre-trained large model, the method comprising: S1. Obtaining an X-ray medical image for which a report is to be generated; S2, obtaining the Chinese medical report generation requirements input by the user; S3, importing the acquired X-ray medical image and the Chinese medical report generation requirements input by the user into a pre-trained large model to obtain the Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder, a medical large language model and a similar report retrieval module; The medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical large language model; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model is used to generate a Chinese medical report for the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder and based on the input Chinese medical report generation requirements and the retrieved medical report.

[0025] To facilitate understanding of the present invention, the following is a further description of the Chinese medical report generation method based on a pre-trained large model provided by the present invention based on the principle of the Chinese medical report generation method based on a pre-trained large model and the process of the Chinese medical report generation method based on a pre-trained large model in the embodiment.

[0026] Specifically, the method for generating a Chinese medical report based on a pre-trained large model includes: Step 110: Obtain an X-ray medical image for which a report is to be generated.

[0027] Specifically, the X-ray medical images taken by the doctor are confirmed and initially judged to see whether the X-ray medical images are clear and contain the required diagnostic information. If the quality of the X-ray medical images meets the requirements, the doctor will upload the X-ray medical images to the public image storage system of the hospital. The X-ray medical images required to generate medical reports are downloaded from the public image storage system.

[0028] Step 120: Obtain the Chinese medical report generation requirement input by the user.

[0029] Step 130: import the acquired X-ray medical image and the Chinese medical report generation requirements input by the user into a pre-trained large model to obtain the Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder, a medical large language model and a similar report retrieval module; The medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical large language model; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model is used to generate a Chinese medical report for the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder and based on the input Chinese medical report generation requirements and the retrieved medical report.

[0030] It should be noted that after the Chinese medical report generation requirements and the retrieved medical reports are input into the medical big language model, the Chinese medical report generation requirements and the retrieved medical reports need to be converted into instruction embedding extraction and similar report embedding extraction through the word segmenter in the medical big language model, and the converted instruction embedding and similar report embedding are used for the trained medical big language model to generate the Chinese medical report of the X-ray medical image.

[0031] Specifically, the medical visual encoder is a visual Transformer model encoder using the ViT architecture. In this embodiment, the visual encoder of BiomedCLIP is used. The medical large language model uses a pre-trained large language model (LLM) with rich medical knowledge and strong Chinese processing capabilities. In this embodiment, Taiyi-LLM is used as the medical LLM component. Taiyi-LLM, or the "Taiyi" biomedical large model, is a Chinese-English bilingual biomedical large model developed by the Information Retrieval Laboratory of the School of Computer Science, Dalian University of Technology based on Qwen-7b and fine-tuned with multi-task instructions.

[0032] Specifically, the medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features, including: The input X-ray medical image is divided into N pre-set non-overlapping image blocks of fixed size, where each image block after segmentation is a one-dimensional vector, and the image block sequence is obtained It should be noted that the divided image blocks are squares with equal length and width.

[0033] Through the fully connected layers in the medical visual encoder The image block sequence Mapped to dimension The vector space of the image block embedding sequence is obtained ; Add image classification embedding at the beginning of the image block embedding sequence , and then add image classification embedding to the first part Each image patch in the image patch embedding sequence is embedded with the image classification embedding Adding positional encoding , get the image block coding sequence of the X-ray medical image ; Encode the image block sequence Import the first layer of the Transformer encoder in the medical vision encoder, use the output of each layer as the input of the next layer of Transformer encoder, perform feature extraction layer by layer, and output the visual features of the X-ray medical image after being encoded by the last layer of Transformer encoder ; in, represents the X-ray medical image, Represents the segmentation of the acquired X-ray medical image image blocks, , Indicates The height of the image block, Indicates The width of the image block, Indicates The number of channels of an image block, , Indicates the number of image blocks obtained by segmenting the acquired X-ray medical image, represents the fully connected layer, represents the dimension of the vector space, represents image classification embedding, represents the image position encoding, Represents an image block coding sequence; represents the visual features of the X-ray medical image, represents the global features of the X-ray medical image, Representing local features of the X-ray medical image; Represents the Transformer encoder.

[0034] Specifically, this paper selects the visual encoder of BiomedCLIP to extract the visual features of X-ray images. BiomedCLIP is a visual language model in the biomedical field trained on the PMC-15M dataset. Because its visual encoder has sufficient ability to extract medical image features, its model weights are frozen throughout the training process.

[0035] As an embodiment of the present invention, before the medical large language model generates a Chinese medical report of the X-ray medical image according to the visual features of the X-ray medical image obtained by the medical visual encoder and according to the input Chinese medical report generation requirements and the retrieved medical report, it further includes: performing visual embedding processing on the visual features of the X-ray medical image: Through the trained projection layer The local features of the input X-ray medical image Mapped to the same dimension as the word embedding space of the medical large language model, the visual embedding of the X-ray medical image is obtained , .

[0036] Specifically, the projection layer The training methods include: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a projection layer sample training set; Image-text alignment training: Use the data in the projection layer sample training set to The multimodal template performs data conversion and uses the converted data to project the layer Perform iterative training, using the converted mixed data Partial data and loss function for projection layer Calculate the loss value and then use the back propagation algorithm to calculate the projection layer The gradient of the parameters is used to update the projection layer using the gradient descent method Parameters until the preset number of iterations of training is completed or the calculated projection layer is The loss value no longer decreases, completing the projection layer Train to get the trained projection layer ; in, The visual features of the images representing the data in the training set of the projection layer samples, A text description of an image representing data from the projection layer sample training set.

[0037] It should be noted that the projection layer It is an MLP neural network, the projection layer Used to transform local features of X-ray medical images Projected to the same dimension as the word embedding space of the medical large language model, the local features of the X-ray medical image are Used to guide the generation of Chinese medical reports for the X-ray medical images.

[0038] Specifically, the MIMIC-CXR dataset is the largest publicly available dataset in the field of medical report generation, and the Vision-FLAN dataset is a dataset that integrates 191 visual question answering (VQA) tasks from 101 open source datasets. In the Vision-FLAN dataset, each visual question answering task is accompanied by instructions written by experts, and the Vision-FLAN dataset contains VFLAN-Caption data and VFLAN-Instruct data. VFLAN-Caption data represents an image caption dataset, and VFLAN-Instruct data represents a visual question answering task instruction dataset.

[0039] The MIMIC-CXR dataset is in English, and the medical reports in the MIMIC-CXR dataset are unstructured data. The MIMIC-CXR dataset used in this example is obtained from the XrayGPT project, which processes the unstructured medical reports through the GPT3.5 model to generate concise and coherent medical summaries that are more in line with human language habits. In addition, in this example, the MIMIC-CXR data of the XrayGPT project is translated into Chinese using Tencent translation service to obtain the Chinese MIMIC-CXR dataset for research on the Chinese medical report generation task.

[0040] The Vision-FLAN dataset is a Chinese version obtained from the ALLaVA project. In this example, the data in the obtained Vision-FLAN dataset is preprocessed to obtain VFLAN-Caption data and VFLAN-Instruct data, specifically including: deleting data in the Vision-FLAN dataset that does not conform to the model training format, data with garbled characters, and data with meaningless symbols.

[0041] Exemplarily, a method for constructing a large medical language model includes: Create the Qwen-7b model as the initial model and initialize the model.

[0042] Exemplarily, the training method of the medical large language model includes: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a model sample training set; Instruction fine-tuning training: After importing the data in the model sample training set into the initial model, use the imported data Multimodal templates are used to convert data, and then the converted data is used The loss value of the medical language model is calculated using partial data and loss function, the gradient of the medical language model is calculated using the back propagation algorithm, and then the model parameters of other linear layers except the lm_head layer of the medical language model are updated using the low-rank adaptation fine-tuning method until the preset number of iterative training times is completed or the calculated loss value of the medical language model no longer decreases, the training of the medical language model is completed, and a trained medical language model is obtained; in, Visual features of images representing the data in the model’s sample training set, A text description of an image representing the data in the model’s training set, Representing the model roles and capabilities of the medical large language model, Indicates the user tag. Indicates the report generation requirements of the user input medical large language model, Represents a helper tag, A stop mark indicating that the large medical language model has finished responding to the user's request for generating a report on the large medical language model.

[0043] Specifically, the principle of the low-rank adaptation fine-tuning method is: Assume that the weight matrix of a layer in the model is , the incremental parameter matrix to be fine-tuned by the low-rank adaptive fine-tuning method The low-rank approximation is two matrices with smaller parameters and , the parameter update process can be regarded as ; At this time, the parameter amount of all parameter fine-tuning changes from the original of Became , the number of parameters that need to be updated will be significantly reduced. When fine-tuning, keep is frozen, and the matrix is ​​distributed with a random Gaussian distribution Initialize the matrix with zero matrix Initialization, which ensures Gradually learn new knowledge during training to ensure the stability of the whole process. Therefore, given an input , for the output , its forward calculation becomes , when inferring, we can Add directly to the original parameters If you need to switch additional matrices to adapt to different application scenarios, you can also subtract Then add different The method is simple to implement.

[0044] in, and is the dimension of the pre-trained model weights, is the rank of the increment matrix, and .

[0045] Specifically, during the training process of the medical large language model, the model sample training set includes X-ray medical images, their corresponding Chinese medical reports, and their corresponding Chinese medical report generation requirements.

[0046] Exemplarily, the method for constructing the preset image memory storage device includes: Collection of historical X-ray medical images; Use a medical visual encoder to encode the collected historical X-ray medical images to obtain the global features of each historical X-ray medical image; Normalizing the global features of each historical X-ray medical image to obtain a global feature vector of each historical X-ray medical image; Set an index number for the global feature vector of each historical X-ray medical image to obtain the feature vector to be queried, and collect all the feature vectors to be queried to obtain a feature vector sequence , , the global feature vector of each historical X-ray medical image and its corresponding index number, that is, the feature vector sequence The image is stored in a preset memory to obtain a preset image memory.

[0047] The method for constructing the preset medical report file to be queried includes: Collect the medical reports corresponding to the historical X-ray medical images, and aggregate them with the index numbers of the global feature vectors of the corresponding historical X-ray medical images to obtain a medical report sequence , , the index number of the global feature vector of the medical report and its corresponding historical X-ray medical image, that is, the medical report sequence The data is stored in a preset file to obtain a preset medical report file to be queried.

[0048] in, Indicates the number The feature vector to be queried, Indicates the number Medical reports, Indicates the number of historical X-ray medical images acquired.

[0049] Specifically, in step 130, according to the global features of the X-ray medical image obtained by encoding, a medical report corresponding to an X-ray medical image similar to the X-ray medical image is retrieved from a preset image memory storage and a medical report file to be queried, including: Normalizing the global features of the X-ray medical image encoded by the medical vision encoder to obtain a global feature vector of the X-ray medical image; Calculate the square of the Euclidean distance between the global feature vector of the processed X-ray medical image and each feature vector in the preset image memory storage, and sort each feature vector in the image memory storage from small to large according to the calculated square of the Euclidean distance, and select the feature vector in the image memory storage with the smallest square of the Euclidean distance calculated from the global feature vector of the X-ray medical image in the sorting, and record it as the feature vector ; The feature vector in the selected image memory The corresponding historical X-ray medical image is an X-ray medical image similar to the X-ray medical image; Find feature vectors in the pre-set medical report files to be queried The corresponding medical report, the found medical report is a medical report of an X-ray medical image similar to the X-ray medical image.

[0050] Specifically, in this embodiment, the IndexFlatL2 structure of the Faiss library is used to retrieve the feature vector of the X-ray image to be queried in the image memory storage according to the processed feature vector, and the Euclidean distance method is used according to the formula , calculate the square of the Euclidean distance between the global feature vector of the processed X-ray medical image and each feature vector in the preset image memory storage, and sort each feature vector in the image memory storage from small to large according to the calculated square of the Euclidean distance. Since the smaller the square of the Euclidean distance between two vectors, the greater the similarity of the images corresponding to the two vectors, the feature vector in the image memory storage with the smallest square of the Euclidean distance calculated with the global feature vector of the X-ray medical image is selected in the sorting, and recorded as the feature vector ; Selected feature vector The corresponding X-ray medical image is an X-ray medical image similar to the X-ray medical image. The index number retrieves the preset medical report file to be queried and obtains the feature vector corresponding to the queried feature vector. The medical report with the same index number is the medical report of the X-ray medical image similar to the X-ray medical image, and the obtained medical report is sent to the medical language model.

[0051] As an embodiment of the present invention, Figure 2 As shown, the present invention also provides a Chinese medical report generation system based on a pre-trained large model, the system 200 includes: an image acquisition module 210, an instruction acquisition module 220 and a report generation module 230; An image acquisition module 210, for acquiring an X-ray medical image for which a report is to be generated; The instruction acquisition module 220 is used to acquire the Chinese medical report generation requirement input by the user; A report generation module 230 is used to import the acquired X-ray medical image and the Chinese medical report generation requirements input by the user into a pre-trained large model to obtain a Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder 231, a medical large language model 232 and a similar report retrieval module 233; The medical visual encoder 231 is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module 233 is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical language model 232; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model 232 is used to generate a Chinese medical report of the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder 231, and based on the input Chinese medical report generation requirements and the retrieved medical report.

[0052] Furthermore, in the report generation module 230, the input X-ray medical image is encoded to obtain visual features of the encoded X-ray medical image, including: The input X-ray medical image is divided into N pre-set non-overlapping image blocks of fixed size to obtain an image block sequence ; Through the fully connected layers in the Medical Vision Encoder 231 The image block sequence Mapped to dimension The vector space of the image block embedding sequence is obtained ; Add image classification embedding at the beginning of the image block embedding sequence , and then add image classification embedding to the first part Each image patch in the image patch embedding sequence is embedded with the image classification embedding Adding positional encoding , get the image block coding sequence of the X-ray medical image ; Encode the image block sequence Import the first layer of the Transformer encoder in the Medical Vision Encoder 231, use the output of each layer as the input of the next layer of Transformer encoder, perform feature extraction layer by layer, and output the visual features of the X-ray medical image after being encoded by the last layer of Transformer encoder ; in, represents the X-ray medical image, Represents the segmentation of the acquired X-ray medical image image blocks, , Indicates The height of the image block, Indicates The width of the image block, Indicates The number of channels of an image block, , Indicates the number of image blocks obtained by segmenting the acquired X-ray medical image, represents the fully connected layer, represents the dimension of the vector space, represents image classification embedding, represents the image position encoding, Represents an image block coding sequence; represents the visual features of the X-ray medical image, represents the global features of the X-ray medical image, Representing local features of the X-ray medical image; Represents the Transformer encoder.

[0053] Furthermore, before generating a Chinese medical report of the X-ray medical image according to the visual features of the X-ray medical image obtained by the medical visual encoder 231 and according to the input Chinese medical report generation requirements and the retrieved medical report, the medical language model 232 further includes: performing visual embedding processing on the visual features of the X-ray medical image: Through the trained projection layer The local features of the input X-ray medical image Mapped to the same dimension as the word embedding space of the medical large language model 232, the visual embedding of the X-ray medical image is obtained , ; The projection layer The training methods include: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a projection layer sample training set; Image-text alignment training: Use the data in the projection layer sample training set to The multimodal template performs data conversion and uses the converted data to project the layer Perform iterative training, using the converted mixed data Partial data and loss function for projection layer Calculate the loss value and then use the back propagation algorithm to calculate the projection layer The gradient of the parameters is used to update the projection layer using the gradient descent method Parameters until the preset number of iterations of training is completed or the calculated projection layer is The loss value no longer decreases, completing the projection layer Train to get the trained projection layer ; in, The visual features of the images representing the data in the training set of the projection layer samples, A text description of an image representing data from the projection layer sample training set.

[0054] Furthermore, the method for constructing the medical big language model 232 includes: Create the Qwen-7b model as the initial model and perform model initialization; The training method of the medical language model 232 includes: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a model sample training set; Instruction fine-tuning training: After importing the data in the model sample training set into the initial model, use the imported data Multimodal templates are used to convert data, and then the converted data is used The loss value of the medical large language model 232 is calculated using partial data and a loss function, the gradient of the medical large language model 232 is calculated using a back propagation algorithm, and then the model parameters of other linear layers except the lm_head layer of the medical large language model 232 are updated using a low-rank adaptation fine-tuning method until a preset number of iterative training times is completed or the calculated loss value of the medical large language model 232 no longer decreases, the training of the medical large language model 232 is completed, and a trained medical large language model 232 is obtained; in, Visual features of images representing the data in the model’s sample training set, A text description of an image representing the data in the model’s training set, Representing the model roles and capabilities of the medical large language model, Indicates the user tag. Indicates the report generation requirements of the user input medical large language model, Represents a helper tag, A stop mark indicating that the large medical language model has finished responding to the user's request for generating a report on the large medical language model.

[0055] Exemplarily, in the report generating module 230, according to the global features of the X-ray medical image obtained by encoding, searching the pre-set image memory storage and the medical report file to be queried for the medical report corresponding to the X-ray medical image similar to the X-ray medical image includes: Normalizing the global features of the X-ray medical image encoded by the medical vision encoder 231 to obtain a global feature vector of the X-ray medical image; The square of the Euclidean distance between the processed global feature vector of the X-ray medical image and each feature vector in the preset image memory storage is calculated, and each feature vector in the image memory storage is sorted from small to large according to the square of the calculated Euclidean distance. Since the smaller the square of the Euclidean distance between two vectors, the greater the similarity of the images corresponding to the two vectors, the feature vector in the image memory storage having the smallest square of the Euclidean distance calculated with the global feature vector of the X-ray medical image is selected in the sorting, and recorded as the feature vector ; The feature vector in the selected image memory The corresponding historical X-ray medical image is an X-ray medical image similar to the X-ray medical image; Find feature vectors in the pre-set medical report files to be queried The corresponding medical report, the found medical report is a medical report of an X-ray medical image similar to the X-ray medical image.

[0056] Exemplarily, the method for constructing the preset image memory storage device includes: Collection of historical X-ray medical images; Encode the collected historical X-ray medical images using the medical vision encoder 231 to obtain the global features of each historical X-ray medical image; Normalizing the global features of each historical X-ray medical image to obtain a global feature vector of each historical X-ray medical image; Setting an index number for the global feature vector of each historical X-ray medical image obtained; The global feature vector of each historical X-ray medical image and its corresponding index number are stored in a preset memory to obtain a preset image memory.

[0057] Exemplarily, the method for constructing the preset medical report file to be queried includes: Medical reports corresponding to the historical X-ray medical images are collected, and the medical reports and the index numbers of the global feature vectors of the corresponding historical X-ray medical images are stored in a preset file to obtain a preset medical report file to be queried.

[0058] Figure 3 A schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the Chinese medical report generation system based on the pre-trained large model provided in an embodiment of the present invention.

[0059] The terminal 300 may include: a processor 310, a memory 320 and a communication module 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0060] The memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can execute some or all of the steps in the following above-mentioned system embodiment.

[0061] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0062] The communication module 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0063] The present invention constructs a large model, sets up similar report retrieval, and realizes accurate matching of medical images and their corresponding reports by combining the image feature processing technology of the medical large language model and the medical visual encoder. The medical reports obtained through retrieval optimize the performance of the medical large language model in the reasoning stage, further alleviate the hallucination problem caused by the medical large language model, and improve the accuracy of the generated medical reports in lesion description and diagnostic suggestions, effectively improving the quality and efficiency of medical reports.

[0064] The present invention extracts the global and local features of X-ray medical images through the medical visual encoder in the constructed large model, thereby improving the accuracy of lesion feature recognition. Secondly, the medical large language model in the large model constructed by the present invention is based on the Qwen-7b model, and performs image-text alignment training and instruction fine-tuning through the constructed mixed data sample training set, thereby improving the accuracy, robustness and stability of the large model's understanding of the visual features of X-ray medical images.

[0065] In addition, the present invention constructs a mixed data sample training set containing rich image-text correspondences, and performs image-text alignment training on the set projection layer. This not only enhances the projection layer's ability to understand the visual features of the X-ray image, but also promotes the deep fusion of image information and text description by the medical big language model, so that the generated medical report can more accurately reflect the lesion condition in the image.

[0066] Moreover, the present invention constructs a mixed data sample training set containing rich image-text correspondences to perform instruction fine-tuning training on the medical language model, guiding the model to learn how to generate standardized, accurate, and logical medical reports based on image features, including detailed lesion descriptions, precise diagnostic suggestions, etc. In addition, the present invention uses a rank adaptation fine-tuning method to perform efficient parameter fine-tuning on the medical language model, which not only effectively reduces the number of fine-tuning parameters and significantly reduces the consumption of computing resources of the medical language model, but also ensures that the fine-tuned medical language model can effectively adapt to downstream tasks. This process effectively alleviates the hallucination problem that may occur when the model generates reports (i.e., generating text that does not match the image content), and improves the credibility and practicality of the report.

[0067] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect.

[0068] The technical effects that can be achieved by this embodiment can be found in the above description and will not be repeated here.

[0069] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the system described in each embodiment of the present invention.

[0070] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the system embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the system embodiment.

[0071] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and systems can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0072] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0074] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, and all of these shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for generating Chinese medical reports based on a pre-trained large model, characterized in that: S1. Obtaining an X-ray medical image for which a report is to be generated; S2, obtaining the Chinese medical report generation requirements input by the user; S3, importing the acquired X-ray medical image and the Chinese medical report generation requirements into a pre-trained large model to obtain the Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder, a medical large language model and a similar report retrieval module; The medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical large language model; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model is used to generate a Chinese medical report for the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder and based on the input Chinese medical report generation requirements and the retrieved medical report.

2. The method for generating Chinese medical reports based on a pre-trained large model according to claim 1, characterized in that: Encode the input X-ray medical image to obtain the visual features of the X-ray medical image, including: The input X-ray medical image is divided into N pre-set non-overlapping image blocks of fixed size to obtain an image block sequence ; The image patch sequence is transformed into Mapped to dimension The vector space of the image block embedding sequence is obtained ; Add image classification embedding at the beginning of the image block embedding sequence , and then add image classification embedding to the first part Each image patch in the image patch embedding sequence is embedded with the image classification embedding Adding positional encoding , get the image block coding sequence of the X-ray medical image ; Encode the image block sequence Import the first layer of the Transformer encoder in the medical vision encoder, use the output of each layer as the input of the next layer of Transformer encoder, perform feature extraction layer by layer, and output the visual features of the X-ray medical image after being encoded by the last layer of Transformer encoder ; in, represents the X-ray medical image, Represents the segmentation of the acquired X-ray medical image image blocks, , Indicates The height of the image block, Indicates The width of the image block, Indicates The number of channels of an image block, , Indicates the number of image blocks obtained by segmenting the acquired X-ray medical image, represents the fully connected layer, represents the dimension of the vector space, represents image classification embedding, represents the image position encoding, Represents an image block coding sequence; represents the visual features of the X-ray medical image, represents the global features of the X-ray medical image, Representing local features of the X-ray medical image; Represents the Transformer encoder.

3. The method for generating Chinese medical reports based on a pre-trained large model according to claim 1, characterized in that: Before generating a Chinese medical report of the X-ray medical image according to the visual features of the X-ray medical image obtained by the medical visual encoder and the input Chinese medical report generation requirements and the retrieved medical report, the medical language model further includes: performing visual embedding processing on the visual features of the X-ray medical image: Through the trained projection layer The local features of the X-ray medical image Mapped to the same dimension as the word embedding space of the medical large language model, the visual embedding of the X-ray medical image is obtained , ; The projection layer The training methods include: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a projection layer sample training set; Image-text alignment training: Use the data in the projection layer sample training set to The multimodal template performs data conversion and uses the converted data to project the layer Perform iterative training, using the converted mixed data Partial data and loss function for projection layer Calculate the loss value and then use the back propagation algorithm to calculate the projection layer The gradient of the parameters is used to update the projection layer using the gradient descent method Parameters until the preset number of iterations of training is completed or the calculated projection layer is The loss value no longer decreases, completing the projection layer Train to get the trained projection layer ; in, The visual features of the images representing the data in the training set of the projection layer samples, A text description of an image representing data from the projection layer sample training set.

4. The method for generating Chinese medical reports based on a pre-trained large model according to claim 1, characterized in that: The method for constructing a large medical language model includes: Create the Qwen-7b model as the initial model and perform model initialization; The training method of the medical large language model includes: Construct a sample training set: obtain data from the MIMIC-CXR dataset as in-domain data, obtain data from the Vision-FLAN dataset, preprocess the data from the Vision-FLAN dataset to obtain VFLAN-Caption data and VFLAN-Instruct data, use the VFLAN-Caption data and VFLAN-Instruct data as common data, aggregate the in-domain data and the common data, and obtain a model sample training set; Instruction fine-tuning training: After importing the data in the model sample training set into the initial model, use the imported data Multimodal templates are used to convert data, and then the converted data is used The loss value of the medical language model is calculated using partial data and loss function, the gradient of the medical language model is calculated using the back propagation algorithm, and then the model parameters of other linear layers except the lm_head layer of the medical language model are updated using the low-rank adaptation fine-tuning method until the preset number of iterative training times is completed or the calculated loss value of the medical language model no longer decreases, the training of the medical language model is completed, and a trained medical language model is obtained; in, Visual features of images representing the data in the model’s sample training set, A text description of an image representing the data in the model’s training set, Representing the model roles and capabilities of the medical large language model, Indicates the user tag. Indicates the report generation requirements of the user input medical large language model, Represents a helper tag, A stop mark indicating that the large medical language model has finished responding to the user's request for generating a report on the large medical language model.

5. The method for generating Chinese medical reports based on a pre-trained large model according to claim 1, characterized in that: In step S3, according to the global features of the X-ray medical image obtained by encoding, a medical report corresponding to an X-ray medical image similar to the X-ray medical image is retrieved from a preset image memory storage and a medical report file to be queried, including: Normalizing the global features of the X-ray medical image encoded by the medical vision encoder to obtain a global feature vector of the X-ray medical image; Calculate the square of the Euclidean distance between the global feature vector of the processed X-ray medical image and each feature vector in the preset image memory storage, and sort each feature vector in the image memory storage from small to large according to the calculated square of the Euclidean distance, and select the feature vector in the image memory storage with the smallest square of the Euclidean distance calculated from the global feature vector of the X-ray medical image in the sorting, and record it as the feature vector ; The feature vector in the selected image memory The corresponding historical X-ray medical image is an X-ray medical image similar to the X-ray medical image; Find feature vectors in the pre-set medical report files to be queried The corresponding medical report, the found medical report is a medical report of an X-ray medical image similar to the X-ray medical image.

6. The method for generating Chinese medical reports based on a pre-trained large model according to claim 5, characterized in that: The method for constructing the preset image memory storage device includes: Collection of historical X-ray medical images; Use a medical visual encoder to encode the collected historical X-ray medical images to obtain the global features of each historical X-ray medical image; Normalizing the global features of each historical X-ray medical image to obtain a global feature vector of each historical X-ray medical image; Setting an index number for the global feature vector of each historical X-ray medical image obtained; The global feature vector of each historical X-ray medical image and its corresponding index number are stored in a preset memory to obtain a preset image memory; The method for constructing the preset medical report file to be queried includes: Medical reports corresponding to the historical X-ray medical images are collected, and the medical reports and the index numbers of the global feature vectors of the corresponding historical X-ray medical images are stored in a preset file to obtain a preset medical report file to be queried.

7. A Chinese medical report generation system based on a pre-trained large model, characterized in that: The system includes an image acquisition module, an instruction acquisition module and a report generation module; An image acquisition module, used to acquire X-ray medical images for generating reports; An instruction acquisition module is used to obtain the Chinese medical report generation requirements input by the user; A report generation module, used for importing the acquired X-ray medical image and the Chinese medical report generation requirements input by the user into a pre-trained large model to obtain a Chinese medical report of the X-ray medical image; the large model includes a medical visual encoder, a medical large language model and a similar report retrieval module; The medical visual encoder is used to encode the input X-ray medical image to obtain the visual features of the X-ray medical image, wherein the visual features include global features and local features; The similar report retrieval module is used to retrieve the medical report corresponding to the X-ray medical image similar to the X-ray medical image in the pre-set image memory storage and the medical report file to be queried according to the global features of the X-ray medical image obtained by encoding, and input the retrieved medical report into the medical large language model; the image memory storage stores the global feature vector of the historical X-ray medical image and its corresponding index number; the medical report file to be queried stores the medical report corresponding to the historical X-ray medical image and the index number of the global feature vector of the historical X-ray medical image; The medical large language model is used to generate a Chinese medical report for the X-ray medical image based on the visual features of the X-ray medical image obtained by the medical visual encoder and based on the input Chinese medical report generation requirements and the retrieved medical report.

8. The Chinese medical report generation system based on the pre-trained large model according to claim 7 is characterized in that: In the report generation module, according to the global features of the X-ray medical image obtained by encoding, a medical report corresponding to an X-ray medical image similar to the X-ray medical image is retrieved from a preset image memory storage and a medical report file to be queried, including: Normalizing the global features of the X-ray medical image encoded by the medical vision encoder to obtain a global feature vector of the X-ray medical image; Calculate the square of the Euclidean distance between the global feature vector of the processed X-ray medical image and each feature vector in the preset image memory storage, and sort each feature vector in the image memory storage from small to large according to the calculated square of the Euclidean distance, and select the feature vector in the image memory storage with the smallest square of the Euclidean distance calculated from the global feature vector of the X-ray medical image in the sorting, and record it as the feature vector ; The feature vector in the selected image memory The corresponding historical X-ray medical image is an X-ray medical image similar to the X-ray medical image; Find feature vectors in the pre-set medical report files to be queried The corresponding medical report, the found medical report is a medical report of an X-ray medical image similar to the X-ray medical image.

9. The Chinese medical report generation system based on the pre-trained large model according to claim 8 is characterized in that: The method for constructing the preset image memory storage device includes: Collection of historical X-ray medical images; Use a medical visual encoder to encode the collected historical X-ray medical images to obtain the global features of each historical X-ray medical image; Normalizing the global features of each historical X-ray medical image to obtain a global feature vector of each historical X-ray medical image; Setting an index number for the global feature vector of each historical X-ray medical image obtained; The global feature vector of each historical X-ray medical image and its corresponding index number are stored in a preset memory to obtain a preset image memory; The method for constructing the preset medical report file to be queried includes: Medical reports corresponding to the historical X-ray medical images are collected, and the medical reports and the index numbers of the global feature vectors of the corresponding historical X-ray medical images are stored in a preset file to obtain a preset medical report file to be queried.

10. A terminal, characterized in that: The terminal includes a memory and a processor; A memory, used for storing a Chinese medical report generation program based on a pre-trained large model; A processor is used to implement the Chinese medical report generation method based on a pre-trained large model described in any one of claims 1-6 when executing the Chinese medical report generation program based on a pre-trained large model.

Images