Diagnosis report generation method and device, storage medium and electronic equipment

Abstract

The invention discloses a diagnosis report generation method and device, a storage medium and electronic equipment, and the method comprises the steps: firstly obtaining a medical image of a user, and then determining a disease analysis result of the medical image through a disease analysis model based on the medical image. Finally, the disease analysis result and the medical image or the image features of the medical image are input into a diagnosis report generation model, a diagnosis report, generated by the diagnosis report generation model, of the user is obtained, and the diagnosis report generation model is obtained by training a large language model based on the sample medical image. The diagnosis report is generated according to the determined disease analysis result, so that the immediate phenomenon can be reduced, and the accuracy of the medical report is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a diagnostic report generation method, device, storage medium and electronic device. Background Art

[0002] With the development of medical technology, users can use Magnetic Resonance Imaging (MRI) equipment to perform brain examinations and obtain multiple sequences of medical images, such as T1, T2, T2-Fluid Inversion Recovery Attenuation Sequence (T2-FLAIR), Diffusion Weighted Imaging (DWI), Apparent Diffusion Coefficient (ADC), etc. Doctors can use multiple sequences of medical images to determine whether the user has a brain disease and write a diagnosis report.

[0003] In order to improve the diagnostic efficiency of doctors, the diagnostic report generation model can be trained through deep learning methods, so that the trained diagnostic report generation model can generate diagnostic reports based on medical images. However, the accuracy of diagnostic reports generated by the currently trained diagnostic report generation model is low, such as the phenomenon of making something out of nothing in the diagnostic report.

[0004] Therefore, how to improve the accuracy of the diagnostic reports generated by the diagnostic report generation model is an urgent problem to be solved.

[0005] Based on this, this specification provides a method for generating a diagnostic report. Summary of the invention

[0006] The present specification provides a diagnostic report generation method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a method for generating a diagnostic report, including:

[0009] Obtain the user's medical images;

[0010] Based on the medical image, determine the disease analysis result of the medical image through a disease analysis model;

[0011] The disease analysis result and the medical image or the image feature of the medical image are input into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model. The diagnosis report generation model is obtained by training a large language model based on sample medical images.

[0012] Optionally, based on the medical image, determining the disease analysis result of the medical image through a disease analysis model specifically includes:

[0013] Determining image features of the medical image, wherein the image features are obtained through an image feature extraction model or a disease analysis model;

[0014] Based on the image features, determining the disease analysis result of the medical image through the disease analysis model;

[0015] Inputting the disease analysis result and the medical image or the image feature of the medical image into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model specifically includes:

[0016] Inputting the disease analysis result and the medical image into a diagnosis report generation model, the diagnosis report generation model extracts image features of the medical image, and generates a diagnosis report for the user based on the disease analysis result and the image features; or

[0017] The disease analysis result and the image features obtained by the image feature extraction model or the disease analysis model are input into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model.

[0018] Optionally, the disease analysis model includes any one of an image detection model, an image segmentation model, an image recognition model, a regression model, and a disease classification model;

[0019] Based on the image features, the disease analysis result of the medical image is determined by the disease analysis model, specifically including:

[0020] Performing image detection on the medical image based on the image features through an image detection model to obtain a detection result, and determining the detection result as a disease analysis result of the medical image; or

[0021] Segment the medical image based on the image features using an image segmentation model to obtain a segmentation result, and determine the segmentation result as a disease analysis result of the medical image; or

[0022] Using an image recognition model, based on the image features, the medical image is recognized to obtain a recognition result, and the recognition result is determined as a disease analysis result of the medical image; or

[0023] Determine the regression result of the medical image based on the image features through a regression model, and determine the regression result as the disease analysis result of the medical image; or

[0024] The medical image is classified into disease categories based on the image features through a disease classification model to obtain a disease classification result, which is then determined as a disease analysis result of the medical image.

[0025] Optionally, when the disease analysis model is a disease classification model, determining the disease analysis result of the medical image based on the image feature through the disease analysis model specifically includes:

[0026] Inputting the medical image into a prompt model to obtain prompt information of the medical image output by the prompt model, wherein the prompt model includes any one of an image detection model, an image segmentation model, an image recognition model, and a regression model;

[0027] Based on the prompt information and the image features, the disease classification result output by the disease classification model is obtained through the disease classification model as the disease analysis result.

[0028] Optionally, before obtaining the disease classification result output by the disease classification model based on the prompt information and the image features, the method further includes:

[0029] Determine a text report of the prompt information according to a preset text conversion template;

[0030] The text report is determined as prompt information.

[0031] Optionally, the method further comprises:

[0032] Acquiring preset disease information, wherein the preset disease information includes at least one of disease category, disease name, and disease symptoms;

[0033] The preset disease information, the disease analysis result and the medical image or the image feature of the medical image are input into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model.

[0034] Optionally, the medical image is classified into disease categories based on the image features by using a disease classification model to obtain a disease classification result, which specifically includes:

[0035] Inputting the acquired preset disease information into a text feature extraction model to obtain text features of the preset disease information output by the text feature extraction model;

[0036] The disease classification model is used to classify the medical image based on the text features and the image features to obtain a disease classification result.

[0037] Optionally, training the disease analysis model specifically includes:

[0038] Obtain sample diagnostic reports and sample medical images;

[0039] Based on the sample medical images, obtaining prediction analysis results and sample image features through the disease analysis model;

[0040] The disease analysis model is trained based on the prediction analysis results, the sample image features and the sample diagnosis report.

[0041] Optionally, training the diagnosis report generation model specifically includes:

[0042] Acquire a medical language big model, wherein the medical language big model is obtained by pre-training a general language big model based on a medical text set and a medical image set;

[0043] Inputting the sample image features and the prediction analysis results into the medical language big model, and the medical language big model outputs a prediction diagnosis report of the sample medical image;

[0044] According to the sample diagnosis report and the predicted diagnosis report, the medical language large model is fine-tuned and trained, and the fine-tuned medical language large model is used as a diagnosis report generation model.

[0045] This specification provides a diagnostic report generating device, the device comprising:

[0046] A medical image acquisition module, used to acquire the user's medical images;

[0047] A disease analysis result determination module, used to determine the disease analysis result of the medical image based on the medical image through a disease analysis model;

[0048] A diagnosis report generation module is used to input the disease analysis results and the medical images or the image features of the medical images into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model. The diagnosis report generation model is obtained by training a large language model based on sample medical images.

[0049] Optionally, the disease analysis result determination module is specifically used to determine the image features of the medical image, wherein the image features are obtained through an image feature extraction model or a disease analysis model; based on the image features, the disease analysis results of the medical image are determined through the disease analysis model; the diagnosis report generation module is specifically used to input the disease analysis results and the medical image into a diagnosis report generation model, the diagnosis report generation model extracts the image features of the medical image, and generates a diagnosis report for the user based on the disease analysis results and the image features; or input the disease analysis results and the image features obtained through the image feature extraction model or the disease analysis model into a diagnosis report generation model to obtain a diagnosis report for the user generated by the diagnosis report generation model.

[0050] Optionally, the disease analysis model includes any one of an image detection model, an image segmentation model, an image recognition model, a regression model, and a disease classification model; the disease analysis result determination module is specifically used to, through the image detection model, based on the image features, perform image detection on the medical image to obtain a detection result, and determine the detection result as the disease analysis result of the medical image; or through the image segmentation model, based on the image features, segment the medical image to obtain a segmentation result, and determine the segmentation result as the disease analysis result of the medical image; or through the image recognition model, based on the image features, identify the medical image to obtain a recognition result, and determine the recognition result as the disease analysis result of the medical image; or through the regression model, based on the image features, determine the regression result of the medical image, and determine the regression result as the disease analysis result of the medical image; or through the disease classification model, based on the image features, perform disease classification on the medical image to obtain a disease classification result, and determine the disease classification result as the disease analysis result of the medical image.

[0051] Optionally, when the disease analysis model is a disease classification model, the disease analysis result determination module is specifically used to input the medical image into a prompt model to obtain prompt information of the medical image output by the prompt model, and the prompt model includes any several of an image detection model, an image segmentation model, an image recognition model, and a regression model; based on the prompt information and the image features, the disease classification model is used to obtain the disease classification result output by the disease classification model as the disease analysis result.

[0052] Optionally, the device further comprises:

[0053] A text conversion module is used to determine a text report of the prompt information according to a preset text conversion template before obtaining the disease classification result output by the disease classification model based on the prompt information and the image features through the disease classification model; and determine the text report as the prompt information.

[0054] Optionally, the diagnostic report generation module is specifically used to obtain preset disease information, which includes at least one of disease category, disease name, and disease symptoms; input the preset disease information, the disease analysis results, and the medical image or the image features of the medical image into a diagnostic report generation model to obtain a diagnostic report for the user generated by the diagnostic report generation model.

[0055] Optionally, the disease analysis result determination module is specifically used to input the acquired preset disease information into a text feature extraction model to obtain text features of the preset disease information output by the text feature extraction model; through the disease classification model, based on the text features and the image features, the medical image is classified into disease categories to obtain disease classification results.

[0056] Optionally, the device further comprises:

[0057] The disease analysis model training module is used to obtain sample diagnosis reports and sample medical images; based on the sample medical images, obtain the prediction analysis results and sample image features through the disease analysis model; based on the prediction analysis results, the sample image features and the sample diagnosis report, train the disease analysis model.

[0058] Optionally, the device further comprises:

[0059] The diagnosis report generation model training module is used to obtain a medical language big model, wherein the medical language big model is obtained by pre-training a general language big model based on a medical text set and a medical image set; the sample image features and the prediction analysis results are input into the medical language big model, and the medical language big model outputs a predictive diagnosis report of the sample medical image; the medical language big model is fine-tuned according to the sample diagnosis report and the predicted diagnosis report, and the fine-tuned medical language big model is used as the diagnosis report generation model.

[0060] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned diagnostic report generation method is implemented.

[0061] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned diagnostic report generating method when executing the program.

[0062] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0063] It can be seen from the diagnostic report generation method provided in this specification that the method first obtains the user's medical image, and then determines the disease analysis result of the medical image based on the medical image through the disease analysis model. Finally, the disease analysis result and the medical image or the image feature of the medical image are input into the diagnostic report generation model to obtain the diagnostic report of the user generated by the diagnostic report generation model. The diagnostic report generation model is obtained by training a large language model based on sample medical images. Generating a diagnostic report by determining the disease analysis result can reduce the phenomenon of making something out of nothing and improve the accuracy of the medical report. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0065] Figure 1 A flowchart of a method for generating a diagnostic report provided in this specification;

[0066] Figure 2 A schematic diagram of a process for determining disease analysis results provided in this manual;

[0067] Figure 3 A flowchart of generating disease analysis results when the disease analysis model provided in this specification is a disease classification model;

[0068] Figure 4 A flowchart of the determination text report provided for this specification;

[0069] Figure 5 A flowchart of extracting text features provided in this specification;

[0070] Figure 6 A flowchart of another text feature extraction method provided in this specification;

[0071] Figure 7 A schematic diagram of the text feature fusion process provided for this manual;

[0072] Figure 8 A flowchart of the training image feature extraction model provided in this specification;

[0073] Fig. 9 A flowchart of a disease classification model training provided in this manual;

[0074] Fig.10 A schematic diagram of another process for training a disease analysis model provided in this specification;

[0075] Fig.11 A flowchart of the training diagnostic report generation model provided in this manual;

[0076] Fig.12 A schematic diagram of a diagnostic report generating device provided in this specification;

[0077] Fig.13 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0079] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0080] The diagnosis of brain diseases usually relies on medical images generated by magnetic resonance imaging equipment. Magnetic resonance imaging equipment can generate medical images of different sequences. Medical images of different sequences have more image information, which helps to improve the accuracy of the diagnostic reports written by doctors. However, the diagnostic report is obtained only by doctors observing and writing medical images of different sequences, which is inefficient and has high requirements on the ability of doctors. Therefore, the diagnostic report generation model can be trained by deep learning methods to improve the efficiency of diagnostic report generation, but the accuracy of the generated diagnostic report cannot be guaranteed. Therefore, this specification provides a diagnostic report generation method.

[0081] The execution subject of this specification may be an image processing device that can process medical images, or other electronic devices that can process medical images, such as servers, distributed clusters, personal computers (PCs), etc. This specification does not limit this. For the sake of convenience, this specification uses the image processing device as the execution subject for description. Various models are deployed in the image processing device, or it can communicate with electronic devices that deploy various models to obtain a diagnosis report. This specification uses various models deployed in the image processing device for description.

[0082] Figure 1 A flowchart of a diagnostic report generation method provided in this specification specifically includes the following steps:

[0083] S100: Obtaining medical images of the user.

[0084] In one or more embodiments of the present specification, the diagnosis report is obtained based on the medical images of the user. Therefore, in order to generate the diagnosis report of the user, the image processing device may first obtain the medical images of the user.

[0085] Specifically, in response to the operation of the magnetic resonance imaging device by doctors and other medical personnel, the magnetic resonance imaging device generates several sequences of medical images, and sends the medical images to the image processing device. The image processing device receives the medical images sent by the magnetic resonance imaging device to obtain the user's medical images.

[0086] It should be noted that this specification does not limit the types of sequences of medical images obtained, and the types of sequences may include at least one of T1, T2, T2-FLAIR, DWI, and ADC. However, medical images of multiple sequences help to improve the accuracy of the generated diagnostic report. For example, for cerebral infarction diseases, medical images of the DWI sequence can be used to quickly identify the infarct area, and the type of infarction can be determined by combining the medical images of the ADC sequence. Medical images of the T2 sequence and the T2-FLAIR sequence can also be used to observe accompanying changes such as cerebral edema, thereby obtaining a diagnostic report with a higher accuracy rate.

[0087] S102: Based on the medical image, determine the disease analysis result of the medical image through a disease analysis model.

[0088] Specifically, the image processing device can first determine the image features of the medical image, and then, based on the image features, determine the disease analysis results of the medical image through the disease analysis model. By extracting image features, image information of different dimensions in the medical image can be obtained to obtain more accurate disease analysis results, thereby improving the accuracy of the diagnosis report.

[0089] The image processing device can determine the image features of the medical image through various methods. Specifically, if the disease analysis model has the function of extracting image features, the image processing device can input the medical image into the disease analysis model, and the disease analysis model extracts the image features of the medical image and determines the disease analysis results of the medical image based on the image features.

[0090] Figure 2 A flowchart for determining the results of disease analysis provided in this manual, such as Figure 2 shown.

[0091] If the disease analysis model does not have the function of extracting image features, the image processing device can input the medical image into the image feature extraction model to obtain the image features of the medical image output by the image feature extraction model. Then the image features are input into the disease analysis model to obtain the disease analysis results of the medical image. It can also be understood that the image features are obtained through the image feature extraction model or the disease analysis model. Among them, the image feature extraction model can be a model based on the Transformer architecture (Vision Transformer, ViT), or it can be other types of models that can be used to extract image features, and this specification does not limit this.

[0092] S104: Input the disease analysis result and the medical image or the image feature of the medical image into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model, wherein the diagnosis report generation model is obtained by training a large language model based on sample medical images.

[0093] In one or more embodiments of the present specification, after determining the disease analysis result, the image processing device may input the disease analysis result and the medical image or the image features of the medical image into a diagnosis report generation model to obtain a diagnosis report for the user generated by the diagnosis report generation model.

[0094] Specifically, the image processing device may input the disease analysis result and the image features obtained by the image feature extraction model or the disease analysis model into the diagnosis report generation model to obtain the diagnosis report of the user generated by the diagnosis report generation model. The disease analysis result and the medical image may also be input into the diagnosis report generation model, and the diagnosis report generation model extracts the image features of the medical image, and generates the diagnosis report of the user according to the disease analysis result and the image features.

[0095] Generally speaking, the accuracy of the diagnosis report obtained directly from medical images is low, but this specification obtains the disease analysis results first, and then obtains the diagnosis report based on the disease analysis results and the medical image or the image features of the medical image, which can improve the accuracy of the diagnosis report. This is because the disease analysis results have a prompting effect on the diagnosis report generation model, so as to reduce the probability of the diagnosis report generated by the diagnosis report generation model being fabricated, thereby improving the accuracy of the diagnosis report.

[0096] based on Figure 1 The diagnostic report generation method shown in the figure first obtains the medical image of the user, and then determines the disease analysis result of the medical image based on the medical image through the disease analysis model. Finally, the disease analysis result and the medical image or the image feature of the medical image are input into the diagnostic report generation model to obtain the diagnostic report of the user generated by the diagnostic report generation model. The diagnostic report generation model is obtained by training a large language model based on sample medical images. Generating a diagnostic report by determining the disease analysis result can reduce the phenomenon of making something out of nothing and improve the accuracy of the medical report.

[0097] For step S102, the disease analysis model includes any one of an image detection model, an image segmentation model, an image recognition model, a regression model, and a disease classification model, and can also be an image processing model that can identify any number of diseases. It is understandable that if the model can analyze medical images or image features of medical images, and the output results of the model have a prompting effect on the diagnosis report generation model, then the model can be a disease analysis model. Then, the image processing device can perform image detection on the medical image based on the image features through different types of disease analysis models to obtain disease analysis results.

[0098] Specifically, when the disease analysis model is an image detection model, the image detection model can be used to perform image detection on the medical image based on the image features to obtain a detection result, and the detection result is determined as the disease analysis result of the medical image. In other words, the image detection model can output the detection result of the abnormal area in the medical image based on the image features, such as detecting the presence of a lesion area in the medical image and marking it, so as to subsequently prompt the diagnosis report generation model to focus on the lesion area to generate a diagnosis report. Among them, the disease analysis model can be a Transformer network model, such as ViT, Bert, etc., and this specification does not limit this.

[0099] When the disease analysis model is an image segmentation model, the image segmentation model can be used to segment the medical image based on the image features to obtain a segmentation result, and the segmentation result is determined as the disease analysis result of the medical image. For example, each pixel of the medical image is classified according to the image features to obtain the classification results of the lesion area and the non-lesion area. According to the classification results, the medical image is segmented to obtain the segmentation result, so that the diagnosis report generation model can process each segmented area, obtain more local information, and be more targeted.

[0100] When the disease analysis model is an image recognition model, the medical image can be recognized based on the image features by the image recognition model to obtain a recognition result, which is determined as the disease analysis result of the medical image. For example, the recognition result is text in the medical image that does not include the lesion area.

[0101] When the disease analysis model is a regression model, the regression result of the medical image can be determined based on the image features through the regression model, and the regression result can be determined as the disease analysis result of the medical image. The regression model can be used to predict the probability of disease occurrence, the degree of disease, etc. For example, the regression result can predict that the user does not currently have disease A, but is developing a trend towards having disease A.

[0102] When the disease analysis model is a disease classification model, the disease classification model can be used to classify the medical image based on the image features to obtain a disease classification result, which is determined as the disease analysis result of the medical image. For example, the disease classification result is that the disease suffered by the user corresponding to the medical image is disease A.

[0103] Figure 3 When the disease analysis model provided in this specification is a disease classification model, a flowchart of generating disease analysis results is shown in FIG. Figure 3 shown.

[0104] Continue to use Figure 2 For example, in addition to inputting medical images into an image feature extraction model to obtain image features, the image processing device can also input the medical images into a prompt model to obtain prompt information of the medical images output by the prompt model. Based on the prompt information and the image features, the disease classification model is used to obtain the disease classification results output by the disease classification model as the disease analysis results. Among them, the prompt model includes any of the image detection model, image segmentation model, image recognition model, and regression model, and the specific model type can be UNet, V-Net, ResNet, etc.

[0105] That is to say, the image processing device can input the medical image into any one or more of the image detection model, image segmentation model, image recognition model, and regression model to obtain some prompt information. Then, some prompt information and image features are input into the disease classification model to obtain the disease classification result. Among them, the prompt information may include any one or more of the detection result, segmentation result, recognition result, and regression result.

[0106] For example, if the prompt information is a test result, and the test result includes a lesion area, the disease classification model can analyze the lesion area, determine the disease category corresponding to the lesion area, and output it. The prompt information can be used to prompt the disease classification model, so that the disease classification result output by the disease classification model is more accurate, thereby improving the accuracy and reliability of the diagnosis report output by the diagnosis report generation model.

[0107] It should be noted that the prompt model can be understood as a small model, that is, a lightweight neural network model, with a parameter level of several thousand to several million, while the parameter level of the diagnosis report generation model is at least in the hundreds of millions. Compared with the diagnosis report generation model, which is a large model, the parameter level of the prompt model is much smaller than that of the diagnosis report generation model.

[0108] Furthermore, Figure 4 A flowchart of determining the text report provided for this manual, such as Figure 4 shown.

[0109] The image processing device can also determine the text report of the prompt information according to the preset text conversion template, and determine the text report as the prompt information. Then, based on the text report and image features, the disease classification result is determined through the disease classification model. Among them, the preset text conversion template can be determined according to the sample diagnosis report, and the text report may include image information, such as what is seen in the image. Generally speaking, the sample diagnosis report will describe the sequence type of medical images and the information of each area of ​​the medical image, such as the lesion area. The sample diagnosis report may include the location information of the lesion, the signal level of the lesion area, the size of the lesion, whether the boundary is clear, the morphology and other image information.

[0110] For example, the text report of a medical image is: FLAIR sequence image, location: left frontal lobe, signal intensity: high signal, morphology: irregular, size: about 1.5cm×1.2cm, boundary: clear, lesion: considered to be a glioma. T2-weighted sequence image, location: right basal ganglia, signal intensity: high signal, morphology: round, size: about 0.8cm×0.6cm, boundary: clear, lesion: considered to be perivascular space dilation. T1-weighted sequence image, location: no obvious abnormality. DWI sequence image, location: no obvious abnormality. Among them, the lesion: considered to be a glioma is only a prediction result and is not the result in the diagnosis report.

[0111] According to the preset text conversion template, the output results of several small models are converted into a clinical report format to obtain a text description, which can not only make the form of prompt information more standardized, but also serve as a supplementary text for the disease classification model, thereby enabling the disease classification model to obtain more accurate classification results.

[0112] Figure 5 The flowchart of extracting text features provided in this manual is as follows: Figure 5 shown.

[0113] In addition, the image processing device can also input the text report into the text feature extraction model, and then obtain the text features of the text report output by the text feature extraction model. Based on the text features and image features, the disease classification result is obtained through the disease classification model. Specifically, the linear layer can be used to obtain the embedding vector of the image feature and the embedding vector of the text feature. The disease classification model uses a multi-head cross-attention mechanism to use each disease category that can be classified by itself as a key and value, aligns the embedding vector of the image feature with the embedding vector of the text feature, and performs multi-head query to obtain the attention vector, which is then superimposed with the image feature, and the disease classification is performed through the fully connected layer to obtain the disease classification result. Among them, the text feature extraction model can be obtained based on a pre-trained Bert network, or it can be obtained based on other neural networks. This specification does not limit this, as long as the text feature extraction model has the ability to extract text features.

[0114] Extracting text features from text reports can obtain text information of more dimensions. The increase of text information can reduce the difference in the amount of input text information during image and text feature alignment and disease classification, which is conducive to improving the accuracy of disease classification results and thus improving the accuracy of diagnosis reports. It is understandable that if Figure 5 In the flowchart shown, the structure that uses both the text feature extraction model and the image feature extraction model is a dual-tower structure. The dual-tower structure can process data of two modes at the same time, and has a high processing efficiency.

[0115] Figure 6 Another flowchart of text feature extraction provided in this specification is as follows: Figure 6 shown.

[0116] If the image processing device does not obtain prompt information, it can also obtain preset disease information, which includes at least one of disease category, disease name, and disease symptoms. The obtained preset disease information is input into the text feature extraction model to obtain the text features of the preset disease information output by the text feature extraction model. Through the disease classification model, based on the text features of the preset disease information and the image features, the medical image is classified into a disease classification result. The preset disease information can be set as needed, and this manual does not limit this. For example, disease symptoms include pathological manifestations such as white matter hyperintensities, acute infarction, and lacunar infarction.

[0117] By obtaining preset disease information, it is possible to obtain only disease categories included in the preset disease information when classifying diseases later, reduce the probability of outputting other disease categories, avoid problems of making something out of nothing and hallucinations, and thus improve the accuracy of the diagnosis report generation model.

[0118] Figure 7 The text feature fusion process diagram provided for this manual is as follows: Figure 7 shown.

[0119] If the prompt information is obtained, the image processing device can also fuse the text features of the preset disease information and the text features of the text report. In order to distinguish the text features of the preset disease information and the text features of the text report, this specification determines the text features of the preset disease information as the first text features, and determines the text features of the text report as the second text features. Specifically, the image processing device inputs the acquired preset disease information and the text report into the text feature extraction model, and obtains the first text features of the preset disease information and the second text features of the text report output by the text feature extraction model. According to the first text feature and the second text feature, the fused text feature is determined. Then, based on the fused text feature and the image feature, the disease classification result output by the disease classification model is obtained through the disease classification model. This specification does not limit the method of text feature fusion, which can be direct feature splicing, or it can be dimensionality reduction of the first text feature and the second text feature, and then text feature splicing. It can also be fused through an attention mechanism and a shared fully connected layer.

[0120] For step S104, the image processing device can also obtain preset disease information, input the preset disease information, the disease analysis result and the medical image or the image feature of the medical image into the diagnosis report generation model, and obtain the diagnosis report of the user generated by the diagnosis report generation model.

[0121] During the diagnosis report generation stage, the preset disease information and disease classification results are used as prompts to provide the diagnosis report generation model with generation templates and restrictions on the generated content, thereby reducing the illusion of the diagnosis report generation model and generating a professional structured report.

[0122] It should be noted that the diagnostic report generated by this manual is a reference for doctors and other medical staff. Doctors and other medical staff can refer to the diagnostic report to obtain the final diagnostic conclusion.

[0123] This specification also provides a method for training a disease analysis model. The executor of the model training can be a computing device with computing capabilities such as a server. This specification describes the method for model training using the server as the executor.

[0124] The server first obtains a sample diagnosis report and a sample medical image, wherein the sample diagnosis report includes information such as age, gender, medical history, imaging report, and diagnosis opinion. Based on the sample medical image, the disease analysis model is used to obtain the prediction analysis results and sample image features. Based on the prediction analysis results, the sample image features, and the sample diagnosis report, the disease analysis model is trained.

[0125] It is understandable that, based on the aforementioned description of the steps of using the trained disease analysis model, image features can be obtained through the image feature extraction model or the disease analysis model. Then, in the process of training the disease analysis model, sample image features can also be obtained through the image feature extraction model or the disease analysis model. In this specification, the sample image features are first obtained through the image feature extraction model.

[0126] In order to obtain more accurate sample image features, the server may first train the image feature extraction model. After the training is completed, the server may input the sample medical image into the image feature extraction model to obtain the sample image features output by the image feature extraction model. Next, the server may also input the sample medical image and the sample image features into the disease analysis model to obtain the prediction analysis results of the sample medical image output by the disease analysis model. Of course, the image feature extraction model may also be trained while the disease analysis model is being trained.

[0127] When training the image feature extraction model, the server can be trained in conjunction with the text features of the sample diagnostic report. Specifically, the server can input the sample diagnostic report into the text feature extraction model to obtain the label text features of the sample diagnostic report output by the text feature extraction model, and input the sample medical image into the image feature extraction model to obtain the sample image features output by the image feature extraction module. Then, the sample medical images and sample diagnostic reports of the same group are used as positive sample pairs, and the sample medical images and sample diagnostic reports of different groups are used as negative sample pairs. The association between the sample medical images and the sample diagnostic reports is learned through methods such as comparative learning, so that the similarity of the positive sample pairs is the highest. Finally, the features extracted by the image feature extraction model integrate the feature information in the text description to obtain image features with higher accuracy. Among them, the same group means that the sample diagnostic reports of the group are obtained based on the sample medical images of the group.

[0128] Figure 8 The flowchart of the training image feature extraction model provided in this manual is as follows: Figure 8 shown.

[0129] The image feature extraction model includes an image encoder and a linear layer, and the text feature extraction model includes a text encoder and a linear layer. The server can input sample medical images of sequences such as T1, T2, T2-FLAIR, DWI, ADC, etc. into the image encoder of the image feature extraction model, and then into the linear layer to obtain a sample image vector of (N, D). At the same time, the sample diagnosis report is input into the text encoder in the text feature extraction model, and then into the linear layer to obtain a label text feature of (N, D). The server can use loss functions such as cosine similarity loss for feature alignment and train the image feature extraction model.

[0130] refer to Figure 5 As shown in the flowchart, the server can also train the disease analysis model in conjunction with the prompt model and the text feature extraction model. At this time, the disease analysis model can be a disease classification model.

[0131] Specifically, the server inputs the sample medical image into the prompt model to obtain sample prompt information. Then, the sample prompt information is converted into text form using a preset text template to obtain text prompt information. Then, the text prompt information is input into the text feature extraction model to obtain sample text features. The sample text features and sample image features are input into the disease classification model to obtain the predicted classification results. The disease classification model is trained based on the predicted classification results and the labeled disease classification results in the sample diagnosis report.

[0132] refer to Figure 7 As shown in the flowchart, the server can also train the disease classification model based on preset disease information and text prompt information.

[0133] Fig. 9 A flowchart of a disease classification model training provided in this manual, such as Fig. 9 shown.

[0134] Specifically, in order to distinguish the sample text features of the preset disease information and the sample text features of the text prompt information, this specification determines the sample text features of the preset disease information as the first sample text features, and determines the sample text features of the text prompt information as the second sample text features. First, the preset disease information and the sample text features are respectively input into the text encoder, and then respectively input into the linear layer to obtain the first sample text features and the second sample text features, and the first sample text features and the second sample text features are fused to obtain the sample fusion features. Through the multi-head cross attention mechanism, the relationship between the sample image features and the sample fusion features is learned to generate an attention vector. Then it is superimposed with the sample image features, and the disease is classified through the fully connected layer and the Sigmoid activation function to obtain the predicted disease classification result. Through the binary cross entropy loss and the hierarchical constraint loss, the disease classification model is trained based on the predicted disease classification results and the labeled disease classification results in the sample diagnosis report. This specification does not limit the type of loss function, nor does it limit the use of which attention mechanism. The aforementioned loss function and multi-head cross attention mechanism are only an example in this specification.

[0135] In addition, when training the disease analysis model, the image feature extraction model can also be fine-tuned. Specifically, some parameters of the image feature extraction model can be adjusted to make the image feature extraction model focus on preset disease categories.

[0136] This specification further illustrates how to obtain sample image features through a disease analysis model. Fig.10 A flowchart of another disease analysis model training method provided in this specification is as follows: Fig.10 As shown, the disease analysis model is taken as a disease classification model as an example for explanation.

[0137] Since the sample image features are obtained through the disease classification model, it can be understood that the disease classification model includes an image feature extraction module for extracting image features. Then, the server can input the sample medical image into the image feature extraction module in the disease analysis model to obtain the sample image features, and then determine the predicted classification results based on the sample image features and output them. The disease classification model is trained based on the predicted classification results, the preset loss function and the labeled disease classification results in the sample diagnosis report.

[0138] Since the disease classification model includes an image feature extraction module, in order to improve the accuracy of the image features extracted by the image feature extraction module, the server can not only train the disease classification model, but also train the image feature extraction module. The aforementioned steps of training the image feature extraction model can also be applied to training the image feature extraction module in the disease analysis model. Specifically, the server can input the sample diagnosis report into the text feature extraction model to obtain the sample text features, and input the sample text features into the disease classification model so that the disease classification model uses the sample text features to train the image feature extraction module. When obtaining the sample image features through the image feature extraction model, it is also stated that the disease classification model is trained using preset disease information, etc., so that the aforementioned process also exists in obtaining the sample image features through the disease classification model, which will not be repeated here.

[0139] This specification also provides a method for training a diagnostic report generation model. When training the diagnostic report generation model, it can be jointly trained with the disease analysis model. The disease analysis model can also be trained first, and after the training is completed, the diagnostic report generation model is trained. This specification does not limit this.

[0140] Fig.11 The flowchart of the training diagnostic report generation model provided in this manual is as follows: Fig.11 shown.

[0141] When training the diagnosis report generation model, the server may first obtain a large medical language model, wherein the large medical language model is obtained by pre-training a general language large model based on a medical text set and a medical image set. The large medical language model may process sample medical images and extract image features of the sample medical images. Afterwards, the sample image features and the prediction analysis results are input into the large medical language model, and the large medical language model outputs a predictive diagnosis report of the sample medical image. The server may fine-tune the large medical language model based on loss functions such as BLEU (Bilingual Evaluation Understudy, BLEU) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation, ROUGE), according to the sample diagnosis report and the predicted diagnosis report, and use the fine-tuned large medical language model as the diagnosis report generation model. Specifically, the parameters of the large medical language model may be fine-tuned through LoRA (Low-Rank Adaptation, LoRA), PEFT (Parameter-Efficient Fine-Tuning, PEFT), Prompt-Tuning, etc., and this specification does not limit this. Among them, Fig.11The contrast loss in can be the aforementioned cosine similarity contrast loss, or can be other contrast loss functions, which is not limited in this specification.

[0142] The above is a diagnostic report generation method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding diagnostic report generation device, such as Fig.12 As shown, the device comprises:

[0143] The medical image acquisition module 1200 is used to acquire the medical image of the user;

[0144] The disease analysis result determination module 1202 is used to determine the disease analysis result of the medical image based on the medical image by using a disease analysis model;

[0145] The diagnosis report generation module 1204 is used to input the disease analysis results and the medical images or the image features of the medical images into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model. The diagnosis report generation model is obtained by training a large language model based on sample medical images.

[0146] Optionally, the disease analysis result determination module 1202 is specifically used to determine the image features of the medical image, where the image features are obtained through an image feature extraction model or a disease analysis model; based on the image features, the disease analysis results of the medical image are determined through the disease analysis model; the diagnosis report generation module 1204 is specifically used to input the disease analysis results and the medical image into a diagnosis report generation model, the diagnosis report generation model extracts the image features of the medical image, and generates a diagnosis report for the user based on the disease analysis results and the image features; or input the disease analysis results and the image features obtained through the image feature extraction model or the disease analysis model into a diagnosis report generation model to obtain a diagnosis report for the user generated by the diagnosis report generation model.

[0147] Optionally, the disease analysis model includes any one of an image detection model, an image segmentation model, an image recognition model, a regression model, and a disease classification model; the disease analysis result determination module 1202 is specifically used to perform image detection on the medical image based on the image features through the image detection model to obtain a detection result, and determine the detection result as the disease analysis result of the medical image; or segment the medical image based on the image features through the image segmentation model to obtain a segmentation result, and determine the segmentation result as the disease analysis result of the medical image; or identify the medical image based on the image features through the image recognition model to obtain a recognition result, and determine the recognition result as the disease analysis result of the medical image; or determine the regression result of the medical image based on the image features through the regression model, and determine the regression result as the disease analysis result of the medical image; or classify the medical image based on the image features through the disease classification model to obtain a disease classification result, and determine the disease classification result as the disease analysis result of the medical image.

[0148] Optionally, when the disease analysis model is a disease classification model, the disease analysis result determination module 1202 is specifically used to input the medical image into a prompt model to obtain prompt information of the medical image output by the prompt model, and the prompt model includes any several of an image detection model, an image segmentation model, an image recognition model, and a regression model; based on the prompt information and the image features, the disease classification model is used to obtain the disease classification result output by the disease classification model as the disease analysis result.

[0149] Optionally, the device further comprises:

[0150] A text conversion module is used to determine a text report of the prompt information according to a preset text conversion template before obtaining the disease classification result output by the disease classification model based on the prompt information and the image features through the disease classification model; and determine the text report as the prompt information.

[0151] Optionally, the diagnostic report generation module 1204 is specifically used to obtain preset disease information, which includes at least one of disease category, disease name, and disease symptoms; input the preset disease information, the disease analysis results, and the medical image or the image features of the medical image into a diagnostic report generation model to obtain a diagnostic report for the user generated by the diagnostic report generation model.

[0152] Optionally, the disease analysis result determination module 1202 is specifically used to input the acquired preset disease information into a text feature extraction model to obtain text features of the preset disease information output by the text feature extraction model; through the disease classification model, based on the text features and the image features, the medical image is classified into disease categories to obtain disease classification results.

[0153] Optionally, the device further comprises:

[0154] The disease analysis model training module is used to obtain sample diagnosis reports and sample medical images; based on the sample medical images, obtain the prediction analysis results and sample image features through the disease analysis model; based on the prediction analysis results, the sample image features and the sample diagnosis report, train the disease analysis model.

[0155] Optionally, the device further comprises:

[0156] The diagnosis report generation model training module is used to obtain a medical language big model, wherein the medical language big model is obtained by pre-training a general language big model based on a medical text set and a medical image set; the sample image features and the prediction analysis results are input into the medical language big model, and the medical language big model outputs a predictive diagnosis report of the sample medical image; the medical language big model is fine-tuned according to the sample diagnosis report and the predicted diagnosis report, and the fine-tuned medical language big model is used as the diagnosis report generation model.

[0157] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides a method for generating diagnostic reports.

[0158] This manual also provides Fig.13 The structural diagram of the electronic device shown in FIG. Fig.13 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0159] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0160] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0161] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0162] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0163] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0169] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0171] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0173] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0174] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A method for generating a diagnostic report, characterized in that: The method comprises: Obtain the user's medical images; Based on the medical image, determine the disease analysis result of the medical image through a disease analysis model; The disease analysis result and the medical image or the image feature of the medical image are input into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model. The diagnosis report generation model is obtained by training a large language model based on sample medical images.

2. The method according to claim 1, characterized in that Based on the medical image, determining the disease analysis result of the medical image through a disease analysis model specifically includes: Determining image features of the medical image, wherein the image features are obtained through an image feature extraction model or a disease analysis model; Based on the image features, determining the disease analysis result of the medical image through the disease analysis model; Inputting the disease analysis result and the medical image or the image feature of the medical image into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model specifically includes: Inputting the disease analysis result and the medical image into a diagnosis report generation model, the diagnosis report generation model extracts image features of the medical image, and generates a diagnosis report for the user based on the disease analysis result and the image features; or The disease analysis result and the image features obtained by the image feature extraction model or the disease analysis model are input into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model.

3. The method according to claim 2, characterized in that The disease analysis model includes any one of an image detection model, an image segmentation model, an image recognition model, a regression model, and a disease classification model; Based on the image features, the disease analysis result of the medical image is determined by the disease analysis model, specifically including: Performing image detection on the medical image based on the image features through an image detection model to obtain a detection result, and determining the detection result as a disease analysis result of the medical image; or Segment the medical image based on the image features using an image segmentation model to obtain a segmentation result, and determine the segmentation result as a disease analysis result of the medical image; or Using an image recognition model, based on the image features, the medical image is recognized to obtain a recognition result, and the recognition result is determined as a disease analysis result of the medical image; or Determine the regression result of the medical image based on the image features through a regression model, and determine the regression result as the disease analysis result of the medical image; or The medical image is classified into disease categories based on the image features through a disease classification model to obtain a disease classification result, which is then determined as a disease analysis result of the medical image.

4. The method according to claim 2, characterized in that When the disease analysis model is a disease classification model, determining the disease analysis result of the medical image based on the image features through the disease analysis model specifically includes: Inputting the medical image into a prompt model to obtain prompt information of the medical image output by the prompt model, wherein the prompt model includes any one of an image detection model, an image segmentation model, an image recognition model, and a regression model; Based on the prompt information and the image features, the disease classification result output by the disease classification model is obtained through the disease classification model as the disease analysis result.

5. The method according to claim 4, characterized in that Based on the prompt information and the image features, before obtaining the disease classification result output by the disease classification model through the disease classification model, the method further includes: Determine a text report of the prompt information according to a preset text conversion template; The text report is determined as prompt information.

6. The method according to claim 1, characterized in that The method further comprises: Acquiring preset disease information, wherein the preset disease information includes at least one of disease category, disease name, and disease symptoms; The preset disease information, the disease analysis result and the medical image or the image feature of the medical image are input into a diagnosis report generation model to obtain a diagnosis report of the user generated by the diagnosis report generation model.

7. The method according to claim 3, characterized in that Using a disease classification model, based on the image features, the medical image is classified into disease categories to obtain a disease classification result, which specifically includes: Inputting the acquired preset disease information into a text feature extraction model to obtain text features of the preset disease information output by the text feature extraction model; The disease classification model is used to classify the medical image based on the text features and the image features to obtain a disease classification result.

8. The method according to claim 1, characterized in that Training the disease analysis model specifically includes: Obtain sample diagnostic reports and sample medical images; Based on the sample medical images, obtaining prediction analysis results and sample image features through the disease analysis model; The disease analysis model is trained based on the prediction analysis results, the sample image features and the sample diagnosis report.

9. The method according to claim 8, characterized in that Training the diagnostic report generation model specifically includes: Acquire a medical language big model, wherein the medical language big model is obtained by pre-training a general language big model based on a medical text set and a medical image set; Inputting the sample image features and the prediction analysis results into the medical language big model, and the medical language big model outputs a prediction diagnosis report of the sample medical image; According to the sample diagnosis report and the predicted diagnosis report, the medical language large model is fine-tuned and trained, and the fine-tuned medical language large model is used as a diagnosis report generation model.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images