Diagnosis result prediction method and device based on medical image description

Abstract

The invention provides a diagnosis result prediction method and device based on medical image description, and the method comprises the steps: carrying out the description information extraction of an obtained medical image report, and extracting image description information used for describing a brain MRI image in the medical image report; the method comprises the following steps: preprocessing image description information, and mapping the image description information into image description data; performing format specification processing on the image description data in combination with a data input format of the first disease prediction model, and converting the image description data into image description features; inputting the image description features into a disease result prediction layer of a first disease prediction model, and determining a result description vector of the medical image report; and inputting the result description vector into a result classification layer of the first disease prediction model, and determining a prediction diagnosis result of the medical image report. Therefore, accurate prediction of the diagnosis result can be realized only by using the image description information describing the brain MRI image, and a doctor can be assisted to improve the disease diagnosis efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for predicting diagnostic results based on medical image description. Background Art

[0002] With the increasing attention paid to human health and the rapid development of science and technology, the cycle of disease discovery and treatment has been gradually shortened. In the process of disease discovery, rapid and accurate disease diagnosis is the basis for proper treatment of the disease. Among them, brain diseases have the characteristics of high incidence and high risk, which makes the medical system have a stronger demand for rapid and accurate diagnosis of brain diseases.

[0003] Clinically, in order to more accurately determine the patient's brain disease, brain MRI images are usually taken, and then the radiologist writes the imaging report and obtains preliminary results. Finally, the clinician makes a final diagnosis by integrating the brain MRI images, image description information, and diagnostic results in the imaging report. Here, in terms of the diagnostic results, since the diagnostic results are the subjective diagnosis made by the doctor based on the brain MRI images and image description information, different doctors may have different diagnostic opinions even for the same brain MRI images and image description information, and in order to ensure the accuracy of the diagnostic results, doctors usually need to spend a lot of time on the diagnostic results, which slows down the diagnostic progress;

[0004] Existing "models" that can be used for disease diagnosis usually need to be combined with brain MRI images to achieve disease diagnosis. Since the "model" can only receive images of fixed size, the current image-based model has poor adaptability to MRI images and can only choose to discard redundant modalities or generate missing modalities to match the current "model", resulting in problems of information waste and inaccurate generation results. Summary of the invention

[0005] In view of this, the purpose of the present application is to provide a method and device for predicting diagnostic results based on medical image description, which can accurately predict diagnostic results by only using the image description information used to describe brain MRI images in the medical imaging report, thereby assisting doctors in improving the efficiency of disease diagnosis.

[0006] The present application embodiment provides a method for predicting a diagnosis result based on medical image description, and the method for predicting a diagnosis result includes:

[0007] Extracting description information from the acquired medical imaging report to extract image description information used to describe the brain MRI image in the medical imaging report;

[0008] By preprocessing the image description information, mapping the image description information into image description data;

[0009] In combination with the data input format of the first disease prediction model, the image description data is formatted and standardized, and the image description data is converted into image description features;

[0010] Inputting the image description feature into the disease result prediction layer of the first disease prediction model, and determining the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by training a Chinese language model pre-trained by using a full-text masking method;

[0011] The result description vector is input into the result classification layer of the first disease prediction model, and the predicted diagnosis result of the medical imaging report is determined through the result classification layer.

[0012] In a possible implementation, extracting description information from the acquired medical imaging report to extract image description information used to describe the brain MRI image in the medical imaging report includes:

[0013] In combination with the image report template of the medical image report, the image description area in the medical image report is determined by analyzing the acquired medical image report;

[0014] By identifying the content in the image description area, image description information for describing the brain MRI image in the medical image report is extracted.

[0015] In a possible implementation manner, mapping the image description information into image description data by preprocessing the image description information includes:

[0016] Performing information screening processing on the image description information, filtering out invalid description information and invalid characters in the image description information, and obtaining concise description information for describing the brain MRI image;

[0017] Performing word segmentation processing on the simplified description information, dividing the simplified description information into a plurality of sub-words; wherein the sub-words include characters and / or words;

[0018] Each subword is encoded one by one, and the simplified description information is mapped into image description data.

[0019] In a possible implementation, combining the data input format of the first disease prediction model, performing format standardization processing on the image description data, and converting the image description data into image description features include:

[0020] Determine, in combination with the data input format of the first disease prediction model, the data input length of the input features received by the first disease prediction model;

[0021] Determining a format processing method of the image description data by comparing the data input length with the data length of the image description data;

[0022] When the format processing method is filling processing, the image description data is filled with preset characters, and the filled image description data is determined as the image description feature;

[0023] When the format processing method is interception processing, data within a specific range of the image description data is intercepted, and the intercepted image description data is determined as the image description feature.

[0024] In a possible implementation manner, determining the format processing method of the image description data by comparing the data input length with the data length of the image description data includes:

[0025] If the data input length is greater than the data length of the image description data, determining that the format processing method of the image description data is padding processing;

[0026] If the data input length is less than the data length of the image description data, it is determined that the format processing method of the image description data is interception processing.

[0027] In a possible implementation, the disease outcome prediction layer of the first disease prediction model is trained by the following steps:

[0028] Acquire multiple medical sample reports and disease label data of each medical sample report; wherein the disease label data is a result description vector of the medical sample report;

[0029] For each medical sample report, information screening is performed on the medical sample report to filter out invalid description information and invalid characters in the medical sample report to obtain simplified sample information for describing the brain MRI image in the medical sample report;

[0030] Performing word segmentation processing on the simplified sample information to split the simplified sample information into multiple subwords;

[0031] Performing encoding processing on each subword one by one, mapping the simplified sample information into image sample data;

[0032] In combination with the data input format of the Chinese language model, the image sample data is formatted and standardized, and the image sample data is converted into image sample features;

[0033] The image sample features of each medical sample report are used as input features, and the disease label data of each medical sample report is used as output features. The parameters of the pre-trained Chinese language model are fine-tuned to obtain a trained disease outcome prediction layer.

[0034] In a possible implementation, inputting the result description vector into the result classification layer of the first disease prediction model, and determining the predicted diagnosis result of the medical imaging report through the result classification layer, includes:

[0035] Inputting the result description vector into the result classification layer of the first disease prediction model, and determining the disease diagnosis category to which the medical imaging report belongs by analyzing the result description vector in the result classification layer;

[0036] The disease diagnosis category is determined as the predicted diagnosis result of the medical imaging report.

[0037] In a possible implementation, after extracting the image description information for describing the brain MRI image in the medical imaging report, the diagnosis result prediction method further includes:

[0038] Determining the type of diagnostic task for the patient to which the medical image report belongs by identifying the medical image report;

[0039] Obtain candidate diagnosis results, result query statements, and disease diagnosis auxiliary templates of the diagnosis task type;

[0040] Fill the image description information, the result query statement and the candidate diagnosis result into the disease diagnosis auxiliary template respectively, and generate a disease diagnosis statement for predicting the predicted diagnosis result of the medical image report;

[0041] The disease diagnosis statement is input into a second disease prediction model to determine the predicted diagnosis result of the medical imaging report; wherein the second disease prediction model is obtained by fine-tuning the parameters of a pre-trained language model; and the pre-trained language model is a basic model pre-trained in a general scenario.

[0042] In a possible embodiment, the diagnostic task types include one or more of an IDH genotype determination task, a WHO grading task, a brain tumor type determination task, and a 1p / 19q co-deletion task.

[0043] The present application also provides a diagnostic result prediction device based on medical image description, the diagnostic result prediction device comprising:

[0044] An information extraction module, used to extract description information from the acquired medical imaging report, and extract image description information used to describe the brain MRI image in the medical imaging report;

[0045] A data mapping module, used for mapping the image description information into image description data word by word by preprocessing the image description information;

[0046] A format adjustment module, used to perform format standardization processing on the image description data in combination with the data input format of the first disease prediction model, and convert the image description data into image description features;

[0047] A result prediction module, used for inputting the image description feature into the disease result prediction layer of the first disease prediction model, and determining the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by training a Chinese language model pre-trained by a full-text masking method;

[0048] The result classification module is used to input the result description vector into the result classification layer of the first disease prediction model, and determine the predicted diagnosis result of the medical imaging report through the result classification layer.

[0049] In a possible implementation, when the information extraction module is used to extract description information from the acquired medical imaging report and extract image description information used to describe the brain MRI image in the medical imaging report, the information extraction module is used to:

[0050] In combination with the image report template of the medical image report, the image description area in the medical image report is determined by analyzing the acquired medical image report;

[0051] By identifying the content in the image description area, image description information for describing the brain MRI image in the medical image report is extracted.

[0052] In a possible implementation manner, when the data mapping module is used to map the image description information into image description data by preprocessing the image description information, the data mapping module is used to:

[0053] Performing information screening processing on the image description information, filtering out invalid description information and invalid characters in the image description information, and obtaining concise description information for describing the brain MRI image;

[0054] Performing word segmentation processing on the simplified description information, dividing the simplified description information into a plurality of sub-words; wherein the sub-words include characters and / or words;

[0055] Each subword is encoded one by one, and the simplified description information is mapped into image description data.

[0056] In a possible implementation, when the format adjustment module is used to perform format standardization processing on the image description data in combination with the data input format of the first disease prediction model and convert the image description data into image description features, the format adjustment module is used to:

[0057] Determine, in combination with the data input format of the first disease prediction model, the data input length of the input features received by the first disease prediction model;

[0058] Determining a format processing method of the image description data by comparing the data input length with the data length of the image description data;

[0059] When the format processing method is filling processing, the image description data is filled with preset characters, and the filled image description data is determined as the image description feature;

[0060] When the format processing method is interception processing, data within a specific range of the image description data is intercepted, and the intercepted image description data is determined as the image description feature.

[0061] In a possible implementation manner, when the format adjustment module is used to determine the format processing method of the image description data by comparing the data input length with the data length of the image description data, the format adjustment module is used to:

[0062] If the data input length is greater than the data length of the image description data, determining that the format processing method of the image description data is padding processing;

[0063] If the data input length is less than the data length of the image description data, it is determined that the format processing method of the image description data is interception processing.

[0064] In a possible implementation, the diagnosis result prediction device further includes a model training module, and the model training module is used to train and obtain a disease result prediction layer of the first disease prediction model through the following steps:

[0065] Acquire multiple medical sample reports and disease label data of each medical sample report; wherein the disease label data is a result description vector of the medical sample report;

[0066] For each medical sample report, information screening is performed on the medical sample report to filter out invalid description information and invalid characters in the medical sample report to obtain simplified sample information for describing the brain MRI image in the medical sample report;

[0067] Performing word segmentation processing on the simplified sample information to split the simplified sample information into multiple subwords;

[0068] Performing encoding processing on each subword one by one, mapping the simplified sample information into image sample data;

[0069] In combination with the data input format of the Chinese language model, the image sample data is formatted and standardized, and the image sample data is converted into image sample features;

[0070] The image sample features of each medical sample report are used as input features, and the disease label data of each medical sample report is used as output features. The parameters of the pre-trained Chinese language model are fine-tuned to obtain a trained disease outcome prediction layer.

[0071] In a possible implementation, when the result classification module is used to input the result description vector into the result classification layer of the first disease prediction model and determine the predicted diagnosis result of the medical imaging report through the result classification layer, the result classification module is used to:

[0072] Inputting the result description vector into the result classification layer of the first disease prediction model, and determining the disease diagnosis category to which the medical imaging report belongs by analyzing the result description vector in the result classification layer;

[0073] The disease diagnosis category is determined as the predicted diagnosis result of the medical imaging report.

[0074] In a possible implementation, after extracting the image description information for describing the brain MRI image in the medical imaging report, the diagnosis result prediction device is further used to:

[0075] Determining the type of diagnostic task for the patient to which the medical image report belongs by identifying the medical image report;

[0076] Obtain candidate diagnosis results, result query statements, and disease diagnosis auxiliary templates of the diagnosis task type;

[0077] Fill the image description information, the result query statement and the candidate diagnosis result into the disease diagnosis auxiliary template respectively, and generate a disease diagnosis statement for predicting the predicted diagnosis result of the medical image report;

[0078] The disease diagnosis statement is input into a second disease prediction model to determine the predicted diagnosis result of the medical imaging report; wherein the second disease prediction model is obtained by fine-tuning the parameters of a pre-trained language model; and the pre-trained language model is a basic model pre-trained in a general scenario.

[0079] In a possible embodiment, the diagnostic task types include one or more of an IDH genotype determination task, a WHO grading task, a brain tumor type determination task, and a 1p / 19q co-deletion task.

[0080] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the diagnostic result prediction method based on medical image description are performed as described above.

[0081] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned method for predicting diagnostic results based on medical image description are executed.

[0082] The embodiment of the present application provides a method and device for predicting diagnostic results based on medical image description, extracting description information from the acquired medical image report, extracting image description information used to describe the brain MRI image in the medical image report; mapping the image description information to image description data by preprocessing the image description information; processing the image description data in a standardized format in combination with the data input format of the first disease prediction model, and converting the image description data into image description features; inputting the image description features into the disease result prediction layer of the first disease prediction model, and determining the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by pre-training the Chinese language model using the full-text masking method; inputting the result description vector into the result classification layer of the first disease prediction model, and determining the predicted diagnostic result of the medical image report through the result classification layer. In this way, accurate prediction of diagnostic results can be achieved by only using the image description information used to describe the brain MRI image in the medical image report, thereby assisting doctors in improving the diagnostic efficiency of diseases.

[0083] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0085] Figure 1 A flowchart of a method for predicting diagnostic results based on medical image description provided in an embodiment of the present application;

[0086] Figure 2 A flowchart of another method for predicting diagnostic results based on medical image description provided in an embodiment of the present application;

[0087] Figure 3 One of the structural schematic diagrams of a diagnostic result prediction device based on medical image description provided in an embodiment of the present application;

[0088] Figure 4 The second structural diagram of a diagnostic result prediction device based on medical image description provided in an embodiment of the present application;

[0089] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0091] Research has found that with the increasing attention paid to human health and the rapid development of science and technology, the cycle of disease discovery and treatment has been gradually shortened. In the process of disease discovery, rapid and accurate disease diagnosis is the basis for proper treatment of the disease. Among them, brain diseases have the characteristics of high incidence and high risk, which makes the medical system have a stronger demand for rapid and accurate diagnosis of brain diseases.

[0092] Clinically, in order to more accurately determine the patient's brain disease, brain MRI images are usually taken, and then the radiologist writes the imaging report and obtains preliminary results. Finally, the clinician makes a final diagnosis by integrating the brain MRI images, image description information, and diagnostic results in the imaging report. Here, in terms of the diagnostic results, since the diagnostic results are the subjective diagnosis made by the doctor based on the brain MRI images and image description information, different doctors may have different diagnostic opinions even for the same brain MRI images and image description information, and in order to ensure the accuracy of the diagnostic results, doctors usually need to spend a lot of time on the diagnostic results, which slows down the diagnostic progress;

[0093] Existing "models" that can be used for disease diagnosis usually need to be combined with brain MRI images to achieve disease diagnosis. Since the "model" can only receive images of fixed size, the current image-based model has poor adaptability to MRI images and can only choose to discard redundant modalities or generate missing modalities to match the current "model", resulting in problems of information waste and inaccurate generation results.

[0094] Based on this, an embodiment of the present application provides a method for predicting diagnostic results based on medical image description, which can accurately predict diagnostic results by only using the image description information used to describe brain MRI images in the medical imaging report, thereby assisting doctors in improving the efficiency of disease diagnosis.

[0095] See also Figure 1 , Figure 1 This is a flowchart of a method for predicting diagnostic results based on medical image description provided in an embodiment of the present application. Figure 1 As shown in , the diagnosis result prediction method provided in the embodiment of the present application includes:

[0096] S101. Extract description information from the acquired medical imaging report to extract image description information used to describe the brain MRI image in the medical imaging report.

[0097] S102: Preprocess the image description information to map the image description information word by word into image description data.

[0098] S103, combining the data input format of the first disease prediction model, performing format standardization processing on the image description data, and converting the image description data into image description features.

[0099] S104. Input the image description features into the disease outcome prediction layer of the first disease prediction model, and determine the result description vector of the medical image report through the disease outcome prediction layer; wherein the disease outcome prediction layer is obtained by pre-training a Chinese language model using full-text masking.

[0100] S105. Input the result description vector into the result classification layer of the first disease prediction model, and determine the predicted diagnosis result of the medical imaging report through the result classification layer.

[0101] A method for predicting diagnostic results based on medical image description is provided in an embodiment of the present application. In the process of diagnosing a disease based on a medical image report, first, description information of the medical image report is extracted to extract the image description information from the medical image report, thereby reducing the amount of data to be processed by the first disease prediction model in the subsequent process, and indirectly improving the efficiency of disease diagnosis; secondly, by pre-processing the image description information, invalid description information and invalid characters in the image description information are filtered out to avoid interference with the disease prediction process; finally, the first disease prediction model is used to accurately predict the predicted diagnostic results of the medical image report.

[0102] In step S101, considering that in the subsequent diagnosis result prediction process, there is no need to pay attention to the brain MRI images and diagnosis results in the medical imaging report, therefore, in order to avoid interference of the brain MRI images and diagnosis results in the prediction process, it is necessary to extract the descriptive information of the acquired medical imaging report, so as to separate the image description information used to describe the brain MRI images in the medical imaging report from the medical imaging report.

[0103] Here, the image description information can be extracted from the medical imaging report by using text recognition to identify the contents recorded in the medical imaging report; considering that the writing formats of medical imaging reports used in the same hospital or even in different hospitals are similar or even the same, in order to improve the efficiency of extracting image description information, the embodiment of the present application uses an image report template of the medical imaging report to assist in the recognition of image description information.

[0104] In one implementation, step S101 includes:

[0105] S1011. In combination with the image report template of the medical image report, the acquired medical image report is analyzed to determine the image description area in the medical image report.

[0106] In this step, by comparing the image report template of the medical image report with the medical image report, the writing layout of the acquired medical image report is analyzed to determine the image description area where the image description information in the medical image report is written.

[0107] S1012: Extract image description information used to describe the brain MRI image in the medical image report by identifying the content in the image description area.

[0108] In this step, after locating the writing area of ​​the image description information (i.e., the image description area), the image description information used to describe the brain MRI images in the medical image report can be extracted from the medical image report simply by identifying in the image description area, which not only reduces the content that needs to be identified, but also ensures the accuracy of the image description information that can be accurately identified; at the same time, it also preliminarily filters out the brain MRI images and diagnostic results that do not need to be concerned about in the diagnosis result prediction stage.

[0109] In step S102, in order to enable the first disease prediction model to focus more on the information contained in words rather than the information contained in single words during the recognition process, the image description information needs to be mapped into the form of image description data when preprocessing the image description information and converting the image description information into image description data that can be recognized by the first disease prediction model.

[0110] Here, considering that the image description information may contain invalid description information such as abnormal descriptions and invalid characters such as spaces, line breaks, and abnormal characters, in order to avoid the impact of invalid description information and invalid description characters on the diagnosis result prediction process, before mapping the image description information into image description data, it is also necessary to filter out the invalid description information and invalid characters in the image description information.

[0111] In one implementation, step S102 includes:

[0112] S1021. Perform information screening processing on the image description information to filter out invalid description information and invalid characters in the image description information to obtain concise description information for describing the brain MRI image.

[0113] In this step, the extracted image description information is screened to filter out invalid description information that has no image in the diagnosis result prediction process (for example, patient information confused in the image description information due to misidentification, erroneously identified image description information, etc.) and invalid characters (for example, spaces, line breaks, abnormal characters, etc.), so as to obtain concise description information that can be used to describe brain MRI images.

[0114] S1022: Perform word segmentation processing on the simplified description information, and split the simplified description information into multiple sub-words.

[0115] In this step, the simplified description information is segmented to split the simplified description information into a plurality of sub-words such as characters and / or words.

[0116] S1023. Encode each subword one by one, and map the simplified description information into image description data.

[0117] In this step, each of the split subwords is encoded one by one, each subword is mapped to a unique integer ID or an embedded vector, and the unique integer ID or embedded vector of each subword is concatenated in sequence to map the simplified description information to image description data.

[0118] Here, considering that the first disease prediction model has certain restrictions on the data input format, that is, the first disease prediction model needs to receive input data within a specified length range, therefore, after obtaining the image description data, it is further combined with the data input format required by the first disease prediction model to perform format standardization on the image description data so that the image description data can be recognized by the first disease prediction model.

[0119] In step S103, the image description data is formatted in a standardized manner in combination with a data input format acceptable to the first disease prediction model, so as to convert the image description data into image description features that can be recognized by the first disease prediction model.

[0120] In one implementation, step S103 includes:

[0121] S1031. Determine the data input length of the input features received by the first disease prediction model in combination with the data input format of the first disease prediction model.

[0122] S1032: Determine a format processing method for the image description data by comparing the data input length with the data length of the image description data.

[0123] In this step, the data input length of the input features received by the first disease prediction model is compared with the data length of the image description data, and the format processing method of the image description data is clarified by determining whether the image description data is in a data input format acceptable to the first disease prediction model.

[0124] In one implementation, step S1032 includes:

[0125] S10321. If the data input length is greater than the data length of the image description data, determine that the format processing method of the image description data is padding processing.

[0126] In this step, if the data input length is greater than the data length of the image description data, it means that the content in the image description data is too little. Therefore, the image description data needs to be padded to obtain input features that can be received by the first disease prediction model. At this time, the format processing method of the image description data is determined to be padding processing.

[0127] S10322: If the input data length is less than the data length of the image description data, determine that the format processing method of the image description data is truncation processing.

[0128] In this step, if the data input length is less than the data length of the image description data, it means that the image description data contains too much content. Therefore, the image description data needs to be truncated to obtain input features that can be received by the first disease prediction model. At this time, the format processing method of the image description data is determined to be truncation processing.

[0129] S1033: When the format processing method is filling processing, the image description data is filled with preset characters, and the filled image description data is determined as an image description feature.

[0130] In this step, when the format processing method of the image description data is padding processing, it means that the data length of the image description data is short. At this time, it is necessary to pad the image description data so that the data length of the image description data is consistent with the data input length of the input feature received by the first disease prediction model. At this time, the image description data can be padded with preset characters (for example, 0) to pad the image description data to make the data length of the image description data consistent with the data input length of the input feature received by the first disease prediction model, thereby obtaining image description features that can be used to make diagnosis results.

[0131] S1034: When the format processing method is interception processing, data within a specific range of the image description data is intercepted, and the intercepted image description data is determined as an image description feature.

[0132] In this step, when the format processing method of the image description data is truncation processing, it means that the data length of the image description data is long. At this time, it is necessary to truncate the data within a specific range of the image description data so that the data length of the image description data is consistent with the data input length of the input feature received by the first disease prediction model;

[0133] At this time, considering that in the image description data, the content that can better describe the brain MRI image is usually recorded at the front end of the image description data part, that is, the "head data" of the image description data contains more effective description information, therefore, when the image description data is intercepted, the "tail data" that exceeds the data input length of the input feature received by the first disease prediction model is intercepted, so as to cut the data length of the image description data to be consistent with the data input length of the input feature received by the first disease prediction model, and then obtain the image description features that can be used to make a diagnosis.

[0134] Exemplarily, taking the data input length of the input features received by the first disease prediction model as "512 characters", if the data length of the image description data is less than "512 characters", the image description data is padded to "512 characters" using the preset character "0"; if the data length of the image description data is longer than "512 characters", the "tail data" after "512 characters" in the image description data is cut off.

[0135] Here, with the continuous development of natural language processing models, the BERT series of models are currently widely recognized natural language models with good comprehensive expressiveness. The BERT series of models are developed based on the Transformer architecture. Considering that the data used in this application for predicting the diagnosis results is Chinese text data, therefore, in the embodiment of this application, Chinese RoBERTa-wwm-ext (hereinafter referred to as RoBERTa), which is more suitable for analyzing Chinese text data, is selected to complete the prediction of the diagnosis results.

[0136] In step S104, the image description features are input into the disease outcome prediction layer of the first disease prediction model, so as to analyze the image description features through the disease outcome prediction layer to determine the result description vector of the medical image report; wherein the disease outcome prediction layer is obtained by training a Chinese language model pre-trained by a full-text masking method.

[0137] Specifically, whole word masking (WWM) is used to pre-train RoBERTa. Compared with the BERT model obtained by WordPiece pre-training, WordPiece decomposes words / sentences into subwords, while WWM is an improvement on WordPiece. By randomly masking the entire word to which the subword belongs, the disease outcome prediction layer obtained by pre-training pays more attention to the information of the entire word during the prediction process, rather than just focusing on the meaning of the single word itself.

[0138] In one embodiment, the disease outcome prediction layer of the first disease prediction model is trained by the following steps:

[0139] Step a: obtain multiple medical sample reports and disease label data of each medical sample report.

[0140] In this step, after collecting a large number of medical sample reports, multiple medical sample reports and disease label data of each medical sample report are obtained by filtering out medical sample reports with abnormal report descriptions; here, the disease label data of each medical sample report is obtained using various widely recognized standard methods.

[0141] The disease label data is a result description vector of the medical sample report.

[0142] Step b: for each medical sample report, perform information screening on the medical sample report, filter out invalid description information and invalid characters in the medical sample report, and obtain simplified sample information for describing the brain MRI image in the medical sample report.

[0143] Step c: performing word segmentation processing on the simplified sample information, and splitting the simplified sample information into multiple sub-words.

[0144] Step d: Encode each subword one by one, and map the simplified sample information into image sample data.

[0145] Step e: combining the data input format of the Chinese language model, performing format standardization processing on the image sample data, and converting the image sample data into image sample features;

[0146] Step f: using the image sample features of each medical sample report as input features, and the disease label data of each medical sample report as output features, fine-tuning the parameters of the pre-trained Chinese language model to obtain a trained disease outcome prediction layer.

[0147] In step S105, after obtaining the result description vector of the medical imaging report, the result description vector of the medical imaging report can be classified through an additional result classification layer to determine the predicted diagnosis result of the medical imaging report according to the disease diagnosis category to which the result description vector belongs.

[0148] In one implementation, step S105 includes:

[0149] S1051. Input the result description vector into the result classification layer of the first disease prediction model, and determine the disease diagnosis category to which the medical imaging report belongs by analyzing the result description vector in the result classification layer.

[0150] In this step, the result description vector (CLS) is input into the result classification layer (i.e., the fully connected layer) of the first disease prediction model, and the disease diagnosis category to which the medical imaging report belongs is determined by analyzing the result description vector (CLS) in the result classification layer (i.e., the fully connected layer).

[0151] S1052: Determine the disease diagnosis category as the predicted diagnosis result of the medical imaging report.

[0152] Here, considering that large language models can also accurately diagnose brain diseases, in the embodiment of the present application, the diagnosis results of the patients can be predicted with the help of a pre-trained large language model (i.e., the second disease prediction model).

[0153] See also Figure 2 , Figure 2 Another flow chart of a method for predicting diagnostic results based on medical image description provided in an embodiment of the present application; Figure 2 As shown, in one embodiment, after extracting the image description information for describing the brain MRI image in the medical image report, the diagnosis result prediction method further includes:

[0154] S201. Determine the type of diagnostic task of the patient to which the medical imaging report belongs by identifying the medical imaging report.

[0155] In this step, the type of disease suffered by the patient to which the medical image report belongs is determined by identifying medical keywords and other information in the medical image report; and the type of disease suffered by the patient is determined as the diagnostic task type of the second disease prediction model when the second disease prediction model is used to diagnose the patient.

[0156] Among them, the diagnostic task types include one or more of an IDH genotype determination task, a WHO grading task, a brain tumor type determination task, and a 1p / 19q co-deletion task.

[0157] S202: Obtain candidate diagnosis results, result query statements, and disease diagnosis auxiliary templates of the diagnosis task type.

[0158] In this step, for different diagnostic task types, the disease diagnosis statements that the second disease prediction model needs to recognize during the prediction process are also different. Therefore, in order to accurately predict the predicted diagnostic results of the medical imaging report for each diagnostic task type, before using the second disease prediction model to predict the diagnostic results, the candidate diagnostic results that can be fed back during the diagnosis of patients with this diagnostic task type, the result query statements for inquiring about the diagnostic results, and the disease diagnosis auxiliary template applicable to this diagnostic task type are obtained.

[0159] S203, filling the image description information, the result query statement and the candidate diagnosis result into the disease diagnosis auxiliary template respectively, and generating a disease diagnosis statement for predicting the predicted diagnosis result of the medical image report.

[0160] In this step, the image description information, result query statements and candidate diagnosis results extracted from the medical imaging report are respectively filled into the acquired disease diagnosis auxiliary template to generate a disease diagnosis statement for predicting the predicted diagnosis result of the medical imaging report.

[0161] Here, in the disease diagnosis statement (i.e., input prompt), first, the image description information of each patient is clarified for analysis by the second disease diagnosis model, and the identity of the second disease diagnosis model is given as "a radiologist with many years of experience in brain diseases", so as to inspire the second disease diagnosis model to make full use of the existing prior knowledge related to brain diseases; secondly, the question being asked (i.e., result query statement) is clarified, for example, analyzing the IDH genotype of the patient; finally, the second disease diagnosis model is required to output specific results, for example, output labels such as "wild" and "mutated", rather than just outputting content without giving specific results; thereby, the second disease diagnosis model has better performance.

[0162] Specifically, when the diagnostic task type is an IDH genotype determination task, the disease diagnosis statement (i.e., the input prompt of the second disease prediction model) is: This is the brain image description information of a patient with brain disease: "Image description information (the patient's real image description information)". Assuming that you are a radiologist with many years of clinical experience in brain diseases, please tell me the IDH genotype of the patient. The label "mutation" indicates that the patient's IDH genotype is mutant, and the label "wild" indicates that the patient's IDH genotype is wild type.

[0163] When the diagnosis task type is a WHO grading task, the disease diagnosis statement (i.e., the input prompt of the second disease prediction model) is: This is the brain image description information of a patient with brain disease: "Image description information (the patient's real image description information)". Assuming that you are a radiologist with many years of clinical experience in brain diseases, please tell me the WHO grade of the patient. The label "low grade" indicates that the patient's brain tumor is low grade, and the label "high grade" indicates that the patient's brain tumor is high grade.

[0164] When the diagnosis task type is a brain tumor type determination task, the disease diagnosis statement (i.e., the input prompt of the second disease prediction model) is: This is the brain image description information of a patient with brain disease: "Image description information (the patient's real image description information)". Suppose you are a radiologist with many years of clinical experience in brain diseases, please tell me the type of brain tumor of the patient. The label "glioblastoma" indicates that the patient's brain tumor type is glioblastoma, and the label "lymphoma" indicates that the patient's brain tumor type is lymphoma.

[0165] When the diagnostic task type is a 1p / 19q co-deletion task, the disease diagnosis statement (i.e., the input prompt of the second disease prediction model) is: This is the brain image description information of a patient with brain disease: "Image description information (the patient's actual image description information)". Assuming that you are a radiologist with many years of clinical experience in brain diseases, please tell me the 1p / 19q co-deletion of this patient. The label "occurred" indicates that the patient has 1p / 19q chromosome co-deletion, and the label "did not occur" indicates that the patient has not 1p / 19q chromosome co-deletion.

[0166] S204: Input the disease diagnosis statement into a second disease prediction model to determine the predicted diagnosis result of the medical imaging report.

[0167] In this step, the disease diagnosis statement is input into the pre-trained second disease prediction model, so as to determine the predicted diagnosis result of the medical imaging report by identifying the disease diagnosis statement using the second disease prediction model.

[0168] Here, the second disease prediction model is obtained by fine-tuning the parameters of the pre-trained language model; that is, the second disease diagnosis model involved in the embodiment of the present application is obtained by fine-tuning the parameters of the pre-trained language model so that the pre-trained language model can be suitable for predicting the diagnosis results.

[0169] The pre-trained language model is a basic model that is pre-trained in a general scenario; that is, the pre-trained language model mentioned in the embodiments of the present application is a basic model that is pre-trained for natural general scenarios.

[0170] Among them, the second disease prediction model can be a dialogue series model such as the Baichuan2 model, which can accept disease diagnosis sentences of various diagnostic task types as input, and make corresponding inferences and responses based on the knowledge learned during its pre-training process.

[0171] In one embodiment, the second disease prediction model is trained by the following steps:

[0172] Step g: obtaining multiple medical sample reports and disease label data of each medical sample report.

[0173] In this step, after collecting a large number of medical sample reports, multiple medical sample reports and disease label data of each medical sample report are obtained by filtering out medical sample reports with abnormal report descriptions; here, the disease label data of each medical sample report is obtained using various widely recognized standard methods.

[0174] The disease label data is the sample diagnosis result of the medical sample report.

[0175] Step h: for each medical sample report, based on the diagnostic task type of the medical sample report, generate a sample diagnostic statement of the medical sample report.

[0176] Step i: taking the sample diagnosis sentence of each medical sample report as the input feature, taking the sample diagnosis result of each medical sample report as the output feature, fine-tuning the parameters of the large language model, and obtaining a trained second disease prediction model.

[0177] In order to comprehensively and fully evaluate the effectiveness of the diagnostic result prediction method based on medical image description in the embodiment of the present application, four parts of experiments are designed to verify respectively: (1) performance comparison between the first disease prediction model and the second disease prediction model; (2) performance comparison between the first disease prediction model using only medical image description and the disease prediction model using only brain MRI images; (3) performance comparison between the first disease prediction model using only medical image description and the first disease prediction model using only brain MRI images in the presence of missing image modalities; (4) comparison between the first disease prediction model using only medical image description and the first disease prediction model using only brain MRI images in terms of data preprocessing time consumption.

[0178] The embodiments of the present application collected medical imaging reports of about 12,980 patients in total. Specifically, the medical imaging report of each patient contains brain MRI images of multiple modalities and their corresponding image description information, among which most patients contain complete images of five modalities, including T1 modality, T2 modality, T1 enhanced modality, Flair modality and ADC modality. In the experiment of the missing modality part, since the number of patients who only lack the Flair modality in the collected data is the largest, the embodiments of the present application use patients who only lack the Flair modality as an example to complete the test when the modality is missing, and the remaining patients who lack other modalities have been excluded in the process of data preprocessing. It should be additionally noted that the medical imaging reports of the patients used in this application only include image description data but not the diagnosis result part, so as to avoid misidentifying the diagnosis result part as image description data.

[0179] The embodiment of the present application was verified on four brain disease-related tasks, namely, IDH genotype determination task, WHO grading determination task, brain tumor type determination task and 1p / 19q co-deletion determination task, and the true label of each patient was the pathological diagnosis gold standard provided by the radiologist. In the IDH genotype identification task, a total of 4699 patients were included, of which 4160 patients did not have modal deletion, and 539 patients lacked Flair modality. The IDH genotype of the patients was wild type and mutant; in the WHO grading determination task, a total of 3549 patients were included, all of whom did not have modal deletion, and the WHO grade of the patients was divided into low grade and high grade; in the brain tumor type determination task, a total of 1228 patients were included, and the brain tumor types of the patients were glioblastoma and lymphoma; in the 1p / 19q co-deletion determination task, a total of 3504 patients were included, and the patients were divided into two types: 1p / 19q co-deletion and no 1p / 19q co-deletion.

[0180] In order to fully and reasonably evaluate the performance of different models, this application uses the area under ROC curve (AUROC), classification accuracy (ACC), and F1-score (F1) as indicators for evaluating all models. The experimental results are given in the form of mean ± standard deviation, and all results are the results of five repeated experiments.

[0181] (1) Performance comparison of the first disease prediction model (i.e., the pre-trained language model) and the second disease prediction model (i.e., the language model of the large model).

[0182] First, the embodiment of the present application fully compares the diagnostic performance of the two language models. Specifically, the effects of the first disease prediction model (RoBERTa) and the second disease prediction model (Baichuan2-13B) after fine-tuning are compared. As shown in Table 1, the performance of the above two models on different tasks is different. For three of the four tasks, the effect of the first disease prediction model (RoBERTa) is better than that of the second disease prediction model (Baichuan2-13B). By calculating the average accuracy of all tasks, it can be found that the effect of the first disease prediction model (RoBERTa) is worse than that of the second disease prediction model (Baichuan2-13B), indicating that although the number of parameters of the first disease prediction model (RoBERTa) is not as good as the number of residuals of the second disease prediction model (Baichuan2-13B), it can still achieve good results under the amount of data collected in this application. However, the advantage of the second disease prediction model (Baichuan2-13B) is that it can complete the prediction of multiple diagnostic tasks with one model, while the first disease prediction model (RoBERTa) needs to train a model for each diagnostic task. Therefore, the cost of using the second disease prediction model (Baichuan2-13B) to achieve disease prediction is lower.

[0183] Table 1: Performance comparison between the first disease prediction model and the second disease prediction model

[0184]

[0185] (2) Comparison of the performance of the first disease prediction model using only medical image descriptions and the first disease prediction model using only brain MRI images on different diagnostic tasks.

[0186] In order to fully compare the performance of the first disease prediction model using only medical image descriptions and the first disease prediction model using only brain MRI images, this application compares the first disease prediction model with four different image models using brain MRI images, specifically, including a 2D model DeepRisk that does not use segmentation priors (a total of 40 slices are taken from five modalities at equal distances and spliced ​​as the input of the 2D model), two 2D models that use segmentation priors 2D MedMINIST (the single slice with the largest tumor area in all modalities is spliced ​​as the input of the model according to the segmentation results) and DenseNet (the 24 slices where the tumor is located in all modalities are spliced ​​as the input of the model according to the segmentation results), and a 3D model 3D MedMNIST that does not use segmentation priors (the original 3D image as a whole is directly used as input). The performance comparison of the two types of models on the two tasks is shown in Table 2.

[0187] Table 2: Comparison of the performance of disease prediction models using only medical image descriptions and only brain MRI images on WHO-level judgment tasks

[0188]

[0189] As can be seen from Table 2, the disease prediction model using medical image descriptions outperforms the disease prediction model using brain MRI images in all evaluation indicators of the two brain disease-related tasks, and the effect is obvious, which fully demonstrates the rationality and effectiveness of the disease prediction model using only medical image descriptions; in addition, compared with the two disease prediction models with segmentation priors, the disease prediction model based on medical image descriptions can still achieve better consistency results. Therefore, it can be proved that the disease prediction model using medical image descriptions can achieve better classification diagnosis performance without increasing additional workload, showing the advantage of the model based on image reports in saving preprocessing processes.

[0190] (3) Comparison of the performance of disease prediction models that only use medical image descriptions and disease prediction models that only use brain MRI images in the presence of missing imaging modalities.

[0191] When using brain MRI images to predict brain diseases, since different imaging modalities contribute to different disease diagnoses to different degrees, and the contribution mechanism is complex and not completely clear, the current disease diagnosis models using brain MRI images generally do not have a unified modality selection rule, so that in actual application, there will be situations where additional modalities are not used or patients are screened based on the number of modalities, which is not conducive to the widespread application of the model. However, the disease diagnosis model based on brain MRI images does not have the limitation of fixing the patient's imaging modality. Therefore, this application takes the IDH genotype identification task as an example to further evaluate the effect of the disease prediction model that only uses medical image descriptions and the disease prediction model that only uses brain MRI images when there is a problem of missing imaging modalities. Considering that the disease prediction model using brain MRI images requires data of fixed imaging modalities as input, taking the complete five imaging modalities as an example, this application first trains a modality missing generation model to supplement the patient's missing modalities. This part is completed based on the pGAN model. For the missing Flair modality images, a model is trained to generate the Flair modality from T1 modality, T2 modality, T1 enhanced modality and ADC modality. After the supplement, all patients contain imaging data of five modalities, and the image description information itself does not contain the description of the Flair modality, so no additional changes are made. The performance comparison of the disease prediction model using medical image descriptions and the disease prediction model using brain MRI images when the modality is missing is shown in Table 3. Table 3: Performance comparison of the disease prediction model using medical image descriptions and the disease prediction model using brain MRI images when the modality is missing

[0192]

[0193] As can be seen from Table 3, when the modality is missing, even if the image part has supplemented the missing modality, the model based on the image report can still achieve significantly better results than the model based on the image. This shows that the effectiveness of using image description information with lower information dimension and more dense information makes the disease prediction model based on image description information have stronger adaptability to missing modalities and information mining capabilities.

[0194] (4) Comparison of the data preprocessing time consumption between the disease prediction model that only uses medical image descriptions and the disease prediction model that only uses brain MRI images.

[0195] In the disease prediction model using brain MRI images, due to the need to ensure the consistency of input data, almost all methods require the original MRI images to be preprocessed before inputting into the model for analysis. Conventional image preprocessing methods mainly include N4 bias field correction, registration, skull stripping and segmentation. Image description information is also preprocessed before inputting into the model for analysis. Conventional image description information preprocessing methods mainly include removing abnormal characters, spaces, and line breaks.

[0196] This application counts the preprocessing time of the MRI images and image description information of a single patient. The comparison of preprocessing time is shown in Table 4, where the time consumption is the average value of 10 patients. The statistical time only reports the execution time of this program, and other irrelevant operations have been excluded.

[0197] Table 4: Comparison of data preprocessing time consumption between disease prediction models using medical image descriptions and disease prediction models using only brain MRI images

[0198]

[0199] As shown in Table 4, due to the high dimensionality and complexity of MRI images, the time for preprocessing MRI images is much longer than the time for preprocessing medical image descriptions. This means that when the amount of image data is huge, the timeliness of model design and development and clinical reasoning applications will be limited. Therefore, the full utilization and mining of disease prediction models based on medical image descriptions is conducive to achieving faster preliminary diagnosis and has great application space in scenarios that require timeliness.

[0200] The diagnostic result prediction method based on medical image description provided in the embodiment of the present application extracts description information from the acquired medical image report, extracts image description information used to describe the brain MRI image in the medical image report; maps the image description information to image description data by preprocessing the image description information; performs format standardization processing on the image description data in combination with the data input format of the first disease prediction model, and converts the image description data into image description features; inputs the image description features into the disease result prediction layer of the first disease prediction model, and determines the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by pre-training the Chinese language model using the full-text masking method; inputs the result description vector into the result classification layer of the first disease prediction model, and determines the predicted diagnostic result of the medical image report through the result classification layer. In this way, accurate prediction of the diagnostic result can be achieved by only using the image description information used to describe the brain MRI image in the medical image report, thereby assisting doctors in improving the diagnostic efficiency of the disease.

[0201] See also Figure 3 , Figure 4 , Figure 3 This is one of the structural schematic diagrams of a diagnostic result prediction device based on medical image description provided in an embodiment of the present application, Figure 4 This is a second structural diagram of a diagnostic result prediction device based on medical image description provided in an embodiment of the present application. Figure 3 As shown in , the diagnostic result prediction device 300 includes:

[0202] An information extraction module 310 is used to extract description information from the acquired medical imaging report, and extract image description information used to describe the brain MRI image in the medical imaging report;

[0203] A data mapping module 320, configured to map the image description information into image description data word by word by preprocessing the image description information;

[0204] The format adjustment module 330 is used to perform format standardization processing on the image description data in combination with the data input format of the first disease prediction model, and convert the image description data into image description features;

[0205] The result prediction module 340 is used to input the image description feature into the disease result prediction layer of the first disease prediction model, and determine the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by training a Chinese language model pre-trained by a full-text masking method;

[0206] The result classification module 350 is used to input the result description vector into the result classification layer of the first disease prediction model, and determine the predicted diagnosis result of the medical imaging report through the result classification layer.

[0207] Furthermore, when the information extraction module 310 is used to extract description information from the acquired medical imaging report and extract image description information for describing the brain MRI image in the medical imaging report, the information extraction module 310 is used to:

[0208] In combination with the image report template of the medical image report, the image description area in the medical image report is determined by analyzing the acquired medical image report;

[0209] By identifying the content in the image description area, image description information for describing the brain MRI image in the medical image report is extracted.

[0210] Furthermore, when the data mapping module 320 is used to map the image description information into image description data by preprocessing the image description information, the data mapping module 320 is used to:

[0211] Performing information screening processing on the image description information, filtering out invalid description information and invalid characters in the image description information, and obtaining concise description information for describing the brain MRI image;

[0212] Performing word segmentation processing on the simplified description information, dividing the simplified description information into a plurality of sub-words; wherein the sub-words include characters and / or words;

[0213] Each subword is encoded one by one, and the simplified description information is mapped into image description data.

[0214] Furthermore, when the format adjustment module 330 is used to perform format standardization processing on the image description data in combination with the data input format of the first disease prediction model and convert the image description data into image description features, the format adjustment module 330 is used to:

[0215] Determine, in combination with the data input format of the first disease prediction model, the data input length of the input features received by the first disease prediction model;

[0216] Determining a format processing method of the image description data by comparing the data input length with the data length of the image description data;

[0217] When the format processing method is filling processing, the image description data is filled with preset characters, and the filled image description data is determined as the image description feature;

[0218] When the format processing method is interception processing, data within a specific range of the image description data is intercepted, and the intercepted image description data is determined as the image description feature.

[0219] Furthermore, when the format adjustment module 330 is used to determine the format processing method of the image description data by comparing the data input length with the data length of the image description data, the format adjustment module 330 is used to:

[0220] If the data input length is greater than the data length of the image description data, determining that the format processing method of the image description data is padding processing;

[0221] If the data input length is less than the data length of the image description data, it is determined that the format processing method of the image description data is interception processing.

[0222] Further, such as Figure 4 As shown, the diagnosis result prediction device 300 further includes a model training module 360, and the model training module 360 ​​is used to train and obtain the disease result prediction layer of the first disease prediction model through the following steps:

[0223] Acquire multiple medical sample reports and disease label data of each medical sample report; wherein the disease label data is a result description vector of the medical sample report;

[0224] For each medical sample report, information screening is performed on the medical sample report to filter out invalid description information and invalid characters in the medical sample report to obtain simplified sample information for describing the brain MRI image in the medical sample report;

[0225] Performing word segmentation processing on the simplified sample information to split the simplified sample information into multiple subwords;

[0226] Performing encoding processing on each subword one by one, mapping the simplified sample information into image sample data;

[0227] In combination with the data input format of the Chinese language model, the image sample data is formatted and standardized, and the image sample data is converted into image sample features;

[0228] The image sample features of each medical sample report are used as input features, and the disease label data of each medical sample report is used as output features. The parameters of the pre-trained Chinese language model are fine-tuned to obtain a trained disease outcome prediction layer.

[0229] Furthermore, when the result classification module 350 is used to input the result description vector into the result classification layer of the first disease prediction model and determine the predicted diagnosis result of the medical image report through the result classification layer, the result classification module 350 is used to:

[0230] Inputting the result description vector into the result classification layer of the first disease prediction model, and determining the disease diagnosis category to which the medical imaging report belongs by analyzing the result description vector in the result classification layer;

[0231] The disease diagnosis category is determined as the predicted diagnosis result of the medical imaging report.

[0232] Furthermore, after extracting the image description information for describing the brain MRI image in the medical image report, the diagnosis result prediction device 300 is further used to:

[0233] Determining the type of diagnostic task for the patient to which the medical image report belongs by identifying the medical image report;

[0234] Obtain candidate diagnosis results, result query statements, and disease diagnosis auxiliary templates of the diagnosis task type;

[0235] Fill the image description information, the result query statement and the candidate diagnosis result into the disease diagnosis auxiliary template respectively, and generate a disease diagnosis statement for predicting the predicted diagnosis result of the medical image report;

[0236] The disease diagnosis statement is input into a second disease prediction model to determine the predicted diagnosis result of the medical imaging report; wherein the second disease prediction model is obtained by fine-tuning the parameters of a pre-trained language model; and the pre-trained language model is a basic model pre-trained in a general scenario.

[0237] Furthermore, the diagnostic task types include one or more of an IDH genotype determination task, a WHO grading task, a brain tumor type determination task, and a 1p / 19q co-deletion task.

[0238] The diagnostic result prediction device based on medical image description provided in the embodiment of the present application extracts description information from the acquired medical image report, and extracts image description information used to describe the brain MRI image in the medical image report; by preprocessing the image description information, the image description information is mapped to image description data; in combination with the data input format of the first disease prediction model, the image description data is format-standardized, and the image description data is converted into image description features; the image description features are input into the disease result prediction layer of the first disease prediction model, and the result description vector of the medical image report is determined through the disease result prediction layer; wherein the disease result prediction layer is obtained by training the Chinese language model pre-trained by the full-text masking method; the result description vector is input into the result classification layer of the first disease prediction model, and the predicted diagnostic result of the medical image report is determined through the result classification layer. In this way, accurate prediction of the diagnostic result can be achieved by only using the image description information used to describe the brain MRI image in the medical image report, thereby assisting doctors in improving the diagnostic efficiency of the disease.

[0239] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .

[0240] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 or Figure 2 The specific implementation of the steps of the method for predicting diagnostic results based on medical image description in the method embodiment shown can be found in the method embodiment, which will not be repeated here.

[0241] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 or Figure 2 The specific implementation of the steps of the method for predicting diagnostic results based on medical image description in the method embodiment shown can be found in the method embodiment, which will not be repeated here.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0243] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0244] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0246] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0247] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for predicting diagnostic results based on medical image description, characterized in that: The diagnostic result prediction method comprises: Extracting description information from the acquired medical imaging report to extract image description information used to describe the brain MRI image in the medical imaging report; By preprocessing the image description information, mapping the image description information into image description data; In combination with the data input format of the first disease prediction model, the image description data is formatted and standardized, and the image description data is converted into image description features; Inputting the image description feature into the disease result prediction layer of the first disease prediction model, and determining the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by training a Chinese language model pre-trained by using a full-text masking method; The result description vector is input into the result classification layer of the first disease prediction model, and the predicted diagnosis result of the medical imaging report is determined through the result classification layer.

2. The diagnostic result prediction method according to claim 1, characterized in that: The extracting of description information from the acquired medical imaging report to extract image description information used to describe the brain MRI image in the medical imaging report includes: In combination with the image report template of the medical image report, the image description area in the medical image report is determined by analyzing the acquired medical image report; By identifying the content in the image description area, image description information for describing the brain MRI image in the medical image report is extracted.

3. The diagnostic result prediction method according to claim 1, characterized in that: The preprocessing of the image description information to map the image description information into image description data includes: Performing information screening processing on the image description information, filtering out invalid description information and invalid characters in the image description information, and obtaining concise description information for describing the brain MRI image; Performing word segmentation processing on the simplified description information, dividing the simplified description information into a plurality of sub-words; wherein the sub-words include characters and / or words; Each subword is encoded one by one, and the simplified description information is mapped into image description data.

4. The diagnostic result prediction method according to claim 1, characterized in that: The step of combining the data input format of the first disease prediction model, performing format standardization processing on the image description data, and converting the image description data into image description features includes: Determine, in combination with the data input format of the first disease prediction model, the data input length of the input features received by the first disease prediction model; Determining a format processing method of the image description data by comparing the data input length with the data length of the image description data; When the format processing method is filling processing, the image description data is filled with preset characters, and the filled image description data is determined as the image description feature; When the format processing method is interception processing, data within a specific range of the image description data is intercepted, and the intercepted image description data is determined as the image description feature.

5. The diagnostic result prediction method according to claim 4, characterized in that: The step of determining the format processing method of the image description data by comparing the data input length with the data length of the image description data includes: If the data input length is greater than the data length of the image description data, determining that the format processing method of the image description data is padding processing; If the data input length is less than the data length of the image description data, it is determined that the format processing method of the image description data is interception processing.

6. The diagnostic result prediction method according to claim 1, characterized in that: The disease outcome prediction layer of the first disease prediction model is trained through the following steps: Acquire multiple medical sample reports and disease label data of each medical sample report; wherein the disease label data is a result description vector of the medical sample report; For each medical sample report, information screening is performed on the medical sample report to filter out invalid description information and invalid characters in the medical sample report to obtain simplified sample information for describing the brain MRI image in the medical sample report; Performing word segmentation processing on the simplified sample information to split the simplified sample information into multiple subwords; Performing encoding processing on each subword one by one, mapping the simplified sample information into image sample data; In combination with the data input format of the Chinese language model, the image sample data is formatted and standardized, and the image sample data is converted into image sample features; The image sample features of each medical sample report are used as input features, and the disease label data of each medical sample report is used as output features. The parameters of the pre-trained Chinese language model are fine-tuned to obtain a trained disease outcome prediction layer.

7. The diagnostic result prediction method according to claim 1, characterized in that: The step of inputting the result description vector into the result classification layer of the first disease prediction model and determining the predicted diagnosis result of the medical imaging report through the result classification layer includes: Inputting the result description vector into the result classification layer of the first disease prediction model, and determining the disease diagnosis category to which the medical imaging report belongs by analyzing the result description vector in the result classification layer; The disease diagnosis category is determined as the predicted diagnosis result of the medical imaging report.

8. The diagnostic result prediction method according to claim 1, characterized in that: After extracting the image description information for describing the brain MRI image in the medical image report, the diagnosis result prediction method further includes: By identifying the medical imaging report, determining the diagnostic task type of the patient to which the medical imaging report belongs; Obtain candidate diagnosis results, result query statements, and disease diagnosis auxiliary templates of the diagnosis task type; Fill the image description information, the result query statement and the candidate diagnosis result into the disease diagnosis auxiliary template respectively, and generate a disease diagnosis statement for predicting the predicted diagnosis result of the medical image report; The disease diagnosis statement is input into a second disease prediction model to determine the predicted diagnosis result of the medical imaging report; wherein the second disease prediction model is obtained by fine-tuning the parameters of a pre-trained language model; and the pre-trained language model is a basic model pre-trained in a general scenario.

9. The diagnostic result prediction method according to claim 8, characterized in that: The diagnostic task types include one or more of an IDH genotype determination task, a WHO grading task, a brain tumor type determination task, and a 1p / 19q co-deletion task.

10. A diagnostic result prediction device based on medical image description, characterized in that: The diagnostic prediction device comprises: An information extraction module, used to extract description information from the acquired medical imaging report, and extract image description information used to describe the brain MRI image in the medical imaging report; A data mapping module, used for mapping the image description information into image description data word by word by preprocessing the image description information; A format adjustment module, used to perform format standardization processing on the image description data in combination with the data input format of the first disease prediction model, and convert the image description data into image description features; A result prediction module, used for inputting the image description feature into the disease result prediction layer of the first disease prediction model, and determining the result description vector of the medical image report through the disease result prediction layer; wherein the disease result prediction layer is obtained by training a Chinese language model pre-trained by a full-text masking method; The result classification module is used to input the result description vector into the result classification layer of the first disease prediction model, and determine the predicted diagnosis result of the medical imaging report through the result classification layer.

Images