A quality assessment method, device and storage medium for imaging diagnostic report

Abstract

The present application discloses a quality assessment method, device and storage medium for an imaging diagnostic report, comprising: determining a diagnostic report corresponding to an imaging examination of a target object, and an audit report corresponding to the diagnostic report; determining first text information that is different between the diagnostic report and the audit report, and generating first prompt information based on the first text information; identifying the first prompt information using a pre-trained difference analysis model, and determining error information corresponding to the first text information; in the case of determining that a target difference type includes a diagnostic error, grading the diagnostic error based on a pre-set disease grade library, and determining the corresponding grading result; and outputting a quality assessment result of the diagnostic report based on the error information and the grading result and using the difference analysis model.

Description

Technical Field

[0001] The present application relates to the field of medical diagnosis technology, and in particular to a quality assessment method, device and storage medium for imaging diagnostic reports. Background Art

[0002] As the number of medical imaging examinations increases, hospitals need to train more diagnostic teachers to cope with the huge number of examinations and generate initial reports based on imaging examinations (i.e., diagnostic reports). How to correctly evaluate the work quality of diagnostic teachers in writing diagnostic reports based on imaging examinations and achieve accurate quality control of each diagnostic report is a difficult problem.

[0003] There are usually two ways to solve the above problem in the prior art. First, a sampling evaluation of the diagnosis report is performed by manual sampling. Second, the quality of the diagnosis report is evaluated by determining the similarity between the diagnosis report and the audit report.

[0004] However, no matter which of the above quality assessment methods is used, there are certain defects. For the first method, the advantage is that it can ensure the professionalism and accuracy of the quality control results. The disadvantage is that the manual efficiency is low and a small amount of sampling will inevitably have deviations, which cannot truly reflect the quality of all diagnostic reports. For the second method, the advantage is that it can achieve quality control for all diagnostic reports. The disadvantage is that the similarity can only be judged at the text level, lacking the professionalism of the image description, and it is impossible to clarify the problems in the diagnostic report. In other words, it can only be evaluated quantitatively, not qualitatively.

[0005] With respect to the technical problem existing in the above-mentioned prior art of how to correctly evaluate the work quality of the diagnostic report written based on the imaging examination, so as to achieve accurate quality control of each diagnostic report, no effective solution has been proposed so far. Summary of the invention

[0006] The embodiments of the present disclosure provide a method, device and storage medium for quality assessment of imaging diagnostic reports, so as to at least solve the technical problem existing in the prior art of how to correctly assess the work quality of diagnostic reports written based on imaging examinations, thereby achieving accurate quality of each diagnostic report.

[0007] According to one aspect of an embodiment of the present disclosure, a quality assessment method for an imaging diagnostic report is provided, comprising: determining a diagnostic report corresponding to an imaging examination of a target object, and an audit report corresponding to the diagnostic report, wherein the diagnostic report is used to indicate a diagnostic opinion for the imaging examination, and the audit report is used to indicate a report generated by auditing the diagnostic report; determining first text information in which there is a difference between the diagnostic report and the audit report, and generating first prompt information based on the first text information; identifying the first prompt information using a pre-trained difference analysis model, and determining error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed site, and the target difference type includes non-diagnostic errors and / or diagnostic errors; in the case of determining that the target difference type includes a diagnostic error, grading the diagnostic errors based on a pre-set disease grade library, and determining corresponding grading results; and outputting a quality assessment result of the diagnostic report based on the error information and the grading result and using the difference analysis model, wherein the quality assessment result includes diagnostic problems existing in the diagnostic report and a quality score of the diagnostic report.

[0008] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a processor executes any one of the methods described above.

[0009] According to another aspect of the embodiments of the present disclosure, there is also provided a quality assessment device for an image diagnosis report, comprising: a report determination module, for determining a diagnosis report corresponding to an image examination of a target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate a diagnosis opinion for the image examination, and the audit report is used to indicate a report generated by auditing the diagnosis report; a first prompt information generation module, for determining first text information that is different between the diagnosis report and the audit report, and generating first prompt information based on the first text information; a difference type return module, for identifying the first prompt information using a pre-trained difference analysis model, and determining error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed site, and the target difference type includes non-diagnostic errors and / or diagnostic errors; a grading result determination module, for grading the diagnostic errors based on a pre-set disease grade library and determining corresponding grading results when it is determined that the difference type includes a diagnostic error; and a result output module, for outputting a quality assessment result of the diagnosis report based on the error information and the grading result and using the difference analysis model, wherein the quality assessment result includes the diagnostic problems existing in the diagnosis report and the quality score of the diagnosis report.

[0010] According to another aspect of an embodiment of the present disclosure, there is also provided a quality assessment device for an image diagnostic report, comprising: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: determining a diagnosis report corresponding to an image examination of a target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate a diagnosis opinion for the image examination, and the audit report is used to indicate a report generated by auditing the diagnosis report; determining first text information in which there is a difference between the diagnosis report and the audit report, and generating first prompt information based on the first text information; identifying the first prompt information using a pre-trained difference analysis model, and determining error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed site, and the target difference type includes non-diagnostic errors and / or diagnostic errors; when it is determined that the difference type includes a diagnostic error, grading the diagnostic errors based on a pre-set disease grade library, and determining the corresponding grading results; and outputting a quality assessment result of the diagnosis report based on the error information and the grading result and using the difference analysis model, wherein the quality assessment result includes diagnostic problems existing in the diagnosis report and a quality score of the diagnosis report.

[0011] The present application provides a quality assessment method for an imaging diagnostic report. First, a processor determines a diagnostic report corresponding to an imaging examination of a target object, and an audit report corresponding to the diagnostic report. Then, the processor determines a first text message in which there is a difference between the diagnostic report and the audit report, and generates a first prompt message based on the first text message. Further, a pre-trained difference analysis model is used to identify the first prompt message, and the error message corresponding to the first text message is determined. Afterwards, when it is determined that the target difference type includes a diagnostic error, the diagnostic error is graded based on a pre-set disease grade library, and the corresponding grading result is determined. Finally, the processor outputs a quality assessment result of the diagnostic report based on the error message and the grading result and using the difference analysis model.

[0012] With reference to the above-described contents, it can be known that in the present application, firstly, a first prompt message is generated based on the first text information that is different between the diagnostic report and the audit report. Then, based on the first prompt message and using a pre-trained difference analysis model, the error message corresponding to the first text message is determined. And further determine the target difference type (for example, non-diagnostic errors and / or diagnostic errors) in the error message corresponding to the first text message. Thus, through the above-mentioned method, not only the error message in the diagnostic report can be accurately determined, but also the difference type to which the error message belongs can be determined, whether it is a non-diagnostic error or a diagnostic error, thereby achieving accurate classification of the error message.

[0013] Secondly, when it is determined that the target difference type contains diagnostic errors, the disease grade library can be used to grade the diagnostic errors and determine the grading results. Then, the difference analysis model is used to output the quality assessment results of the diagnostic report based on the error information and the grading results. Among them, the quality assessment results include not only the diagnostic problems in the diagnostic report, but also the overall score of the diagnostic report. Thus, the technical effect of quantifying the quality of the diagnostic report based on the evaluation results and clarifying the diagnostic error information is achieved.

[0014] This solves the technical problem in the prior art of how to correctly evaluate the work quality of diagnostic reports written based on imaging examinations, thereby achieving accurate quality control of each diagnostic report. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0016] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0017] Figure 2 is a schematic diagram of a quality assessment system for imaging diagnostic reports according to Embodiment 1 of the present disclosure;

[0018] Figure 3 is a schematic diagram of another quality assessment system for imaging diagnostic reports according to Embodiment 1 of the present disclosure;

[0019] Figure 4 is a flow chart of the quality assessment method of the imaging diagnostic report according to Embodiment 1 of the present disclosure;

[0020] Figure 5A to Figure 5E is a schematic diagram of splitting the first task into an interaction process for interacting with a large language model according to Embodiment 1 of the present disclosure;

[0021] Figure 6 is a schematic diagram of fine-tuning a large language model through reinforcement learning according to Embodiment 1 of the present disclosure;

[0022] Figure 7 is a schematic diagram of a quality assessment device for an imaging diagnostic report according to Embodiment 2 of the present disclosure;

[0023] Figure 8 It is a schematic diagram of a quality assessment device for an imaging diagnostic report according to Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0026] Example 1

[0027] According to this embodiment, a method embodiment of quality assessment of an imaging diagnostic report is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing a quality assessment method for an imaging diagnostic report. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, the transmission device, and the input / output interface are connected to the processor via a bus. In addition, it may also include: a display, a keyboard, and a cursor control device connected to the input / output interface. A person skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computing device. As involved in the embodiments of the present disclosure, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0030] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the quality assessment method of the image diagnosis report in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the quality assessment method of the image diagnosis report of the above-mentioned application is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0031] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.

[0033] It should be noted that, in some optional embodiments, the above Figure 1 The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0034] Figure 2 is a schematic diagram of a quality assessment system for imaging diagnostic reports according to this embodiment. Figure 2 As shown, the system includes: a data receiving module, a text comparison module, a prompt information generation module and a pre-trained difference analysis model.

[0035] Wherein, the data receiving module is connected to the terminal device for communication, and is used to receive the diagnosis report and audit report corresponding to the target object through the terminal device. The text comparison module is connected to the data receiving module for communication, and is used to compare the text of the diagnosis report and the audit report, and determine the first text information that is different between the diagnosis report and the audit report in combination with the image knowledge base. The prompt information generation module is used to fill the received first text information into the pre-set second template and generate the first prompt information. The difference analysis model is used to determine and return the error information corresponding to the first text information according to the first prompt information. Wherein, the error information includes the target difference type, the misdiagnosis site and the error content. And wherein the target difference type includes non-diagnostic errors and / or diagnostic errors. In addition, it is worth noting that the difference analysis model in the present application is a large model.

[0036] In addition, the system also includes a grading module and a data output module. The grading module is used to determine whether the target difference type corresponding to the error information contains a diagnostic error, and if it contains a diagnostic error, the grading result of the diagnostic error is determined using the disease grade library. The prompt information generation module is used to fill the received error information and the grading result into a pre-set first template and generate a second prompt information. The difference analysis model is used to output a quality assessment result corresponding to the diagnostic report based on the second prompt information. The data output module is connected to the terminal device for communication, and is used to send the quality assessment result corresponding to the diagnostic report to the terminal device.

[0037] Furthermore, the system also includes a model training module, wherein the model training module is used to train the difference analysis model based on the training data in the database.

[0038] Figure 3 is a schematic diagram of another quality assessment system for imaging diagnostic reports according to this embodiment. Figure 3As shown, when the grading module determines that the target difference type does not include diagnostic errors and only includes non-diagnostic errors, the grading module sends the data information that does not include diagnostic errors to the prompt information generation module. The prompt information generation module fills the error information into the pre-set first template and generates a second prompt information. Thus, the difference analysis model can output the quality assessment result corresponding to the diagnostic report based on the second prompt information, and finally the data output module sends the quality assessment result to the terminal device.

[0039] It should be noted that each template in the system can be applied to the hardware structure described above.

[0040] Under the above operating environment, according to the first aspect of this embodiment, a method for evaluating the quality of an imaging diagnosis report is provided. Figure 2 The system implementation shown in . Figure 4 A schematic diagram showing the process of the method is shown in FIG. Figure 4 As shown, the method includes:

[0041] S402: Determine a diagnosis report corresponding to the imaging examination of the target object and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate a diagnosis opinion for the imaging examination, and the audit report is used to indicate a report generated by auditing the diagnosis report;

[0042] S404: Determine first text information that is different between the diagnosis report and the audit report, and generate first prompt information based on the first text information;

[0043] S406: using a pre-trained difference analysis model to identify the first prompt information, and determining error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed part, and the target difference type includes a non-diagnostic error and / or a diagnostic error;

[0044] S408: When it is determined that the target difference type includes a diagnosis error, grading the diagnosis error based on a preset disease grade library, and determining a corresponding grading result; and

[0045] S410: Based on the error information and the classification result and using the difference analysis model, output the quality assessment result of the diagnosis report, wherein the quality assessment result includes the diagnosis problem existing in the diagnosis report and the quality score of the diagnosis report.

[0046] Specifically, refer to Figure 2As shown, first, the operator sends the diagnosis report corresponding to the imaging examination of the target object and the audit report corresponding to the diagnosis report to the data receiving module through the terminal device, so that the data receiving module receives the diagnosis report and the audit report corresponding to the target object (S402). Among them, the diagnosis report is used to indicate the diagnostic opinion for the imaging examination of the target object. The audit report is used to indicate the report generated after the diagnosis report is audited. In actual situations, the diagnosis report is generally written by doctors with lower seniority or less experience, and the audit report is generally written by doctors with more experience.

[0047] In addition, the diagnostic report specifically includes: the gender of the target subject, the age of the target subject, the site examined, the type of equipment examined, the examination findings, and the diagnostic impression.

[0048] Example of diagnostic report structure:

[0049] Gender: Female. Age: 7 years old. Examination site: Chest. Examination equipment: CT.

[0050] Examination findings: The bilateral thorax is roughly symmetrical, the bronchial vessels of both lungs are clear, and strip-like high-density shadows can be seen in the middle lobe of the left lung, with a relatively uniform density. No enlarged lymph nodes were found in the bilateral hilum and mediastinum. The mediastinum did not show widening, and the position was centered without displacement. There were no abnormalities in the heart and large blood vessels, and no pleural effusion was found on both sides. No obvious abnormalities were found in the bone and soft tissue structure of the thorax within the scanning range. The bilateral cerebral hemispheres were symmetrical, and spots and patchy low-density lesions were found in the center of the bilateral semi-oval, the radiating crown, and the basal ganglia. The boundaries of some lesions were clear, some were blurred, and some were watery. There was no obvious expansion of the ventricular system, no widening of the cerebral sulci, cerebral columns, and cerebral cisterns, no displacement of the midline structure, and no abnormal density shadows were found in the brainstem and cerebellum. No abnormal changes were found in the skull bones.

[0051] Diagnostic impression: Atelectasis of the right middle lobe, the cause is unknown, and CT enhancement or bronchoscopy is recommended for further examination. Multiple ischemic lesions in the bilateral semiovale centrum, corona radiata, and base segment areas, and senile cerebral infarction. No abnormalities were found in the corpus cavernosum.

[0052] When the data receiving module receives the diagnosis report and audit report corresponding to the target object, the diagnosis report and audit report are sent to the text comparison template. The text comparison template uses the open source algorithm JAVA-Diff-Utils to perform text comparison on the diagnosis report and the audit report, and combines it with the pre-set imaging knowledge base to determine the whole sentence (i.e., the first text information) that is different between the diagnosis report and the audit report. Among them, the whole sentence refers to a complete meaning expression statement with a period and a semicolon as separators. After that, the text comparison module sends the first text information to the prompt information generation module, so that the prompt information generation module fills the first text information into the pre-set second template and generates the first prompt information (S404).

[0053] Further, the prompt information generation module sends the generated first prompt information to the pre-trained difference analysis model, and uses the difference analysis model to identify the first prompt information, thereby determining the error information corresponding to the first text information (S406). The error information includes the target difference type, the misdiagnosis site, and the error content. The target difference type includes non-diagnostic errors and / or diagnostic errors. And the diagnostic errors include missed diagnosis, overdiagnosis, misdiagnosis, lesion localization error, orientation error, lesion grading error, measurement error, and irregular description.

[0054] Table 1 shows the difference types, the scores corresponding to the respective difference types, and the descriptions of the difference types.

[0055] Table 1

[0056]

[0057] Referring to Table 1, it can be seen that, for example, the difference analysis model determines that the target difference type includes non-diagnostic errors and diagnostic errors based on the first prompt information corresponding to the first text information, and the diagnostic error is missed diagnosis. That is, in the diagnosis report, not only is "mediastinum" written as "septum" (i.e., non-diagnostic error), but also the description of the "lung" lesion is missing (i.e., missed diagnosis in diagnostic errors).

[0058] Further, for example, the difference analysis model receives the following first prompt information:

[0059] {"role":"system",

[0060] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0061] {"role":"user","content":"A:null;B:interstitial lesions in the left upper lobe;"}.

[0062] And the difference analysis model outputs the following error types of diagnostic errors corresponding to the error information based on the first prompt information: {"role":"assistant",

[0063] "content":"Diagnostic error type: minor missed diagnosis;

[0064] Missed diagnosis site: left lung;

[0065] Error content: {A: null; B: interstitial lesions}"}.

[0066] Thus, when the difference analysis model receives the first prompt message, it can determine whether the error message belongs to a non-diagnostic error or a diagnostic error, thereby achieving the technical effect of accurately classifying the difference type to which the error message belongs.

[0067] After that, the difference analysis model sends the target difference type corresponding to the error message to the grading module, and the grading module determines whether the target difference type contains a diagnostic error. When the grading module determines that the target difference type contains a diagnostic error, it grades the diagnostic error based on a pre-set disease grading library and determines the corresponding grading result (S408). Among them, the disease grading library includes a disease difference grading table. The disease difference grading table shows the severity of the disease difference type.

[0068] Table 2 shows the disease difference grading table.

[0069] Table 2

[0070]

[0071] Referring to Table 2, for example, if the disease name is "pneumonia", the corresponding lesion grade is "mild".

[0072] Furthermore, for example, if the target difference type output by the difference analysis model includes a diagnostic error, and the diagnostic error includes "misdiagnosis", then the grading module can determine the grading result as "mild" based on "misdiagnosis + pneumonia" sent by the text comparison module.

[0073] Then, the prompt message generation module fills the error message sent by the difference analysis model and the grading result sent by the grading module into a pre-set first template and generates a second prompt message.

[0074] Finally, the prompt message generation module inputs the second prompt message into the difference analysis model and uses the difference analysis model to output the quality assessment result of the diagnostic report (S410). Among them, the quality assessment result includes the diagnostic problems existing in the diagnostic report and the quality score of the diagnostic report. For example, input the second prompt message into the difference analysis model, and the difference analysis model outputs: Total score: 80; Diagnostic problem: There is a misdiagnosis in the left lung, it should be interstitial change instead of pneumonia; Typo problem: "bianjian" is written as "bianjian".

[0075] As described in the background technology, in this application, firstly, a first prompt message is generated based on the first text information that is different between the diagnostic report and the audit report. Then, based on the first prompt message and using a pre-trained difference analysis model, the error message corresponding to the first text message is determined. And further determine the target difference type (for example, non-diagnostic errors and / or diagnostic errors) in the error message corresponding to the first text message. Therefore, through the above method, not only can the error information in the diagnostic report be accurately determined, but also it can be determined whether the difference type to which the error information belongs is a non-diagnostic error or a diagnostic error, thereby achieving accurate classification of the error information.

[0076] Secondly, when it is determined that the target difference type contains diagnostic errors, the disease grade library can be used to grade the diagnostic errors and determine the grading results. Then, the difference analysis model is used to output the quality assessment results of the diagnostic report based on the error information and the grading results. Among them, the quality assessment results include not only the diagnostic problems in the diagnostic report, but also the overall score of the diagnostic report. Thus, the technical effect of quantifying the quality of the diagnostic report based on the evaluation results and clarifying the diagnostic error information is achieved.

[0077] This solves the technical problem in the prior art of how to correctly evaluate the work quality of diagnostic reports written based on imaging examinations, thereby achieving accurate quality control of each diagnostic report.

[0078] also, Figure 2 The above content is the case where the classification module determines that the target difference type includes diagnostic errors, or diagnostic errors and non-diagnostic errors. In addition to the above situation, the target difference type of the error information may only include non-diagnostic errors, but not diagnostic errors.

[0079] refer to Figure 3 As shown, when the grading module receives the target difference type sent by the difference analysis model, it determines whether the target difference type contains diagnostic errors. Further, the grading module determines that the target difference type does not contain diagnostic errors, but only contains non-diagnostic errors, and sends the data information corresponding to "only contains non-diagnostic errors" to the prompt information generation module. The prompt information generation module fills the error information sent by the difference analysis model into the first template, thereby generating a second prompt information. When the difference analysis model receives the second prompt information, it outputs the diagnostic problems existing in the diagnostic report and the quality score of the diagnostic report.

[0080] Optionally, the operation of outputting the quality assessment results of the diagnostic report based on the error information and grading results and using a difference analysis model also includes: filling the error information and grading results into a pre-set first template and generating a second prompt information; and inputting the second prompt information into the difference analysis model, and using the difference analysis model to output the quality assessment results of the diagnostic report.

[0081] Specifically, when the prompt information generation module receives the target difference type corresponding to the error information and the grading result corresponding to the error information sent by the large model, the error information and the grading result are filled into the preset first template, and the second prompt information is generated.

[0082] The first template may be, for example:

[0083] {"role":"system",

[0084] "content": "You are a medical report expert in the field of medical imaging. Your task is to summarize the differences between the provided diagnostic report description A and the audit report description B - merge the differences in the same part and quantify the diagnostic score of A according to the defined score"},

[0085] {"role":"user",

[0086] "content": "***"}.

[0087] When the prompt information generation module fills the error information and the classification result into the preset first template, for example, the second prompt information shown below may be generated:

[0088] {"role":"system",

[0089] "content": "You are a medical report expert in the field of medical imaging. Your task is to summarize the differences between the provided diagnostic report description A and the audit report description B - merge the differences in the same part and quantify the diagnostic score of A according to the defined score"},

[0090] {"role":"user",

[0091] "content": Diff1{"Diagnostic error type: slight misdiagnosis; Misdiagnosed site: left lung; Error content: {A: flake-like increased density; B: grid-like changes}"}

[0092] Diff2{"Diagnostic error type: minor misdiagnosis; Misdiagnosed site: left lung; Error content: {A: pneumonia; B: interstitial lesions}"}

[0093] Diff3{"Diagnostic error type: typo; Error content: {A: bian jian; B bian jian}"}.

[0094] After that, the prompt information generation module inputs the second prompt information into the difference analysis model and uses the difference analysis model to output the quality evaluation result of the diagnostic report. For example, the quality evaluation result corresponding to the diagnostic report is:

[0095] {"role": "assistant",

[0096] "content": "Total score: 100 - 15 - 5 = 80;

[0097] Diagnostic problem: There is a misdiagnosis in the left lung, which should be interstitial change instead of pneumonia. Typo problem, writing 'bian jian' as 'bian jian'"}.

[0098] Thus, from the above content, it can be seen that since the prompt information generation module has preset the first template, the prompt information generation module can generate the second prompt information as long as it fills the received error information and grading result into the first template. That is, due to the generality of the first template, the prompt information generation module can not only generate the second prompt information quickly, but also further increase the applicable scope of the difference analysis model.

[0099] Optionally, the operation of inputting the second prompt information into the difference analysis model and using the difference analysis model to output the quality evaluation result of the diagnostic report includes: inputting the second prompt information into the difference analysis model and using the difference analysis model to determine whether the second prompt information contains multiple prompt sub-information, where the grading results and misdiagnosis parts of each prompt sub-information are the same; in the case that the second prompt information contains multiple prompt sub-information, retaining any one of the prompt sub-information in the second prompt information and removing the remaining prompt sub-information; and using the difference analysis model and based on the processed second prompt information, outputting the quality evaluation result of the diagnostic report.

[0100] Specifically, when the prompt information generation module generates the second prompt information and inputs the second prompt information into the difference analysis model, the difference analysis model determines whether the second prompt information contains multiple prompt sub-information. Among them, the grading results and misdiagnosis parts of each prompt sub-information are the same.

[0101] For example, the second prompt information in the above example contains two prompt sub-information for the same misdiagnosed part, and the misdiagnosed part of the two prompt sub-information is the left lung. That is, Diff1: Diagnostic error type: slight misdiagnosis; misdiagnosed part: left lung; error content: A: increased density in sheets; B: grid-like changes. Diff2: Diagnostic error type: slight misdiagnosis; misdiagnosed part: left lung; error content: A: pneumonia; B: interstitial lesions.

[0102] Therefore, when the second prompt information includes a plurality of prompt sub-information, any one prompt sub-information in the second prompt information is retained, and the remaining prompt sub-information in the second prompt information is deleted.

[0103] For example, the second prompt information contains two prompt sub-information, and the diagnostic site corresponding to the two prompt sub-information is the left lung. Further, the difference analysis model determines that the target difference type and the grading results of the two prompt sub-information are the same. That is, both are slight misdiagnoses. Therefore, the difference analysis model retains any one of the two prompt sub-information in the second prompt information, and removes the other prompt sub-information, thereby generating the processed second prompt information.

[0104] Finally, the differential analysis model uses the processed second prompt information and outputs the quality assessment results of the diagnostic report. For example, when it is determined that the misdiagnosed part corresponding to the two prompt sub-information is the left lung, and the grading results are both slight misdiagnoses, the output quality assessment results only show "the left lung is misdiagnosed, it should be interstitial changes instead of pneumonia", and do not show "it should be grid-like changes, not sheet-like increased density shadows".

[0105] Therefore, through the above method, the technical effect of solving the problem of misdiagnosis of the same part and avoiding repeated deductions can be achieved.

[0106] Optionally, the operation of generating the first prompt information based on the first text information includes: filling the first text information into a preset second template, and generating the first prompt information.

[0107] Specifically, when the prompt information generation module receives the first text information sent by the text comparison module, the first text information is filled into the preset second template, and the first prompt information is generated.

[0108] The second template may be, for example:

[0109] {"role":"system",

[0110] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0111] {"role":"user",

[0112] "content": "***"}.

[0113] Then, the prompt information generation module inputs the first prompt information into the difference analysis model, and uses the difference analysis model to output the target difference type corresponding to the first text information. For example, the target difference type corresponding to the first text information is:

[0114] {"role":"assistant",

[0115] "content":"Diagnostic error type: misdiagnosis;

[0116] Misdiagnosed site: left lung;

[0117] Incorrect content: {A: pneumonia; B: interstitial lesions}.

[0118] Therefore, it can be seen from the above content that since the prompt information generation module pre-sets the second template, the prompt information generation module can generate the first prompt information by simply filling the received first text information into the second template. That is, since the second template is universal, the prompt information generation module can not only quickly generate the first prompt information, but also further increase the scope of application of the difference analysis model.

[0119] Optionally, the operation of determining the first text information that differs between the diagnosis report and the audit report includes: extracting the second text information in the diagnosis report and the third text information in the audit report; and comparing the second text information and the third text information, and determining the first text information that differs in combination with a pre-set imaging knowledge base.

[0120] Specifically, when the text comparison module receives the diagnosis report and the audit report, the second text information in the diagnosis report and the third text information in the audit report are first extracted. Then, the text comparison module performs a text comparison on the second text information of the diagnosis report and the third text information of the audit report, and combines the image knowledge base to find the whole sentence (i.e., the first text information) in which there are differences between the second text information and the third text information. Among them, the differences are mainly divided into three categories. First, the description content of the second text information of the diagnosis report and the third text information of the audit report are different. For example, the second text information is: The bone at the anterior edge of the cervical vertebral body is slightly changed. The third text information is: The bone at the anterior edge of the cervical vertebral body is slightly pointed.

[0121] Second, the second text information of the diagnosis report has some descriptions, but the third text information of the audit report lacks corresponding descriptions. For example, the second text information is: the bone at the front edge of the cervical vertebral body is slightly pointed. The third text information is none.

[0122] Third, the third text information of the audit report has some descriptions, but the second text information of the diagnosis report lacks corresponding descriptions. For example, the second text information is none, and the third text information is: the bone at the front edge of the cervical vertebral body is slightly pointed.

[0123] Therefore, through the above method, it is possible to quickly determine the difference text between the diagnostic report and the audit report, roughly judge whether there is a difference between the diagnostic report and the audit report, and provide the necessary technical basis for the subsequent accurate determination of the target difference type between the diagnostic report and the audit report.

[0124] Optionally, the method also includes training a difference analysis model through the following steps: pre-training a preset large language model using an imaging report; fine-tuning the pre-trained large language model using samples corresponding to the first task, so that the large language model can be used to implement the first task, wherein the first task is used to determine the difference type, misdiagnosed part and erroneous content corresponding to the difference complete sentence corresponding to the diagnosis report and the audit report; and performing a second fine-tuning on the fine-tuned large language model using samples corresponding to the second task, so as to obtain a difference analysis model, wherein the second task solves the problem of merging the same misdiagnosed parts in the diagnosis report to avoid repeated deductions, and summarizes the diagnosis problems at the same time.

[0125] Specifically, the difference analysis model is obtained by further training on the basis of a pre-set large language model (hereinafter referred to as the "large language model"). Preferably, the large language model can be, for example, a large language model based on llama. According to an embodiment of the present application, the training of the difference analysis model is divided into three stages:

[0126] In the first stage, the large language model is pre-trained using the imaging report, including: pre-training the large language model using the imaging report and the homophone dictionary to enable the large language model to recognize typos in the imaging report; and pre-training the large language model using the imaging report to enable it to understand the description paradigm of the imaging report and recognize the typical features of each description, such as anatomical parts, signs of lesions, lesions, etc. Since this part of the content has been disclosed by many existing technologies, it is not the focus of the present invention and will not be repeated here.

[0127] In the second stage, the pre-trained large language model is fine-tuned using samples corresponding to the first task so that the large language model can be used to implement the first task, wherein the first task is to determine the difference type, misdiagnosis location, and error content corresponding to the difference complete sentence corresponding to the diagnosis report and the audit report based on the difference complete sentence. After the fine-tuning in the second stage, the large language model can achieve the following functions: after inputting the difference complete sentence of the diagnosis report and the audit report, it can feedback the corresponding difference type (excluding confirmation of the grading result), misdiagnosis location, and error content. For example:

[0128] Input the following information to the large language model:

[0129] {"role":"system",

[0130] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0131] {"role":"user","content":"A:null;B:interstitial lesions in the left upper lobe;"}.

[0132] The large language model can provide feedback on the following information:

[0133] {"role":"assistant",

[0134] "content":"Diagnostic error type: minor missed diagnosis;

[0135] Missed diagnosis site: left lung;

[0136] Error content: {A: null; B: interstitial lesions}"}.

[0137] The specific training process of the second stage will be described in detail below.

[0138] In the third stage, the fine-tuned large language model is fine-tuned again using samples corresponding to the second task to obtain a differential analysis model. The second task solves the problem of merging the same misdiagnosed parts in the diagnosis report to avoid repeated deductions and summarize the diagnosis problems. Specifically, after the second task training, the following information is input into the differential analysis model:

[0139] {"role":"system",

[0140] "content": "You are a medical report expert in the field of medical imaging. Your task is to summarize the differences between the provided diagnostic report description A and the review report description B - merge the differences in the same location, and quantify the diagnostic score of A according to the defined score."

[0141] {"role": "user",

[0142] "content": "Diff1 {Diagnostic error type: minor misdiagnosis; Misdiagnosed location: left lung; Error content: {A: patchy increased density shadow; B: reticular change}}"

[0143] Diff2 {Diagnostic error type: minor misdiagnosis; Misdiagnosed location: left lung; Error content: {A: pneumonia; B: interstitial lesion}}"

[0144] Diff3 {Diagnostic error type: typo; Error content: {A: bian jian; B: bian jian}}"

[0145] Then the large language model after fine-tuning in the third stage can return the following information:

[0146] {"role": "assistant",

[0147] "content": "Total score: 100 - 15 - 5 = 80 Diagnostic problem: There is a misdiagnosis in the left lung, it should be interstitial change instead of pneumonia. Typo problem, writing 'bian jian' as 'bian jian'}"

[0148] The fine-tuning in the third stage will be described in detail below.

[0149] Optionally, the operation of fine-tuning the pre-trained large language model using the samples corresponding to the first task includes: using the trajectory samples corresponding to the interaction process after splitting the first task, fine-tuning the large language model based on reinforcement learning; and using the input-output pair samples corresponding to the first task to fine-tune the large language model.

[0150] Specifically, in the second stage, the operation of fine-tuning the pre-trained large language model can be further divided into two stages. Specifically as follows:

[0151]

Stage 2 - 1

[0152] First, split the specific execution process of task 1 to obtain the interaction process corresponding to interacting with the large language model for task 1. Specifically, Figure 5A to Figure 5E is the specific flowchart for splitting task 1 according to the embodiments of the present application.

[0153] The following is a brief description of the steps in the flowchart. The prompt information that needs to be entered into the large language model can refer to the template and prompt information format described above, which will not be repeated here. Figure 5A to Figure 5E As shown in the figure, the interaction process after task 1 is disassembled is as follows:

[0154] S502: Obtain the difference complete sentences corresponding to the diagnosis report and the audit report, that is, the complete sentence A of the diagnosis report and the complete sentence B of the audit report.

[0155] S504: Request the large language model to determine whether there is a complete sentence difference, that is, whether the complete sentence A or the complete sentence B is empty (NULL).

[0156] S506: Receive feedback information from the large language model.

[0157] S508: When the feedback information of the large language model is "multiple sentences", that is, the complete sentence A is not empty and the complete sentence B is empty, if the feedback information of the large language model is wrong, the task is judged to have failed, otherwise the large language model is requested to determine the entities in the complete sentence A of the diagnosis report and the corresponding entity types. Specifically, in this solution, entity types corresponding to medical terms and diagnosis conclusions are predefined. Table 3 below shows some examples of entity types:

[0158] Table 3 Entity Type

[0159]

[0160] The large language model then performs NLP analysis on the complete sentence to determine its entities and corresponding entity types, for example:

[0161] The complete sentence "The bone at the front edge of the cervical vertebrae becomes slightly pointed" can be decomposed into the following entities:

[0162] 1) "cervical vertebra", part entity;

[0163] 2) “front edge”, directional entity;

[0164] 3) "The bone becomes slightly pointed", a positive sign entity.

[0165] Thus, the large language model can determine the entities and corresponding entity types in the complete sentence of the diagnosis report according to the input prompt information.

[0166] S520: When the feedback information from the large language model is "missing sentence", that is, the complete sentence A is empty but the complete sentence B is not empty, if the feedback information from the large language model is incorrect, the task is determined to have failed, otherwise the large language model is requested to determine the entities in the complete sentence B of the audit report and the corresponding entity types.

[0167] S532: When the feedback information from the large language model is "incomplete sentence difference", that is, both the complete sentence A and the complete sentence B are not empty, if the feedback information from the large language model is incorrect, the task is determined to have failed. Otherwise, the large language model is requested to determine the entities in the complete sentence A of the diagnosis report and the complete sentence B of the audit report as well as the corresponding entity types.

[0168] S560: When the feedback information from the large language model is other information, it is directly determined that the task has failed.

[0169] refer to Figure 5B As shown, following step S508, the entity and entity type of the complete sentence A returned by the large language model are received.

[0170] S510: If the feedback information from the large language model is incorrect, the task is directly deemed to have failed. Otherwise, the large language model is requested to determine whether the diagnosis report has an "overdiagnosis" error, that is, the complete sentence A contains a positive sign entity. If the large language model determines that it is "overdiagnosed", the process proceeds to step S512. If the large language model determines that it is "not overdiagnosed", the process proceeds to step S516.

[0171] S512: When the large language model determines that the diagnosis is "overdiagnosed", if the feedback information is incorrect, the task is directly deemed to have failed. Otherwise, the large language model is requested to return the difference type, the misdiagnosed part, and the error content.

[0172] S514: Receive the difference type, misdiagnosis part and error content returned by the large language model. If the feedback information of the large language model is wrong, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0173] S516: When the large language model determines that the diagnosis is not overdiagnosed, if the feedback information is wrong, the task is directly considered to have failed, otherwise the large language model is requested to return the difference type, the misdiagnosis part, and the error content. The difference type includes: description redundancy, etc.

[0174] S518: Receive the difference type, misdiagnosis part and error content returned by the large language model. If the feedback information from the large language model is incorrect, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0175] refer to Figure 5C As shown, following step S520, the entity and entity type of the complete sentence B returned by the large language model are received.

[0176] S522: If the feedback information from the large language model is incorrect, the task is directly deemed to have failed. Otherwise, the large language model is requested to determine whether the diagnosis report has a "missed diagnosis" error, that is, the complete sentence B contains a positive sign entity. If the large language model determines that it is "missed diagnosis", the process proceeds to step S524. If the large language model determines that it is "not missed diagnosis", the process proceeds to step S528.

[0177] S524: When the large language model determines that a diagnosis is “missed”, if the feedback information is incorrect, the task is directly deemed to have failed. Otherwise, the large language model is requested to return the difference type, misdiagnosis site, and error content.

[0178] S526: Receive the difference type, misdiagnosis site and error content returned by the large language model. If the feedback information from the large language model is incorrect, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0179] S528: When the large language model determines that the diagnosis is not missed, if the feedback information is wrong, the task is directly considered to have failed. Otherwise, the large language model is requested to return the difference type, misdiagnosis part, and error content. The difference types include: lack of measurement, lack of suggestion, lack of comparison, incomplete description, etc.

[0180] S530: Receive the difference type, misdiagnosis part and error content returned by the large language model. If the feedback information of the large language model is wrong, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0181] refer to Figure 5D As shown, following step S532, the entities and entity types of the complete sentence A and the complete sentence B returned by the large language model are received.

[0182] S534: If the feedback information of the large language model is wrong, the task is directly judged as failed. Otherwise, the large language model is requested to determine the positive difference type of the difference complete sentence according to the positive entities of complete sentence A and complete sentence B. If complete sentence A and complete sentence B are one yin and one yang, then go to step S536. If complete sentence A and complete sentence B are both negative (two yin), then go to step S542. If complete sentence A and complete sentence B are both positive (two yang), then go to Figure 5E Step S550 is shown.

[0183] S536: When the positive difference type returned by the large language model is one negative and one positive, if the feedback information of the large language model is incorrect, the task is directly determined to have failed. Otherwise, the large language model is requested to determine whether the difference type is over-diagnosis or under-diagnosis.

[0184] S538: If the feedback information from the large language model is incorrect, the task is directly judged as failed. Otherwise, the large language model is requested to return the difference type, misdiagnosis part, and error content.

[0185] S540: Receive the difference type, misdiagnosis part and error content returned by the large language model. If the feedback information of the large language model is wrong, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0186] S542: When the positive difference type returned by the large language model is two negatives, if the feedback information of the large language model is incorrect, the task is directly judged to have failed. Otherwise, the large language model is requested to judge the difference type of the diagnosis report from the following items: no error; irregular description; typo...

[0187] S544: When the large language model determines that the difference type is no difference, if the feedback information of the large language model is incorrect, the task is directly determined to have failed; otherwise, the task is determined to have succeeded.

[0188] S546: When the large language model determines that the difference type is an irregular description or typo, if the feedback information from the large language model is incorrect, the task is directly deemed to have failed. Otherwise, the large language model is requested to return the difference type, the misdiagnosed part, and the error content.

[0189] S548: If the feedback information from the large language model is incorrect, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0190] refer to Figure 5E As shown, following step S534, the positive difference type returned by the large language model is received as two positives.

[0191] S550: When the positive difference type returned by the large language model is two positives, if the feedback information of the large language model is incorrect, the task is directly judged as failed. Otherwise, the difference type of the diagnosis report is determined from the following items: misdiagnosis, lesion localization error, orientation error, lesion grading error, measurement error, etc.

[0192] S552: When the difference type returned by the large language model is "misdiagnosis", that is, the positive sign entities of sentence A and sentence B are different, if the feedback information of the large language model is wrong, the task is directly judged as failed, otherwise the large language model is requested to return the difference type, misdiagnosis part and error content.

[0193] S554: Receive the difference type, misdiagnosis part and error content returned by the large language model. If the feedback information of the large language model is incorrect, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0194] S556: When the difference type returned by the large language model is lesion localization error, orientation error, lesion grading error or measurement error, if the feedback information of the large language model is incorrect, the task is directly judged as failed. Otherwise, the large language model is requested to return the difference type, misdiagnosis site and error content.

[0195] S558: Receive the difference type, misdiagnosis part and error content returned by the large language model. If the feedback information of the large language model is incorrect, the task is directly deemed to have failed; otherwise, the task is deemed to have succeeded.

[0196] Therefore, through the above process, the large language model can be trained to have reasoning capabilities related to Task 1 through the training model.

[0197] Then, based on the above execution process, trajectory samples for training the large language model are generated.

[0198] For example, the complete sentence A of the diagnosis report and the complete sentence B of the audit report are as follows:

[0199] A: Slight bone changes are seen at the anterior edge of the cervical vertebral body;

[0200] B: The bone at the front edge of the cervical vertebral body becomes slightly pointed.

[0201] Based on the difference complete sentence shown above, the following trajectory can be generated:

[0202]

[0203] According to the above process, the following trajectory samples can be formed:

[0204] S={Q1,V1; Q2,V2;... ; Q5,V5}.

[0205] Thus, referring to the method described above, a sample set for training a large language model can be constructed, wherein the sample set includes multiple trajectory samples {S1, S2, S3, .......}.

[0206] Then, Figure 6 is a schematic diagram of fine-tuning a large language model through reinforcement learning according to an embodiment of the present disclosure. Figure 6 As shown, by using sample trajectories, an environment unit that executes the above-mentioned interaction process is constructed in the training module, and then the environment unit is used to fine-tune the large language model through reinforcement learning based on a preset reward mechanism.

[0207] During fine-tuning, the environment unit follows Figures 5A to 5E The interactive process shown is to input the request Q in the trajectory sample to the large language model iThe large language model is based on the Q input of the environment unit i Generate Feedback A i The environment unit will then feedback A i With the corresponding answer V i Compare and confirm whether the task has failed one by one. If the task has not failed, the environment unit will give a corresponding reward R for each interaction. i , and issue the next request Q i+1 This cycle repeats until the task is determined to be successful.

[0208] Regarding the reward mechanism, for example, it can be set that, for a trajectory sample, only the reward R returned in the last interaction round in which the task is judged to be successful is 1, and the rewards returned in other rounds are 0.

[0209] Then, on this basis, the training module can fine-tune the large language model according to known reinforcement learning methods. For example, value learning or strategy learning, or the Actor-Critic algorithm can also be used. I will not go into details here.

[0210] [Phase 2-2]: Fine-tune the large language model using a sample of input-output pairs corresponding to the first task.

[0211] Specifically, an example of a sample pair is shown below:

[0212] Sample pair 1: Misdiagnosis error:

[0213] [{"role":"system",

[0214] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0215] {"role":"user",

[0216] "content": "A: pneumonia in the left upper lobe; B: interstitial lesions in the left upper lobe;"},

[0217] {"role":"assistant",

[0218] "content":"Diagnostic error type: minor misdiagnosis; Misdiagnosed site: left lung; Error content: {A: pneumonia; B: interstitial lesions}"}]}.

[0219] Sample pair 2: missed diagnosis error:

[0220] [{"role":"system",

[0221] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0222] {"role":"user",

[0223] "content": "A: null; B: interstitial lesions in the left upper lobe;"},

[0224] {"role":"assistant",

[0225] "content":"Diagnostic error type: minor missed diagnosis; Missed diagnosis site: left lung; Error content: {A:; B: interstitial lesions}"}]}.

[0226] Sample pair 3: Typos data:

[0227] [{"role":"system",

[0228] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0229] {"role":"user",

[0230] "content": "A: The bone at the front edge of the cervical vertebral body is slightly changed; B: The bone at the front edge of the cervical vertebral body is slightly pointed;"},

[0231] {"role":"assistant",

[0232] "content": "Diagnostic error type: typo; Error content: {A: becomes visible; B becomes sharp}"}]}. ......

[0233] Among them, System is the system prompt word, that is, the role definition of the model, user is the user input content, and assistant is the standard answer.

[0234] Therefore, when training the large model, the prompt information in the sample is input into the large model, and the loss is calculated based on the difference between the predicted output of the large model and the actual result, for example:

[0235] Taking the above missed diagnosis training sample 1 as an example, the prompt information (i.e. input sample) for inputting into the large model is:

[0236] {"role":"system",

[0237] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0238] {"role":"user",

[0239] "content": "A: pneumonia in the left upper lobe; B: interstitial lesions in the left upper lobe;"},

[0240] {"role":"assistant",

[0241] "content":"Diagnostic error type: minor misdiagnosis; Misdiagnosed site: left lung; Error content: {A: pneumonia; B: interstitial lesions}"}]}.

[0242] Among them, "diagnosis error type: slight misdiagnosis" is the diagnosis error type sample ST, "misdiagnosis site: left lung" is the misdiagnosis site sample SL, and "error content: {A: pneumonia; B: interstitial lesions}" is the error content sample SC. The above is the output sample.

[0243] During training, input the following prompt information to the large model:

[0244] {"role":"system",

[0245] "content": "You are a medical reporting expert in the field of medical imaging. Your task is to identify the diagnostic error types of A and B based on the provided diagnostic report description A and review report description B"},

[0246] {"role":"user",

[0247] "content":"A: Pneumonia in the left upper lobe; B: Interstitial lesions in the left upper lobe;"}.

[0248] Then, the big model will predict and output the difference type, misdiagnosis site, and error content based on the above input information, which are denoted as ET, EL, and EC.

[0249] Therefore, the loss function can be constructed based on the loss between the diagnostic error type sample ST and the predicted diagnostic difference type ET, the loss between the misdiagnosed part sample SL and the predicted misdiagnosed part EL, and the loss between the error content sample SC and the predicted error content EC:

[0250] .

[0251] Where m is the number of samples. T( ) is the diagnostic error type loss function, L L ( ) is the misdiagnosis site loss function and L C ( ) is the error content loss function. k T , k L and k C are the weight values ​​corresponding to the three losses respectively.

[0252] Thus, through the fine-tuning process of the above-mentioned stage 2-1 and stage 2-2, the fine-tuning of the difference analysis model is completed.

[0253] At present, for Task 1 described in this application, the conventional training method is still to fine-tune the large language model in the form of input-output pairs. However, there is a problem with the conventional training method, that is, the trained large model has weak reasoning ability. For Task 1, if the large model has weak reasoning ability, once a difference sentence different from the training sample appears during the application process, the accuracy of the judgment will decrease accordingly. Therefore, it is necessary to strengthen the reasoning ability of the large model used to perform Task 1.

[0254] If you want to enhance the reasoning ability of the large model, you need to construct trajectory samples to enhance the reasoning ability of the large model. However, since the execution process of Task 1 only involves one interaction, that is, inputting the difference sentence between the diagnosis report and the audit report, and outputting information such as the type of diagnosis difference, the misdiagnosis site, and the error content, it is difficult to introduce trajectory samples during the training process to enhance the reasoning ability of the large model.

[0255] In view of this, the inventor further refined and split the process of Task 1, thereby further splitting Task 1 into an interactive process that interacts with the large model. And this interactive process is consistent with the reasoning process of completing Task 1. Therefore, by fine-tuning in this way, the large language model has the reasoning ability corresponding to Task 1. On this basis, the input-output sample pairs are used to fine-tune the large language model, so that the large language model can more accurately determine the difference type of the diagnosis report.

[0256] Optionally, the operation of performing secondary fine-tuning on the fine-tuned large language model using samples corresponding to the second task includes: fine-tuning the large language model using input-output pair samples corresponding to the second task.

[0257] Specifically, examples of input-output pair samples corresponding to the second task are as follows:

[0258] [{"role":"system",

[0259] "content": "You are a medical report expert in the field of medical imaging. Your task is to summarize the differences between the provided diagnostic report description A and the review report description B - merge the differences in the same location and quantify the diagnostic score of A according to the defined score"}

[0260] {"role": "user",

[0261] "content": "Diff1{Diagnostic error type: minor misdiagnosis; Misdiagnosed location: left lung; Error content: {A: patchy increased density shadow; B: reticular change}}"}

[0262] Diff2{Diagnostic error type: minor misdiagnosis; Misdiagnosed location: left lung; Error content: {A: pneumonia; B: interstitial lesion}}

[0263] Diff3{Diagnostic error type: typo; Error content: {A: bian jian; B bian jian}}

[0264] {"role": "assistant",

[0265] "content": "Total score: 100 - 15 - 5 = 65;

[0266] Diagnostic problem: There is a misdiagnosis in the left lung. It should be interstitial change instead of pneumonia. Typo problem, writing 'bian jian' as 'bian jian'}"}}

[0267] Among them, the following part is the input sample:

[0268] {"role": "system",

[0269] "content": "You are a medical report expert in the field of medical imaging. Your task is to summarize the differences between the provided diagnostic report description A and the review report description B - merge the differences in the same location and quantify the diagnostic score of A according to the defined score"}

[0270] {"role": "user",

[0271] "content": "Diff1{Diagnostic error type: minor misdiagnosis; Misdiagnosed location: left lung; Error content: {A: patchy increased density shadow; B: reticular change}}"}

[0272] Diff2{Diagnostic error type: minor misdiagnosis; Misdiagnosed location: left lung; Error content: {A: pneumonia; B: interstitial lesion}}

[0273] Diff3{Diagnostic error type: typo; Error content: {A: bian jian; B bian jian}}

[0274] The following part is the output sample:

[0275] {"role": "assistant",

[0276] "content": "Total score: 100 - 15 - 5 = 80;

[0277] Diagnostic problem: Misdiagnosis occurred in the left lung, which should be interstitial change instead of pneumonia. There is a typo problem, writing 'bianjian' as 'bianjian'}}.

[0278] Among them, 80 is the score sample SS. 'Diagnostic problem: Misdiagnosis occurred in the left lung, which should be interstitial change instead of pneumonia. There is a typo problem, writing 'bianjian' as 'bianjian'' is the diagnostic problem sample SP.

[0279] When making fine-tuning, input the following prompt information into the large model:

[0280] {"role": "system":,

[0281] "content": "You are a medical report expert in the field of medical imaging. Your task is to summarize the difference content between the provided diagnostic report description A and the review report description B - merge the differences in the same part, and quantify the diagnostic score of A according to the defined score"}

[0282] {"role": "user",

[0283] "content": "Diff1{Diagnostic error type: Minor misdiagnosis; Misdiagnosed part: Left lung; Error content: {A: Patchy increased density shadow; B: Reticular change}}"}

[0284] Diff2{Diagnostic error type: Minor misdiagnosis; Misdiagnosed part: Left lung; Error content: {A: Pneumonia; B: Interstitial lesion}}

[0285] Diff3{Diagnostic error type: Typo; Error content: {A: Bianjian; B Bianjian}}.

[0286] Thus, the predicted score ES and the predicted diagnostic problem EP output by the large model are obtained.

[0287] Thus, a loss function can be constructed based on the loss between the score sample SS and the predicted score ES and the loss between the diagnostic problem sample SP and the predicted diagnostic problem EP:

[0288] .

[0289] Among them, m is the number of samples. L S( ) is the score loss function and L P ( ) is the loss function of the diagnosis problem. k S and k P are the weight values ​​corresponding to the two losses respectively.

[0290] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0291] Therefore, according to this embodiment, the technical effect of being able to quantify the quality of the diagnosis report based on the evaluation results and clarify the diagnosis error information is achieved.

[0292] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0293] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0294] Example 2

[0295] Figure 7 FIG. 7 shows a quality assessment device 700 for an image diagnosis report according to this embodiment, and the device 700 corresponds to the method according to embodiment 1. Figure 7As shown, the device 700 includes: a report determination module 710, which is used to determine a diagnosis report corresponding to the imaging examination of the target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate the diagnosis opinion for the imaging examination, and the audit report is used to indicate the report generated by auditing the diagnosis report; a first prompt information generation module 720, which is used to determine the first text information that is different between the diagnosis report and the audit report, and generate the first prompt information based on the first text information; a difference type return module 730, which is used to identify the first prompt information using a pre-trained difference analysis model, and determine the error information corresponding to the first text information, wherein the error information includes the target difference type and the misdiagnosis site, and the target difference type includes non-diagnostic errors and / or diagnostic errors; a grading result determination module 740, which is used to grade the diagnostic errors based on a pre-set disease grade library when it is determined that the target difference type contains diagnostic errors, and determine the corresponding grading results; and a result output module 750, which is used to output the quality assessment result of the diagnosis report based on the error information and the grading result and using the difference analysis model, wherein the quality assessment result includes the diagnostic problems existing in the diagnosis report and the quality score of the diagnosis report.

[0296] Optionally, the result output module 750 includes: a second prompt information generation module, used to fill the error information and grading results into a pre-set first template and generate second prompt information; and a result output sub-module, used to input the second prompt information into the difference analysis model, and use the difference analysis model to output the quality assessment results of the diagnostic report.

[0297] Optionally, the result output submodule includes: a judgment module, which is used to input the second prompt information into a differential analysis model, and use the differential analysis model to determine whether the second prompt information contains multiple prompt sub-information, wherein the grading results and misdiagnosed locations of each prompt sub-information are the same; an elimination module, which is used to retain any one of the prompt sub-information in the second prompt information and eliminate the remaining prompt sub-information when the second prompt information contains multiple prompt sub-information; and a quality assessment result output module, which is used to output the quality assessment result of the diagnosis report using the differential analysis model and based on the processed second prompt information.

[0298] Optionally, the first prompt information generating module 720 includes: a first prompt information generating submodule, configured to fill the first text information into a preset second template and generate the first prompt information.

[0299] Optionally, the first prompt information generation module 720 includes: a text information extraction module, used to extract the second text information in the diagnosis report and the third text information in the audit report; and a first text information determination module, used to perform text comparison on the second text information and the third text information, and determine the first text information with differences in combination with a pre-set image knowledge base.

[0300] Therefore, according to this embodiment, the technical effect of being able to quantify the quality of the diagnosis report based on the evaluation results and clarify the diagnosis error information is achieved.

[0301] Example 3

[0302] Figure 8 FIG. 8 shows a quality assessment device 800 for an image diagnosis report according to this embodiment, and the device 800 corresponds to the method according to embodiment 1. Figure 8 As shown, the device 800 includes: a processor 810; and a memory 820, which is connected to the processor 810 and is used to provide the processor 810 with instructions for processing the following processing steps: determining a diagnosis report corresponding to the imaging examination of the target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate the diagnostic opinion for the imaging examination, and the audit report is used to indicate the report generated by auditing the diagnosis report; determining a first text information in which there is a difference between the diagnosis report and the audit report, and generating a first prompt information based on the first text information; using a pre-trained difference analysis model to identify the first prompt information, and determining error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed site, and the target difference type includes non-diagnostic errors and / or diagnostic errors; in the case of determining that the target difference type includes a diagnostic error, grading the diagnostic errors based on a pre-set disease grade library, and determining the corresponding grading results; and outputting a quality assessment result of the diagnosis report based on the error information and the grading result and using the difference analysis model, wherein the quality assessment result includes the diagnostic problems existing in the diagnosis report and the quality score of the diagnosis report.

[0303] Optionally, the operation of outputting the quality assessment result of the diagnostic report based on the error information and grading results and using a difference analysis model includes: filling the error information and grading results into a pre-set first template and generating a second prompt information; and inputting the second prompt information into the difference analysis model, and using the difference analysis model to output the quality assessment result of the diagnostic report.

[0304] Optionally, the operation of inputting the second prompt information into the differential analysis model and using the differential analysis model to output the quality assessment result of the diagnosis report includes: inputting the second prompt information into the differential analysis model, and using the differential analysis model to determine whether the second prompt information contains multiple prompt sub-information, wherein the grading results and misdiagnosed locations of each prompt sub-information are the same; when the second prompt information contains multiple prompt sub-information, retaining any one prompt sub-information in the second prompt information and discarding the remaining prompt sub-information; and using the differential analysis model and based on the processed second prompt information, outputting the quality assessment result of the diagnosis report.

[0305] Optionally, the operation of generating the first prompt information based on the first text information includes: filling the first text information into a preset second template, and generating the first prompt information.

[0306] Optionally, the operation of determining the first text information that has differences in the diagnosis report and the audit report includes: extracting the second text information in the diagnosis report and the third text information in the audit report; and performing a text comparison on the second text information and the third text information, and determining the first text information that has differences in combination with a pre-set imaging knowledge base.

[0307] Therefore, according to this embodiment, the technical effect of being able to quantify the quality of the diagnosis report based on the evaluation results and clarify the diagnosis error information is achieved.

[0308] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0309] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0310] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0311] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0312] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0313] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0314] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of an imaging diagnostic report, characterized in that: include: Determine a diagnosis report corresponding to the imaging examination of the target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate a diagnosis opinion for the imaging examination, and the audit report is used to indicate a report generated by auditing the diagnosis report; Determine first text information that is different between the diagnosis report and the audit report, and generate first prompt information based on the first text information; Using a pre-trained difference analysis model to identify the first prompt information, and determine error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed part, and the target difference type includes a non-diagnostic error and / or a diagnostic error; In the case where it is determined that the target difference type includes the diagnostic error, grading the diagnostic error based on a preset disease grade library and determining a corresponding grading result; and Based on the error information and the grading result and using the difference analysis model, a quality assessment result of the diagnosis report is output, wherein the quality assessment result includes the diagnosis problem existing in the diagnosis report and the quality score of the diagnosis report.

2. The method according to claim 1, characterized in that The operation of outputting the quality assessment result of the diagnosis report based on the error information and the classification result and using the difference analysis model includes: Filling the error information and the classification result into a preset first template, and generating a second prompt information; and The second prompt information is input into the difference analysis model, and the difference analysis model is used to output a quality assessment result of the diagnosis report.

3. The method according to claim 2, characterized in that The operation of inputting the second prompt information into the difference analysis model, and outputting the quality assessment result of the diagnosis report using the difference analysis model includes: Inputting the second prompt information into the difference analysis model, and using the difference analysis model to determine whether the second prompt information includes multiple prompt sub-information, wherein the grading results and the misdiagnosed parts of the respective prompt sub-information are the same; In the case where the second prompt information includes the plurality of prompt sub-information, retaining any one prompt sub-information in the second prompt information, and removing the remaining prompt sub-information; and The difference analysis model is used and based on the processed second prompt information, a quality assessment result of the diagnosis report is output.

4. The method according to claim 1, characterized in that The operation of generating first prompt information based on the first text information includes: The first text information is filled into a preset second template, and the first prompt information is generated.

5. The method according to claim 4, characterized in that The operation of determining the first text information having a difference between the diagnosis report and the audit report includes: extracting the second text information in the diagnosis report and the third text information in the audit report; and A text comparison is performed on the second text information and the third text information, and the first text information having differences is determined in combination with a preset image knowledge base.

6. The method according to claim 1, characterized in that The difference analysis model is also obtained by training through the following steps: Use image reports to pre-train a preset large language model; Fine-tune the pre-trained large language model using samples corresponding to the first task so that the large language model can be used to implement the first task, wherein the first task is to determine the difference type, misdiagnosis part, and error content corresponding to the difference complete sentence corresponding to the diagnosis report and the audit report according to the difference complete sentence; as well as The fine-tuned large language model is fine-tuned again using samples corresponding to the second task to obtain the difference analysis model. The second task solves the problem of merging misdiagnoses of the same part in the diagnosis report, avoids repeated deductions, and summarizes the diagnosis problems.

7. The method according to claim 6, characterized in that The operation of fine-tuning the pre-trained large language model using the sample corresponding to the first task includes: Using the trajectory samples corresponding to the interaction process after splitting the first task, fine-tune the large language model based on reinforcement learning; and The large language model is fine-tuned using a sample of input-output pairs corresponding to the first task, and wherein The operation of performing secondary fine-tuning on the fine-tuned large language model using samples corresponding to the second task includes: fine-tuning the large language model using input-output pair samples corresponding to the second task.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. A quality assessment device for imaging diagnostic reports, characterized in that: include: A report determination module, used to determine a diagnosis report corresponding to the imaging examination of the target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate a diagnostic opinion for the imaging examination, and the audit report is used to indicate a report generated by auditing the diagnosis report; A first prompt information generating module, used for determining first text information that is different between the diagnosis report and the audit report, and generating first prompt information based on the first text information; a difference type returning module, configured to identify the first prompt information by using a pre-trained difference analysis model, and determine error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed part, and the target difference type includes a non-diagnostic error and / or a diagnostic error; a grading result determination module, for grading the diagnostic errors based on a preset disease grade library and determining corresponding grading results when it is determined that the target difference type includes the diagnostic errors; and A result output module is used to output the quality assessment result of the diagnostic report based on the error information and the grading result and using the difference analysis model, wherein the quality assessment result includes the diagnostic problems existing in the diagnostic report and the quality score of the diagnostic report.

10. A quality assessment device for imaging diagnostic reports, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Determine a diagnosis report corresponding to the imaging examination of the target object, and an audit report corresponding to the diagnosis report, wherein the diagnosis report is used to indicate a diagnosis opinion for the imaging examination, and the audit report is used to indicate a report generated by auditing the diagnosis report; Determine first text information that is different between the diagnosis report and the audit report, and generate first prompt information based on the first text information; Using a pre-trained difference analysis model to identify the first prompt information, and determine error information corresponding to the first text information, wherein the error information includes a target difference type and a misdiagnosed part, and the target difference type includes a non-diagnostic error and / or a diagnostic error; In the case where it is determined that the target difference type includes the diagnostic error, grading the diagnostic error based on a preset disease grade library and determining a corresponding grading result; and Based on the error information and the grading result and using the difference analysis model, a quality assessment result of the diagnosis report is output, wherein the quality assessment result includes the diagnosis problem existing in the diagnosis report and the quality score of the diagnosis report.

Images