Artificial intelligence automatic report evaluation method and system

Abstract

The invention discloses an artificial intelligence automatic report evaluation method and system, and the method comprises the steps: carrying out the focus information extraction of a to-be-evaluated text report outputted by a medical artificial intelligence model and a pre-established gold standard text report through a trained large language model, and converting the focus information into structured information; and obtaining a to-be-evaluated text structured report and a gold standard text structured report. In order to solve the problem that the lesions between the two reports need to establish a corresponding relationship for evaluation, the method further introduces a structured report matching algorithm, so that a corresponding matching relationship is established for the lesions in the two reports. On the basis, whether the attributes of the single focus are consistent or not can be evaluated, so that the performance of the text report is subjected to multi-level quantification.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence automated report evaluation method and system. Background Art

[0002] With the rapid development of artificial intelligence technology, a series of medical artificial intelligence models dedicated to clinical diagnosis have emerged in the medical field, aiming to help doctors reduce report writing time and improve clinical diagnosis efficiency. Compared with text generation models in conventional applications, the generated content in the medical field has its own particularity, mainly because the artificial intelligence models in this field need to output more professional, accurate, and stable diagnostic texts. Therefore, the evaluation of medical artificial intelligence is also more complex and challenging than models in general fields.

[0003] For general text generation tasks, the evaluation criteria of artificial intelligence models are to analyze the completeness, consistency, and diversity of the generated text. For the medical field, this general evaluation method obviously cannot fully reflect the performance of medical artificial intelligence. In the process of medical diagnosis, the text report needs to describe the various attribute information of the lesion in detail, and the ultimate goal is to accurately evaluate the nature of the lesion, which is often described as short texts such as "benign" and "malignant". However, the existing evaluation methods are mainly quantified based on text similarity. The difference of a single word does not affect the overall similarity of the text. Therefore, this type of method cannot keenly capture the brief but important description of the lesion. For medical artificial intelligence, a more reasonable evaluation method needs to be designed to quickly and accurately analyze the performance of different models in practical applications. Summary of the invention

[0004] The present application provides an artificial intelligence automated report evaluation method and system to solve the problems of existing medical artificial intelligence report evaluation methods that cannot fully reflect the performance of medical artificial intelligence, cannot keenly capture brief but important lesion descriptions, and cannot quickly and accurately analyze the performance of different models in practical applications.

[0005] According to the first aspect, an embodiment provides an artificial intelligence automated report evaluation method, the method comprising:

[0006] The text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report are used to extract lesion information through the trained large language model and convert it into structured information to obtain the structured text report to be evaluated and the structured standard text report;

[0007] Based on the lesion structured information in the text structured report to be evaluated and the standard text structured report, a pre-trained text encoder is used to encode the lesion information to obtain a feature vector;

[0008] The similarity of the feature vectors was calculated and used as the edge weight. The lesion information in the two reports was optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports. The text report to be evaluated was evaluated based on the matching results.

[0009] Furthermore, the text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report are used to extract lesion information through the trained large language model and convert it into structured information, specifically including:

[0010] Based on the constructed knowledge graph, a large language model is used to extract lesion information in the text and convert it into structured output information to generate a structured report. The extracted lesion information includes lesion location, lesion type, lesion nature and related necessary attribute information.

[0011] Furthermore, the construction of the knowledge graph specifically includes:

[0012] The construction process of the knowledge graph includes data collection, entity recognition and relationship extraction steps. The data used to construct the knowledge graph comes from the medical database in the application field. Then, the named entity recognition technology is used to extract the relevant named entities from the data. At the same time, natural language processing technology is used to extract the relationship words between entities from the text, so as to convert the relationship between entities into a graphical structure for representation.

[0013] Furthermore, based on the constructed knowledge graph, a large language model is used to extract the lesion information in the text and convert it into structured output information to generate a structured report, which includes:

[0014] Entity recognition and matching: Use a large language model to extract entities from text, and verify the extracted entity information with the help of a knowledge graph;

[0015] Relationship-based reasoning: Based on the entity relationships in the knowledge graph, further reasoning is performed on the extracted structured attribute information to correct the output of the large language model;

[0016] Supplement missing information: For some key attributes that may be missing in the generated structured report, the knowledge graph is used to supplement the missing attribute information based on existing entities and relationships.

[0017] Furthermore, based on the lesion structured information in the text structured report to be evaluated and the standard text structured report, the pre-trained text encoder is used to encode the feature vector of each lesion information, specifically including:

[0018] Each line of lesion structured information in the text structured report to be evaluated and the standard text structured report is used to generate a unique description text for each lesion, wherein the unique description text is generated by splicing multiple structured attribute information of the lesion;

[0019] The pre-trained Transformer-based text encoder is used to encode the unique description text of the lesion to obtain the feature vector of the lesion.

[0020] Furthermore, the pre-training of the text encoder includes:

[0021] The pre-training process of the text encoder uses paired image-text pairs to train a deep neural network, where the image is generated by cropping the corresponding lesion area from the original image, and the text is a natural language description of the lesion attributes. The input of the text encoder comes from the structured attribute information of the lesion, and the output is a high-dimensional feature vector;

[0022] Using T k ,I k Represents the feature vector of the kth image-text pair of the input. The loss function in the pre-training process is based on the generated feature vector T k ,I k The similarity is calculated to promote higher similarity between matching image-text pairs and lower similarity between unmatched image-text pairs; the similarity function includes cosine similarity, and the calculation formula is as follows:

[0023]

[0024] Where T represents the feature vector of the text, I represents the feature vector of the image, and n represents the length of the feature vector.

[0025] Furthermore, the feature vector similarity is calculated and used as the edge weight. The lesion information in the two reports is optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports, including:

[0026] Each line of lesions in the two structured reports is abstracted into a single node in a bipartite graph, and the text encoder encodes and converts the unique description text of the lesion in the corresponding line of the input to obtain a feature vector;

[0027] Then, the Kuhn-Munkres algorithm is used to solve the optimal match. The optimal match will select multiple pairs of lesions with the highest sum of feature vector similarities in the two reports, establish a one-to-one correspondence between the lesions in the two reports, and thus form an edge connection in the bipartite graph. The edge weight between nodes is set to the similarity of the two feature vectors.

[0028] Then, a suitable similarity threshold is set to delete the edge connections with weights lower than the threshold in the weighted bipartite graph, thereby reducing the complexity of the bipartite graph.

[0029] Furthermore, the similarity calculation of the feature vectors adopts the cosine similarity function.

[0030] Furthermore, the text report to be evaluated is evaluated according to the matching results, specifically including:

[0031] Evaluation indicators include the detection rate of lesions, that is, what proportion of lesions in the standard text report can be mentioned in the report output by the artificial intelligence model;

[0032] Evaluate the similarity of the attribute information of the lesions in the report, convert the structured attribute information of the lesions into relevant short texts through structured conversion and evaluate them, including calculating the TF-IDF similarity of each attribute information;

[0033] The structured information is used to supplement the prediction accuracy of the artificial intelligence model for lesion attributes. Attributes with fixed values ​​are quantitatively evaluated through classification indicators, including the recall rate and precision of each structured attribute.

[0034] According to the first aspect, an embodiment provides an artificial intelligence automated report evaluation system, the system comprising:

[0035] A structured report generation module is used to extract lesion information from the text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report through a trained large language model and convert it into structured information to obtain a structured text report to be evaluated and a standard text structured report;

[0036] A structured report matching module is used to obtain a feature vector of each lesion information by encoding the lesion structured information in the text structured report to be evaluated and the standard text structured report using a pre-trained text encoder;

[0037] The similarity of the feature vectors was calculated and used as the edge weight. The lesion information in the two reports was optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports. The text report to be evaluated was evaluated based on the matching results.

[0038] The present application provides an artificial intelligence automated report evaluation method and system. This method combines existing large language models, structured reports, text matching and other technologies to quantitatively analyze the performance of medical artificial intelligence models in text generation tasks from another perspective. In this method, the lesions involved in the diagnostic report are regarded as single entities, and then the structured information of the single lesion is extracted from the original text, thereby converting the text consistency evaluation task into an accuracy evaluation of each element of the lesion. It has the following beneficial effects:

[0039] 1) Artificial intelligence report evaluation method: Compared with the conventional long text evaluation method, this method first extracts structured lesion information from the predicted text. Based on the structured report corresponding to the text, the consistency evaluation between texts can be converted into the detection and classification evaluation of all lesions in the structured results, thereby quantifying the performance of medical artificial intelligence in text generation tasks.

[0040] 2) Structured report generation algorithm: The large language model can understand and generate natural language and has strong context perception capabilities; the knowledge graph can help computers understand and represent concepts, entities and the relationships between them in the real world, thereby providing more accurate information retrieval and semantic understanding. In the structured report generation task, the combination of the large language model and the knowledge graph can improve the professionalism and reliability of the report content.

[0041] 3) Text similarity matching algorithm: In order to align the lesions in two structured results, the present invention further proposes a text matching algorithm based on a feature encoder, which uses a multimodal pre-trained text encoder to express the semantic information of lesion attributes as a feature vector, and then calculates the similarity between each vector as the criterion for lesion matching. This helps to identify the same lesion description from the perspective of text semantics, thereby avoiding the influence of text length on lesion matching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of an artificial intelligence automated report evaluation method provided by one embodiment of the present invention;

[0043] Figure 2 A specific implementation flow chart of an artificial intelligence automated report evaluation method provided by one embodiment of the present invention;

[0044] Figure 3 A flow chart of a structured report generation algorithm in an artificial intelligence automated report evaluation method provided by one embodiment of the present invention;

[0045] Figure 4 An example diagram of a structured report in an artificial intelligence automated report evaluation method provided by one embodiment of the present invention;

[0046] Figure 5 An example of a knowledge graph construction process in an artificial intelligence automated report evaluation method provided by an embodiment of the present invention;

[0047] Figure 6 A flowchart of text encoder pre-training in an artificial intelligence automated report evaluation method provided by one embodiment of the present invention;

[0048] Figure 7A flow chart of a structured report matching algorithm in an artificial intelligence automated report evaluation method provided by one embodiment of the present invention;

[0049] Figure 8 A schematic diagram of the logical structure of an artificial intelligence automated report evaluation system provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0051] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0052] The first embodiment of the present invention provides an artificial intelligence automated report evaluation method, which is combined with Figure 1 and Figure 2 Provide detailed explanation.

[0053] For the text report output by the medical artificial intelligence model, the method proposed in the present invention adopts the following method: Figure 2 The process shown in the figure is automatically evaluated to obtain more effective evaluation indicators. First, the artificial intelligence report and the gold standard report will respectively decompose the mentioned lesions into structured reports through the structured report generation algorithm. The report usually contains the location, type, nature and other necessary attributes of the lesions. Since there is still a problem of lesion misalignment between the two reports, this method further introduces a structured report matching algorithm to establish a corresponding matching relationship for the lesions in the two reports. On this basis, it is possible to evaluate whether the various attributes of a single lesion are consistent, thereby performing multi-level quantification of the performance of the text report, such as indicators such as the completeness of the description and the classification accuracy of the model.

[0054] like Figure 1 As shown, in step S100, the text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report are subjected to lesion information extraction and converted into structured information through a trained large language model to obtain a structured text report to be evaluated and a standard text structured report.

[0055] Structured report generation algorithms, such as Figure 3 As shown:

[0056] Specifically, the large language model is usually trained using diversified texts in a general knowledge base, which enables the model to better understand the complex structure and semantics in natural language, and the context-based information extraction capability helps to identify attribute relationships in text reports. By setting appropriate prompt word sentences, the present invention uses the large language model to extract lesion information in the text and converts it into structured output information. Figure 4 Taking PET / CT report as an example, the AI ​​report, gold standard report and corresponding structured results are shown. In the figure, only "lesion location", "lesion type" and "lesion nature" are used as required attributes. In actual application, more structured attributes can be extracted for evaluation from different angles. Figure 4 The structured results shown can determine whether the artificial intelligence report has problems such as missing lesions or incorrect description of lesion properties.

[0057] Based on a specific dedicated knowledge base, the present invention further constructs a knowledge graph within the application field. The construction process of the knowledge graph is as follows: Figure 5 As shown in the figure, it mainly includes three steps: data collection, entity recognition, and relationship extraction. The data used to build the knowledge graph comes from the medical database in the application field. Then, the named entity recognition technology is used to extract the relevant named entities from these data. At the same time, natural language processing technology is also needed to extract the relationship words between entities from the text, so as to convert the relationship between entities into a graph structure for representation.

[0058] The knowledge graph expresses the associations between entities in a structured way, thereby verifying whether the structured information generated by the large language model is consistent with the existing domain knowledge, preventing the generation of erroneous information, and improving the accuracy and rationality of imaging report generation. Constructing a knowledge graph in the corresponding scenario helps to correct and supplement the knowledge learned by the large model, so that the generated report is more in line with actual clinical needs. The knowledge graph has three main functions in the process of generating structured reports:

[0059] 1. Entity recognition and matching: When generating a report, the large language model first extracts entities in the text (such as lesion location and lesion type) and verifies these entity information with the help of the knowledge graph. For example, when the model generates "right lower lobe solid tumor", it queries the knowledge graph to see whether the relationship between the two entities "right lower lobe" and "solid tumor" is reasonable.

[0060] 2. Reasoning based on relationships: Based on the entity relationships in the knowledge graph, the structured attribute information can be further reasoned. For example, if the metabolic activity of the lesion increases, the relationship of "high metabolic lesion-associated-malignant tumor" in the knowledge graph can prompt the model to be more inclined to generate a "malignant" diagnosis, thereby correcting the output of the large language model.

[0061] 3. Supplement missing information: The structured report generated by the model may still lack certain key attributes (such as metabolic information, diagnostic description), and the knowledge graph can supplement these attribute information based on existing entities and relationships. For example, if the model generates "left upper lobe lesion" but no metabolic information, the knowledge graph can prompt the model to supplement "increased metabolic activity" and other statements based on common imaging features.

[0062] like Figure 1 As shown, in step S200, based on the lesion structured information in the text structured report to be evaluated and the standard text structured report, a pre-trained text encoder is used to encode to obtain a feature vector of each lesion information.

[0063] Structured Report Matching Algorithm:

[0064] Although the corresponding structured report is output by combining the large language model and the knowledge graph, the lesion information in the structured result is not output in a fixed order, and there may be missed lesions in the predicted text. In order to match the lesions in the two structured reports, this embodiment uses a text encoder and a weighted bipartite graph matching algorithm to solve this problem.

[0065] The text encoder is responsible for encoding the unique description text of the lesion. The unique description text is generated by splicing multiple structural attributes of the lesion, such as the location and type attributes of the lesion, to generate a description such as "right lower lobe of the lung - ground glass nodule" to determine the uniqueness of the lesion in the examination report. This strategy matches multiple structured data from the perspective of semantic encoding, thereby avoiding the impact of text length on matching performance. Referring to the training paradigm of the multimodal model, the text encoder can be pre-trained using the lesion text description and the corresponding image, which helps to supplement the image information lacking in the plain text description.

[0066] The pre-training process uses paired images and texts to train the deep neural network, where the image is generated by cropping the corresponding lesion area from the original image, and the text is a natural language description of the lesion properties. In this patent, the input of the encoder comes from the structured properties of the lesion. The specific process is as follows Figure 6 As shown in Figure 2, image encoders and text encoders are usually deep neural networks based on the Transformer structure, which convert the input image and text into high-dimensional feature vectors respectively. Figure 6 Use T k ,I k Represents the feature vector of the kth image-text pair of the input. The loss function in the pre-training process is based on the feature vector T k ,I k The similarity is calculated to promote higher similarity between matching image-text pairs and lower similarity between unmatched image-text pairs. The commonly used similarity function is cosine similarity, which is calculated as shown in the following formula (1), where T represents the feature vector of the text, I represents the feature vector of the image, and n represents the length of the feature vector.

[0067]

[0068] The text encoder processes the lesions in each line of the gold standard report and the artificial intelligence report, converts the unique description text of these lesions into corresponding feature vectors through the text encoder, and then calculates the feature similarity between the lesions as the weight of the two. The feature similarity is also calculated using formula (1) in the pre-training process, and then constructs a weighted bipartite graph of the matching relationship between each lesion for the structured results of the two reports.

[0069] like Figure 1 As shown, in step S300, the feature vector similarity is calculated and used as the edge weight, and the lesion information in the two reports is optimally matched by constructing a weighted bipartite graph of the matching relationship between each lesion in the two reports, and the text report to be evaluated is evaluated based on the matching result.

[0070] The weighted bipartite graph abstracts the lesions in the structured report into nodes in the graph, and the weight of the weighted bipartite graph is calculated by the similarity of the feature vectors, thereby converting the two structured reports into the same graph structure. In this embodiment, for the weighted bipartite graph, the lesions in each row of the structured report are abstracted into a single node in the graph ( Figure 7 The orange and green nodes in the figure are then converted into feature vectors using the unique description text of the lesion in the row as input by the text encoder. Figure 7 As shown in Figure 2, constructing a weighted bipartite graph also requires establishing a one-to-one correspondence between the lesions in the two reports, thus forming an edge connection in the bipartite graph ( Figure 7The dotted line in the figure), the edge weights between the nodes are set to the similarity of the two eigenvectors, and then setting a suitable similarity threshold can delete the edge connections with lower weights in the weighted bipartite graph, reducing the complexity of the bipartite graph. This strategy naturally merges the lesions and corresponding relationships in the two reports into a weighted bipartite graph, and finally the optimal match of each lesion in the two reports can be established through the matching algorithm of the graph. In this embodiment, matching refers to automatically aligning the lesions in the gold standard report to the artificial intelligence report according to the attribute description of the lesion. A successful match represents that the lesion exists in both the gold standard report and the artificial intelligence report, and a failed match represents that the lesion does not exist in the artificial intelligence report. This matching process can be solved using the Kuhn-Munkres algorithm, and the optimal match will select multiple pairs of lesions with the highest sum of similarities in the two reports.

[0071] Through the structured matching algorithm, even if the lesions in the two reports are arranged in any order, the correspondence between the lesions in the two reports can be automatically established in the end.

[0072] Evaluation indicators:

[0073] Through the above two steps, the long text description in the artificial intelligence report is converted into a structured lesion detection, and a corresponding relationship with the gold standard report is established. Based on these lesion attributes and matching relationships, the consistency assessment of the text can be evaluated by the consistency of the lesion attributes. The evaluation method proposed in the present invention does not require too much manual participation and automatically quantifies the performance of the medical artificial intelligence model in text generation.

[0074] Commonly used indicators include the detection rate of lesions, that is, what proportion of lesions in the gold standard report can be mentioned in the AI ​​report; at the same time, the similarity of the attribute values ​​of the lesions can be further evaluated. These attribute values ​​are converted into relevant short texts through structured generation algorithms. Evaluating such short texts is more accurate than directly evaluating long texts, such as the TF-IDF similarity of various attributes. In addition, the structured information can also be used to supplement the prediction accuracy of the model for lesion attributes. For attributes with fixed values, quantitative evaluation can be performed through classification indicators. These indicators include the recall rate (Recall) and precision (Precision) of various structured attributes, as shown in formula (2) and formula (3). Assuming that the prediction accuracy of the benign and malignant lesions reported by the AI ​​is evaluated, TP in the formula represents the number of lesions predicted to be malignant in both reports, FP represents the number of benign lesions predicted to be malignant, and FN represents the number of malignant lesions predicted to be benign.

[0075]

[0076] These quantitative indicators are often more intuitive and accurate than conventional long text evaluation methods, and are not just about evaluating the character similarity of two long texts. Based on the framework proposed by the present invention, the quantitative indicators finally output can reflect in detail whether various artificial intelligence models have high completeness, accuracy, and professionalism in generating medical text tasks.

[0077] In addition, in addition to being applied to medical reports, the method proposed in the present invention is also applicable to evaluating any other structured text, and the structured tables output in different application scenarios are also different. In the structured matching algorithm, multiple attributes need to be selected as the unique description of the lesion. In practical applications, there is no limit to the number and type of specific structured attributes.

[0078] Corresponding to the above-disclosed artificial intelligence automated report evaluation method, the embodiment of the present invention also discloses an artificial intelligence automated report evaluation system, such as Figure 8 As shown, it specifically includes:

[0079] A structured report generation module is used to extract lesion information from the text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report through a trained large language model and convert it into structured information to obtain a structured text report to be evaluated and a standard text structured report;

[0080] A structured report matching module is used to obtain a feature vector of each lesion information by encoding the lesion structured information in the text structured report to be evaluated and the standard text structured report using a pre-trained text encoder;

[0081] The similarity of the feature vectors was calculated and used as the edge weight. The lesion information in the two reports was optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports. The text report to be evaluated was evaluated based on the matching results.

[0082] It should be noted that for a detailed description of a sandstone and mudstone fine characterization system based on waveform phase control technology and joint model inversion provided in an embodiment of the present invention, reference can be made to the relevant description of an artificial intelligence automated report evaluation method provided in an embodiment of the present application, which will not be repeated here.

[0083] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0084] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. An artificial intelligence automated report evaluation method, characterized in that: The method comprises: The text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report are used to extract lesion information through the trained large language model and convert it into structured information to obtain the structured text report to be evaluated and the structured standard text report; Based on the lesion structured information in the text structured report to be evaluated and the standard text structured report, a pre-trained text encoder is used to encode the lesion information to obtain a feature vector; The similarity of the feature vectors was calculated and used as the edge weight. The lesion information in the two reports was optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports. The text report to be evaluated was evaluated based on the matching results.

2. An artificial intelligence automated report evaluation method as claimed in claim 1, characterized in that: The text reports to be evaluated output by the medical artificial intelligence model and the pre-established standard text reports are used to extract lesion information through the trained large language model and convert it into structured information, including: Based on the constructed knowledge graph, a large language model is used to extract lesion information in the text and convert it into structured output information to generate a structured report. The extracted lesion information includes lesion location, lesion type, lesion nature and related necessary attribute information.

3. An artificial intelligence automated report evaluation method as claimed in claim 2, characterized in that: The construction of the knowledge graph specifically includes: The construction process of the knowledge graph includes data collection, entity recognition and relationship extraction steps. The data used to construct the knowledge graph comes from the medical database in the application field. Then, the named entity recognition technology is used to extract the relevant named entities from the data. At the same time, natural language processing technology is used to extract the relationship words between entities from the text, so as to convert the relationship between entities into a graphical structure for representation.

4. An artificial intelligence automated report evaluation method as claimed in claim 3, characterized in that: Based on the constructed knowledge graph, a large language model is used to extract the lesion information in the text and convert it into structured output information to generate a structured report, including: Entity recognition and matching: Use a large language model to extract entities from text, and verify the extracted entity information with the help of a knowledge graph; Relation-based reasoning: Based on the entity relationships in the knowledge graph, further reasoning is performed on the extracted structured attribute information to correct the output of the large language model; Supplement missing information: For some key attributes that may be missing in the generated structured report, the knowledge graph is used to supplement the missing attribute information based on existing entities and relationships.

5. The artificial intelligence automated report evaluation method according to claim 1, characterized in that: Based on the lesion structured information in the text structured report to be evaluated and the standard text structured report, the pre-trained text encoder is used to encode the feature vector of each lesion information, including: Each line of lesion structured information in the text structured report to be evaluated and the standard text structured report is used to generate a unique description text for each lesion, wherein the unique description text is generated by splicing multiple structured attribute information of the lesion; The pre-trained Transformer-based text encoder is used to encode the unique description text of the lesion to obtain the feature vector of the lesion.

6. An artificial intelligence automated report evaluation method as claimed in claim 5, characterized in that: The pre-training of the text encoder includes: The pre-training process of the text encoder uses paired image-text pairs to train a deep neural network, where the image is generated by cropping the corresponding lesion area from the original image, and the text is a natural language description of the lesion attributes. The input of the text encoder comes from the structured attribute information of the lesion, and the output is a high-dimensional feature vector; Using T k ,I k Represents the feature vector of the kth image-text pair of the input. The loss function in the pre-training process is based on the generated feature vector T k ,I k The similarity is calculated to promote higher similarity between matching image-text pairs and lower similarity between unmatched image-text pairs; the similarity function includes cosine similarity, and the calculation formula is as follows: Where T represents the feature vector of the text, I represents the feature vector of the image, and n represents the length of the feature vector.

7. An artificial intelligence automated report evaluation method as claimed in claim 1, characterized in that: The feature vector similarity is calculated and used as the edge weight. The lesion information in the two reports is optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports. Specifically, the following steps are performed: Each line of lesions in the two structured reports is abstracted into a single node in a bipartite graph, and the text encoder encodes and converts the unique description text of the lesion in the corresponding line of the input to obtain a feature vector; Then, the Kuhn-Munkres algorithm is used to solve the optimal match. The optimal match will select multiple pairs of lesions with the highest sum of feature vector similarities in the two reports, establish a one-to-one correspondence between the lesions in the two reports, and thus form an edge connection in the bipartite graph. The edge weight between nodes is set to the similarity of the two feature vectors. Then, a suitable similarity threshold is set to delete the edge connections with weights lower than the threshold in the weighted bipartite graph, thereby reducing the complexity of the bipartite graph.

8. An artificial intelligence automated report evaluation method as claimed in claim 7, characterized in that: The similarity of feature vectors is calculated using the cosine similarity function.

9. The artificial intelligence automated report evaluation method according to claim 1, characterized in that: Evaluate the text report to be evaluated based on the matching results, including: Evaluation indicators include the detection rate of lesions, that is, what proportion of lesions in the standard text report can be mentioned in the report output by the artificial intelligence model; Evaluate the similarity of the attribute information of the lesions in the report, convert the structured attribute information of the lesions into relevant short texts through structured conversion and evaluate them, including calculating the TF-IDF similarity of each attribute information; The structured information is used to supplement the prediction accuracy of the artificial intelligence model for lesion attributes. Attributes with fixed values ​​are quantitatively evaluated through classification indicators, including the recall rate and precision of each structured attribute.

10. An artificial intelligence automated report evaluation system, characterized in that: The system comprises: A structured report generation module is used to extract lesion information from the text report to be evaluated output by the medical artificial intelligence model and the pre-established standard text report through a trained large language model and convert it into structured information to obtain a structured text report to be evaluated and a structured standard text report; A structured report matching module is used to obtain a feature vector of each lesion information by encoding the lesion structured information in the text structured report to be evaluated and the standard text structured report using a pre-trained text encoder; The similarity of the feature vectors was calculated and used as the edge weight. The lesion information in the two reports was optimally matched by constructing a weighted bipartite graph of the matching relationship between the lesions in the two reports. The text report to be evaluated was evaluated based on the matching results.

Images