Artificial Intelligence Automated Report Evaluation Methods and Systems

Abstract

This invention discloses an automated report evaluation method and system based on artificial intelligence. The method extracts lesion information from the text report to be evaluated output by a medical AI model and a pre-established gold standard text report, converting them into structured information using a trained large language model. This yields a structured report of the text to be evaluated and a structured report of the gold standard text. Addressing the issue of establishing a correspondence between lesions in the two reports for evaluation, this method further introduces a structured report matching algorithm, thereby establishing a corresponding matching relationship for lesions in the two reports. Based on this, the consistency of various attributes of a single lesion can be evaluated, thus enabling multi-level quantification of the text report performance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an automated report evaluation method and system based on artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, a series of medical AI models dedicated to clinical diagnosis have emerged in the medical field, aiming to help doctors reduce report writing time and improve clinical diagnostic efficiency. Compared to text generation models in conventional applications, the content generated in the medical field has its own unique characteristics, mainly because AI models in this field need to output more professional, accurate, and stable diagnostic texts. Therefore, the evaluation of medical AI is more complex and challenging than that of models in other fields.

[0003] For general text generation tasks, the evaluation criteria for AI models focus on analyzing the completeness, consistency, and diversity of the generated text. However, in the medical field, this general evaluation method clearly cannot fully reflect the performance of medical AI. In medical diagnosis, text reports need to describe in detail the various attributes of lesions, with the ultimate goal of accurately assessing their nature, often described concisely as 'benign' or 'malignant.' However, existing evaluation methods primarily quantify text similarity; differences in individual characters do not affect the overall similarity of the text. Therefore, these methods cannot accurately capture concise but crucial lesion descriptions. For medical AI, a more reasonable evaluation method needs to be designed to quickly and accurately analyze the performance of different models in practical applications. Summary of the Invention

[0004] This application provides an automated report evaluation method and system for artificial intelligence to address the problems of existing medical artificial intelligence report evaluation methods, which cannot fully reflect the performance of medical artificial intelligence, cannot accurately capture brief but important lesion descriptions, and cannot quickly and accurately analyze the performance of different models in practical applications.

[0005] According to the first aspect, one embodiment provides an AI-automated report evaluation method, the method comprising:

[0006] The lesion information of the text report to be evaluated and the pre-established standard text report output by the medical artificial intelligence model are extracted and converted into structured information by the trained large language model, resulting in the structured report of the text to be evaluated and the structured report of the standard text.

[0007] Based on the lesion structure information in the text structure report to be evaluated and the standard text structure report, the feature vector of each lesion information is obtained by encoding using a pre-trained text encoder.

[0008] The similarity of feature vectors is calculated and used as edge weights. By constructing a weighted bipartite graph of the matching relationship between lesions in the two reports, the lesion information in the two reports is optimally matched. The text report to be evaluated is evaluated based on the matching results.

[0009] Furthermore, the text reports to be evaluated output by the medical artificial intelligence model and the pre-established standard text reports are used to extract lesion information and convert it into structured information through a trained large language model, specifically including:

[0010] Based on the constructed knowledge graph, a large language model is used to extract lesion information from the text and convert it into structured output information to generate a structured report. The extracted lesion information includes lesion location, lesion type, lesion nature, and related necessary attribute information.

[0011] Furthermore, the construction of knowledge graphs specifically includes:

[0012] The process of building a knowledge graph includes data collection, entity recognition, and relation extraction. The data used to build the knowledge graph comes from medical databases in the application domain. Then, named entity recognition technology is used to extract relevant named entities from the data, and natural language processing technology is used to extract relation words between entities from the text, thereby converting the relationships between entities into a graph structure for representation.

[0013] Furthermore, based on the constructed knowledge graph, a large language model is used to extract lesion information from the text and convert it into structured output information to generate a structured report, specifically including:

[0014] Entity recognition and matching: Entities are extracted from text using a large language model, and the extracted entity information is verified using a knowledge graph;

[0015] Relation-based reasoning: Based on entity relations in the knowledge graph, further reasoning is performed on the extracted structured attribute information to correct the output of the large language model;

[0016] Supplementing missing information: For certain key attributes that may be missing in the generated structured report, knowledge graphs are used to supplement the missing attribute information based on existing entities and relationships.

[0017] Furthermore, based on the lesion structure information in the structured report of the text to be evaluated and the standard structured report, a pre-trained text encoder is used to encode the feature vectors of each lesion, specifically including:

[0018] Each line of lesion structure information in the structured report of the text to be evaluated and the standard text structured report is used to generate a unique description text for each lesion. The unique description text is generated by splicing together multiple structured attribute information of the lesion.

[0019] The feature vector of the lesion is obtained by encoding the lesion unique description text using a pre-trained Transformer-based text encoder.

[0020] Furthermore, the pre-training of the text encoder includes:

[0021] The pre-training process of the text encoder uses paired image-text to train the deep neural network. The image is generated by cropping the corresponding lesion region from the original image, and the text is a natural language description of the lesion attributes. The input of the text encoder comes from the structured attribute information of the lesion, and the output is a high-dimensional feature vector.

[0022] Use T k I k This represents the k-th image-text feature vector from the input. The loss function during pre-training is based on the generated feature vector T. k I k The similarity is calculated to ensure that matching image-text pairs have higher similarity, while mismatched image-text pairs have lower similarity. The similarity function includes cosine similarity, and the calculation formula is as follows:

[0023]

[0024] Where T represents the feature vector of the text, I represents the feature vector of the image, and n represents the length of the feature vector.

[0025] Furthermore, the similarity of feature vectors is calculated and used as edge weights. Optimal matching of lesion information in the two reports is achieved by constructing a weighted bipartite graph representing the matching relationships between lesions in the two reports. Specifically, this includes:

[0026] Each lesion in both structured reports is abstracted as a single node in a bipartite graph. The text encoder encodes and transforms the text based on the unique description of the lesion in the corresponding line of the input to obtain a feature vector.

[0027] Then, the Kuhn-Munkres algorithm is used to solve the optimal matching problem. The optimal matching will select the multiple pairs of lesions with the highest sum of feature vector similarity between the two reports, establish a one-to-one correspondence between the lesions in the two reports, and thus form the edge connection in the bipartite graph. The edge weight between nodes is set to the similarity between the two feature vectors.

[0028] Then, an appropriate similarity threshold is set to delete edges in the weighted bipartite graph whose weights are below the threshold, thereby reducing the complexity of the bipartite graph.

[0029] Furthermore, the similarity of feature vectors is calculated using the cosine similarity function.

[0030] Furthermore, the text report to be evaluated is evaluated based on the matching results, specifically including:

[0031] Evaluation metrics include the lesion detection rate, which is the proportion of lesions in the standard text report that can be mentioned in the report output by the artificial intelligence model.

[0032] The similarity of various attribute information of lesions in the assessment report is evaluated by converting the structured attribute information of lesions into relevant short texts and then evaluating them, including calculating the TF-IDF similarity of various attribute information.

[0033] The structured information is used to supplement the AI ​​model's prediction accuracy of lesion attributes. For attributes with fixed values, they are quantitatively evaluated using classification indicators, including the recall and precision of each structured attribute.

[0034] According to the first aspect, one embodiment provides an artificial intelligence-automated report evaluation system, the system comprising:

[0035] The structured report generation module is used to extract lesion information from the medical artificial intelligence model's output text report to be evaluated and the pre-established standard text report, and convert it into structured information to obtain the structured report of the text to be evaluated and the structured report of the standard text.

[0036] The structured report matching module is used to encode the feature vectors of each lesion based on the lesion structured information in the structured report to be evaluated and the standard text structured report using a pre-trained text encoder.

[0037] The similarity of feature vectors is calculated and used as edge weights. By constructing a weighted bipartite graph of the matching relationship between lesions in the two reports, the lesion information in the two reports is optimally matched. The text report to be evaluated is evaluated based on the matching results.

[0038] This application provides an automated report evaluation method and system based on artificial intelligence. This method combines existing technologies such as large language models, structured reporting, and text matching to quantitatively analyze the performance of medical AI models in text generation tasks from a different perspective. In this method, lesions mentioned in the diagnostic report are treated as individual entities. The structured information of each lesion is then extracted from the original text, thereby transforming the text consistency assessment task into an accuracy assessment of each element of the lesion. It has the following beneficial effects:

[0039] 1) Artificial Intelligence Report Evaluation Method: Compared with conventional long text evaluation methods, this method first extracts structured lesion information from the predicted text. Based on the structured report corresponding to the text, the consistency evaluation between texts can be transformed into the detection and classification evaluation of all lesions in the structured results, thereby quantifying the performance of medical artificial intelligence in text generation tasks.

[0040] 2) Structured Report Generation Algorithm: Large language models can understand and generate natural language, and have strong context awareness capabilities; knowledge graphs can help computers understand and represent concepts, entities, and relationships between them in the real world, thus providing more accurate information retrieval and semantic understanding. In the task of structured report generation, the combination of large language models and knowledge graphs can improve the professionalism and reliability of the report content.

[0041] 3) Text Similarity Matching Algorithm: To align lesions in two structured results, this invention further proposes a text matching algorithm based on a feature encoder. A multimodal pre-trained text encoder is used to express the semantic information of lesion attributes as feature vectors, and then the similarity between these vectors is calculated as the criterion for lesion matching. This helps to identify identical lesion descriptions from a textual semantic perspective, thereby avoiding the influence of text length on the lesion matching results. Attached Figure Description

[0042] Figure 1 A flowchart illustrating an AI-automated report evaluation method according to an embodiment of the present invention;

[0043] Figure 2 A flowchart illustrating a specific implementation of an AI-automated report evaluation method according to an embodiment of the present invention;

[0044] Figure 3 This is a flowchart of a structured report generation algorithm in an AI-automated report evaluation method according to an embodiment of the present invention;

[0045] Figure 4 This is an example diagram of a structured report in an AI-automated report evaluation method provided in one embodiment of the present invention;

[0046] Figure 5 This invention provides an example of a knowledge graph construction process in an AI-automated report evaluation method, as provided in one embodiment of the present invention.

[0047] Figure 6 A flowchart of the text encoder pre-training process in an AI-based automated report evaluation method provided in one embodiment of the present invention;

[0048] Figure 7This is a flowchart of a structured report matching algorithm in an AI-based automated report evaluation method according to an embodiment of the present invention;

[0049] Figure 8 This is a schematic diagram of the logical structure of an artificial intelligence automated report evaluation system provided in one embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0051] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0052] The first embodiment of this invention provides an artificial intelligence-automated report evaluation method, which will be described below in conjunction with... Figure 1 and Figure 2 Please provide a detailed explanation.

[0053] For text reports output by medical artificial intelligence models, the method proposed in this invention employs the following... Figure 2 The illustrated process automates evaluation to obtain more effective assessment metrics. First, the AI ​​report and the gold standard report are processed using a structured report generation algorithm to decompose the mentioned lesions into structured reports, which typically include the lesion's location, type, nature, and other necessary attributes. Because lesion misalignment exists between the two reports, this method further introduces a structured report matching algorithm to establish corresponding matching relationships between the lesions in the two reports. Based on this, the consistency of various attributes of individual lesions can be evaluated, thus enabling multi-level quantification of the text report's performance, such as the completeness of the description and the model's classification accuracy.

[0054] like Figure 1 As shown, in step S100, the lesion information of the text report to be evaluated and the pre-established standard text report output by the medical artificial intelligence model are extracted and converted into structured information by the trained large language model, so as to obtain the structured report of the text to be evaluated and the structured report of the standard text.

[0055] Structured report generation algorithms, such as Figure 3 As shown:

[0056] Specifically, large language models are typically trained using diverse texts from general knowledge bases. This allows the model to better understand the complex structures and semantics of natural language, and the ability to extract contextual information helps identify attribute relationships in text reports. By setting appropriate prompt words, this invention uses large language models to extract lesion information from text and convert it into structured output information. Figure 4 Using a PET / CT report as an example, this diagram illustrates an AI-generated report, a gold standard report, and their corresponding structured results. The diagram only uses "lesion location," "lesion type," and "lesion nature" as required attributes. In practical applications, more structured attributes can be extracted for evaluation from different perspectives. Unlike long-text evaluation methods, this approach... Figure 4 The structured results shown can help determine whether the AI ​​report contains issues such as missing lesions or incorrect descriptions of lesion characteristics.

[0057] Based on a specific specialized knowledge base, this invention further constructs a knowledge graph within the application domain. The construction process of the knowledge graph is as follows: Figure 5 As shown, the process mainly includes three steps: data acquisition, entity recognition, and relation extraction. The data used to construct the knowledge graph comes from medical databases in the application domain. Subsequently, named entity recognition technology is used to extract relevant named entities from this data. At the same time, natural language processing technology is needed to extract relation words between entities from the text, thereby converting the relationships between entities into a graphical structure for representation.

[0058] Knowledge graphs express the relationships between entities in a structured way, thereby verifying whether the structured information generated by large language models is consistent with existing domain knowledge, preventing the generation of erroneous information, and improving the accuracy and rationality of image report generation. Constructing knowledge graphs for specific scenarios helps to correct and supplement the knowledge learned by large models, making the generated reports more aligned with actual clinical needs. Knowledge graphs primarily serve three functions in the structured report generation process:

[0059] 1. Entity Recognition and Matching: When generating a report, the large language model first extracts entities from the text (such as lesion location and lesion type), and then uses a knowledge graph to verify this entity information. For example, when the model generates "right lower lobe solid tumor", it queries the knowledge graph to see if the relationship between the entities "right lower lobe" and "solid tumor" is reasonable.

[0060] 2. Relationship-based reasoning: Based on entity relationships in the knowledge graph, further reasoning can be applied to the structured attribute information. For example, if the metabolic activity of a lesion increases, the relationship "high metabolic lesion - associated - malignant tumor" in the knowledge graph can suggest that the model is more inclined to generate a "malignant" diagnosis, thereby correcting the output of the large language model.

[0061] 3. Supplementing Missing Information: The structured report generated by the model may still lack certain key attributes (such as metabolic information and diagnostic descriptions). The knowledge graph can supplement this attribute information based on existing entities and relationships. For example, if the model generates "lesion in the upper lobe of the left lung" but lacks metabolic information, the knowledge graph can prompt the model to supplement statements such as "increased metabolic activity" by combining common imaging features.

[0062] like Figure 1 As shown, in step S200, based on the lesion structure information in the text structure report to be evaluated and the standard text structure report, a pre-trained text encoder is used to encode and obtain the feature vector of each lesion information.

[0063] Structured report matching algorithm:

[0064] Although a structured report was generated by combining a large language model and a knowledge graph, the lesion information was not output in a fixed order in the structured results, and there might be omitted lesions in the predicted text. To match the lesions in the two structured reports, this embodiment uses a text encoder and a weighted bipartite graph matching algorithm to solve this problem.

[0065] The text encoder is responsible for encoding the unique descriptive text of the lesion. This unique descriptive text is generated by concatenating various structural attributes of the lesion, such as its location and type, resulting in a description like "right lower lobe - ground-glass nodule," used to determine the uniqueness of the lesion in the examination report. This strategy matches multiple pieces of structured data from a semantic encoding perspective, thus avoiding the impact of text length on matching performance. Referring to the training paradigm of multimodal models, the text encoder can be pre-trained using lesion text descriptions and corresponding images, helping to supplement the image information lacking in pure text descriptions.

[0066] The pre-training process uses paired image-text pairs to train the deep neural network. The images are generated by cropping corresponding lesion regions from the original images, and the text is a natural language description of the lesion attributes. In this patent, the encoder's input comes from the structured attributes of the lesions. The specific process is as follows: Figure 6 As shown, image encoders and text encoders are typically based on deep neural networks with a Transformer structure, which convert the input image and text into high-dimensional feature vectors, respectively. Figure 6 Using T k I k This represents the feature vector of the k-th image-text pair in the input. The loss function during pre-training is based on the feature vector T. k I k The similarity is calculated to ensure that matching image-text pairs have higher similarity, while mismatched image-text pairs have lower similarity. The commonly used similarity function is cosine similarity, which is calculated as shown in the following formula (1), where T represents the feature vector of the text, I represents the feature vector of the image, and n represents the length of the feature vector.

[0067]

[0068] The text encoder processes the lesions in each line of the gold standard report and the artificial intelligence report, converts the unique description text of these lesions into corresponding feature vectors through the text encoder, and then calculates the feature similarity between each lesion as the weight of the two. The feature similarity is also calculated using formula (1) in the pre-training process, and then constructs a weighted bipartite graph of the lesion matching relationship for the structured results of the two reports.

[0069] like Figure 1 As shown, in step S300, the similarity of feature vectors is calculated and used as edge weights. By constructing a weighted bipartite graph of the matching relationship between lesions in the two reports, the lesion information in the two reports is optimally matched. The text report to be evaluated is evaluated based on the matching results.

[0070] Weighted bipartite graphs abstract lesions from structured reports as nodes in a graph. The weights of the weighted bipartite graph are calculated using the similarity of feature vectors, thus transforming two structured reports into the same graph structure. In this embodiment, for a weighted bipartite graph, the lesions in each row of the structured report are abstracted as a single node in the graph. Figure 7 (The orange and green nodes in the text), and then the text encoder uses the unique descriptive text of the lesions in that line as input to convert it into a feature vector. For example... Figure 7 As shown, constructing a weighted bipartite graph also requires establishing a one-to-one correspondence between the lesions in the two reports, thus forming the edge connections in the bipartite graph. Figure 7The edge weights between nodes (as shown by the dashed lines in the graph) are set to the similarity between two feature vectors. A suitable similarity threshold can then be used to remove edges with lower weights in the weighted bipartite graph, reducing its complexity. This strategy naturally integrates the lesions and their corresponding relationships from two reports into a single weighted bipartite graph. Finally, an optimal match for each lesion in the two reports can be established using a graph matching algorithm. In this embodiment, matching refers to automatically aligning lesions from the gold standard report to the AI ​​report according to their attribute descriptions. A successful match means the lesion exists in both the gold standard report and the AI ​​report; a failed match means the lesion does not exist in the AI ​​report. This matching process can be solved using the Kuhn-Munkres algorithm, and the optimal match will select the pairs of lesions with the highest sum of similarity between the two reports.

[0071] Using a structured matching algorithm, even if the lesions in the two reports are arranged in any order, the correspondence between the lesions in the two reports can be automatically established.

[0072] Evaluation indicators:

[0073] Through the two steps described above, the long text descriptions in the AI ​​report are transformed into structured lesion detections, while establishing a correspondence with the gold standard report. Based on these lesion attributes and matching relationships, the consistency of the text can be evaluated through the consistency of lesion attributes. The evaluation method proposed in this invention requires minimal human intervention and automatically quantifies the performance of medical AI models in text generation.

[0074] Commonly used metrics include the lesion detection rate, i.e., the proportion of lesions in the gold standard report that can be mentioned in the AI ​​report; at the same time, the similarity of various attribute values ​​of the lesions can also be further evaluated. These attribute values ​​are converted into relevant short texts through a structured generation algorithm. Evaluating such short texts is more accurate than directly evaluating long texts, such as the TF-IDF similarity of various attributes. In addition, the structured information can also be used to supplement the model's prediction accuracy of lesion attributes. For attributes with fixed values, they can be quantitatively evaluated through classification metrics, including the recall and precision of various structured attributes, as shown in formulas (2) and (3). Suppose we evaluate the accuracy of the AI ​​report in predicting the benign or malignant nature of lesions. In the formula, TP represents the number of lesions predicted as malignant in both reports, FP represents the number of benign lesions predicted as malignant, and FN represents the number of malignant lesions predicted as benign.

[0075]

[0076] These quantitative metrics are often more intuitive and accurate than conventional long-text evaluation methods, going beyond simply assessing the character similarity between two long texts. Based on the framework proposed in this invention, the final output quantitative metrics can reflect in detail whether various artificial intelligence models possess high completeness, accuracy, and professionalism in generating medical text tasks.

[0077] Furthermore, besides its application in medical reports, the method proposed in this invention is also applicable to evaluating any other structured text, and the output structured tables will vary depending on the application scenario. The structured matching algorithm requires selecting multiple attributes as unique descriptions of the lesion; however, in practical applications, there are no restrictions on the number and type of specific structured attributes.

[0078] Corresponding to the aforementioned disclosed AI-automated report evaluation method, this invention also discloses an AI-automated report evaluation system, such as... Figure 8 As shown, it specifically includes:

[0079] The structured report generation module is used to extract lesion information from the medical artificial intelligence model's output text report to be evaluated and the pre-established standard text report, and convert it into structured information to obtain the structured report of the text to be evaluated and the structured report of the standard text.

[0080] The structured report matching module is used to encode the feature vectors of each lesion based on the lesion structured information in the structured report to be evaluated and the standard text structured report using a pre-trained text encoder.

[0081] The similarity of feature vectors is calculated and used as edge weights. By constructing a weighted bipartite graph of the matching relationship between lesions in the two reports, the lesion information in the two reports is optimally matched. The text report to be evaluated is evaluated based on the matching results.

[0082] It should be noted that for a detailed description of the fine characterization system for sandstone and mudstone based on waveform phase control technology joint model inversion provided in the embodiments of the present invention, please refer to the relevant description of the artificial intelligence automated report evaluation method provided in the embodiments of this application, which will not be repeated here.

[0083] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0084] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. An AI-automated report evaluation method, characterized in that, The method includes: The text report to be evaluated and the pre-established standard text report output by the medical artificial intelligence model are used to extract lesion information and convert it into lesion structured information through a trained large language model, so as to obtain the structured report of the text to be evaluated and the structured report of the standard text. Each line of lesion structure information in the structured report to be evaluated and the standard structured report is used to generate a unique description text for each lesion. The unique description text is generated by splicing together multiple structured information of the lesion. The unique description text of the lesion is encoded by a pre-trained Transformer-based text encoder to obtain the feature vector of the lesion. Each lesion in both structured reports is abstracted as a single node in a bipartite graph. The text encoder encodes and transforms the text based on the unique description of the lesion in the corresponding line of the input to obtain a feature vector. Then, the Kuhn-Munkres algorithm is used to solve for optimal matching. The optimal matching selects the multiple pairs of lesions with the highest sum of feature vector similarity between the two reports, and establishes a one-to-one correspondence between the lesions in the two reports, thus forming the edge connection in the bipartite graph. The edge weight between nodes is set to the similarity between the two feature vectors. Subsequently, an appropriate similarity threshold is set to delete the edge connection in the weighted bipartite graph with a weight lower than the threshold, thereby reducing the complexity of the bipartite graph. The text report to be evaluated is evaluated based on the matching results.

2. The AI-automated report evaluation method as described in claim 1, characterized in that, The evaluation text report output by the medical artificial intelligence model and the pre-established standard text report are used to extract lesion information and convert it into structured information through a trained large language model, specifically including: Based on the constructed knowledge graph, a large language model is used to extract lesion information from the text and convert it into structured output information to generate a structured report. The extracted lesion information includes lesion location, lesion type, lesion nature, and related necessary attribute information.

3. The AI-automated report evaluation method as described in claim 2, characterized in that, The construction of a knowledge graph specifically includes: The process of building a knowledge graph includes data collection, entity recognition, and relation extraction. The data used to build the knowledge graph comes from medical databases in the application domain. Then, named entity recognition technology is used to extract relevant named entities from the data, and natural language processing technology is used to extract relation words between entities from the text, thereby converting the relationships between entities into a graph structure for representation.

4. The AI-automated report evaluation method as described in claim 3, characterized in that, Based on the constructed knowledge graph, a large language model is used to extract lesion information from the text and convert it into structured output information to generate a structured report, specifically including: Entity recognition and matching: Entities are extracted from text using a large language model, and the extracted entity information is verified using a knowledge graph; Relation-based reasoning: Based on entity relations in the knowledge graph, further reasoning is performed on the extracted structured attribute information to correct the output of the large language model; Supplementing missing information: For key attributes missing in the generated structured report, knowledge graphs are used to supplement the missing attribute information based on existing entities and relationships.

5. The AI-automated report evaluation method as described in claim 1, characterized in that, The pre-training of the text encoder includes: The pre-training process of the text encoder uses paired image-text to train the deep neural network. The image is generated by cropping the corresponding lesion region from the original image, and the text is a natural language description of the lesion attributes. The input of the text encoder comes from the structured attribute information of the lesion, and the output is a high-dimensional feature vector. use This represents the k-th image-text pair of inputs as a feature vector, and the loss function during pre-training is based on the generated feature vectors. The similarity is calculated to ensure that matching image-text pairs have higher similarity, while mismatched image-text pairs have lower similarity. The similarity function includes cosine similarity, and the calculation formula is as follows: Where T represents the feature vector of the text, I represents the feature vector of the image, and n represents the length of the feature vector.

6. The AI-automated report evaluation method as described in claim 1, characterized in that, The similarity of feature vectors is calculated using the cosine similarity function.

7. An AI-automated report evaluation system, characterized in that, The system includes: The structured report generation module is used to extract lesion information from the medical artificial intelligence model's output text report to be evaluated and the pre-established standard text report, and convert it into structured information through a trained large language model to obtain the structured report of the text to be evaluated and the structured report of the standard text. The structured report matching module is used to generate unique descriptive text for each lesion from the structured information of each line of the structured report to be evaluated and the standard structured report. The unique descriptive text is generated by splicing multiple structured information of the lesion. The unique descriptive text of the lesion is encoded by a pre-trained Transformer-based text encoder to obtain the feature vector of the lesion. Each lesion in both structured reports is abstracted as a single node in a bipartite graph. The text encoder encodes and transforms the text based on the unique description of the lesion in the corresponding line of the input to obtain a feature vector. Then, the Kuhn-Munkres algorithm is used to solve for optimal matching. The optimal matching selects the multiple pairs of lesions with the highest sum of feature vector similarity between the two reports, and establishes a one-to-one correspondence between the lesions in the two reports, thus forming the edge connection in the bipartite graph. The edge weight between nodes is set to the similarity between the two feature vectors. Subsequently, an appropriate similarity threshold is set to delete the edge connection in the weighted bipartite graph with a weight lower than the threshold, thereby reducing the complexity of the bipartite graph. The text report to be evaluated is evaluated based on the matching results.

Images