A method, system, and storage medium for visualizing medical study protocol cross-validation

Abstract

This application provides a method, system, and storage medium for cross-validation of a visualized medical research scheme. The method includes: acquiring a natural language research question; parsing the research question using a first pre-trained large language model to obtain a set of research elements; constructing a retrieval strategy based on the research element set; using the retrieval strategy to retrieve an initial set of documents; extracting features from the initial set of documents using a second pre-trained large language model to obtain structured feature data; converting the structured feature data into semantic vectors; performing cross-document feature matching and alignment based on the similarity between semantic vectors to obtain a feature alignment relationship set; performing logical consistency verification on the feature alignment relationship set to identify and output a set of contradictory feature pairs; generating a visualization interface based on the feature alignment relationship set and the set of contradictory feature pairs; and generating and outputting a cross-validation report based on the comparison and contradiction information integrated in the visualization interface. This application can efficiently and accurately achieve cross-validation of medical literature.

Description

Technical Field

[0001] This application relates to the field of cross-validation technology for medical protocols, and in particular to a method, system and storage medium for cross-validation of visual medical research protocols. Background Technology

[0002] Current medical research design and conclusion validation heavily rely on systematic evaluation and cross-comparison of existing literature. In practice, to assess the reliability and consistency of conclusions across different research protocols for a specific research question—such as evaluating the efficacy of a particular intervention—researchers typically need to manually review a large amount of relevant medical literature, extract key research features, and perform point-by-point comparisons and inconsistencies analysis. This process faces multiple challenges, including the sheer volume of medical literature, complex terminology, and diverse research design types. While literature retrieval tools exist, their functions are largely limited to text matching and list-based presentation, failing to deeply understand the semantics of the literature and automatically extract, align, and validate structured research features. Therefore, current technological solutions primarily rely on manual operation by researchers, resulting in inefficient, subjective, error-prone, and unsystematic cross-document feature extraction, alignment, and inconsistency detection processes, failing to provide efficient, objective, and comprehensive intelligent assistance for research protocol design and validation. Summary of the Invention

[0003] In order to efficiently and accurately cross-validate complex research protocols in multiple medical papers, this application provides a method, system and storage medium for visualizing cross-validation of medical research protocols.

[0004] Firstly, this embodiment provides a method for cross-validation of a visual medical research protocol, the method comprising: The system obtains a natural language research question input by the user, and parses the natural language research question using a first pre-trained large language model to obtain a structured set of research elements. A retrieval strategy is constructed based on the aforementioned set of research elements, and the retrieval strategy is used to search in a multi-source medical literature database to obtain an initial set of literature. The second pre-trained large language model is used to perform multi-task feature extraction on the full text of the documents in the initial document set to obtain the structured feature data corresponding to each document; The structured feature data of each document is converted into semantic vectors, and cross-document feature matching and alignment are performed based on the similarity between the semantic vectors to obtain a feature alignment relationship set; The feature alignment relationship set is input into a preset contradiction detection rule base for logical consistency verification to identify and output a set of contradictory feature pairs; A visualization interface containing a feature comparison matrix and a research design map is generated based on the feature alignment relationship set and the contradictory feature pair set; A cross-validation report is generated and output based on the comparison and contradiction information integrated by the visualization interface.

[0005] In some embodiments, the step of parsing the natural language research question using a first pre-trained large language model to obtain a structured set of research elements includes: The core medical entities are obtained by using the first pre-trained large language model to identify medical entities in the natural language research question. A structured PICO element set is generated based on the core medical entities and preset templates; The structured PICO element set is normalized by calling the medical knowledge graph to obtain the normalized PICO element set. Based on the standardized PICO element set, corresponding medical subject terms are generated to obtain a structured set of research elements.

[0006] In some embodiments, obtaining an initial set of documents by using the retrieval strategy in a multi-source medical literature database includes: The aforementioned search strategies were submitted to a multi-source medical literature database and parallel search was performed through the official AOI interface to obtain a candidate literature set. A pre-defined abstract model is used to score the relevance of the abstracts of each document in the candidate document set, and documents with scores higher than a pre-defined score threshold are selected as the initial document set. The citation frequency, journal impact factor, and research type weight of the initial screening literature set are obtained. A comprehensive quality assessment is conducted based on the citation frequency, journal impact factor, and research type weight of the initial screening literature set to obtain a weighted score. The initial document set is sorted according to the weighted score to select a preset number of documents to form the initial document set.

[0007] In some embodiments, the step of using a second pre-trained large language model to perform multi-task feature extraction on the full text of the documents in the initial document set to obtain structured feature data corresponding to each document includes: The full text of each document is divided into paragraphs to obtain the paragraph sequence of each document; For each paragraph in the paragraph sequence of each document, the second pre-trained large language model is used to perform named entity recognition, entity relation extraction, and research design classification tasks to extract the entity set, entity relation graph, and research type label of the paragraph. The entity sets, entity relationship graphs, and research type labels extracted from each paragraph of the same document are fused and conflict-resolved to construct a feature map representing the full text structure of the document. Multi-dimensional feature data, including descriptions of research subjects, details of intervention measures, control design schemes, definitions of outcome indicators, and statistical results, are extracted and structured from the feature maps of the same document to serve as the structured feature data of the document.

[0008] In some embodiments, the step of obtaining a feature alignment relationship set by cross-document feature matching and alignment based on the similarity between the semantic vectors includes: Obtain the descriptive text corresponding to the same type of feature in the structured feature data of each document, and convert each descriptive text into a semantic vector through a pre-trained sentence encoding model; The similarity matrix is ​​obtained by calculating the similarity between semantic vectors of the same type of feature among various documents; A multi-pair matching algorithm is used to process the similarity matrix to perform optimal matching of features and obtain a preliminary set of aligned feature pairs. Cluster analysis is performed on the preliminary aligned feature pair set to group feature pairs with similarity higher than a preset similarity threshold into the same feature group and generate representative feature descriptions within the group; Based on the feature groups and representative feature descriptions within the groups, a feature alignment relationship set containing feature correspondences between documents is constructed.

[0009] In some embodiments, the step of inputting the feature alignment relationship set into a preset contradiction detection rule base for logical consistency verification to identify and output a set of contradictory feature pairs includes: Obtain the feature groups that have been matched in the feature alignment relationship set, and extract the key parameters corresponding to each document in each feature group; The key parameters are input into a preset contradiction detection rule base, and the statistical conclusion contradiction rule, the dosage range contradiction rule, and the population characteristic contradiction rule are applied sequentially to perform logical verification to obtain the verification result. Based on the verification results, document feature pairs that violate any rule are identified and assigned corresponding contradiction level labels. Aggregate all document feature pairs with contradiction level labels to generate a set of contradictory feature pairs.

[0010] In some embodiments, generating a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the contradictory feature pair set includes: Extract representative feature descriptions within each feature group and corresponding literature sources from the feature alignment relationship set to construct a feature comparison matrix with literature as rows and representative feature descriptions within each group as columns. Based on the contradictory features, each feature pair and its contradiction level label in the set are highlighted in the corresponding matrix unit of the feature comparison matrix; Based on the feature correspondence between documents in the feature alignment relationship set, an initial research design graph is constructed with documents as nodes and feature similarity as weights. The contradictory feature set is mapped to the initial research design graph to identify the conflict between the edges of contradictory document nodes, thereby obtaining a revised research design graph; The feature comparison matrix and the revised research design map are integrated into an interactive visualization interface.

[0011] In some embodiments, the method further includes configuring hyperlinks to the original literature data for each matrix unit and graph node.

[0012] Secondly, this embodiment provides a cross-validation system for visualized medical research schemes. The system includes: a parsing module, a retrieval module, an extraction module, an alignment module, a validation module, a visualization module, and an output module; wherein, The parsing module is used to obtain the natural language research question input by the user, and to parse the natural language research question through the first pre-trained large language model to obtain a structured set of research elements. The retrieval module is used to construct a retrieval strategy based on the set of research elements, and use the retrieval strategy to search in a multi-source medical literature database to obtain an initial set of literature. The extraction module is used to perform multi-task feature extraction on the full text of the documents in the initial document set using the second pre-trained large language model to obtain the structured feature data corresponding to each document. The alignment module is used to convert the structured feature data of each document into semantic vectors, and to perform cross-document feature matching and alignment based on the similarity between the semantic vectors to obtain a feature alignment relationship set; The verification module is used to input the feature alignment relationship set into a preset contradiction detection rule base for logical consistency verification in order to identify and output a set of contradictory feature pairs; The visualization module is used to generate a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the contradictory feature pair set. The output module is used to generate and output a cross-validation report based on the comparison and contradiction information integrated by the visualization interface.

[0013] Thirdly, this embodiment provides a computer-readable storage medium storing a computer program that can run on a processor, wherein the computer program, when executed by the processor, implements a cross-validation method for a visual medical research scheme as described in the first aspect.

[0014] By employing the above method, this application obtains the natural language research question input by the user, and parses the natural language research question using a first pre-trained large language model to obtain a structured set of research elements. A retrieval strategy is constructed based on the research element set, and the retrieval strategy is used to search a multi-source medical literature database to obtain an initial set of documents. A second pre-trained large language model is used to perform multi-task feature extraction on the full text of the documents in the initial set to obtain structured feature data corresponding to each document. The structured feature data of each document is converted into semantic vectors, and cross-document feature matching and alignment are performed based on the similarity between semantic vectors to obtain a feature alignment relationship set. The feature alignment relationship set is input into a preset contradiction detection rule base for logical consistency verification to identify and output a set of contradictory feature pairs. A visualization interface containing a feature comparison matrix and a research design map is generated based on the feature alignment relationship set and the set of contradictory feature pairs. A cross-validation report is generated and output based on the comparison and contradiction information integrated in the visualization interface. This enables efficient and accurate cross-validation of complex research schemes in multiple medical documents. Attached Figure Description

[0015] Figure 1 This is a flowchart of a cross-validation method for a visual medical research scheme provided in this application.

[0016] Figure 2 This application provides a flowchart of a method for obtaining a structured set of research elements by parsing a natural language research problem using a first pre-trained large language model.

[0017] Figure 3 This application provides a method flowchart for identifying and outputting a set of contradictory feature pairs by inputting a set of feature alignment relationships into a preset contradiction detection rule base for logical consistency verification.

[0018] Figure 4 This is a flowchart of a method provided in this application for generating a visualization interface containing a feature comparison matrix and a research design map based on a feature alignment relationship set and a set of contradictory feature pairs.

[0019] Figure 5 This is a schematic diagram of the connection of a cross-validation system for a visual medical research scheme provided in this application. Detailed Implementation

[0020] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but is consistent with the broadest scope claimed in this application.

[0021] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a cross-validation method for a visual medical research protocol provided in this application. Figure 1 As shown, a cross-validation method for a visual medical research protocol includes the following steps: Step S100: Obtain the natural language research question input by the user, and parse the natural language research question using the first pre-trained large language model to obtain a structured set of research elements.

[0023] This application describes the verification process from the perspective of the verification end. The verification end provides an interactive input interface that allows users to input complete research questions in natural language, i.e., input natural language research questions, thereby enabling the verification end to obtain the natural language research questions input by the user.

[0024] In order to transform natural language research questions into structured elements that can be retrieved and compared, a semantic parsing and structuring approach based on a medical large language model was adopted. Figure 2 This is a flowchart illustrating the method provided in this application for parsing natural language research problems using a first pre-trained large language model to obtain a structured set of research elements. For example... Figure 2 As shown, the process of obtaining a structured set of research elements by analyzing a natural language research question using the first pre-trained large language model includes the following steps: Step S101: Use the first pre-trained large language model to identify medical entities in the natural language research problem to obtain the core medical entities.

[0025] Step S102: Generate a structured PICO element set based on the core medical entity and the preset template.

[0026] Step S103: Call the medical knowledge graph to standardize each element in the structured PICO element set to obtain a standardized PICO element set.

[0027] Step S104: Generate corresponding medical subject terms based on the standardized PICO element set to obtain a structured research element set.

[0028] The first pre-trained large language model is a sequence labeling model pre-trained and fine-tuned with medical text, such as BioBERT or ClinicalBERT based on the BERT architecture. This model receives natural language research questions input by the user, such as whether metformin can reduce the risk of cardiovascular events in patients with type 2 diabetes. Its embedding layer converts the input text into a sequence of word vectors, which are then processed by multiple Transformer encoders to obtain contextual semantic representations. The model's output layer uses conditional random field decoding to label each token in the input sequence with a medical entity type, identifying entity fragments belonging to predefined types such as diseases, drugs, populations, and outcome indicators. These identified entities form the basis for subsequent structured extraction.

[0029] PICO stands for Population, Intervention, Comparison, and Outcome. A pre-defined template conforming to the PICO framework contains four slots: P (Population), I (Intervention), C (Comparison), and O (Outcome). This step maps the entity types identified in step S101 to their corresponding slots. The mapping rules are based on a table of correspondences between entity types and PICO slots, supplemented by a lightweight rule engine for disambiguation and completion. For example, type 2 diabetes is mapped to slot P, metformin to slot I, and cardiovascular events to slot O. If C (Comparison) is not explicitly mentioned in the question, a default control description is automatically filled in based on the intervention type. Finally, a preliminary, structured set of PICO elements is generated and stored in key-value pairs, for example: {"P": "Type 2 diabetes patients", "I": "Metformin", "C": "Placebo", "O": ["Incidence of cardiovascular events"]}.

[0030] To avoid retrieval bias caused by different expressions of the same concept, medical knowledge graphs, such as UMLS and MeSH (Medical Subject Headings), are further introduced to semantically normalize PICO elements. The descriptive text of each element is matched with concept nodes in the knowledge graph. By calculating text similarity or using a graph embedding model, the corresponding standard concept identifier or preferred term is found. For example, metformin is normalized to the standard drug name "Metformin" in the knowledge graph, and cardiovascular events are normalized to the standard outcome term "Major Adverse Cardiovascular Events (MACE)". For population descriptions, such as patients with type 2 diabetes, the knowledge graph can further associate them with their standard diagnostic codes and typical population characteristics. Each normalized element is associated with a unique standard concept identifier, forming a standardized set of PICO elements, ensuring semantic consistency in subsequent retrieval and comparison.

[0031] To adapt to the search syntax of mainstream medical literature databases, the standardized PICO element set is further converted into corresponding medical subject heading terms or Emtree terms. By querying the mapping relationship between concepts and subject headings in the knowledge graph, the most relevant subject headings and their entry terms are recommended for each standardized element. For example, the standardized drug "etformin" is searched for the MeSH term "etformin". Simultaneously, relevant hyponyms or synonyms can be automatically expanded according to the needs of the search strategy to construct a more comprehensive search expression. The final output is a structured set of research elements, which not only contains the original PICO elements but also includes corresponding standard concept identifiers, subject headings, extended synonym lists, and keyword combinations that can be used to construct Boolean search expressions, providing standardized input for subsequent efficient and accurate literature retrieval.

[0032] First, by using a pre-trained large language model for medical entity recognition, accurate semantic analysis of complex and diverse natural language research questions is achieved, overcoming the shortcomings of keyword matching methods in ambiguity resolution and contextual understanding. Second, structured generation based on PICO templates and knowledge graph-driven normalization processing enables automatic conversion from unstructured text to standardized, unambiguous research elements, significantly improving the recall and precision of subsequent literature searches. Finally, by outputting a structured set of research elements containing medical subject terms, high-quality search strategies can be automatically generated, reducing the time and expertise required for researchers to manually construct search queries. This provides a reliable, efficient, and standardized input foundation for the entire cross-validation process, improving the automation and accuracy of cross-validation of medical research protocols from the source.

[0033] Step S200: Construct a retrieval strategy based on the research element set, and use the retrieval strategy to search in a multi-source medical literature database to obtain an initial literature set.

[0034] After completing the structured analysis of natural language research questions and obtaining a set of research elements including standardized PICO elements and medical subject terms, the next step is to construct a retrieval strategy. This automatically and intelligently transforms the standardized research elements into efficient and accurate literature retrieval instructions, laying the foundation for obtaining a high-quality, highly relevant initial literature set. Specifically, the structured research element set obtained above is used to construct a composite retrieval strategy that can be directly submitted to various medical literature databases, balancing recall and precision. This retrieval strategy is not simply a collection of keywords, but a set of instructions encompassing multiple retrieval formats and execution logic.

[0035] Specifically, the process begins by generating a core Boolean logic search expression. This expression is constructed based on the standardized medical subject terms and their extended synonym lists corresponding to the P (population), I (intervention), C (control), and O (outcome) elements in the research element set. Pre-defined search syntax templates for different databases are provided. For example, for the intervention element, the corresponding MeSH terms and their entry terms are connected using OR to form the intervention part of the search expression; the same applies to population and outcome elements. For the control element, if a default value is used, the corresponding subject terms are automatically added. The AND operator is used to connect the four parts—P, I, C, and O—to form the core Boolean logic search expression. This ensures the standardization of the search, effectively utilizes the controlled terminology of the database, and improves precision.

[0036] Secondly, semantically enhanced natural language search queries are generated to compensate for the shortcomings of relying solely on keyword terms, which may miss new concepts or non-standard expressions in literature. This is achieved by leveraging the semantic understanding capabilities of a pre-trained large language model. The standardized PICO element descriptions from the research element set are input into a finely tuned query expansion generation model, such as a model based on T5 or GPT architecture. This model, trained on a large number of medical search query-related document pairs, can generate a series of semantically relevant and diverse natural language keyword or phrase queries based on the input standardized elements. These generated queries are closer to the free text expressions in the literature abstract or full text, helping to improve recall, especially for the latest research not yet fully included in the keyword system.

[0037] Finally, a search strategy is generated. This involves integrating the core Boolean search expression generated above with the natural language semantic enhancement search expression to form the final composite search strategy. During this process, weights or execution priorities are assigned to different components based on the characteristics of the target database and historical search performance data. For example, in PubMed, keyword searches may be given higher weight; while in full-text databases or preprint platforms, the weight of natural language queries will be correspondingly increased. Furthermore, built-in heuristic rules, such as automatically adding corresponding research design filters for specific research types, further enhance the relevance of search results. The final generated search strategy is a structured instruction object containing specific query strings adapted to different databases, filter settings, and weight information for each component.

[0038] This approach, combining core Boolean retrieval based on standardized subject terms with free text retrieval based on semantic generation from a large language model, and supplemented by intelligent weight configuration, achieves an effective balance between retrieval scope and precision—that is, a balance between recall and precision. Furthermore, the retrieval strategy is designed to consider the characteristics of different medical literature databases and can automatically optimize based on research type, ensuring consistency and efficiency in cross-database retrieval. This provides a reliable guarantee for obtaining a high-quality, comprehensive initial literature set. Fundamentally, it improves the efficiency and quality of the starting point of the cross-validation process for the entire medical research protocol.

[0039] After completing the structured analysis of the natural language research question and obtaining a set of research elements containing standardized PICO elements and medical subject terms, the next step is to construct a search strategy. This automatically and intelligently transforms the standardized research elements into efficient and accurate literature search instructions, laying the foundation for obtaining a high-quality, highly relevant initial literature set. Specifically, the structured research element set obtained above is used to construct a composite search strategy that can be directly submitted to various medical literature databases, balancing recall and precision. This search strategy is not simply a collection of keywords, but a set of instructions containing multiple search formats and execution logics. The process of using this search strategy to retrieve the initial literature set from multiple source medical literature databases includes the following steps: Step S201: Submit the search strategy to the multi-source medical literature database and perform parallel retrieval through the official API interface to obtain a candidate literature set.

[0040] Step S202: Use a preset abstract model to score the relevance of the abstracts of each document in the candidate document set to select documents with scores higher than the preset score threshold as the initial document set.

[0041] Step S203: Obtain the citation frequency, journal impact factor, and research type weight of the initial screening literature collection. Based on the citation frequency, journal impact factor, and research type weight of the initial screening literature collection, conduct a comprehensive quality assessment to obtain a weighted score.

[0042] Step S204: Sort the initial document set according to the weighted score to select the first preset number of documents to form the initial document set.

[0043] Specifically, a configurable database connection pool is maintained first, pre-configured with official API interfaces for several mainstream medical literature databases, including PubMed, Cochrane Library, EMBASE, and CNKI. Upon receiving the constructed search strategy, the strategy is parsed, and the unified search logic is converted into specific query requests adapted to each database, taking into account their different query syntax and interface specifications. For example, for PubMed, which supports MeSH search, the keyword part of the strategy is converted into a Mesh field query; for CNKI, it is converted into the corresponding Chinese academic keyword search formula. Subsequently, an asynchronous task scheduling framework is used to simultaneously send search requests to all configured databases, achieving true multi-source parallel retrieval. The search results returned by each database are deduplicated and merged to form a preliminary candidate document set. The search results returned by each database include metadata such as document title, author, abstract, DOI, and publication year. This parallel processing mechanism significantly reduces the waiting time of traditional serial retrieval. For each document in the initial document set, the full text is obtained from an open access repository (such as PubMed Central) or through a legally authorized channel subscribed by the institution (such as accessing a subscribed full-text database via an institutional API) based on its DOI or PMID. If the full text cannot be obtained, the document is excluded from subsequent steps requiring full-text processing, and only the abstract information is used or the document is discarded during subsequent feature extraction. This step ensures the legality and feasibility of subsequent full-text feature extraction.

[0044] To quickly identify the most relevant literature to the research question from a massive pool of candidate documents, a pre-defined, specially trained abstract relevance scoring model is introduced. This model is built using supervised learning, with its training data consisting of a large number of triples annotated by medical experts: research question, document abstract, and relevance label. The model typically uses pre-trained language models such as BERT or RoBERTa as encoders, encoding the structured description of the research question (e.g., the set of research elements) and the document abstract text into semantic vectors. A multilayer perceptron then calculates the semantic matching score between the two, outputting a relevance score between 0 and 1. This model is applied to each document in the candidate document set, automatically calculating its abstract relevance score to the research question. Subsequently, the scores of all documents are compared to a pre-defined scoring threshold, automatically selecting all documents with scores higher than the threshold to form an initial screening set. This automated scoring and screening process replaces the arduous task of traditional manual abstract reading for initial screening, ensuring the objectivity and consistency of the initial screening results.

[0045] Next, to further evaluate the academic impact and level of evidence of the literature based on relevance, a multi-dimensional quality assessment is conducted on the initial screening literature set. First, by accessing academic bibliometric databases or parsing locally cached knowledge graphs, the citation frequency and impact factor of the journals in which each initial screening literature is published are obtained in batches. Simultaneously, based on the research question type parsed in step S100, different evidence weights are assigned to different research design types from a pre-defined mapping table. Then, a pre-defined weighted scoring formula is used to comprehensively calculate the above three indicators for each literature, generating a weighted score. The weighted scoring formula is, for example: Weighted Score = α * Normalized Citation Frequency + β * Normalized Impact Factor + γ * Research Type Weight, where α, β, and γ are adjustable weight coefficients, and the normalization process aims to eliminate the influence of different indicator dimensions. This evaluation mechanism provides a quantitative and comprehensive assessment of literature quality from two levels: academic impact and methodological rigor.

[0046] Finally, after obtaining the weighted score for each initially screened document, the entire initial screening document set is sorted in descending order of weighted score. Then, based on a preset number N, the top N documents in the sorted list are automatically selected. These N documents are considered to be highly relevant to the research question and relatively optimal in terms of academic quality and level of evidence; they together constitute the initial document set for subsequent deep feature extraction and cross-validation. This ensures that the document set entering the subsequent computationally expensive analysis process is a high-quality, highly relevant core component, effectively optimizing overall resource utilization and analysis efficiency.

[0047] Step S300: Use the second pre-trained large language model to perform multi-task feature extraction on the full text of the documents in the initial document set to obtain the structured feature data corresponding to each document.

[0048] After efficiently searching and filtering multi-source medical literature databases to obtain a high-quality, highly relevant initial literature set, in order to further achieve accurate cross-document comparison and verification, it is necessary to extract in-depth, structured research features from these documents. This involves introducing a second pre-trained large language model to perform refined multi-task feature extraction on the full text of the documents, converting unstructured text into machine-understandable and computable structured feature data, laying the foundation for subsequent feature alignment and contradiction detection. Specifically, the process of using the second pre-trained large language model to perform multi-task feature extraction on the full text of the documents in the initial literature set to obtain the structured feature data corresponding to each document includes the following steps: Step S301: Divide the full text of each document into paragraphs to obtain the paragraph sequence of each document.

[0049] Step S302: For each paragraph in the paragraph sequence of each document, the second pre-trained large language model is used to perform named entity recognition, entity relation extraction, and research design classification tasks to extract the entity set, entity relation graph, and research type label of the paragraph.

[0050] Step S303 involves fusing and resolving conflicts among the entity sets, entity relationship graphs, and research type labels extracted from each paragraph of the same document to construct a feature map representing the full-text structure of the corresponding document.

[0051] Step S304: Extract and structure the multi-dimensional feature data, including the description of the research subjects, details of the intervention measures, control design, definition of the outcome indicators, and statistical results data, from the feature maps of the same document to serve as the document's structured feature data.

[0052] Specifically, the first step employs an intelligent paragraph segmentation method combining rule-based and semantic understanding. This involves identifying the document's macro-structure based on pre-defined regular expressions for medical literature chapter titles, dividing the entire text into main chapters such as background, methods, results, and discussion. Then, within each chapter, a sentence boundary detection model fine-tuned for medical texts, such as a spaCy or BERT-based model, is used to identify sentence endings. Sentences are aggregated based on semantic coherence analysis, calculating the cosine similarity between the embedding vectors of adjacent sentences. When the similarity falls below a pre-defined threshold, it is considered the end of a paragraph. This results in a paragraph sequence that reflects the document's logical structure while ensuring internal semantic coherence, providing suitable input for subsequent fine-grained analysis at the paragraph level.

[0053] The second pre-trained large-scale language model is a model pre-trained on a large-scale corpus in the biomedical field and jointly fine-tuned for the aforementioned multi-tasks, such as a model based on the PubMedBERT architecture. For the named entity recognition task, the model identifies entities belonging to predefined types within a paragraph, including research subjects, interventions, control groups, outcome indicators, and statistical results. For the entity relation extraction task, the model identifies semantic relationships between these entities, such as intervention-applied to-research subject, and constructs a local entity relation graph with entities as nodes and relations as edges. For the research design classification task, the model performs multi-label classification on the paragraph content, determining whether it involves specific research design elements such as randomization, blinding, assignment hiding, and follow-up, and outputs the corresponding design type label and its confidence level. Thus, through parallel extraction of multiple tasks within a single paragraph, a deep, structured analysis of local information about the research design is achieved.

[0054] Then, information scattered throughout the text is integrated to form a unified document-level structured representation. First, entity fusion is performed, which involves resolving textual references based on entity mentions and normalizing concepts based on medical knowledge graphs. Entities pointing to the same real-world concept from different paragraphs are merged, and all their contexts and attributes are summarized. Second, relationship graph fusion is performed, which involves stitching together the local entity relationship graphs of each paragraph. By identifying and linking core entities across paragraphs, a connected knowledge graph covering the entire text—the feature graph—is constructed. In this graph, nodes represent normalized entities, and edges represent relationships between entities. Finally, conflict resolution is performed. When different paragraphs describe the same entity or relationship differently, a rule-based or model-confidence-weighted method is used to adjudicate the discrepancies, ensuring the consistency of the internal logic of the feature graph.

[0055] Finally, based on the global feature map constructed in step S303, the core feature dimensions used for cross-validation are systematically extracted. This involves locating research object nodes from the map, extracting and integrating all their associated attributes to form standardized research object descriptions. Intervention and control nodes are located, and their detailed parameters and design details are extracted to form detailed descriptions of the intervention and control. Outcome indicator nodes are located, and definition information such as indicator name, measurement time point, and measurement method is extracted. Statistical result nodes are located, and specific effect estimates, confidence intervals, and p-values ​​are extracted and associated with the corresponding intervention, control, and outcome indicators. Ultimately, all extracted information is organized into a structured data object containing all key, unambiguous feature fields and their values ​​used for subsequent cross-document comparisons, serving as the final structured feature data for the document. This achieves automated conversion from unstructured full-text documents to high-precision structured feature data, significantly improving the accuracy and completeness of feature extraction. Secondly, by employing a two-stage strategy combining paragraph-level fine-grained extraction with full-text-level graph fusion, not only were local details captured, but the overall logical structure and contextual relationships of the research scheme were also reconstructed and understood, resulting in extracted feature data with rich semantic information and inherent consistency. Finally, the standardized, multi-dimensional structured feature data produced provides direct and reliable input for subsequent automated feature matching and alignment across documents, as well as rule-based logical contradiction detection.

[0056] Step S400: Convert the structured feature data of each document into semantic vectors, and perform cross-document feature matching and alignment based on the similarity between semantic vectors to obtain a feature alignment relationship set.

[0057] After completing multi-task feature extraction of the full text of each document in the initial document set and obtaining structured and standardized multi-dimensional feature data, in order to achieve automated and accurate cross-document comparison, these discrete, text-based feature data need to be transformed into machine-computable and measurable numerical representations for subsequent semantic similarity calculation and feature matching. This involves efficiently and accurately converting the structured feature data of each document into vector representations with rich semantic information, laying the foundation for building comparability between features.

[0058] Specifically, firstly, for the structured feature data of each document, descriptive text corresponding to each feature dimension is extracted. This descriptive text consists of normalized natural language sentences or phrases generated in the preceding steps, directly carrying the complete semantic information of the features. Subsequently, a pre-trained sentence encoding model is used to encode each feature descriptive text. This model is preferably pre-trained on both large-scale general corpora and biomedical professional corpora. The model receives a feature descriptive text as input, obtains a context-aware semantic representation of the text through its internal Transformer encoder, and aggregates this representation into a fixed-dimensional, dense semantic vector through a pooling layer. This process maps the originally unstructured text semantics to a continuous vector space, where semantically similar texts have corresponding vectors that are closer in distance within the space. Finally, to maintain consistency in the transformation and facilitate subsequent cross-document matching, the same sentence encoding model is used to vectorize the descriptive text of the same type of feature in all documents. For example, feature descriptions of intervention details in all documents are converted into semantic vectors in the same vector space using this model. Thus, the structured feature data of each document is transformed into a set of multiple semantic vectors, each representing the specific content of a particular type of feature. By employing a pre-trained medical-specific sentence encoding model, the standardized feature description text is efficiently and accurately mapped into semantically rich numerical vectors, achieving precise numerical representation of the deep semantics of complex medical features. This lays a comparability foundation for subsequent automated feature alignment. This approach not only significantly improves the efficiency and consistency of cross-document feature matching but also enhances the method's versatility and scalability by not relying on specific rules. Overall, it provides a reliable and computable semantic comparison basis for intelligent cross-validation of medical research protocols.

[0059] After completing multi-task feature extraction of the full text of each document in the initial document set and obtaining structured, standardized, multi-dimensional feature data, to achieve automated and accurate cross-document comparison, these discrete, text-based feature data need to be transformed into machine-computable and measurable numerical representations for subsequent semantic similarity calculation and feature matching. This involves efficiently and accurately converting the structured feature data of each document into vector representations with rich semantic information, laying the foundation for building comparability between features.

[0060] After completing the semantic vectorization transformation of the structured feature data of each document, in order to achieve accurate and automated cross-document comparison, it is necessary to perform systematic feature matching and alignment based on these semantic vectors to construct a structured data foundation that can clearly reflect the feature correspondence and differences between different documents. The process of obtaining a feature alignment relationship set through cross-document feature matching and alignment based on the similarity between semantic vectors includes the following steps: Step S401: Obtain the descriptive text corresponding to the same type of feature in the structured feature data of each document, and convert each descriptive text into a semantic vector through a pre-trained sentence encoding model.

[0061] Step S402: Calculate the similarity between semantic vectors of the same type of features among the documents to obtain a similarity matrix.

[0062] Step S403: Use a multi-pair matching algorithm to process the similarity matrix to perform optimal matching of features and obtain a preliminary set of aligned feature pairs.

[0063] Step S404: Perform cluster analysis on the preliminary aligned feature pair set to group feature pairs with similarity higher than a preset similarity threshold into the same feature group and generate representative feature descriptions within the group.

[0064] Step S405: Construct a feature alignment relationship set containing feature correspondences between documents based on feature groups and representative feature descriptions within the groups.

[0065] Specifically, the structured feature data includes various predefined feature types, such as descriptions of research subjects and details of intervention measures. For a specific feature type requiring comparison, such as details of intervention measures, all documents in the initial literature set are traversed, and the descriptive text corresponding to that feature type is extracted from the structured feature data of each document. These descriptive texts are relatively standard natural language sentences after the aforementioned feature extraction and normalization processing. Subsequently, a dedicated sentence encoding model, which has been fully pre-trained on biomedical corpora, such as a model based on the Sentence-BERT architecture and fine-tuned using medical texts like PubMed abstracts, is used to independently encode each descriptive text. This model takes variable-length text input, obtains contextual semantic representations through its internal Transformer encoder, and outputs a fixed-dimensional high-dimensional dense vector, i.e., a semantic vector. This ensures that feature descriptions in different documents, which may have different expressions but similar semantics, are mapped to close positions in the same semantic vector space, laying the mathematical foundation for subsequent similarity calculations.

[0066] After obtaining the set of semantic vectors of all documents for a certain feature type, the similarity between the semantic vectors of any two documents on that feature is calculated. Specifically, cosine similarity is used as the metric, calculating the cosine of the angle between two semantic vectors in their respective high-dimensional space. This value ranges from [-1, 1], with values ​​closer to 1 indicating greater semantic similarity. By iterating through all document pairs, for a set of M documents, an M×M symmetric similarity matrix is ​​calculated. Each element (i, j) of this matrix represents the degree of semantic similarity between document i and document j on the specific feature type. This quantifies the semantic relationships of text features into sortable and thresholdable numerical relationships.

[0067] Because different documents may emphasize or detail different aspects of the same research approach, a feature description in one document may be semantically partially similar to feature descriptions in multiple documents. To establish accurate correspondences, it is necessary to solve the many-to-many optimal matching problem. This is addressed using an improved Hungarian algorithm or a maximum weight matching algorithm. First, the similarity matrix is ​​treated as a weight matrix of a bipartite graph. The algorithm aims to find a set of feature pairs that match such that the sum of the total similarity weights of these matches is maximized, while satisfying specific matching constraints. By solving this optimization problem, a preliminary aligned feature pair set is output. Each element in the set is a tuple recording the feature identifiers of two documents deemed most likely to describe the same core fact or concept, along with their calculated similarity. This achieves preliminary cross-document feature association.

[0068] The initial alignment of feature pairs may still contain redundancy. For example, if the features of document A are aligned with those of document B, and the features of document B are aligned with those of document C, then the features of A, B, and C should belong to the same semantic cluster. Therefore, a clustering algorithm is used for merging. Based on the aforementioned initial alignment of feature pairs and their similarities, a similarity graph is constructed using features as nodes and feature pairs and their similarities as edges. Then, hierarchical clustering or community detection algorithms are applied to aggregate feature nodes in the graph that are tightly connected by high-similarity edges into a feature group. A preset similarity threshold is used as the criterion for clustering to ensure that the features within a group have a high degree of semantic consistency. For each generated feature group, a representative feature description is generated by extracting the common parts of all feature description texts within the group, or by selecting the feature description with the highest average similarity to other features within the group. This description summarizes the core semantics commonly expressed by the features in the group.

[0069] Finally, the clustering results are structurally integrated to form the final feature alignment relation set. This relation set is a structured dataset where each entry corresponds to a feature group. Each entry contains at least the following information: a unique identifier for the feature group; the feature type corresponding to the feature group; a description of the representative features generated within the group; and a list of feature identifiers for all documents belonging to that group. The feature alignment relation set clearly records the semantic correspondence of features across documents in a machine-readable form, indicating which features in different documents describe the same or highly similar content, providing a direct and clear input basis for subsequent contradiction detection and visualization comparison.

[0070] Step S500: Input the feature alignment relationship set into the preset contradiction detection rule base for logical consistency verification to identify and output the contradictory feature pair set.

[0071] Figure 3 This application provides a method flowchart for identifying and outputting a set of contradictory feature pairs by inputting a set of feature alignment relationships into a pre-defined contradiction detection rule base for logical consistency verification. Figure 3 As shown, the process of inputting the feature alignment relationship set into a preset contradiction detection rule base for logical consistency verification to identify and output a set of contradictory feature pairs includes the following steps: Step S501: Obtain the feature groups that have been matched in the feature alignment relationship set, and extract the key parameters corresponding to each document in each feature group.

[0072] Step S502: Input the key parameters into the preset contradiction detection rule library and apply the statistical conclusion contradiction rule, the dosage range contradiction rule, and the population characteristic contradiction rule in sequence to perform logical verification and obtain the verification result.

[0073] Step S503: Identify document feature pairs that violate any rule based on the verification results, and assign them corresponding contradiction level labels.

[0074] Step S504: Aggregate all document feature pairs with contradiction level labels to generate a set of contradictory feature pairs.

[0075] Specifically, the feature alignment set is a structured data set formed after the aforementioned feature matching and alignment. Each entry represents a feature group, which contains feature identifiers from multiple documents identified as describing the same core concept. First, the feature alignment set is traversed to locate each matched feature group. Then, based on the document feature identifiers recorded in the feature group, the structured feature data corresponding to each document is traced back to accurately extract specific parameter values ​​related to that feature group. The types of these key parameters depend on the feature type corresponding to the feature group. For example, if the feature group corresponds to details of intervention measures, the extracted key parameters include drug name, specific dosage, dosing frequency, and treatment duration; if it corresponds to statistical results data, the extracted key parameters include effect size, confidence interval, and p-value; if it corresponds to descriptions of research subjects, the extracted key parameters may include quantitative or categorical information such as the age range, gender ratio, and disease stage of the population. The extraction process ensures that the parameters contributed by each document strictly correspond to its role in the feature group, forming a set of key parameters with feature groups as units and documents as dimensions, preparing structured input data for subsequent logical verification.

[0076] The pre-defined contradiction detection rule base embeds formal rules based on medical research logic and statistical principles. After inputting the key parameter set extracted in step S501, three core rules—statistical conclusion contradiction rules, dosage range contradiction rules, and population characteristic contradiction rules—are invoked sequentially for automated verification. The statistical conclusion contradiction rule targets feature groups containing statistical results such as effect size and p-value. The rule's judgment logic is: compare the effect direction and statistical significance reported by different literature within the same feature group. If the effect directions of two literature reports are opposite, and their respective p-values ​​are both less than the preset significance threshold, the rule is triggered, indicating a statistical conclusion contradiction. The dosage range contradiction rule targets feature groups describing the same intervention. The rule's judgment logic is: compare the drug dosage parameters reported by different literature within this feature group. Calculate the difference rate between the highest and lowest doses. If the difference rate exceeds a preset reasonable threshold and cannot be reasonably explained by differences in medication frequency or treatment duration, the rule is triggered, indicating a dosage range contradiction. The population characteristic contradiction rule targets feature groups describing the research subjects. The rule's judgment logic is: compare the key population characteristic parameters reported by different literature within this feature group. Statistical tests are used to determine whether these parameters differ significantly across different documents. If the p-value of the difference is less than a preset significance threshold, a rule is triggered, indicating a contradiction in population characteristics. For each rule applied, pairwise checks are performed on all possible document pairs within the current feature group, and a structured check result is output. This result records whether a contradiction was triggered, the type of rule triggered, and the identifiers of the document pairs involved.

[0077] The verification results output in step S502 are analyzed to filter out all document feature pairs that violate any of the aforementioned contradictory rules. Each identified contradictory document feature pair is assigned a predefined contradiction level label based on the type and severity of the rule violation. For example, a contradiction in statistical conclusions is labeled as a Level 1 contradiction, indicating a fundamental conflict in the conclusions; a contradiction in dosage ranges is labeled as a Level 2 contradiction, indicating significant inconsistencies in key parameters of the intervention protocol; and a contradiction in population characteristics is labeled as a Level 3 contradiction, indicating significant differences in the baseline population of the study. The contradiction level label is bound to the document feature pair as metadata and used for subsequent ranking and visualization of contradiction severity.

[0078] Finally, all document feature pairs identified and labeled with contradiction level tags in step S503 are aggregated, deduplicated, and structurally integrated to generate the final set of contradictory feature pairs. This set is a structured data list, where each entry records a specific contradiction instance, including but not limited to the following information: the identifiers of the two contradictory documents, the specific feature group or feature type involved in the contradiction, the name of the triggering contradiction rule, the assigned contradiction level tag, and relevant parameters extracted from the original documents as evidence. The set of contradictory feature pairs is the final output of the contradiction detection process, providing clear, machine-readable factual evidence of contradictions for subsequent visualization highlighting and cross-validation report generation, thereby significantly enhancing the automation, intelligence, and reliability of cross-validation of medical research protocols.

[0079] Step S600: Generate a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the contradictory feature pair set.

[0080] After completing the logical processing of the feature alignment relationship set and the contradictory feature pair set, in order to enable researchers to intuitively and efficiently grasp the results of multi-document comparisons and contradictory information, the above-mentioned structured data needs to be transformed into a highly interactive and insightful visualization. That is, by constructing a feature comparison matrix and a research design map, and integrating them into a unified interactive interface, intuitive support is provided for core analysis and decision-making. Figure 4 This is a flowchart illustrating the method provided in this application for generating a visualization interface containing a feature comparison matrix and a research design map based on a set of feature alignment relationships and a set of contradictory feature pairs. For example... Figure 4 As shown, the process of generating a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the set of contradictory feature pairs includes the following steps: Step S601: Extract the representative feature descriptions within each feature group and the corresponding literature sources from the feature alignment relationship set to construct a feature comparison matrix with literature as rows and representative feature descriptions within the group as columns.

[0081] Step S602: Based on the contradictory features, highlight the corresponding matrix units in the feature comparison matrix for each feature pair and its contradictory level label in the set.

[0082] Step S603: Based on the feature correspondence between documents in the feature alignment relationship set, construct an initial research design graph with documents as nodes and feature similarity as weights.

[0083] Step S604: Map the set of contradictory feature pairs to the initial research design graph to identify the conflict between the edges of contradictory document nodes to obtain the revised research design graph.

[0084] Step S605: Integrate the feature comparison matrix and the revised research design map into an interactive visualization interface.

[0085] Specifically, the feature alignment relationship set is traversed, and the representative feature descriptions within each feature group and all document identifiers belonging to that group are read. Using the representative feature descriptions generated in step S404 as the benchmark column for comparison, and each document in the initial document set as an independent row, a two-dimensional matrix is ​​initialized. For each cell in the matrix, i.e., the intersection of a specific document and a specific feature description, if the document belongs to the current feature group, the cell is filled with the specific parameter value or original description fragment of that document on this feature; if it does not belong, it is marked as not involved or left blank. The final generated feature comparison matrix clearly displays the specific values ​​or descriptions of each document on different core feature dimensions in tabular form, making the feature distribution and differences across documents readily apparent.

[0086] Then, the set of contradictory feature pairs is traversed. For each contradictory feature pair, the identifiers of the two documents, the feature groups involved, and the contradiction level label are parsed. Subsequently, in the feature comparison matrix constructed in step S601, the rows containing the two documents and the columns containing the representative feature descriptions corresponding to the contradictory feature groups are located, thus identifying two target matrix units. According to the predefined visual mapping rules for contradiction level labels, these two target units are highlighted. For example, a first-level contradiction can be mapped to a red background, a second-level contradiction to an orange background, and a third-level contradiction to a yellow background. This directly and prominently presents the logically detected contradictions in the comparison matrix, guiding researchers to quickly focus on key data points with conflicts or inconsistencies.

[0087] Next, each document in the initial document set is abstracted as an independent node. The feature alignment relationship set is traversed. For any two documents with a feature matching relationship, a comprehensive feature similarity weight is calculated based on their semantic similarity across all aligned feature groups, such as by taking the average or weighted sum of the similarities across feature groups. This weight is then used as the attribute value of the edge connecting the two document nodes. This constructs an undirected weighted graph, i.e., the initial research design graph. In this graph, the size of the nodes can be set according to the weighted score or citation frequency of the documents, while the thickness or color depth of the edges is proportional to the feature similarity weight, thus intuitively revealing the overall correlation strength and community structure of the document cluster in terms of research methods and design.

[0088] The contradictory relationships identified in step S602 are then overlaid onto the initial research design map. For each pair of contradictory documents recorded in the contradictory feature pair set, the edge connecting these two document nodes is found in the map. Based on the degree of contradiction of the contradictory pair, additional conflict markers are added to this edge. For example, specific labels can be added to the edge, or the line type of the edge can be changed. This marking, along with the visual encoding of the original edge representing the strength of the association, allows the revised research design map to not only display the feature similarity associations between documents but also highlight the specific logical conflicts within these associations, presenting a more comprehensive network of inter-document relationships.

[0089] Finally, in a unified interactive web interface, the feature comparison matrix generated in step S601 and the revised research design map generated in step S604 are displayed side-by-side or in tabs. This interface is equipped with rich interactive functions: it supports users to sort documents or features by clicking the matrix header; it supports displaying detailed feature values, contradiction explanations, or document metadata when the mouse hovers over a matrix cell or map node / edge; it supports users to filter specific document subsets through the map view, with the matrix view updating synchronously; and it supports zooming, panning, and node dragging to adjust the map layout. Ultimately, this interactive visualization interface, as an integrated analysis workbench, provides users with a multi-faceted and explorable cross-validation view, from macro-level correlations to micro-level details, from consistency comparisons to contradiction warnings, significantly enhancing user participation, exploration depth, and decision support capabilities in the cross-validation process.

[0090] Preferably, each matrix unit and graph node is configured with a hyperlink to the original literature data, which allows for one-click tracing from macroscopic comparison results to microscopic original text evidence, significantly improving the tracing efficiency, interaction depth, and user operation convenience of the cross-validation process.

[0091] Step S700: Generate and output a cross-validation report based on the comparison and contradiction information integrated by the visualization interface.

[0092] Specifically, based on the structured data contained in the feature comparison matrix and the revised research design map integrated in the interactive visualization interface, a structured cross-validation report is automatically generated. First, all content is extracted from the feature comparison matrix, with literature as rows and representative feature descriptions within a group as columns, forming a literature-feature comparison table. Simultaneously, feature similarity weight edges between nodes and contradictory edge relationships marked with conflict indicators are extracted from the revised research design map. Next, a preset report template is invoked to integrate and organize the extracted information with the contradictory feature pair set and its contradiction level labels generated in step S500. The report content automatically includes: an overview of the research question and included literature, a summary of feature consistency analysis based on the feature alignment relationship set, a detailed list of contradictory points based on the contradictory feature pair set, and an overall validation conclusion and credibility assessment after comprehensive comparison and contradictory information. Finally, the completed structured report content is rendered into a standard document format and provided for download or direct output to the interface preview. By automating the aggregation and interpretation of core comparisons and contradictions found in the visualization interface, the interactive analysis results are transformed into systematic and conclusive text reports, significantly improving the efficiency and standardization from analysis to conclusion output. This provides users with authoritative verification documents that can be directly used for evaluation, decision-making, or archiving.

[0093] Figure 5 This is a schematic diagram of the connection of a cross-validation system for a visual medical research protocol provided in this application. Figure 5 As shown, a cross-validation system for a visualized medical research scheme includes: a parsing module, a retrieval module, an extraction module, an alignment module, a validation module, a visualization module, and an output module.

[0094] The system comprises the following modules: A parsing module acquires the user-input natural language research question and parses it using a first pre-trained large language model to obtain a structured set of research elements. A retrieval module constructs a retrieval strategy based on the research element set and uses this strategy to retrieve an initial set of documents from a multi-source medical literature database. A extraction module uses a second pre-trained large language model to perform multi-task feature extraction on the full text of the documents in the initial set, obtaining structured feature data for each document. An alignment module converts the structured feature data of each document into semantic vectors and performs cross-document feature matching and alignment based on the similarity between semantic vectors to obtain a feature alignment relationship set. A validation module inputs the feature alignment relationship set into a pre-defined contradiction detection rule base for logical consistency validation to identify and output a set of contradictory feature pairs. A visualization module generates a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the set of contradictory feature pairs. An output module generates and outputs a cross-validation report based on the comparison and contradiction information integrated in the visualization interface.

[0095] The other functions performed by the above-mentioned parsing module, retrieval module, extraction module, alignment module, verification module, visualization module, and output module, as well as the technical details of each function, are the same as or similar to the corresponding features in the cross-validation method of a visualization medical research scheme described above, so they will not be repeated here.

[0096] This application also provides a computer storage medium storing a computer program that, when run on a computer, enables the computer to execute the steps in the cross-validation method for a visual medical research scheme described above.

[0097] It should be understood that although the steps in the flowcharts in the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be performed in other orders.

[0098] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for cross-validation of a visual medical research protocol, characterized in that, The method includes: The system obtains a natural language research question input by the user, and parses the natural language research question using a first pre-trained large language model to obtain a structured set of research elements. A retrieval strategy is constructed based on the aforementioned set of research elements, and the retrieval strategy is used to search in a multi-source medical literature database to obtain an initial set of literature. The second pre-trained large language model is used to perform multi-task feature extraction on the full text of the documents in the initial document set to obtain the structured feature data corresponding to each document; The structured feature data of each document is converted into semantic vectors, and cross-document feature matching and alignment are performed based on the similarity between the semantic vectors to obtain a feature alignment relationship set; The feature alignment relationship set is input into a preset contradiction detection rule base for logical consistency verification to identify and output a set of contradictory feature pairs; A visualization interface containing a feature comparison matrix and a research design map is generated based on the feature alignment relationship set and the contradictory feature pair set; A cross-validation report is generated and output based on the comparison and contradiction information integrated by the visualization interface.

2. The method of claim 1, wherein, The structured set of research elements obtained by parsing the natural language research question through the first pre-trained large language model includes: The core medical entities are obtained by using the first pre-trained large language model to identify medical entities in the natural language research question. A structured PICO element set is generated based on the core medical entities and preset templates; The structured PICO element set is normalized by calling the medical knowledge graph to obtain the normalized PICO element set. Based on the standardized PICO element set, corresponding medical subject terms are generated to obtain a structured set of research elements.

3. The method of claim 1, wherein, The initial literature set obtained by using the retrieval strategy in a multi-source medical literature database includes: The aforementioned search strategies were submitted to a multi-source medical literature database for parallel retrieval via the official API interface to obtain a candidate literature set. A pre-defined abstract model is used to score the relevance of the abstracts of each document in the candidate document set, and documents with scores higher than a pre-defined score threshold are selected as the initial document set. The citation frequency, journal impact factor, and research type weight of the initial screening literature set are obtained. A comprehensive quality assessment is conducted based on the citation frequency, journal impact factor, and research type weight of the initial screening literature set to obtain a weighted score. The initial document set is sorted according to the weighted score to select a preset number of documents to form the initial document set.

4. The method of claim 3, wherein, The step of using a second pre-trained large language model to perform multi-task feature extraction on the full text of the documents in the initial document set to obtain the structured feature data corresponding to each document includes: The full text of each document is divided into paragraphs to obtain the paragraph sequence of each document; For each paragraph in the paragraph sequence of each document, the second pre-trained large language model is used to perform named entity recognition, entity relation extraction, and research design classification tasks to extract the entity set, entity relation graph, and research type label of the paragraph. The entity sets, entity relationship graphs, and research type labels extracted from each paragraph of the same document are fused and conflict-resolved to construct a feature map representing the full-text structure of the document. The structured feature data of the same document is extracted and structured from the feature map, including descriptions of research subjects, details of intervention measures, control design, definition of outcome indicators, and statistical results.

5. The method of claim 1, wherein, The feature alignment relationship set obtained by cross-document feature matching and alignment based on the similarity between the semantic vectors includes: Obtain the descriptive text corresponding to the same type of feature in the structured feature data of each document, and convert each descriptive text into a semantic vector through a pre-trained sentence encoding model; The similarity matrix is ​​obtained by calculating the similarity between semantic vectors of the same type of feature among various documents; A multi-pair matching algorithm is used to process the similarity matrix to perform optimal matching of features and obtain a preliminary set of aligned feature pairs. Cluster analysis is performed on the preliminary aligned feature pair set to group feature pairs with similarity higher than a preset similarity threshold into the same feature group and generate representative feature descriptions within the group; Based on the feature groups and representative feature descriptions within the groups, a feature alignment relationship set containing feature correspondences between documents is constructed.

6. The method of claim 5, wherein, The step of inputting the feature alignment relationship set into a preset contradiction detection rule base for logical consistency verification to identify and output a set of contradictory feature pairs includes: Obtain the feature groups that have been matched in the feature alignment relationship set, and extract the key parameters corresponding to each document in each feature group; The key parameters are input into a preset contradiction detection rule base, and the statistical conclusion contradiction rule, the dosage range contradiction rule, and the population characteristic contradiction rule are applied sequentially to perform logical verification to obtain the verification result. Based on the verification results, document feature pairs that violate any rule are identified and assigned corresponding contradiction level labels. Aggregate all document feature pairs with contradiction level labels to generate a set of contradictory feature pairs.

7. The method of claim 6, wherein, The step of generating a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the contradictory feature pair set includes: Extract representative feature descriptions within each feature group and corresponding literature sources from the feature alignment relationship set to construct a feature comparison matrix with literature as rows and representative feature descriptions within each group as columns. Based on the contradictory features, each feature pair and its contradiction level label in the set are highlighted in the corresponding matrix unit of the feature comparison matrix; Based on the feature correspondence between documents in the feature alignment relationship set, an initial research design graph is constructed with documents as nodes and feature similarity as weights. The contradictory feature set is mapped to the initial research design graph to identify the conflict between the edges of contradictory document nodes, thereby obtaining a revised research design graph; The feature comparison matrix and the revised research design map are integrated into an interactive visualization interface.

8. The method of claim 7, wherein, The method also includes configuring hyperlinks to the original literature data for each matrix unit and graph node.

9. A visualizing medical study protocol cross-validation system, characterized in that, The system includes: a parsing module, a retrieval module, an extraction module, an alignment module, a verification module, a visualization module, and an output module; wherein, The parsing module is used to obtain the natural language research question input by the user, and to parse the natural language research question through the first pre-trained large language model to obtain a structured set of research elements. The retrieval module is used to construct a retrieval strategy based on the set of research elements, and use the retrieval strategy to search in a multi-source medical literature database to obtain an initial set of literature. The extraction module is used to perform multi-task feature extraction on the full text of the documents in the initial document set using the second pre-trained large language model to obtain the structured feature data corresponding to each document. The alignment module is used to convert the structured feature data of each document into semantic vectors, and to perform cross-document feature matching and alignment based on the similarity between the semantic vectors to obtain a feature alignment relationship set; The verification module is used to input the feature alignment relationship set into a preset contradiction detection rule base for logical consistency verification in order to identify and output a set of contradictory feature pairs; The visualization module is used to generate a visualization interface containing a feature comparison matrix and a research design map based on the feature alignment relationship set and the contradictory feature pair set. The output module is used to generate and output a cross-validation report based on the comparison and contradiction information integrated by the visualization interface.

10. A computer readable storage medium having stored thereon a computer program, capable of running on a processor, characterized in that, When the computer program is executed by the processor, it implements a cross-validation method for a visual medical research scheme as described in any one of claims 1 to 8.

Images