A medical corpus dataset construction method based on multi-stage large model evaluation and paragraph-level labeling

Abstract

The application relates to a medical corpus dataset construction method based on multi-stage large model evaluation and paragraph-level labeling, which comprises the following steps: collecting multi-source medical pre-data, performing standardization preprocessing, and obtaining a medical candidate text set; performing privacy and ethics risk control detection and desensitization gating on the medical candidate text set to obtain a compliance medical text set with sensitive information removed; outputting the scores and veto marks of each compliance medical text based on a first large language model and a second large language model; determining whether to delete the current compliance medical text according to the veto mark, and calculating the final score of the non-deleted compliance medical text; segmenting the text by paragraphs and performing paragraph scoring; establishing a hierarchical data warehouse to store the text with a comprehensive score higher than a preset text storage threshold or the paragraph with a paragraph score higher than a preset paragraph storage threshold, and completing medical corpus dataset construction. Compared with the prior art, the application can construct a high-quality corpus dataset for medical and clinical large model training.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical information processing technology, and in particular to a method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation. Background Technology

[0002] Large-scale medical models have significant application value in tasks such as medical question answering, clinical decision support, medical literature understanding, and medical record summary generation. Their effectiveness is highly dependent on the size, factual reliability, and security compliance of the training corpus. Existing medical corpus systems, primarily composed of academic papers and some guidelines / textbooks, generally suffer from the following shortcomings: (1) Scale and quality are hard to achieve at the same time: A large amount of corpus from the Internet or unstructured medical texts has problems such as noise, repetition, logical omissions and factual errors; (2) The privacy and ethical risks of medical corpora are prominent: case records, test reports, image descriptions, etc. often contain identifiable personal information (PHI / PII), and direct use for training poses compliance risks; (3) Lack of reproducible quality assessment standards: Assessments based on a single large model or a single indicator are easily affected by cue drift and model bias, and are difficult to reproduce stably; (4) The granularity of the corpus is too coarse: the traditional approach mainly uses "document-level" tags, ignoring the quality differences between paragraphs within a document, resulting in high-value paragraphs being dragged down by noisy paragraphs and being discarded as a whole, or low-value paragraphs being mistakenly included. (5) High cost of manual labeling: High-quality paragraph-level labeling (such as clinical points, evidence levels, treatment recommendations, complications / contraindications, etc.) relies on professionals and is difficult to scale up.

[0003] Chinese patent application CN120472484A discloses a processing method for extracting structured information and protecting privacy in medical reports. By integrating advanced image preprocessing, dynamic layout analysis, OCR technology, and a dual-channel privacy detection model, it can efficiently and accurately extract text information from tables and establish coordinate mapping relationships between image description paragraphs and corresponding images. The generated JSON structure, containing text, tables, and image-text mapping relationships, ensures data integrity and accuracy. However, this invention only addresses the privacy and ethical risks of medical corpora. In practical use, it can only extract table text information, remove private information, and store it, lacking quality assessment and having overly coarse granular corpora.

[0004] Therefore, there is a need for a scalable, highly reliable, and auditable corpus construction method for medical scenarios that can achieve high-quality screening and fine-grained paragraph-level annotation at a low cost while ensuring privacy compliance, and form a structured dataset that can be used for training. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation, the method comprising: Collect medical texts from multiple sources and perform standardized preprocessing to obtain a set of medical candidate texts; The medical candidate text set is subjected to privacy and ethical risk control detection and desensitization gating to obtain a compliant medical text set with sensitive information removed; Based on the first large language model, standardized evaluation rules including scoring and annotation specifications are generated for the set of compliant medical texts. Based on the scoring specifications, the second large language model is used to output the score and rejection mark of each compliant medical text. Based on the output results of the second large language model, the annotation specifications are improved, and the review score and review rejection mark of each compliant medical text are output. The two output results of the second large language model are compared. If the score difference exceeds the preset score threshold or the rejection mark is inconsistent, the review score and review rejection mark are output. The current compliant medical text and its two output results are added to the disputed sample pool for subsequent retraining of the large language model. Otherwise, the average score and rejection mark of the two output results are output. The decision to delete the current compliant medical text is based on the veto flag. The confidence score is calculated and the score is fused based on the score of the compliant medical text that has not been deleted, and the final score is obtained. The non-deleted compliant medical text is segmented into paragraphs, and the second language model is used to output the score and rejection flag for each paragraph. The rejection flag determines whether to delete the current paragraph. The non-deleted paragraphs are structurally labeled according to the annotation specifications, and paragraph scores are calculated. Establish a hierarchical data warehouse to store non-deleted compliant medical texts with a comprehensive score higher than a preset text storage threshold, or paragraphs with a paragraph score higher than a preset paragraph storage threshold, thus completing the construction of the medical corpus dataset.

[0007] Furthermore, the multi-source medical data includes text data from clinical guidelines, textbooks, academic papers, case summaries, laboratory reports, drug instructions, clinical pathways, and knowledge Q&A. The standardized preprocessing includes text cleaning, language detection and standardization, rule-based structure parsing and hierarchical decomposition, and initial screening for medical relevance.

[0008] Furthermore, the rule-based structure parsing process includes: A structure parser is used to hierarchically split the multi-source medical texts after text cleaning, language detection, and standardization, outputting a node tree and a collection of paragraphs. The structure parser is executed based on preset rules, which include title keyword rules, Chinese numbering hierarchy rules, Arabic numeral hierarchy rules, list item rules, blank line and paragraph boundary rules, sentence segmentation rules, figure and caption rules, reference rules, dose continuity rules, structural anomaly handling rules, and length constraint rules. The process of the initial screening for medical relevance includes: Medical feature vectors are extracted from the node tree and paragraph set. The medical feature vectors include TF-IDF features, medical terminology dictionary hit features, medical entity recognition result features, structural features, and statistical features. The medical relevance probability of the medical feature vectors is obtained based on the pre-trained logistic regression model. Medical feature vectors below the dynamic threshold are removed, and medical feature vectors above the dynamic threshold are output as a set of medical candidate texts.

[0009] Furthermore, when segmenting the undeleted compliant medical text by paragraph, the segmentation is based on the node tree output by the structure parser.

[0010] Furthermore, the logistic regression model is trained based on a training dataset containing positive and negative samples and a log loss function with L2 regularization. The training dataset is divided into a training set, a validation set, and a test set. The dynamic threshold is selected from the validation set based on the principle of recall priority or optimal F1 score.

[0011] Furthermore, the privacy and ethical risk control detection and desensitization gating process includes: Based on the rule engine and entity recognition model, PHI / PII detection is performed on the medical candidate text set to identify patient sensitive information and entities in each medical candidate text fragment, and the corresponding entity category and confidence level are output to obtain multiple risk type sets; The risk score in the risk type set is calculated using a weighted composite method; Perform standardized desensitization and replacement on the replaceable information in the risk type set, replacing the replaceable information with standardized markers, and perform PHI / PII detection again. If there are still high-confidence patient sensitive information and entities, then the corresponding patient sensitive information and entities are judged as uncontrollable risks. The risk type set is subjected to hard-gating screening. If the risk score of the risk type set is greater than the preset risk threshold or there is an uncontrollable risk in the risk type set, the corresponding medical candidate text is removed to obtain a set of compliant medical texts with sensitive information removed.

[0012] Furthermore, both the first and second large language models use the Transformer decoder architecture as the base model and are pre-trained on medical and clinical texts. The first large language model is fine-tuned by generating evaluations, defining output fields, and providing veto rules and boundary example data to train its output to stabilize standardized evaluation rules. The second large language model is fine-tuned by using quality assessment and security audit instruction data to train its output to structured scores and veto markers.

[0013] Furthermore, after the compliant medical text or paragraphs are stored in the hierarchical data warehouse, the second major language model is also trained by distillation model. The student model is trained based on minimizing the multi-task distillation loss, and the evaluation and annotation capabilities of the second major language model are transferred to the student model. The minimization of multi-task distillation loss includes rating regression loss, multi-label loss and risk loss.

[0014] Furthermore, the process of transferring the evaluation and annotation capabilities of the second largest language model to the student model includes: A distillation training set was constructed using compliant medical texts and paragraphs already stored in a hierarchical data warehouse; A Transformer encoder with small to medium parameters is used as the backbone of the student model, and the output multi-task head includes a rating regression head, a risk prediction head, and a label prediction head. Using the second largest language model as the teacher model, the student model is trained based on minimizing the multi-task distillation loss and the distillation training set; The trained student model is used to replace the second language model. Based on the scoring criteria, the compliant medical texts are evaluated and labeled for the first time. The score, rejection mark and student confidence score of each compliant medical text are output. If the student confidence score is lower than the student confidence score threshold or the score is lower than the preset threshold, the second language model is used to output the review score and review rejection mark of each compliant medical text. The review results and text are stored in the distillation training set to iteratively update the student model.

[0015] Furthermore, when medical text contains factual errors, fails to meet quality standards, has structural abnormalities that cannot be corrected, contains dangerous advice and safety risks, contains non-medical related content, contains repetitive or meaningless content, or violates scoring criteria, the second language model outputs a rejection flag.

[0016] Compared with the prior art, the beneficial effects of the present invention include: 1. This invention discloses a dataset construction method capable of building high-quality corpus datasets for training large-scale medical and clinical models. While ensuring privacy compliance, it achieves high-quality screening and fine-grained paragraph-level annotation, forming a structured dataset suitable for training. This dataset can be used for pre-training, continuous pre-training, and fine-tuning of downstream tasks for large-scale medical models. In this invention, a first semantic model generates a structured unified evaluation, followed by a second large language model for two-stage evaluation and dual-dimensional confidence fusion. This effectively avoids evaluation fluctuations caused by cue drift and model bias, unifying the quality judgment standards for corpora across different batches and scenarios. It overcomes the limitations of traditional single-model / single-index evaluation, achieving stable output and reproducible verification of quality scores. Furthermore, this invention segments compliant medical text by paragraph and scores each paragraph. During storage, even if the overall text score does not meet the storage standard, paragraphs that meet the storage standard can still be stored in the dataset, maximizing the potential of the corpus and reducing the material collection and construction costs of the corpus dataset.

[0017] 2. This invention uses a rule engine and entity recognition model to jointly detect PHI / PII. After standardized desensitization and secondary verification, high-risk text can be directly removed. Any text that fails privacy risk control cannot enter the subsequent evaluation, making the model training data both high-quality and safe and compliant. Privacy desensitization and ethical risk control are embedded in the entire process of corpus construction, rather than being remedied afterward. This ensures the compliance of training data from the source and reduces compliance risks when large medical models are deployed.

[0018] 3. This invention also replicates the scoring, risk identification, and annotation capabilities of the second large language model by training a small-parameter distillation model (student model). The entire corpus is first quickly screened by the distillation model, and the second large language model is only triggered to review low-confidence, high-risk, and boundary samples. This avoids the high cost caused by relying on the large model for the entire corpus, and achieves a balance between high quality and low cost for large-scale processing. Under the premise of ensuring evaluation quality, the evaluation capabilities of the large model are transferred to the small model through model distillation technology, which significantly reduces the computational and time costs of processing the entire corpus. This enables high-quality screening and paragraph-level fine annotation at a lower cost while ensuring privacy compliance. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] Example 1 A method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation is as follows: Figure 1 As shown, steps S1-S7 are included, and the specific details of each step are as follows: Step S1: Collection and standardization preprocessing of multi-source medical corpus.

[0022] Step S1 specifically involves: collecting multi-source medical texts and performing standardized preprocessing to obtain a set of medical candidate texts.

[0023] Multi-source medical data includes textual data such as clinical guidelines, textbooks, academic papers, case summaries, laboratory reports, drug instructions, clinical pathways, and Q&A documents. Specifically, text is collected from publicly available medical resources, authorized data sources, and compliant data warehouses within institutions. Structured fields are simultaneously saved during collection, including but not limited to: title, source, publication date, subject, disease classification, document type, language, URL, or library index.

[0024] Standardized preprocessing includes text cleaning, language detection and standardization, rule-based structure parsing and hierarchical decomposition, and initial screening for medical relevance.

[0025] Text cleaning: Remove HTML tags, garbled characters, duplicate whitespace, and meaningless headers and footers; standardize Chinese and English punctuation and spaces; standardize unit and dosage representations to a standardized format (e.g., mg, ml, mmHg, mmol / L, etc.); transcribe table content into text using "row and column keys" and mark it as TableBlock type for subsequent weight reduction or removal.

[0026] Language detection and standardization: Perform language detection on each text or document; unify encoding to UTF-8; unify line breaks and paragraph separators; perform standardized mapping for common medical terms (such as disease names, generic drug names, and test indicator names), and retain comparison information before and after mapping for auditing purposes.

[0027] The process of rule-based structure parsing includes: A structure parser is used to perform hierarchical splitting of the multi-source medical texts after text cleaning, language detection, and standardization, outputting a node tree and a collection of paragraphs. The structure parser executes based on preset rules, which include title keyword rules, Chinese numbering hierarchy rules, Arabic numeral hierarchy rules, list item rules, blank line and paragraph boundary rules, sentence segmentation rules, figure and caption rules, reference rules, dose continuity rules, structural anomaly handling rules, and length constraint rules.

[0028] The structure parser is specifically a rule-based structured parsing component that splits the original text into a node tree based on heading levels, lists, and paragraph boundaries: Document → Section → Subsection → Paragraph → Sentence, and outputs a collection of paragraphs { } and the heading path for each paragraph.

[0029] The structure parser executes based on at least the following rules (the order of the rules can be fixed, and in case of conflict, the rule with higher priority takes effect): (1) Title keyword rules: When a line of text matches one of the following title keywords, it is determined to be the starting node of the section (Section / Subsection), including but not limited to: Abstract, Introduction / Background, Purpose, Methods / Materials and Methods, Results, Discussion, Conclusion, Case Data, Diagnosis and Treatment Process, Medication Regimen, Follow-up, References, Acknowledgments.

[0030] (2) Chinese numbering hierarchy rules: Level 1: Titles beginning with "Chapter X" (where X is a Chinese numeral) are classified as Level 1 headings; Level 2: Headings beginning with "X," (where X is a Chinese numeral) are classified as Level 2 headings. Level 3: Headings beginning with "(X)" are classified as Level 3 headings; Level 4: Items starting with "1)" or "(1)" are classified as Level 4 headings or list items.

[0031] (3) Arabic numeral hierarchy rules: Level 1: Lines that begin with "1" or "1." followed by heading text are classified as Level 1 headings; Level 2: Headings beginning with "1.1" are considered Level 2 headings; Level 3: Headings starting with "1.1.1" are classified as Level 3 headings.

[0032] (4) List item rules: rows starting with “”, “•”, “*”, “1.”, “1)”, etc. are considered list items, and consecutive list items are merged into subsequences of the same parent node.

[0033] (5) Blank lines and paragraph boundary rules: consecutive blank lines ≥ 1 are considered paragraph boundaries; soft line breaks within the same paragraph are merged according to the sentence end punctuation and length threshold to avoid breaking semantic units.

[0034] (6) Sentence segmentation rules: Chinese is segmented by “。!?;”, and English is segmented by “.?!;”; a whitelist is set for medical abbreviations and unit combinations (such as “mg / dL”, “mmHg”, “No.”, etc.) to avoid missegmentation.

[0035] (7) Chart and Caption Rules: Match “Table 1 / Figure 1 Rows starting with " / Table1 / Figure1" and their following descriptions are categorized as Table / Figure Blocks, which are weighted down by default and are not included in the high-confidence training set.

[0036] (8) Reference rules: After the “References” node appears, the content thereafter is classified as the References block and is either not included in the database or is significantly downgraded by default.

[0037] (9) Dosage continuity rule: Short lines containing “numerical value + unit” (e.g., “5mg”, “0.9% NaCl”, “120mmHg”) should be merged with adjacent lines to ensure the continuity of dosage, indications, contraindications and other information.

[0038] (10) Structural anomaly handling rules: When the numbering level is not continuous or the title sequence is abnormal, the node is attached to the nearest legal parent node and a structural anomaly flag struct_error=1 is written for subsequent quality scoring deduction.

[0039] (11) Length constraint rules: If the paragraph length is less than L_min (e.g., 30 words), it will be merged with the adjacent paragraph; if the paragraph length is greater than L_max (e.g., 800 words), it will be divided into multiple paragraph segments by sentence so that each segment can be evaluated and labeled.

[0040] The initial screening process for medical relevance includes: Self-node tree and paragraph collection Extract medical feature vectors, which include TF-IDF features, medical terminology dictionary hit features, medical entity recognition result features, structural features, and statistical features; The medical relevance probability of medical feature vectors is obtained based on a pre-trained logistic regression model. Medical feature vectors below the dynamic threshold are removed, and medical feature vectors above the dynamic threshold are output as a set of medical candidate texts.

[0041] The logistic regression model is trained on a training dataset containing positive and negative samples and a log loss function with L2 regularization. The training dataset is divided into a training set, a validation set, and a test set. The dynamic threshold is selected from the validation set based on the principle of recall priority or optimal F1 score.

[0042] Feature vector construction: Extract feature vector x from text segment t, defined as: in: : Calculate TF-IDF features on a pre-constructed vocabulary; Hit characteristics of the medical terminology dictionary, including the hit count and percentage of disease names, drug names, examination indicator names, anatomical locations, etc. Characteristics of medical entity identification results, including the number and density of diseases / symptoms / drugs / dosages / examination items; Structural characteristics, including the type of Heading Path, whether it is a chart block, whether it is a References block, whether it is a list item, etc. Statistical characteristics, including length, number ratio, unit ratio, abbreviation ratio, punctuation density, etc.

[0043] Probability Calculation: The logistic regression model outputs the probability of medical relevance as follows: Where y=1 represents medical relevance, and w and b are model parameters.

[0044] Training data construction and training process: Positive samples: fragments of medical texts such as guidelines, textbooks, the main body of papers, drug instructions, and clinical pathways; Negative samples: noise from non-medical web pages, copyright notices, navigation menus, advertisements, general news, and technical text snippets unrelated to medicine; Divide the dataset into training, validation, and test sets. Train the model using log loss with L2 regularization: Choose a threshold τ on the validation set (e.g., recall priority or F1 optimal), and determine the text as a medical candidate text when p(y=1|x)≥τ.

[0045] Step S2: Privacy and ethical risk control detection and desensitization gating.

[0046] Step S2 specifically involves: performing privacy and ethical risk control checks and desensitization gating on the medical candidate text set to obtain a compliant medical text set with sensitive information removed.

[0047] The process of privacy and ethics risk control detection and desensitization gating includes: Based on the rule engine and entity recognition model, PHI / PII detection is performed on the medical candidate text set to identify patient sensitive information and entities in each medical candidate text fragment, and the corresponding entity category and confidence level are output to obtain multiple risk type sets; The risk score in the risk type set is calculated using a weighted composite method; Perform standardized desensitization and replacement on the replaceable information in the risk type set, replace the replaceable information with standardized labels, and perform PHI / PII detection again. If there are still high-confidence patient sensitive information and entities, then the corresponding patient sensitive information and entities are judged as uncontrollable risks. The risk type set is subjected to strict gating. If the risk score of the risk type set is greater than the preset risk threshold or there is an uncontrollable risk in the risk type set, the corresponding medical candidate text is removed to obtain a set of compliant medical texts with sensitive information removed.

[0048] PHI / PII detection can identify personal health information / personal identification information, such as name, ID number, mobile phone number, address, doctor's number, bed number, etc. It aims to remove sensitive personal information from text documents.

[0049] The rules engine is specifically used to match ID card numbers, mobile phone numbers, landline numbers, email addresses, bank card numbers, complete addresses, postal codes, doctor's numbers / hospitalization numbers / examination numbers, bed numbers, etc. The entity recognition model is specifically used to identify entities such as names, medical institutions, geographical locations, time location information, and unique identifiers, and outputs the entity category and confidence level.

[0050] Risk scoring: Calculate a risk score for the set R of detected risk types. A weighted synthesis method is used: in, The confidence level (or normalized occurrence count) of the j-th risk factor. For risk type weights, satisfying Furthermore, strong identifiers (ID card number / medical appointment number / mobile phone number) have a greater weight than weak identifiers (institution name / city name).

[0051] Desensitization: Perform standardized desensitization and replacement on replaceable information while preserving type tags, for example: Name → [NAME]; Phone → [PHONE]; Address → [ADDR]; ID Card → [IDNO]; Medical Appointment Number → [VISIT_ID]; Examination Number → [EXAM_ID].

[0052] After desensitization, run the PHI / PII test again for verification. If strong identifiers still exist, it is determined to be an uncontrollable risk.

[0053] Compliance hard gating: setting thresholds ,when If an uncontrollable strong identifier is detected, the text fragment will be directly removed; otherwise, it will be allowed to proceed to the subsequent quality assessment. The system outputs an audit log, including: text ID, risk type, and risk score. Processing action (de-identification / removal / release), timestamp, rule version, model version, and de-identification digest hash.

[0054] Step S3: Construct a multi-stage evaluation task (TaskSpec) and perform multi-model evaluation.

[0055] Step S3 specifically involves: generating a standardized evaluation rule TaskSpec containing the scoring specification EvalSpec and the annotation specification LabelSpec for the compliant medical text set based on the first large language model LLM-A; outputting the score and rejection flag for each compliant medical text based on the scoring specification using the second large language model LLM-B; improving the annotation specification based on the output results of the second large language model; outputting the review score and review rejection flag for each compliant medical text; comparing the two output results of the second large language model; if the score difference exceeds a preset score threshold or the rejection flag is inconsistent, then outputting the review score and review rejection flag; and adding the current compliant medical text and its two output results to the disputed sample pool for subsequent retraining of the large language model; otherwise, outputting the average score and rejection flag of the two output results; and determining whether to delete the current compliant medical text based on the rejection flag.

[0056] EvalSpec: Set of scoring dimensions D, rubric for each dimension's score definition, deduction points and veto conditions veto_rules, output field schema; LabelSpec: Paragraph label set L (information type, evidence level, available task type, risk marker), entity type set E (disease / symptom / drug / dosage / examination item / indicator value, etc.), output field schema and example.

[0057] Both the first and second largest language models use the Transformer decoder architecture as their base model and are pre-trained on medical and clinical texts. The first largest language model is fine-tuned by generating evaluations, defining output fields, and providing veto rules and boundary example data to train its output to stabilize standardized evaluation rules. The second largest language model is fine-tuned by using quality assessment and security audit instruction data to train its output to structured scores and veto markers.

[0058] The second major language model outputs: scores for each dimension. (e.g., 0~5 or 0~10); Overall score Error and Risk Labels ; veto mark The triggering reason; the evaluation basis (short text).

[0059] When medical text contains factual errors, fails to meet quality standards, has structural abnormalities that cannot be corrected, contains dangerous advice or safety risks, contains non-medical content, contains repetitive or meaningless content, or violates scoring criteria, the second language model outputs a rejection flag.

[0060] In another embodiment, the second-stage review is triggered only for the following texts: texts with high first-stage scores but containing risk labels; borderline texts with first-stage scores near the entry threshold; and texts with a structural anomaly flag struct_error=1.

[0061] Step S4: Calculate the confidence score based on log-likelihood normalization and perform weighted fusion scoring.

[0062] Step S4 specifically involves: calculating confidence and fusing scores based on the scores of compliant medical texts that have not been deleted, and obtaining the final score.

[0063] Specifically, the token-level log-likelihood of the output of the second largest language model is calculated and normalized to obtain... Its expression is: Where x represents the input text and EvalSpec, To output the token.

[0064] Consistency confidence calculation: Perform K independent evaluations on the same text under a fixed EvalSpec (changing the sampling random seed or using slight equivalent cue perturbations), and obtain K comprehensive scores. } Calculate the variance and map it to consistency confidence. .

[0065] Fusion confidence: Weighted Fusion Score: The two-stage scores are weighted and fused to obtain the final score. Overall: in, For a rough overall score, For review of the comprehensive score; , For the corresponding confidence level; > This reflects the higher weight given to the review process. If If the value is 1, it is determined that the data will not be entered into the database.

[0066] Step S5: Segment-level segmentation and fine-grained semantic annotation.

[0067] Step S5 specifically involves: segmenting the non-deleted compliant medical text into paragraphs, using the second language model to output the scores and rejection flags for each paragraph, determining whether to delete the current paragraph based on the rejection flags, performing structured annotation on the non-deleted paragraphs based on annotation specifications, and scoring the paragraphs.

[0068] When segmenting compliant medical text that has not been deleted by paragraph, the segmentation is based on the node tree output by the structure parser. Specifically, the Paragraph nodes output by the structure parser are used as paragraph segments. It also records the HeadingPath, paragraph number, document ID, and context window (heading path and adjacent sentences) for each paragraph.

[0069] Fine-grained annotation task definition: The LabelSpec in TaskSpec generated by the first large language model is used as the sole annotation specification.

[0070] Specifically, LabelSpec stipulates: Information type tags: definition / etiology / symptoms / diagnosis / differential diagnosis / treatment / dosage / contraindications / prognosis / complications / interpretation of examination indicators, etc.; Level of evidence: Strong recommendation from guidelines / Weak recommendation from guidelines / Expert consensus / Case experience / Speculation; Available task types: question answering training, summary training, knowledge triple extraction, retrieval enhancement documents, etc. Risk markers: potentially misleading, outdated information, lack of source, conflict with guidelines, etc. Entity types: disease, symptoms, drug, dosage, frequency of administration, examination items, range of indicator values, indications, contraindications, etc. It also explicitly outputs JSON fields and their value ranges.

[0071] Paragraph annotation and paragraph scoring have different output objectives: scoring outputs numerical quality evaluation, while annotation outputs semantic structured information; they can share risk marker fields, but they are not variations of each other.

[0072] Step S6: Screening and storing highly reliable samples based on hard constraints and comprehensive scores.

[0073] Step S6 specifically involves: establishing a hierarchical data warehouse to store non-deleted compliant medical texts with a comprehensive score higher than a preset text storage threshold, or paragraphs with a paragraph score higher than a preset paragraph storage threshold, thereby completing the construction of the medical corpus dataset.

[0074] Hard constraint filtering: Text fragments must meet the following requirements: passing privacy and ethical risk control, i.e. Furthermore, the proposal should be rejected if there are no strong identifiers remaining, no dangerous suggestions, and no obvious factual conflicts.

[0075] The overall score is as follows: in, These are the weighting coefficients. For the final score, This represents the average quality of the top-M high-scoring paragraphs in the current document, i.e., the average paragraph score. To achieve high-score paragraph coverage, As a risk penalty coefficient, The privacy risk score is calculated in step S2.

[0076] in, The number of qualified paragraphs whose paragraph scores exceed a preset paragraph storage threshold. This represents the total number of paragraphs in the text.

[0077] Hierarchical data entry: Establish hierarchical data warehouses by domain (disease system / department), language, level of evidence, and task type; Output: Document-level records: doc_id, source, publish_time, overall_score, S_final, risk_log, model version, and rule version; Paragraph-level records: para_id, doc_id, text, HeadingPath, Q_k, labels, entities, evidence_level, task_type, confidence; Audit logs include: evaluation prompt version, threshold version, de-identification rule version, training data snapshot hash, etc.

[0078] Example 2 This embodiment, based on Embodiment 1 above, discloses a method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation, which distills the second large language model into a student model. The difference between this embodiment and Embodiment 1 above is that: a new process of training a lightweight quality model by quality distillation is added, and the dataset is actively learned, feedback is provided, and the dataset is continuously updated iteratively.

[0079] Specifically, after compliant medical texts or paragraphs are stored in the hierarchical data warehouse, the second language model is also trained by distillation model. The student model is trained based on minimizing the multi-task distillation loss, and the evaluation and annotation capabilities of the second language model are transferred to the student model. Among them, minimizing the multi-task distillation loss includes rating regression loss, multi-label loss and risk loss.

[0080] The student model specifically includes: Rating Regression Head: Output Predicted Rating Combined with prediction ; Risk prediction header: Outputs risk probability ; Label prediction head: Outputs multi-label probabilities .

[0081] Rating regression loss: in, The number of rating dimensions, The d-th dimension score output by the teacher model. For the student model's predicted d-th dimension score, The overall score output by the teacher model. The overall score predicted by the student model. The Huber loss function (robust to outliers, balancing MSE and MAE).

[0082] Multi-label loss: in, The total number of label categories. The m-th class label (0 or 1) output by the teacher model. The predicted label value for the m-th class of the student model.

[0083] Risk of loss: in, This represents the total number of risk types. The true value (probability or binary label) of the k-th type of risk output by the teacher model. The predicted value of the k-th type of risk for the student model.

[0084] Total distillation loss: in, For teacher confidence (suppressing negative transfer of uncertain teacher signals). , and Weighting coefficients for ratings, tags, and risk tasks.

[0085] The process of transferring the evaluation and annotation capabilities of the second-largest language model to the student model includes: A distillation training set was constructed using compliant medical texts and paragraphs already stored in a hierarchical data warehouse; A Transformer encoder with small to medium parameters is used as the backbone of the student model, and the output is a multi-task head, which includes a rating regression head, a risk prediction head, and a label prediction head. The second largest language model is used as the teacher model, and the student model is trained based on minimizing the multi-task distillation loss and the distillation training set. The trained student model is used to replace the second language model. Based on the scoring criteria, the compliant medical texts are evaluated and labeled for the first time. The score, rejection mark and student confidence score of each compliant medical text are output. If the student confidence score is lower than the student confidence score threshold or the score is lower than the preset threshold, the second language model is used to output the review score and review rejection mark of each compliant medical text. The review results and texts are stored in the distillation training set to iteratively update the student model.

[0086] Specifically, after training, temperature calibration is performed on the output probabilities of the student model to make them usable as cascaded thresholds; and the student confidence threshold is determined on the validation set. The boundary interval ε is used for the rollback trigger of subsequent full inference.

[0087] Example 3 Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation.

[0088] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0089] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0090] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0091] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation, characterized in that, The method includes: Collect medical texts from multiple sources and perform standardized preprocessing to obtain a set of medical candidate texts; The medical candidate text set is subjected to privacy and ethical risk control detection and desensitization gating to obtain a compliant medical text set with sensitive information removed; Based on the first large language model, standardized evaluation rules including scoring and annotation specifications are generated for the set of compliant medical texts. Based on the scoring specifications, the second large language model is used to output the score and rejection mark of each compliant medical text. Based on the output results of the second large language model, the annotation specifications are improved, and the review score and review rejection mark of each compliant medical text are output. The two output results of the second large language model are compared. If the score difference exceeds the preset score threshold or the rejection mark is inconsistent, the review score and review rejection mark are output. The current compliant medical text and its two output results are added to the disputed sample pool for subsequent retraining of the large language model. Otherwise, the average score and rejection mark of the two output results are output. The decision to delete the current compliant medical text is based on the veto flag. The confidence score is calculated and the score is fused based on the score of the compliant medical text that has not been deleted, and the final score is obtained. The non-deleted compliant medical text is segmented into paragraphs, and the second language model is used to output the score and rejection flag for each paragraph. The rejection flag determines whether to delete the current paragraph. The non-deleted paragraphs are structurally labeled according to the annotation specifications, and paragraph scores are calculated. Establish a hierarchical data warehouse to store non-deleted compliant medical texts with a comprehensive score higher than a preset text storage threshold, or paragraphs with a paragraph score higher than a preset paragraph storage threshold, thus completing the construction of the medical corpus dataset.

2. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 1, characterized in that, The multi-source medical data includes text data from clinical guidelines, textbooks, academic papers, case summaries, laboratory reports, drug instructions, clinical pathways, and knowledge Q&A. The standardized preprocessing includes text cleaning, language detection and standardization, rule-based structure parsing and hierarchical decomposition, and initial screening for medical relevance.

3. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 2, characterized in that, The rule-based structure parsing process includes: A structure parser is used to hierarchically split the multi-source medical texts after text cleaning, language detection, and standardization, outputting a node tree and a collection of paragraphs. The structure parser is executed based on preset rules, which include title keyword rules, Chinese numbering hierarchy rules, Arabic numeral hierarchy rules, list item rules, blank line and paragraph boundary rules, sentence segmentation rules, figure and caption rules, reference rules, dose continuity rules, structural anomaly handling rules, and length constraint rules. The process of the initial screening for medical relevance includes: Medical feature vectors are extracted from the node tree and paragraph set. The medical feature vectors include TF-IDF features, medical terminology dictionary hit features, medical entity recognition result features, structural features, and statistical features. The medical relevance probability of the medical feature vectors is obtained based on the pre-trained logistic regression model. Medical feature vectors below the dynamic threshold are removed, and medical feature vectors above the dynamic threshold are output as a set of medical candidate texts.

4. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 3, characterized in that, When segmenting the non-deleted compliant medical text by paragraph, the segmentation is based on the node tree output by the structure parser.

5. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 3, characterized in that, The logistic regression model is trained on a training dataset containing positive and negative samples and a log loss function with L2 regularization. The training dataset is divided into a training set, a validation set, and a test set. The dynamic threshold is selected from the validation set based on the principle of recall priority or optimal F1 score.

6. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 1, characterized in that, The privacy and ethics risk control detection and de-identification gating process includes: Based on the rule engine and entity recognition model, PHI / PII detection is performed on the medical candidate text set to identify patient sensitive information and entities in each medical candidate text fragment, and the corresponding entity category and confidence level are output to obtain multiple risk type sets; The risk score in the risk type set is calculated using a weighted composite method; Perform standardized desensitization and replacement on the replaceable information in the risk type set, replacing the replaceable information with standardized markers, and perform PHI / PII detection again. If there are still high-confidence patient sensitive information and entities, then the corresponding patient sensitive information and entities are judged as uncontrollable risks. The risk type set is subjected to hard-gating screening. If the risk score of the risk type set is greater than the preset risk threshold or there is an uncontrollable risk in the risk type set, the corresponding medical candidate text is removed to obtain a set of compliant medical texts with sensitive information removed.

7. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 1, characterized in that, Both the first and second large language models use the Transformer decoder architecture as the base model and are pre-trained on medical and clinical texts. The first large language model is fine-tuned by generating evaluations, defining output fields, and providing veto rules and boundary example data to train its output to stabilize standardized evaluation rules. The second large language model is fine-tuned by using quality evaluation and security audit instruction data to train its output to structured scores and veto marks.

8. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 1, characterized in that, After the compliant medical text or paragraphs are stored in the hierarchical data warehouse, the second major language model is also trained by distillation model. The student model is trained based on minimizing the multi-task distillation loss, and the evaluation and annotation capabilities of the second major language model are transferred to the student model. The minimization of multi-task distillation loss includes rating regression loss, multi-label loss and risk loss.

9. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation as described in claim 8, characterized in that, The process of transferring the evaluation and annotation capabilities of the second largest language model to the student model includes: A distillation training set was constructed using compliant medical texts and paragraphs already stored in a hierarchical data warehouse; A Transformer encoder with small to medium parameters is used as the backbone of the student model, and the output multi-task head includes a rating regression head, a risk prediction head, and a label prediction head. Using the second largest language model as the teacher model, the student model is trained based on minimizing the multi-task distillation loss and the distillation training set; The trained student model is used to replace the second language model. Based on the scoring criteria, the compliant medical texts are evaluated and labeled for the first time. The score, rejection mark and student confidence score of each compliant medical text are output. If the student confidence score is lower than the student confidence score threshold or the score is lower than the preset threshold, the second language model is used to output the review score and review rejection mark of each compliant medical text. The review results and text are stored in the distillation training set to iteratively update the student model.

10. The method for constructing a medical corpus dataset based on multi-stage large model evaluation and paragraph-level annotation according to claim 1, characterized in that, When medical text contains factual errors, fails to meet quality standards, has structural abnormalities that cannot be corrected, contains dangerous advice or safety risks, contains non-medical content, contains repetitive or meaningless content, or violates scoring criteria, the second language model outputs a rejection flag.

Images