Radiotherapy record automatic examination method and system based on multi-modal pre-training model, and storage medium

Abstract

The invention discloses an automatic examination method and system for radiotherapy records based on a multi-modal pre-training model, and a storage medium, and belongs to the field of medical artificial intelligence and radiooncology quality control. The method comprises the following steps: acquiring and standardizing multi-modal data of the whole process of radiotherapy; extracting and fusing image, text and dose features through an improved vision-language model and an enhanced ClinicalBERT (ClinicalBERT); retrieving related clinical evidences from the radiotherapy knowledge base based on a retrieval enhancement generation framework; inputting the fusion features and the retrieval evidence into a locally deployed multi-modal model to generate a structured review report; and executing six functional modules of target coverage evaluation, organ protection analysis, DVH interpretation, quality scoring, guide compliance inspection and anomaly detection based on a report, outputting a grading decision result, and supporting full-link audit tracing. According to the invention, automatic and standardized examination of radiotherapy records is realized, the examination efficiency and consistency are improved, and the labor burden and subjective difference are reduced.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence and radiation oncology quality control technology, and particularly relates to an automatic review method, system and storage medium for radiotherapy records based on a multimodal pre-trained model. Background Technology

[0002] Radiotherapy is an important treatment for cancer, and its process includes multiple stages such as CT simulation localization, target area and organ at risk delineation, treatment plan development, dose assessment, verification imaging, and treatment execution and record management. Radiotherapy records are a key foundation for ensuring treatment quality and traceability, encompassing structured data such as DICOM RT Plan, RT Structure, and RTDose, as well as clinical texts and PDF documents such as radiotherapy localization forms, radiotherapy planning request forms, and planning reports. These records are characterized by complex data types and significant cross-modal features.

[0003] Currently, the review of radiotherapy records in clinical practice mainly relies on manual work by radiation oncologists, physicists, and treatment technicians. The review covers target area naming and delineation consistency, DVH dose parameters, irradiation field parameters, positioning and imaging verification records, and prescription execution. This process is labor-intensive, cumbersome, and prone to oversights and subjective biases. Because the radiotherapy process generates a large amount of structured data (DICOM RT Plan / RT Dose / RT Structure, DVH), text documents (radiotherapy positioning forms, radiotherapy planning request forms, radiotherapy planning reports, etc.), PDF documents, and image information such as CT / MR / verification films, existing systems struggle to uniformly process and comprehensively evaluate this cross-modal data. This results in limited coverage for automated reviews, and manual cross-modal verification is also prone to omissions.

[0004] Furthermore, differing interpretations of clinical guidelines, institutional quality control standards, and historical experience among reviewers lead to significant subjectivity in the scale and conclusions of manual reviews, hindering the standardization and quantifiable evaluation of radiotherapy quality control. While some radiotherapy quality control software can perform rule-based checks on structured parameters, it still struggles to handle complex text and image information, and is unable to determine clinical logical consistency at the multimodal semantic level. Consequently, current automated review methods remain significantly insufficient in both depth and breadth.

[0005] In recent years, with the development of large language models (LLM) and multimodal artificial intelligence technologies, models have acquired the ability to jointly understand and semantically reason about multi-source information such as text and images, making multimodal intelligent analysis possible. However, current technologies lack a systematic solution for recording the entire radiotherapy process (including structured data, text, and images), capable of secure deployment in a hospital intranet environment, and integrating clinical guidelines, institutional standards, and historical experience to achieve automated and intelligent review and decision support of radiotherapy records. Therefore, there is an urgent need for an automated review method and system for radiotherapy records that supports multimodal data alignment, incorporates domain knowledge retrieval, and utilizes large language models for reasoning, in order to improve review efficiency and accuracy, reduce human error, and enhance the overall level of radiotherapy quality control. Summary of the Invention

[0006] This invention provides an automated review method, system, and storage medium for radiotherapy records based on a multimodal pre-trained model. It aims to address shortcomings in existing technologies, such as reliance on manual review, difficulty in uniformly processing cross-modal data, limited coverage of automated tools, and a lack of semantic-level logical consistency judgment. This invention constructs a technical solution that integrates medical images, structured dose parameters, and clinical text through joint modeling and a radiotherapy-related knowledge base. This solution utilizes knowledge retrieval and large language model reasoning to automatically review radiotherapy records. This automates, standardizes, and makes the radiotherapy record review process traceable, improving review efficiency and consistency while reducing the burden and subjectivity of manual review.

[0007] The technical solution of this invention is implemented as follows:

[0008] An automated review method for radiotherapy records based on a multimodal pre-trained model includes the following steps:

[0009] (1) Data acquisition and standardization: Acquire clinical text and storage documents of multimodal data during radiotherapy, and standardize the multimodal data;

[0010] (2) Multimodal feature extraction and fusion: The structural features of medical images are extracted through the Vision-Language Model (VLM), and the semantic features of text and dosage data are extracted through the enhanced ClinicalBERT model. The cross-attention mechanism is used to align and fuse multimodal features to generate a fused medical semantic representation vector.

[0011] (3) Enhanced knowledge retrieval: Based on the fused medical semantic representation vector, relevant clinical guidelines, dose constraints and disease examples are retrieved from the preset radiotherapy knowledge base to form a knowledge evidence set;

[0012] (4) Large language model review and reasoning: The fused medical semantic representation vector and knowledge evidence set are input into the locally deployed multimodal model, and a structured review report is generated through structured review prompts, including target coverage assessment, organ at risk protection analysis, DVH interpretation, plan quality scoring, guideline compliance check and anomaly detection;

[0013] (5) Intelligent analysis and hierarchical decision-making: Based on the structured review report, perform intelligent analysis and hierarchical decision-making, output the review results, and support human-computer interaction for review, confirmation or adding notes;

[0014] (6) Audit traceability and privacy security: The entire review process is audited and traced, and input data, search items, model version, output results and interaction behavior are recorded.

[0015] The multimodal data mentioned in step (1) includes multimodal data such as computed tomography images (DICOM CT images), radiotherapy structures (RT structures), radiotherapy plans (RT plans), radiotherapy doses (RT doses), dose-volume histogram data (DVH data), radiotherapy positioning forms, radiotherapy planning request forms, radiotherapy planning reports, and digital texts of clinical treatment protocols. The digital texts of clinical treatment protocols include diagnostic criteria, radiotherapy pathways, radiotherapy protocols, follow-up plans, etc., and exist in the form of structured documents or database records.

[0016] In step (2), the visual-language model incorporates structural overlap and spatial relationships as supervisory signals during training, and uses the Dice coefficient and Hausdorff distance as composite loss functions to optimize the extraction of three-dimensional structural information, spatial topology, and morphological distribution features. Furthermore, the visual-language model constructs a radiotherapy-specific visual-language pair dataset, incorporating structural overlap and spatial relationships as supervisory signals. It not only uses image-text pairs but also employs medical image segmentation metrics such as the Dice coefficient and Hausdorff distance as additional supervisory signals for the automatic extraction of three-dimensional structural information, spatial topology, and morphological distribution features of target areas (GTV, CTV, PTV) and organs at risk (OARs).

[0017] The enhanced ClinicalBERT model described in step (2) uses a dose symbolic encoding mechanism to construct the dose value, volume fraction, and limit threshold in the text as a "term-value-unit-organ" quadruple token, and improves the robustness of the model through numerical noise enhancement and unit perturbation training.

[0018] The radiotherapy knowledge base mentioned in step (3) includes international / domestic radiotherapy guidelines, organ-at-risk dose constraint datasets, and best practice cases for various diseases, supporting semantic retrieval based on a retrieval enhancement generation framework. Preferably, the international / domestic radiotherapy guidelines include the International Commission on Radiation Units and Measurements (ICRU), the National Comprehensive Cancer Network (NCCN), the European Society for Radiation Oncology (ESTRO), and the American Academy of Medical Physics (AAPM), etc.; the organ-at-risk dose constraint datasets include data compiled based on QUANTEC / RION data.

[0019] The structured review prompts in step (4) include the following fields: patient basic information, planned site and prescription dose, target coverage status, organ at risk exceeding limits, DVH curve anomaly detection, plan quality score and improvement suggestions.

[0020] The intelligent analysis described in step (5) specifically involves the comprehensive implementation of six functional modules in the intelligent analysis layer, including target coverage assessment, organ protection analysis, in-depth interpretation of DVH, quality score calculation, guideline compliance check and anomaly detection, while providing evidence citations.

[0021] The tiered decision-making process includes a four-level mechanism: automatic approval, conditional approval, post-modification review, and escalation expert review, and is integrated with a closed-loop clinical workflow.

[0022] Regarding audit traceability and privacy security in step (6), the entire chain of input data, search entries, model versions, prompts, output results and human-computer interaction behavior during the review process is recorded to form a replayable audit trail. Medical data security and compliance requirements are met through hospital intranet deployment, data anonymization and differential privacy update strategies.

[0023] An automated review system for radiotherapy records based on a multimodal pre-trained model, used to implement the aforementioned automated review method for radiotherapy records, includes:

[0024] The data input module is used to acquire and standardize multimodal radiotherapy data;

[0025] A multimodal fusion module is used to extract and fuse image, text, and dose features;

[0026] The knowledge retrieval module is used to retrieve relevant clinical evidence from the knowledge base;

[0027] The LLM review engine is used to generate structured review reports;

[0028] The intelligent analysis layer is used to perform target coverage assessment, organ protection analysis, DVH interpretation, quality scoring, guideline compliance checks, and anomaly detection.

[0029] The decision output layer is used to output review results and support human-computer interaction.

[0030] The audit traceability module is used to record the entire review process.

[0031] The multimodal fusion module includes a visual-language encoder, an enhanced ClinicalBERT encoder, and a cross-attention fusion unit.

[0032] A storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for automatic review of radiotherapy records.

[0033] Compared with the prior art, the technical solution of the present invention has the following advantages:

[0034] 1. This invention achieves deep fusion and integrated review of cross-modal data, overcoming the limitations of traditional tools. By constructing a radiotherapy-specific VLM model and an enhanced ClinicalBERT model, and utilizing a cross-attention mechanism for feature alignment, this invention maps CT images, 3D structures, dose parameters (DVH), and clinical text into a unified medical semantic space. This enables the system to understand and associate the complex relationships between "anatomical structure-dose distribution-text description" like a clinical expert, thereby achieving comprehensive, semantic-level review of radiotherapy records, covering unstructured text and image logic judgments that traditional rule-based systems cannot handle.

[0035] 2. Enhanced the knowledge-driven nature and interpretability of review decisions, improving clinical credibility. By integrating a Retrieval Enhancement Generation (RAG) framework that incorporates international / domestic guidelines, dose constraints, and best practices, the system can perform real-time correlation and matching of actual data for each specific case with authoritative knowledge bases. When generating review conclusions (such as "OAR exceeded"), the large language model can simultaneously output the specific guideline clauses cited, the source of the threshold, and evidence fragments, making the AI ​​decision-making process transparent and traceable, greatly improving the clinical acceptability and guiding value of the review results.

[0036] 3. A standardized and modular intelligent analysis system was constructed, promoting the standardization of quality control processes. This invention designed analysis modules covering six core functions: target coverage, organ protection, DVH interpretation, quality scoring, guideline compliance, and anomaly detection, and formed a quantifiable quality scoring system. This eliminates subjective differences among different reviewers, achieves the unification and objectification of radiotherapy quality control evaluation standards, and is conducive to the standardization of internal quality management and horizontal comparison within institutions.

[0037] 4. A tiered decision-making and workflow closed-loop mechanism has been established, balancing efficiency and safety. The system's proposed four-level decision-making mechanism—automatic approval, conditional approval, post-modification review, and escalation expert review—intelligently embeds AI review results into existing clinical workflows. For high-quality routine plans, rapid automatic approval improves efficiency; for cases with questions or complexities, automatic review or escalation processes are triggered to ensure key cases receive focused human review, forming a human-machine collaborative, efficient, and safe quality control closed loop.

[0038] 5. The system deployment is secure, compliant, and auditable, meeting the stringent requirements of the healthcare industry. By supporting fully localized deployment within the hospital intranet and employing data anonymization and differential privacy update strategies, this invention fundamentally safeguards the security of sensitive patient data, complying with medical data privacy protection regulations. Simultaneously, the end-to-end audit traceability function records every step from data input and knowledge retrieval to final decision-making, making the entire review process traceable and analyzable. This not only meets the auditing needs of medical quality management but also provides a data foundation for continuous model optimization.

[0039] In summary, this invention not only achieves automated and intelligent review of multimodal radiotherapy records from a technical perspective, but also provides a systematic solution from multiple dimensions such as clinical applicability, decision credibility, process integration, and safety compliance, demonstrating significant practicality, advanced nature, and promotional value. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall architecture of the system according to an embodiment of the present invention;

[0041] Figure 2 This is a detailed block diagram of each functional module of the system in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0044] Currently, existing technologies suffer from several shortcomings, including reliance on manual review of radiotherapy records, difficulty in uniformly processing cross-modal data, limited coverage of automated tools, and a lack of semantic-level logical consistency judgment. To address these technical issues, this invention proposes an automated review method, system, and storage medium for radiotherapy records based on a multimodal pre-trained model.

[0045] Example 1

[0046] An automated review method for radiotherapy records based on a multimodal pre-trained model includes the following steps:

[0047] 1. Data processing and standardization: Acquire clinical text and storage documents of multimodal data during radiotherapy, and perform standardization processing on the multimodal data;

[0048] Specifically: The system automatically imports the patient's DICOM CT images, RT Structure (including GTV, CTV, PTV and multiple OAR structures), RT Plan, RT Dose and DVH files from TPS, and synchronizes the digital text of the clinical treatment protocol from the HIS / RIS / radiotherapy information management platform: the patient's prescription information, fractionated doses, pathological diagnosis and staging, as well as related treatment plan description documents and medical orders. After completing the data integrity and consistency check, the dose units, structure names, coordinate systems, etc. are standardized and unified.

[0049] 2. Multimodal Feature Extraction and Fusion: Structural features of medical images are extracted using a Vision-Language Model (VLM), and semantic features of text and dosage data are extracted using an enhanced ClinicalBERT model. A cross-attention mechanism is then used for multimodal feature alignment and fusion to generate a fused medical semantic representation vector; details are as follows:

[0050] 2.1 Radiation Therapy-Specific Vision-Language Model for Constructing and Improving VLM Training:

[0051] Based on CT images from TPS and their corresponding target areas (GTV, CTV, PTV) and organs at risk (OARs) segmentation results, and combined with textual information such as structural and dose descriptions from radiotherapy planning request forms and planning reports, a radiotherapy-specific image-text pair dataset is constructed. In the visual branch, structural overlap and spatial relationships are introduced as additional supervisory signals to encode the inclusion, adjacency, and distance relationships between different structures. Simultaneously, a composite loss function based on the Dice coefficient and Hausdorff distance is introduced into the training loss, enabling the improved VLM to explicitly optimize segmentation overlap and boundary distance when extracting 3D structural information, spatial topology, and morphological distribution features, thereby obtaining a structural representation that better meets the requirements of radiotherapy.

[0052] 2.2 Structural overlap and spatial topological feature encoding:

[0053] For each target region and OAR structure, a structural embedding vector is automatically generated by improving the VLM visual branch. The embedding explicitly or implicitly includes spatial topological information such as volume, shape, boundary smoothness, and distances and overlap ratios between components. Medical image segmentation evaluation metrics such as the Dice coefficient and Hausdorff distance are used as supervision labels during the training phase to constrain the consistency between the structural embedding and the actual anatomical relationship. This enables the model to distinguish between delineation results of different quality, providing criteria for identifying insufficient target region coverage or abnormal OAR delineation during subsequent automatic review.

[0054] The Dice similarity coefficient is defined as follows: Let A and B be the sets of voxels corresponding to the two sets of segmentation results.

[0055]

[0056] Hausdorff distance is defined as follows: Given two sets of dividing boundaries or point sets A and B, the Hausdorff distance is:

[0057] H(A,B) = max(h(A,B), h(B,A));

[0058] The one-way Hausdorff distance is:

[0059]

[0060] II·II represents the Euclidean distance. h(A, B) describes the "worst-case boundary deviation" between the two partitions; the smaller the value, the closer the boundaries are.

[0061] 2.3 Dose Numerical Understanding and Symbolic Encoding Based on ClinicalBERT:

[0062] For radiotherapy texts (including DVH reports, treatment plans, and application forms), an enhanced ClinicalBERT model for understanding dose and volume numerical values ​​is constructed. A dose symbolic encoding mechanism is introduced to structurally label information such as dose values, volume fractions, and limit thresholds appearing in the text. For key DVH points (including but not limited to D95, V20, Dmax, Dmean, volume values, dose values, and their units), a "term-value-unit-organ" quadruple token is constructed. Numerical noise perturbation and unit perturbation (such as Gy / cGy, percentage / absolute volume) are added during training to augment the data, making ClinicalBERT more robust to numerical nearest neighbors, unit conversions, and writing differences, achieving semantic embedding of DVH parameters and fine-grained modeling of dose semantics.

[0063] 2.4 Multimodal feature alignment and cross-attention fusion:

[0064] The structural embedding vector obtained from the improved VLM, the text semantic embedding obtained from ClinicalBERT, and the dose parameter embedding constructed from the DVH quadruple are input into the multimodal fusion module. A cross-attention mechanism is implemented in the fusion module, allowing text tokens to selectively focus on the corresponding structural embeddings and DVH embeddings based on attention weights. Simultaneously, the structural and DVH embeddings are allowed to perform fine-grained alignment based on textual cues, thereby encoding the relationship between "anatomical structure-dose distribution-text description" in a unified high-dimensional representation space.

[0065] 2.5 Generation of Medical Semantic Representation Vectors:

[0066] Pooling or aggregation operations are performed on the multimodal feature sequences after cross-attention fusion to obtain the fused medical semantic representation vector of the current case. This vector is used for subsequent semantic retrieval of the radiotherapy knowledge base, guideline clause matching, and automatic review and reasoning driven by a large language model. The fused medical semantic representation vector simultaneously preserves three-dimensional structural morphology, spatial topology, DVH numerical features, and textual semantic information, providing a unified feature foundation for subsequent comprehensive evaluation of target coverage, organ-at-risk protection, DVH curve morphology, and plan quality.

[0067] 3. Enhanced Knowledge Retrieval: Based on the fused medical semantic representation vector, relevant clinical guidelines, dose constraints, and disease examples are retrieved from the preset radiotherapy knowledge base to form a knowledge evidence set;

[0068] Specifically, the system retrieves target dosimetry assessment indicators and their recommended thresholds applicable to the current plan from the guideline and OAR constraint databases based on the patient's disease characteristics, including the pathologically confirmed disease type (e.g., lung adenocarcinoma, nasopharyngeal undifferentiated carcinoma) and clinical stage determined according to the TNM staging system, treatment technical parameters, including the planned irradiation site (e.g., "whole brain", "prostate and pelvic lymphatic drainage area"), total prescribed dose (e.g., 70 Gy), fractionation mode (e.g., 35 fractions, 2.0 Gy per fraction), and selected radiotherapy technology type (e.g., fixed-field intensity-modulated radiotherapy (IMRT), volumetric intensity-modulated radiotherapy (VMAT), or pencil-beam proton therapy (PBS)). These indicators include recommended target coverage thresholds (e.g., PTV D95 (the lowest dose received by 95% of the target volume) should not be less than the percentage of the prescribed dose (e.g., ≥95%)), maximum permissible dose in OAR, dose-volume histogram (DVH) constraints, and upper limit of average dose, as well as corresponding clinical background descriptions. This constitutes a set of external knowledge evidence for subsequent reasoning.

[0069] 4. Large Language Model Review and Reasoning: The fused medical semantic representation vector and knowledge evidence set are input into a locally deployed multimodal model. Through structured review prompts, a structured review report is generated, including target coverage assessment, organ at risk protection analysis, DVH interpretation, plan quality scoring, guideline compliance checks, and anomaly detection. Details are as follows:

[0070] The system constructs structured prompt templates, clearly defining the fields that the model needs to output, such as "whether the patient's basic information is correct," "whether the planned site and prescription dosage are correct," "whether the PTV coverage meets the standard," "whether the key OAR exceeds the limit and its specific value and source of constraint," "whether there is dose exceeding the limit or abnormal distribution in the DVH curve," and "overall plan quality score and improvement suggestions," etc. Multimodal fusion representation vectors and knowledge evidence fragments are embedded in the prompts and input into the locally deployed medical big language model. The model generates structured review results, in which each conclusion includes the cited guideline entry or OAR threshold and provides an uncertainty score.

[0071] 5. Intelligent Analysis and Hierarchical Decision-Making: Based on the structured review report, intelligent analysis and hierarchical decision-making are performed, review results are output, and human-computer interaction is supported for review, confirmation, or adding remarks; specifically:

[0072] The system, based on fused semantic vectors and structured dose information generated by a multimodal model, comprehensively executes six functional modules: target coverage assessment, organ protection analysis, in-depth interpretation of DVH curves, overall plan quality scoring, clinical and institutional guideline consistency checks, and potential anomaly detection. Each module independently generates scores, judgment results, and corresponding evidence citations, forming a traceable review record. The system further weights and aggregates the results from each module, outputting an overall quality level, which is presented through a human-computer interaction interface. This allows operators to view evidence, review AI judgments, confirm or modify results, and add notes to form a complete review and decision log.

[0073] 6. Audit Traceability and Privacy Security: The entire review process is audited and traced, recording input data, search entries, model versions, output results, and interaction behaviors; specifically:

[0074] The system records the entire process of input data summaries, knowledge retrieval results, model version information, review results, uncertainty scores, and final clinical decisions. If abnormalities are found during subsequent clinical follow-ups or quality control reviews, the system can trace the causes and analyze model behavior based on the audit records, thereby achieving continuous performance improvement and risk control.

[0075] Through the above specific implementation methods, the present invention can provide radiation oncologists, radiation physicists, and radiation technicians with a multimodal, knowledge-driven large language model-assisted review tool without changing the existing clinical radiotherapy process, significantly improving the efficiency, consistency, and safety of radiotherapy record review.

[0076] like Figure 1 and Figure 2 As shown, this embodiment provides an automatic review system for radiotherapy records based on a multimodal pre-trained model, including:

[0077] The data input module is used to acquire and standardize multimodal radiotherapy data;

[0078] A multimodal fusion module is used to extract and fuse image, text, and dose features;

[0079] The knowledge retrieval module is used to retrieve relevant clinical evidence from the knowledge base;

[0080] The LLM review engine is used to generate structured review reports;

[0081] The intelligent analysis layer is used to perform target coverage assessment, organ protection analysis, DVH interpretation, quality scoring, guideline compliance checks, and anomaly detection.

[0082] The decision output layer is used to output review results and support human-computer interaction.

[0083] The audit traceability module is used to record the entire review process.

[0084] The multimodal fusion module includes a visual-language encoder, an enhanced ClinicalBERT encoder, and a cross-attention fusion unit.

[0085] This embodiment also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described automatic review method for radiotherapy records.

[0086] The automatic review method for radiotherapy records based on a multimodal pre-trained model in this embodiment has the following advantages:

[0087] (1) A vision-language pair dataset for radiotherapy was constructed using the VLM (Vision-Language Model) model. Structural overlap and spatial relationship were introduced as supervision signals. In addition to using image-text pairs, medical image segmentation indicators such as Dice coefficient and Hausdorff distance were also used as additional supervision signals to automatically extract the three-dimensional structural information, spatial topology and morphological distribution features of target areas (GTV, CTV, PTV) and organs at risk (OARs).

[0088] (2) Construct an enhanced ClinicalBERT model for understanding dose and volume numerical values. Introduce a dose symbolic encoding mechanism to structurally label information such as dose values, volume fractions, and limit thresholds appearing in the text. Create DVH keypoint quadruple tokens. This makes ClinicalBERT more robust to numerical nearest neighbors, unit conversions, and writing differences, and enables semantic embedding of DVH parameters and fine-grained modeling of dose semantics.

[0089] (3) Enhanced knowledge retrieval: The system retrieves applicable entries in the guideline library and OAR constraint library based on the patient site, prescription dose, fractionation scheme and technology type (such as IMRT / VMAT / proton therapy, etc.), including recommended target coverage index thresholds (such as a certain proportion of PTV D95 ≥ prescription dose), maximum allowable dose and average dose limit of OAR, as well as corresponding clinical background descriptions, which constitute a set of external knowledge evidence for subsequent reasoning.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automatic review method for radiotherapy records based on a multimodal pre-trained model, characterized in that: Includes the following steps: (1) Data acquisition and standardization: Acquire clinical text and storage documents of multimodal data during radiotherapy, and standardize the multimodal data; (2) Multimodal feature extraction and fusion: The structural features of medical images are extracted through a visual-language model, and the semantic features of text and dosage data are extracted through an enhanced ClinicalBERT model. Multimodal feature alignment and fusion are performed using a cross-attention mechanism to generate a fused medical semantic representation vector. (3) Enhanced knowledge retrieval: Based on the fused medical semantic representation vector, relevant clinical guidelines, dose constraints and disease examples are retrieved from the preset radiotherapy knowledge base to form a knowledge evidence set; (4) Large language model review and reasoning: The fused medical semantic representation vector and knowledge evidence set are input into the locally deployed multimodal model, and a structured review report is generated through structured review prompts, including target coverage assessment, organ at risk protection analysis, DVH interpretation, plan quality scoring, guideline compliance check and anomaly detection; (5) Intelligent analysis and hierarchical decision-making: Based on the structured review report, perform intelligent analysis and hierarchical decision-making, output the review results, and support human-computer interaction for review, confirmation or adding notes; (6) Audit traceability and privacy security: The entire review process is audited and traced, and input data, search items, model version, output results and interaction behavior are recorded.

2. The automatic review method for radiotherapy records based on a multimodal pre-trained model according to claim 1, characterized in that: The multimodal data mentioned in step (1) includes multimodal data such as computed tomography images, radiotherapy structure sets, radiotherapy plans, radiotherapy doses, dose-volume histogram data, radiotherapy positioning forms, radiotherapy plan request forms, radiotherapy plan reports, and digital texts of clinical treatment protocols.

3. The automatic review method for radiotherapy records based on a multimodal pre-trained model according to claim 1, characterized in that: In step (2), the visual-language model introduces structural overlap and spatial relationship as supervision signals during training, and uses Dice coefficient and Hausdorff distance as composite loss functions to optimize the extraction of three-dimensional structural information, spatial topology and morphological distribution features.

4. The automatic review method for radiotherapy records based on a multimodal pre-trained model according to claim 1, characterized in that: The enhanced ClinicalBERT model described in step (2) uses a dose symbolic encoding mechanism to construct the dose value, volume fraction, and limit threshold in the text as a "term-value-unit-organ" quadruple token, and improves the robustness of the model through numerical noise enhancement and unit perturbation training.

5. The automatic review method for radiotherapy records based on a multimodal pre-trained model according to claim 1, characterized in that: The radiotherapy knowledge base mentioned in step (3) includes international / domestic radiotherapy guidelines, dose constraint datasets for organs at risk, and best practice cases for diseases, and supports semantic retrieval based on the retrieval enhancement generation framework.

6. The automatic review method for radiotherapy records based on a multimodal pre-trained model according to claim 1, characterized in that: The structured review prompts in step (4) include the following fields: patient basic information, planned site and prescription dose, target coverage status, organ at risk exceeding limits, DVH curve anomaly detection, plan quality score and improvement suggestions.

7. The automatic review method for radiotherapy records based on a multimodal pre-trained model according to claim 1, characterized in that: The intelligent analysis described in step (5) specifically involves the comprehensive implementation of six functional modules in the intelligent analysis layer, including target coverage assessment, organ protection analysis, in-depth interpretation of DVH, quality score calculation, guideline compliance check and anomaly detection, while providing evidence citations. The tiered decision-making process includes a four-level mechanism: automatic approval, conditional approval, post-modification review, and escalation expert review, and is integrated with a closed-loop clinical workflow.

8. An automatic review system for radiotherapy records based on a multimodal pre-trained model, used to implement the automatic review method for radiotherapy records according to any one of claims 1-7, characterized in that: include: The data input module is used to acquire and standardize multimodal radiotherapy data; A multimodal fusion module is used to extract and fuse image, text, and dose features; The knowledge retrieval module is used to retrieve relevant clinical evidence from the knowledge base; The LLM review engine is used to generate structured review reports; The intelligent analysis layer is used to perform target coverage assessment, organ protection analysis, DVH interpretation, quality scoring, guideline compliance checks, and anomaly detection. The decision output layer is used to output review results and support human-computer interaction. The audit traceability module is used to record the entire review process.

9. The automatic review system for radiotherapy records based on a multimodal model according to claim 8, characterized in that: The multimodal fusion module includes a visual-language encoder, an enhanced ClinicalBERT encoder, and a cross-attention fusion unit.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the automatic review method for radiotherapy records as described in any one of claims 1-8.

Images