Method and system for generating medical image report by field constraint visual language model

Abstract

The application provides a medical image report generation method and system of field constraint visual language model, obtains non-enhanced computer tomography image data and corresponding electronic health record data, respectively extracts visual features and text features and performs cross-modal alignment and fusion to generate joint multi-modal features; calls a structured report template, inputs the joint multi-modal features into a visual-language model decoder to perform self-recurrence decoding, and when decoding operation is performed on each independent field, only probability sampling is performed within a corresponding legal value range or candidate value set range to generate a structured report; key diagnostic feature combinations in the structured report are extracted as query sentences, similarity retrieval is performed in a medical literature vector database, and after relevant academic literature returned is converted into reference labels embedded in corresponding fields, a complete report with evidence-based evidence chain is output. The application overcomes the description inaccuracy defect caused by full word table unordered search, and guarantees the reliability of a diagnostic conclusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and artificial intelligence technology, in particular to a medical image report generation method and system based on field constraint visual language model. BACKGROUND

[0002] In medical clinical diagnosis, the writing of medical image reports is the core work link of radiologists. With the application of deep learning technology, computer-aided diagnosis systems based on artificial intelligence are introduced into this business scenario.

[0003] Existing image report generation systems usually adopt an encoder-decoder architecture, use a visual model to extract image features of medical images, and then pass the image features to a language model to convert the image features into a natural language sequence. A large parameter scale visual language model automatically outputs an imaging report text covering the lesion location, morphological features and preliminary diagnostic opinions through end-to-end text generation capability, assisting physicians to reduce work burden and improve reading efficiency.

[0004] For example, Chinese patent application CN120412875A discloses a medical image report generation method and system based on a large language model, which extracts image embedding vectors and background embedding vectors by obtaining patient original image data and clinical background text, retrieves relevant historical case reports based on the embedding vectors, and generates a medical image report by integrating structured visual evidence, clinical background text and historical case reports using a large language model.

[0005] For another example, Chinese patent application CN121687361A discloses a medical report automatic analysis and structured generation method based on a multi-modal visual-language model, which obtains a lesion-description cluster through cross-modal semantic association, assembles and generates content according to a pre-defined clinical report logic template, and implements logical constraints by limiting the context information and the range of selectable vocabulary accessible to the model during the decoding process, while performing confidence evaluation on the generated structured fields.

[0006] However, the above-mentioned solutions mainly improve the generation quality of medical image reports from the aspects of multi-modal information fusion, template assembly, historical case retrieval or report logic constraints, and have not directly limited the probability sampling space of the visual-language model according to the legal value domain or candidate value set corresponding to each independent field in the structured report. At the same time, for the generated diagnostic conclusion, there is still a lack of a collaborative mechanism to extract key diagnostic features from the structured report, retrieve external academic literature in a medical literature vector database, and embed the corresponding reference numbers back into the relevant fields to form a traceable evidence-based evidence chain. Therefore, how to reduce the fact deviation beyond the clinical rule boundary during the report generation stage and establish a field-level traceable evidence-based support during the report output stage is still a technical problem to be solved. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and system for generating medical image reports using a field-constrained visual language model. This solves the problem that existing visual language models, when generating medical image reports, are prone to deviating from the facts of the images due to their reliance on a full-vocabulary unconstrained decoding mechanism, leading to factual deviations such as inaccurate pathological descriptions and self-contradictions.

[0008] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a method for generating medical image reports using a field-constrained visual language model, comprising the following steps:

[0009] Acquire non-contrast computed tomography (CT) images and corresponding electronic health record data of the subjects to be screened. Preprocess the non-contrast computed tomography images to convert them into a standardized three-dimensional voxel matrix and organize the electronic health record data into a clinical timeline text sequence.

[0010] Multi-scale visual feature vectors of standardized three-dimensional voxel matrices and clinical semantic text feature vectors of clinical timeline text sequences are extracted separately. The multi-scale visual feature vectors and clinical semantic text feature vectors are then aligned and fused to generate joint multimodal features.

[0011] The joint multimodal features are input into the visual-language model decoder. The visual-language model decoder performs autoregressive decoding according to the field order of the pre-called structured report template. The structured report template contains a preset set of fields, and each independent field is configured with a legal value range or a set of candidate values. When performing decoding operations for each independent field, the visual-language model decoder only performs probability sampling within the corresponding legal value range or set of candidate values ​​to generate a structured report.

[0012] Key diagnostic features are extracted from the structured report and combined into a query statement. The query statement is encoded into a query vector and a similarity retrieval is performed in a medical literature vector database. The relevant academic literature returned by the retrieval is converted into citation numbers and automatically embedded into the corresponding fields in the structured report. Finally, a complete report with a chain of evidence-based data is output.

[0013] Preferably, the multi-scale visual feature vectors are aligned and fused with the clinical semantic text feature vectors to generate joint multimodal features, specifically including:

[0014] A cross-attention mechanism is introduced to calculate the semantic correspondence between image regions and clinical text descriptions, obtain the attention weight matrix, and use the attention weight matrix to locate the image spatial region corresponding to the diagnostic description field to complete semantic alignment.

[0015] The multi-scale visual feature vectors with semantic alignment and the clinical semantic text feature vectors are input into the gating fusion unit. The gating fusion unit dynamically adjusts the proportion of the two types of modal information through learnable gating weights, and outputs joint multimodal features after cross-modal alignment and adaptive fusion.

[0016] Preferably, when performing decoding operations for each individual field, the visual-language model decoder only performs probability sampling within the corresponding legal value range or candidate value set to generate a structured report, specifically including:

[0017] For discrete fields, the output layer dimension of the visual-language model decoder is dynamically adjusted to the size of the set of legal candidate values ​​for the current generated field. The output values ​​are processed by the activation function to output the probability distribution of each legal candidate value. The candidate value with the highest probability is selected as the generated result of the current generated field.

[0018] For numerical fields, the visual-language model decoder outputs continuous values ​​that are processed by a mapping function to a preset numerical range;

[0019] For outputs with values ​​exceeding the preset range, an automatic interception operation is performed to eliminate the generation of non-predefined values; and when generating the target field value, the confidence score of the target field output is calculated simultaneously. In the final structured report, target fields with confidence scores below the preset threshold are highlighted with a color mark attribute.

[0020] Preferably, the medical literature vector database only includes academic journal articles with an impact factor greater than a preset impact factor threshold, and a vector index is established in the medical literature vector database using an inverted file index and a product quantization structure;

[0021] The similarity retrieval in the medical literature vector database includes: using inner product similarity as a metric in the medical literature vector database, calculating the inner product similarity score between the query vector and each document vector; sorting the documents in descending order based on the inner product similarity score, filtering out documents with inner product similarity scores below a preset similarity threshold, and extracting relevant academic documents that meet the threshold conditions.

[0022] Preferably, the method further includes performing visual-language cross-validation and conflict warning:

[0023] Input the clinical timeline text sequence into the trained electronic health record big language model and output the text inference conclusion;

[0024] Obtain the imaging diagnostic conclusions output by the visual-language model decoder, and perform semantic similarity calculation based on medical ontology between the imaging diagnostic conclusions and the text reasoning conclusions;

[0025] When the imaging diagnosis conclusion and the text reasoning conclusion point to different disease entities, or when the imaging diagnosis conclusion and the text reasoning conclusion are in opposite directions, a conflict is determined, and a conflict warning signal containing the specific content of both conflicting parties is generated at the top of the report interface of the human-computer collaborative interaction terminal.

[0026] Preferably, the trained electronic health record large language model is obtained through an offline training process using a self-evolutionary algorithm. The offline training process of the self-evolutionary algorithm includes:

[0027] A large language model evaluator with three core judgment capabilities is constructed based on a training dataset containing patients' longitudinal clinical timeline text and corresponding labels. The three core judgment capabilities include: judging the correctness of the answer by verifying the consistency between the model's predicted answer and the actual label; judging the consistency of the reasoning-answer by analyzing the logical chain of the reasoning process to confirm whether it supports the model's predicted answer; and judging the authenticity of the reasoning by checking whether the clinical evidence cited in the reasoning process is completely faithful to the input data.

[0028] The general large language model is called to infer the training samples to generate the corresponding reasoning process and predicted answer, which serve as the initial reasoning samples. After the large language model evaluator completes three judgments and screenings, all high-quality samples that pass the verification are included in the seed dataset.

[0029] Based on the seed dataset, supervised fine-tuning of the basic large language model is carried out to obtain the initial inference model. The model obtained from the previous round of training generates new inference results on the training samples and is verified by the large language model evaluator. The high-quality samples that pass the current round of verification are used to build the current round dataset. The model is fine-tuned and iterated until the area under the curve of the model on the validation set no longer improves or the number of iterations reaches the preset maximum value. The final electronic health record large language model is then output.

[0030] Preferably, the method further includes performing interpretable attribution analysis based on a trained electronic health record large language model:

[0031] Structured clinical feature vectors are extracted from the original clinical timeline text. The structured clinical feature vectors are used as the model input, and the risk margin output by the original large language model is used as the regression training target to construct and train a regression alternative model.

[0032] By adjusting the tree depth, number of leaf nodes, and learning rate hyperparameter of the regression substitution model, we ensure that the fitting accuracy of the regression substitution model on the validation set is greater than a preset threshold.

[0033] The attribution algorithm is used to quantify the local contribution of each structured feature to the prediction result. The local contribution values ​​of derived features belonging to the same clinical indicator and the same diagnostic concept are accumulated and fused to output the positive and negative contribution values ​​of various clinical concepts to the judgment of the patient's disease risk.

[0034] Preferably, before acquiring the non-contrast computed tomography (CT) image data and corresponding electronic health record data of the subject to be screened, the method further includes:

[0035] Rule-based risk screening was performed using routine clinical indicators to identify high-risk populations. The routine clinical indicators included 13 blood biomarkers, 4 types of diagnostic records, and 1 composite lifestyle factor.

[0036] Thirteen blood biomarkers were included, such as glycated hemoglobin, total bilirubin, total bile acids, aspartate aminotransferase, alanine aminotransferase, gamma-glutamyl transferase, alkaline phosphatase, amylase, prothrombin time, D-dimer, creatinine, fasting blood glucose, and serum sodium. Four types of diagnostic records were included, such as diabetes mellitus, cirrhosis, pancreatitis, and jaundice. The composite lifestyle factor was defined as a history of both smoking and alcohol consumption. If any indicator was abnormal, the examinee was identified as a high-risk group, and the steps of obtaining non-contrast computed tomography (CT) images and corresponding electronic health record data for the high-risk group were performed.

[0037] Preferably, the structured report template covers the basic information area, clinical information area, imaging findings area, TNM staging area, and diagnostic opinion area;

[0038] For the TNM staging region, the lesion measurement information output by the visual-language model, including the maximum diameter, number, vascular invasion signs, lymph node and distant metastasis, is obtained. Each component is judged according to the established cancer staging standards, and the final clinical staging conclusion is generated and filled into the corresponding staging field.

[0039] A second aspect of the present invention provides a medical image report generation system based on a field-constrained visual language model, comprising:

[0040] The data acquisition and preprocessing unit is used to acquire non-contrast computed tomography (CT) image data and corresponding electronic health record data of the subjects to be screened, preprocess the non-contrast computed tomography image data into a standardized three-dimensional voxel matrix, and organize the electronic health record data into a clinical timeline text sequence.

[0041] The electronic health record reasoning unit is used to receive clinical timeline text sequences, predict the risk probability of the subject to be screened developing liver, gallbladder and pancreatic cancer within a preset time period, and output the diagnostic reasoning process.

[0042] The image analysis unit receives non-enhanced computed tomography (CT) image data and electronic health record data corresponding to the high-risk prediction results output by the electronic health record inference unit. It extracts multi-scale visual feature vectors of the standardized three-dimensional voxel matrix and clinical semantic text feature vectors of the clinical timeline text sequence, respectively. The multi-scale visual feature vectors and clinical semantic text feature vectors are aligned and fused to generate joint multimodal features. The image analysis unit contains a vision-language model decoder, which takes the joint multimodal features as input and performs autoregressive decoding according to the field order of the pre-called structured report template. When decoding each independent field, probability sampling is performed only within the corresponding legal value range or candidate value set to generate a structured report.

[0043] The evidence-based retrieval unit is used to extract key diagnostic features from the structured report, splice and combine them into a query statement, encode the query statement into a query vector and perform a similarity retrieval in the medical literature vector database, convert the relevant academic literature returned by the retrieval into citation numbers and automatically embed them into the corresponding fields in the structured report, and then output a complete report with an evidence-based evidence chain.

[0044] The cross-validation and conflict warning unit is used to obtain the imaging diagnostic conclusions output by the image analysis unit and the text reasoning conclusions output by the electronic health record reasoning unit, compare the semantic consistency of the imaging diagnostic conclusions and the text reasoning conclusions, and generate a conflict warning signal when a logical contradiction occurs.

[0045] The human-computer collaborative interaction terminal is used to aggregate and present structured reports, evidence-based literature citations, and conflict warning signals to the operation interface.

[0046] This invention provides a method and system for generating medical image reports using a field-constrained visual language model. It offers the following advantages:

[0047] 1. This invention calls a structured report template during the visual language model decoding process and configures a legal value range or candidate value set for independent fields. This allows the decoding operation to perform probability sampling only within a limited preset range and automatically intercepts outputs that exceed the boundary. By restricting the model's output space from the source of generation at the clinical rule level, this invention overcomes the technical defects of conventional networks that lead to inaccurate lesion feature descriptions due to unordered searching of the entire vocabulary. This achieves the effect of improving the consistency and authenticity of medical image report content and ensuring the reliability of the final diagnostic conclusion.

[0048] 2. This invention extracts key diagnostic features from structured reports, concatenates them into query statements, and performs feature similarity retrieval in a medical literature vector database. After converting the returned academic literature into citation numbers and automatically embedding them into the corresponding fields in the report, it changes the single mode of conventional medical auxiliary diagnostic models that only output isolated conclusions. It supplements the clinical interpretation conclusions generated by the system with reliable external evidence-based medical data support, and achieves the effect of building a directly traceable chain of diagnostic evidence and improving the interpretability of the auxiliary diagnostic model.

[0049] 3. This invention extracts multi-scale visual features from non-contrast computed tomography (CT) images and clinical semantic features from electronic health record texts, and combines cross-attention mechanisms and gating fusion units to perform semantic alignment and adaptive fusion of the two modalities. This compensates for the lack of pathological details in non-contrast images due to the absence of contrast imaging, enriches the auxiliary diagnostic basis in the model coding process, and achieves the effect of improving the completeness of feature extraction of complex lesions and the reliability of structured reports in non-invasive screening scenarios. Attached Figure Description

[0050] Figure 1 This is a system architecture diagram of the present invention;

[0051] Figure 2 This is a flowchart of the method of the present invention;

[0052] Figure 3 This is a schematic diagram of the field hierarchy of the structured report template of the present invention;

[0053] Figure 4 This is a diagram of the constraint decoding mechanism of the present invention;

[0054] Figure 5 This is a schematic diagram of the evidence chain retrieval based on RAG according to the present invention;

[0055] Figure 6 This is a schematic diagram of the visual-language cross-validation and conflict warning system of the present invention. Detailed Implementation

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] See attached document Figure 1 , Figure 1This invention presents the overall architecture and data flow diagram of a medical image structured report generation system based on a field-constrained visual-language model, according to an embodiment of the present invention. The medical image structured report generation system based on a field-constrained visual-language model provided by the present invention is mainly applied to liver, gallbladder, and pancreatic cancer screening scenarios based on risk stratification funnels. The medical image structured report generation system integrates multiple functional modules, which work together to complete multimodal data processing, inference verification, and report generation tasks.

[0058] The data acquisition and preprocessing unit acquires non-contrast computed tomography (CT) images and corresponding electronic health record (EHR) data of the subjects to be screened. The unit performs window width and level adjustments, resampling normalization, and region of interest extraction on the CT images, and organizes the EHR data into a clinical timeline text in chronological order. The EHR data includes demographic information, lifestyle factors, body mass index, blood test results, and diagnostic records. The clinical timeline text is organized into a free text sequence based on timestamps, event types, and event content.

[0059] The electronic health record inference unit is connected to the output of the data acquisition and preprocessing unit. The electronic health record inference unit deploys a large language model trained with a self-evolutionary algorithm. It receives clinical timeline text, predicts the risk probability of the subject being screened developing liver, gallbladder, or pancreatic cancer within a preset future time period, and outputs the diagnostic inference process. The large language model uses the Qwen3-32B architecture, with an input context window configured with 3000 tags, fine-tuned using low-rank adaptation, and a temperature parameter set to 0.0 during inference.

[0060] The image analysis unit is connected to the output of the electronic health record inference unit. Internally, the image analysis unit contains a visual-language model based on the Qwen3-32B-VL architecture. The image analysis unit receives images of individuals predicted to be high-risk for screening, along with their corresponding electronic health record data. Under the constraints of predefined structured report template fields, the image analysis unit detects suspected hepatobiliary and pancreatic cancers and outputs a structured report. The structured report includes at least the following fields: lesion location, lesion size, margin morphology, density characteristics, preliminary diagnostic opinion, and differential diagnosis. Each field has a pre-configured range of valid values ​​or a set of candidate values.

[0061] The evidence-based retrieval unit connects to the output of the image analysis unit, extracting key diagnostic features from the structured report. It then performs an approximate nearest neighbor search within a pre-built medical literature vector database. This database includes only academic journal articles with an impact factor greater than 10 and uses an inverted document index and a product quantization structure to build a vector index. After automatically embedding the search results into the corresponding fields of the structured report as citation numbers, the evidence-based retrieval unit generates a complete report with a chain of evidence.

[0062] The first input of the cross-validation and conflict warning unit is connected to the output of the image analysis unit, and the second input is connected to the output of the electronic health record inference unit. The cross-validation and conflict warning unit acquires the imaging diagnostic conclusion and the textual inference conclusion, and performs a semantic consistency comparison between them. When the imaging diagnostic conclusion and the textual inference conclusion point to different disease categories or contain logical contradictions, the cross-validation and conflict warning unit generates a conflict warning signal.

[0063] The human-computer collaborative interaction terminal is connected to the outputs of the image analysis unit, the evidence-based retrieval unit, and the cross-validation and conflict warning unit. The interaction terminal aggregates and presents structured reports, evidence-based literature citations, and conflict warning signals to the user interface for physicians to review and revise.

[0064] See attached document Figure 2 , Figure 2 This is a flowchart of a method according to an embodiment of the present invention. The medical image structured report generation system performs the following steps based on the above architecture.

[0065] Step S1: Multimodal Data Acquisition and Preprocessing. Acquire the upper abdominal non-contrast computed tomography (CT) images and corresponding electronic health records of the subjects to be screened. Perform window width and level adjustment and resampling normalization on the non-contrast CT images. Extract the regions of interest containing abdominal organs based on the pre-trained model and convert them into a standardized three-dimensional voxel matrix. Organize the electronic health record data into a clinical timeline text sequence.

[0066] Step S2: Multimodal Feature Extraction and Preliminary Inference. The clinical timeline text is input into a large language model of electronic health records trained with a self-evolutionary algorithm to predict the cumulative incidence of hepatobiliary and pancreatic cancer in the subjects to be screened over the next five years, and outputs a textualized diagnostic inference process. A standardized three-dimensional voxel matrix is ​​input into a visual encoder to extract multi-scale visual feature vectors. The clinical timeline text is input into a text encoder to extract clinical semantic text feature vectors.

[0067] Step S3: Cross-modal feature alignment and fusion. A cross-attention mechanism is introduced to calculate the semantic correspondence between image regions and clinical text descriptions, aligning multi-scale visual feature vectors with clinical semantic text feature vectors. A gated fusion unit adaptively adjusts the ratio of multi-scale visual feature vectors to clinical semantic text feature vectors to generate joint multi-modal features.

[0068] Step S4: Loading the structured report template. The system calls a structured report template containing a preset set of fields. The structured report template covers the basic information area, clinical information area, imaging findings area, TNM staging area, and diagnostic opinion area. The T, N, and M components are determined according to established cancer staging standards and filled into the corresponding staging fields.

[0069] Step S5: Constraint Decoding to Generate a Structured Report. The joint multimodal features are input into the visual-language model decoder. The visual-language model decoder performs autoregressive decoding according to the field order of the structured report template. For each individual field, the decoding operation only performs probability sampling within a preset range of legal values ​​or candidate value sets. Calculation outputs with values ​​exceeding preset boundaries are automatically intercepted until all fields are generated and a structured report is output.

[0070] Step S6: Key Diagnostic Feature Extraction and Attribution Analysis. Extract key diagnostic features from the structured report, including disease entities, descriptions of key imaging features, and abnormal laboratory test results. Perform interpretability attribution analysis to calculate the local contribution of structured clinical features to risk prediction results, and output the positive and negative contributions of each clinical concept to patient risk assessment.

[0071] Step S7: Evidence-Based Literature Retrieval. Key diagnostic features are concatenated into a query statement, which is then encoded into a dense query vector. In the medical literature vector database, the inner product similarity score between the dense query vector and each document vector is calculated. The vectors are sorted in descending order based on their inner product similarity scores, and relevant academic literature meeting a preset threshold is returned.

[0072] Step S8: Automatic embedding of citation numbers. Relevant academic literature retrieved from the search is converted into citation numbers in a standardized format. These citation numbers are then inserted at the end of the diagnostic conclusion or symptom description fields within the structured report, and a reference list containing links to external source texts is generated at the bottom of the structured report.

[0073] Step S9: Human-Machine Collaborative Review and Output. The complete structured report with evidence-based evidence chains and confidence level markers is pushed to the physician's interactive terminal, and corresponding linked display views at the imaging level are provided simultaneously. The final version is generated after the operator confirms or revises it.

[0074] The medical image structured report generation system includes specific processes such as data preprocessing, feature extraction, model training, and attribution analysis when performing multimodal feature fusion and electronic health record inference mechanisms. The standardized implementation steps of each step are as follows.

[0075] Step S1: Multimodal data acquisition and preprocessing

[0076] This step mainly completes the standardized preprocessing of abdominal CT image data and electronic health record data, providing a high-quality data source for subsequent multimodal feature extraction. It is divided into two sub-steps.

[0077] Step S101: Acquisition and preprocessing of non-contrast computed tomography (CT) image data

[0078] Dedicated CT value display windows are set for abdominal organs such as the liver and pancreas to accurately adapt to the imaging characteristics of different organs. Specifically, the liver window level is set to 50-60 HU, and the window width to 150-200 HU; the pancreas window level is set to 40-50 HU, and the window width to 250-350 HU. After adjusting the window level and width, the data acquisition and preprocessing unit resamples the original image data to achieve spatial resolution resampling, uniformly resampling the inter-slice spacing to 1.0 mm × 1.0 mm × 1.0 mm. The data acquisition and preprocessing unit truncates the CT values, limiting the range to -200 to 300 HU, and performs Z-score normalization to eliminate interference from differences in the dimensions and values ​​of the image data. The data acquisition and preprocessing unit utilizes a 3D segmentation network to accurately extract regions of interest containing the liver, gallbladder, pancreas, and biliary system, outputting a standardized three-dimensional voxel matrix to complete the image data preprocessing.

[0079] Step S102: Electronic Health Record Data Preprocessing

[0080] The raw electronic health record data undergoes time-series standardization, organizing various medical event data in chronological order and standardizing the event record format. Each record includes three core elements: timestamp, event type, and event content. The event types comprehensively cover four categories of clinical data: laboratory test results, diagnostic records, medication records, and surgical records. By screening, sorting, and integrating the scattered and disordered raw health record data, standardized patient clinical timeline texts are generated, providing data support for subsequent text feature extraction and model inference.

[0081] Step S2: Multimodal Feature Extraction and Preliminary Inference

[0082] The clinical timeline text is input into a pre-trained electronic health record large language model to predict the cumulative incidence of hepatobiliary and pancreatic cancer in the next five years for the subjects to be screened, and outputs a textualized diagnostic reasoning process. A standardized three-dimensional voxel matrix is ​​input into a visual encoder to extract multi-scale visual feature vectors. The clinical timeline text is input into a text encoder to extract clinical semantic text feature vectors.

[0083] Step S2 involves the specific implementation of multimodal feature extraction and preliminary inference.

[0084] Step S201: Perform preliminary reasoning from electronic health records. Input the clinical timeline text into the trained electronic health record large language model to predict the risk probability of the subject to be screened developing liver, gallbladder, and pancreatic cancer within the next five years and output the diagnostic reasoning process.

[0085] Step S202: Perform feature extraction from unenhanced computed tomography (CT) image data. The normalized 3D voxel matrix output from step S101 is divided into blocks, resulting in 16×16×16 voxel-sized 3D image blocks. A linear mapping transformation is performed on the segmented 3D image blocks, and spatial location encoding is superimposed to preserve the spatial location information of the image. The processed image features are then input into a visual encoder.

[0086] The visual encoder was pre-trained on a large-scale computed tomography dataset containing various types of abdominal tumors. The pre-training tasks included 3D mask image modeling, organ anatomical structure recognition, and lesion region reconstruction. The visual encoder ultimately outputs a 768-dimensional multi-scale visual feature vector set to complete image modality feature extraction.

[0087] Step S203: Feature Extraction of Electronic Health Record Data

[0088] The standardized clinical timeline text generated in step S102 is input into a text encoder built on the Transformer architecture. The text encoder uses a pre-trained language model based on the Transformer architecture, preferably a Qwen3 series model or a medical language model with equivalent parameter levels that has been further pre-trained on biomedical corpora. For the sequence-type text features output by the text encoder, two aggregation methods are used to complete feature dimensionality reduction and integration: mean pooling can be used to aggregate global text features, or the feature vector corresponding to the [CLS] marker can be directly selected as the overall sequence representation. Finally, a unified 768-dimensional clinical semantic text feature vector is output, completing the text modality feature extraction.

[0089] Step S3: Cross-modal feature alignment and fusion

[0090] This step completes cross-modal alignment and adaptive fusion of image visual features and clinical text semantic features, establishes semantic associations between multimodal data, and generates joint multimodal features that combine image spatial features and clinical semantic features.

[0091] Step S301: Cross-modal semantic alignment

[0092] A cross-attention mechanism is introduced to achieve accurate semantic alignment in both modalities. By calculating the association weights between the spatial location of visual features and the clinical semantic text features, a semantic correspondence between image regions and clinical text is established. The formula for calculating the attention weights is as follows:

[0093]

[0094] In the formula, Image spatial location The corresponding visual feature vector, For the first in the text sequence The semantic feature vector of each tag, For attention scoring function, This is the final calculated cross-modal attention weight. Based on the generated attention weight matrix, the image spatial region corresponding to the clinical diagnostic description field is accurately located, completing the semantic alignment between the visual modality and the text modality.

[0095] Step S302: Adaptive Joint Fusion

[0096] The semantically aligned visual feature vectors and clinical semantic text feature vectors are simultaneously input into the gating fusion unit, which relies on the gating fusion mechanism to achieve adaptive fusion of multimodal information. The gating fusion unit adjusts the information ratio of the image visual modality and the clinical text modality in real time through learnable dynamic gating weights, adaptively adapting to the diagnostic feature requirements of different patients and different lesions. Finally, it outputs joint multimodal features that have completed cross-modal alignment and adaptive fusion, providing core feature support for the subsequent generation of structured reports.

[0097] See attached document Figure 3 , Figure 3 This is a schematic diagram of the field hierarchy of a structured report template according to an embodiment of the present invention. The medical image structured report generation system includes specific processes such as predefined template construction and constraint rule configuration when loading a structured report template.

[0098] Step S4 involves the specific implementation of loading the structured report template. Step S401: Construct a predefined structured report template. The structured report template is divided into a basic information area, a clinical information area, an imaging findings area, a TNM staging area, and a diagnostic opinion area. The basic information area includes fields for report number, patient ID, examination date, and examination type. The examination type is non-contrast computed tomography (CT) scan. The fields in the basic information area are automatically filled by the medical imaging structured report generation system. The clinical information area includes fields for chief complaint, past medical history, and a summary of relevant laboratory test results. The content in the clinical information area is automatically extracted and filled from electronic health record data. The imaging findings area includes sub-regions for liver, gallbladder and bile ducts, pancreas, abdominal cavity, and retroperitoneum. The liver sub-region includes fields for size and shape, liver parenchymal density, and space-occupying lesion. The space-occupying lesion field is further subdivided into sub-fields for location, size, number, margin, density characteristics, and relationship with blood vessels. The lesion location sub-field is located according to the liver segment and pancreatic head, body, and tail boundaries. The gallbladder and bile duct sub-region includes fields for gallbladder size, wall thickness, contents, common bile duct diameter, and whether there is dilation of intrahepatic or extrahepatic bile ducts. The pancreas sub-region includes fields for pancreatic size and shape, whether there is dilation of pancreatic ducts, and whether there are space-occupying lesions. The abdominal cavity and retroperitoneal sub-region includes fields for the presence or absence of ascites and enlarged lymph nodes. The diagnostic opinion area includes fields for preliminary diagnostic opinions ranked by probability, differential diagnoses, and recommended follow-up plans.

[0099] Step S402: Configure field constraint rules and automatic staging logic. For independent fields within the report template, preset corresponding legal value ranges or candidate value sets to constrain the subsequent decoding output space of the model. Discrete fields use candidate value set constraints; the candidate value set for the liver morphology field is configured as normal, diffuse lesion, localized lesion, and postoperative changes; the candidate value set for the lesion edge morphology field is configured as clear, blurred, lobulated, and spiculated. Numerical fields use legal value range constraints; the legal numerical range for the common bile duct diameter field is configured as 1 to 20 mm. In the TNM staging area, the medical image structured report generation system obtains lesion measurement information output by the visual-language model. Lesion measurement information includes maximum diameter, number, vascular invasion signs, lymph node and distant metastasis. The medical image structured report generation system refers to the TNM staging criteria for hepatobiliary and pancreatic tumors in the 8th edition of the American Joint Committee on Cancer Staging Manual, determines the T, N, and M components, and combines them to generate the final clinical staging conclusion to fill the corresponding fields.

[0100] See attached document Figure 4 , Figure 4 This is a diagram illustrating a constraint decoding mechanism according to an embodiment of the present invention. The medical image structured report generation system, when performing constraint decoding to generate a structured report, includes specific processes such as model initialization, probability sampling decoding, and confidence calculation.

[0101] Step S5 relates to the specific implementation of constraint decoding to generate a structured report. Step S501: Initialize the visual-language model sequence decoder. Input the joint multimodal features, which are the output of cross-modal feature alignment and fusion, into the visual-language model decoder. The visual-language model decoder is based on the Transformer architecture and contains 12 decoding layers. Each decoding layer integrates a mask self-attention module, an encoder-decoder cross-attention module, and a feedforward neural network. The visual-language model decoder uses the joint multimodal features as the initial hidden state and combines them with the previously generated field content as the context state.

[0102] Step S502: Perform field-by-field constrained probability sampling and autoregressive decoding. The visual-language model decoder performs autoregressive decoding field by field according to the field order of the structured report template. For discrete fields, the output layer dimension of the visual-language model decoder is dynamically adjusted to the size of the set of legal candidate values ​​for the current generated field, for example, the size of the set of legal candidate values ​​is 3. The output value of the visual-language model decoder is activated by the Softmax function and outputs the probability distribution of each legal candidate value. The candidate value corresponding to the highest probability is selected as the generation result of the current generated field. For numerical fields, the output of the visual-language model decoder is mapped to a continuous value within a preset numerical range by the Sigmoid function. For textual fields, the visual-language model decoder outputs text descriptions using a word-by-word generation method, allowing a certain degree of free text supplementation. The decoding operation is strictly limited to probability sampling within the legal value range or candidate value set corresponding to the current generated field. Outputs with values ​​exceeding the preset range are automatically intercepted by the medical image structured report generation system to eliminate the generation of non-predefined values. The visual-language model decoder performs autoregressive decoding operations cyclically until all preset fields of the structured report template are generated.

[0103] Step S503: Label field-level confidence scores. The visual-language model decoder simultaneously calculates the confidence score of the target field output while generating the target field value. The medical image structured report generation system compares the confidence scores of all target fields with a preset threshold. Target fields with confidence scores below the preset threshold are marked with a yellow background and highlighted in the final structured report, prompting physicians to pay close attention to and review the target field content. The preset threshold value is configured to 0.7.

[0104] Step S6 involves the specific implementation of key diagnostic feature extraction and interpretability attribution analysis.

[0105] Step S601: Extract key diagnostic features from the structured report. Key diagnostic features include the disease entity in the diagnostic conclusion, descriptions of key imaging features, and abnormal laboratory test results.

[0106] Step S602: Perform interpretability attribution analysis. Since the input to the electronic health record inference big language model is a long clinical timeline text sequence, the electronic health record inference big language model cannot directly perform attribution analysis on each clinical feature in the original input text using the traditional SHAP method. Therefore, this step adopts an alternative model strategy to achieve interpretability attribution. Structured clinical feature vectors are extracted from the original clinical timeline text. Structured features include: continuous indicators, such as the most recent value, maximum value, number of abnormalities, proportion of abnormalities, and time trend of each laboratory test indicator; discrete evidence, such as the existence of each diagnostic record, number of mentions, and the time interval between the most recent mention and the current time; and composite indicators, such as body mass index and comorbidity index. Using the structured clinical feature vectors as model input and the risk margin output by the original big language model as the regression training objective, a LightGBM regression alternative model is constructed and trained.

[0107]

[0108] In the formula, For the LightGBM regression model, This is the risk margin predicted by the alternative model.

[0109] By adjusting the tree depth, number of leaf nodes, and learning rate hyperparameters of the LightGBM regression substitution model, the fitting accuracy of the LightGBM regression substitution model on the validation set is ensured. Greater than 0.85. The TreeSHAP algorithm is used to quantify the local contribution of each structured feature to the prediction result:

[0110]

[0111] In the formula, For the set of all features, To use only a subset of features Alternative model predictions at that time This is the local contribution value (SHAP value) for the corresponding feature.

[0112]

[0113] In the formula, This is a clinical concept. The set of all derived feature indexes, The overall attribution contribution value for the corresponding clinical concept.

[0114] The contribution values ​​of derived features belonging to the same clinical indicator and the same diagnostic concept are accumulated and fused to output the positive and negative contribution values ​​of various clinical concepts to the judgment of patients' disease risk, forming a traceable individualized evidence chain.

[0115] See attached documentFigure 5 , Figure 5 This is a schematic diagram of evidence-based chain retrieval based on RAG according to an embodiment of the present invention. The medical image structured report generation system, when performing evidence-based literature retrieval and automatic citation embedding, includes specific processes such as high-quality medical literature knowledge base construction, feature extraction, vector retrieval, and citation generation.

[0116] Step S7 involves the specific implementation of evidence-based literature retrieval.

[0117] Step S701: Perform medical literature data collection and quality screening. Metadata for academic papers in the biomedical field is acquired in batches through the public application programming interface (API) of the biomedical literature database. Metadata includes PubMed identifier, PubMed Central identifier, title, abstract, introduction, authors, journal name, and publication year. The medical image structured report generation system performs quality screening of academic papers based on the journal's impact factor. The system retrieves the latest impact factor corresponding to the journal name of the academic paper and only retains academic papers with an impact factor greater than a preset threshold for inclusion in the knowledge base. The preset impact factor threshold is configured to be 10. Academic papers whose impact factor cannot be obtained or does not reach the preset threshold are not included in the knowledge base.

[0118] Step S702: Perform literature corpus cleaning and vector database construction. For academic papers that have passed quality screening, extract the title, abstract, and introduction. For academic papers missing abstracts or introductions, mark the data fields corresponding to the missing content as empty. For academic papers that are too long, retain the first 20,000 characters of the text to avoid redundant text interfering with the semantic representation. Concatenate the abstract and introduction texts of the academic papers and input them into a pre-trained semantic embedding model to generate a fixed-dimensional dense vector representation. The pre-trained semantic embedding model uses the Qwen3-Embedding-8B model. The dense vector representation is associated and stored with the corresponding PubMed identifier, title, journal name, and publication year metadata to construct a literature vector database. In the literature vector database, FAISS is used as the vector retrieval engine to build an inverted file index and a product quantization index to support approximate nearest neighbor retrieval.

[0119] Step S703: Extract key diagnostic features and encode query vectors. Extract key diagnostic features from the generated structured report. Extraction rules cover identifying core disease entities, key imaging features, and associated laboratory abnormalities in the diagnostic opinion area. Core disease entities include hepatocellular carcinoma and pancreatic ductal adenocarcinoma. Key imaging features include portal vein tumor thrombus and bile duct dilatation. Laboratory abnormalities include CA19-9 greater than 100 U / mL and AFP greater than 400 ng / mL. Concatenate all extracted core disease entities, key imaging features, and laboratory abnormalities into a natural language query statement, for example, generating a query statement with the content "left lobe of liver space-occupying lesion, indistinct margins, portal vein tumor thrombus formation, significantly elevated AFP, suspected hepatocellular carcinoma". Input the natural language query statement into the aforementioned semantic embedding model, and encode and output the query vector. . The dimension parameter for the embedded vector. Query vector. It maps to the same semantic space as the document vectors in the document vector database.

[0120] Step S704: Perform similarity retrieval and related document ranking. Transfer the query vector... Approximate nearest neighbor retrieval is performed in a document vector database. Approximate nearest neighbor retrieval uses inner product similarity as a metric to calculate the query vector. The system calculates similarity scores between the document vectors and all document vectors in the document vector database. The search results are then sorted in descending order of similarity score, and the top K documents with the highest similarity are extracted. K is set to 5. Documents with similarity scores below a preset similarity threshold are filtered out by the medical image structured report generation system. The preset similarity threshold is set to 0.65. For academic documents that meet the threshold, the PubMed identifier, title, journal name, and publication year are extracted to form a citation candidate dataset.

[0121] Step S8 relates to a specific implementation of the automatic embedding of reference numbers.

[0122] Step S801: Perform citation number generation and document end structure construction. Convert relevant literature from the citation candidate dataset into standardized citation numbers according to the medical correspondence between relevant literature and the structured report content. Standardized citation numbers are configured as numerical superscript formats such as [1], [2], or specific bracket formats including the PubMed identifier such as (PubMed:12345678). A structured reference list is generated at the end of the structured report text. The reference list contains complete bibliographic information for each cited article and a link to the original database for accessing the full text.

[0123] Step S802: Perform the labeling embedding operation. The medical image structured report generation system locates the specific diagnostic conclusion field or key sign description field in the structured report that corresponds to the source of the natural language query. The medical image structured report generation system inserts standardized citation labels after the text content of the specific diagnostic conclusion field or key sign description field, constructing an evidence linking the diagnostic conclusion to external authoritative literature within the structured report.

[0124] Step S9 involves the specific implementation of human-machine collaborative review and output, and multi-source diagnostic consistency detection operation is performed before or during the execution of step S9.

[0125] Step S901: Perform visual-language cross-validation and conflict warning. The medical image structured report generation system acquires the imaging diagnostic conclusion output by the visual-language model and simultaneously acquires the text reasoning conclusion output by the large language model reasoning module based on electronic health record text. The medical image structured report generation system performs semantic similarity calculation based on medical ontology on the imaging diagnostic conclusion and the text reasoning conclusion. When both the imaging diagnostic conclusion and the text reasoning conclusion point to the same disease entity, such as both pointing to hepatocellular carcinoma, the medical image structured report generation system determines that the conclusions are consistent and does not generate a warning signal. When the imaging diagnostic conclusion and the text reasoning conclusion point to different disease entities, such as the imaging diagnostic conclusion indicating hepatocellular carcinoma while the text reasoning conclusion indicates pancreatic cancer; or when the imaging diagnostic conclusion and the text reasoning conclusion are in opposite directions, such as the visual-language model determining the imaging diagnostic conclusion as suspicious for malignancy while the text reasoning conclusion is highly likely to be benign, the medical image structured report generation system determines that the conclusions conflict.

[0126] Step S902: Output a conflict warning signal. When a conflict is detected in the conclusions, the medical image structured report generation system generates a conflict warning signal at the top of the report interface of the human-computer collaborative interaction terminal in the form of a banner or pop-up window. The conflict warning signal includes a conflict type identifier indicating that the image and clinical conclusions are inconsistent, the specific content of both conflicting parties, and a suggested operation text that the physician should focus on reviewing the correspondence between the image and the medical record. The conflict warning signal continues to be displayed during the report review period until the operator confirms that it has been read or performs a manual closing operation.

[0127] Step S903: Perform multi-dimensional tabular display of the structured report. The medical imaging structured report generation system displays the structured report in tabular form on the main interface of the human-computer collaborative interaction terminal. Each row of the table corresponds to an independent field in the structured report template, and the presented content includes the field name, field value, confidence level, and evidence citation number. Fields with confidence levels below a preset threshold are highlighted with a yellow background. The comprehensive diagnostic credibility score is displayed in the upper right corner of the report interface. The comprehensive diagnostic credibility score is calculated by the medical imaging structured report generation system by combining the confidence levels of each field and the intermodal consistency parameter.

[0128] Step S904: Perform interactive visualization of images and reports. The human-computer collaborative interaction terminal integrates a medical image browser, supporting simultaneous viewing of original computed tomography images, organ and lesion segmentation masks, and 3D reconstructed views on the report review interface. When the operator clicks on a specific field, such as the lesion location field, in the report form, the medical image structured report generation system automatically highlights the corresponding lesion's two-dimensional or three-dimensional region with a colored outline in the corresponding image layer of the medical image browser, based on the alignment relationship between the spatial coordinates of the lesion segmentation mask and the physical coordinate system of the image. The medical image structured report generation system supports multi-planar reconstruction linkage display. When the operator selects a specific lesion in the report, the medical image browser automatically switches to a synchronous display mode of transverse, coronal, and sagittal planes, marking the same lesion with a colored area in each of the transverse, coronal, and sagittal planes.

[0129] Step S905: Present the Evidence-Based Literature Tracing Panel. When the operator clicks on a citation number in the structured report, the medical imaging structured report generation system generates and displays a literature details panel on the side of the interface or in a pop-up window. The literature details panel displays the corresponding academic paper's title, author list, journal name, publication year, full abstract, and a hyperlink to the original database. For academic papers belonging to the open access category, a PDF preview function is embedded within the literature details panel. The bottom of the literature details panel displays a similarity score between the academic paper and the diagnostic features of the current case, used to help determine the applicability of the literature evidence.

[0130] Step S906: Perform physician revision and feedback data collection. The human-computer collaborative interaction terminal provides a revision interface, allowing operators to directly modify any field value in the structured report or add additional descriptive text. The medical image structured report generation system uses revision trace tracking technology to record detailed information about the modification operations. This detailed information includes the content before modification, the content after modification, the modifier's identifier, and the modification timestamp. The medical image structured report generation system periodically exports revision records. After anonymizing the revision records, they are converted into training data for continuous model optimization and iteration.

[0131] See attached document Figure 1 , Figure 1 This is a system architecture and data flow diagram according to an embodiment of the present invention. The medical image structured report generation system based on a field-constrained visual-language model includes a first module, a second module, a third module, a fourth module, a fifth module, and a sixth module.

[0132] The first module, the data acquisition and preprocessing unit, is used to acquire non-contrast computed tomography (CT) image data and corresponding electronic health record data of the subjects to be screened. The data acquisition and preprocessing unit performs window width and level adjustments, resampling normalization, and region of interest extraction on the non-contrast CT image data. The data acquisition and preprocessing unit organizes the electronic health record data into a clinical timeline text in chronological order.

[0133] The second module, the electronic health record inference unit, connects to the output of the data acquisition and preprocessing unit. The electronic health record inference unit contains a large language model trained using a self-evolutionary algorithm. The large language model adopts the Qwen3-32B architecture, with an input context window configured with 3000 tags, fine-tuned using low-rank adaptation, and a temperature parameter set to 0.0 during inference. The electronic health record inference unit receives clinical timeline text, predicts the risk probability of the subject to be screened developing liver, gallbladder, and pancreatic cancer within a preset future time period, and outputs the diagnostic inference process. The self-evolutionary algorithm training process involved in the electronic health record inference unit includes: generating initial inference samples using a general large language model; filtering the initial inference samples by an LLM evaluator based on answer correctness, inference-answer consistency, and inference authenticity to obtain a seed dataset; fine-tuning the base model based on the seed dataset to obtain the initial inference model; iteratively generating new samples and selecting new samples to add to the training set.

[0134] The third module, the image analysis unit, connects to the output of the electronic health record inference unit. The image analysis unit includes a visual-language model. This model takes non-contrast computed tomography (CT) image data and the corresponding electronic health record trajectory as input. The image analysis unit receives CT image data of individuals predicted to be at high risk for screening, along with their corresponding electronic health record data. The image analysis unit detects suspected hepatobiliary and pancreatic cancers and outputs organ segmentation masks, tumor segmentation masks, cancer type differential diagnoses, and structured reports.

[0135] The fourth module, the evidence-based retrieval unit, connects to the output of the image analysis unit. This unit extracts key diagnostic features from the structured report. It then performs an approximate nearest neighbor search in the medical literature vector database. This database only includes journal articles with an impact factor greater than 10 and uses FAISS as its search engine to build a vector index. Finally, the unit embeds the search results into the structured report using citation numbers.

[0136] The fifth module, the human-computer collaborative interaction terminal, connects to the outputs of the image analysis unit, the evidence-based retrieval unit, and the cross-validation and conflict warning unit. The human-computer collaborative interaction terminal aggregates and presents structured reports and evidence-based literature citations to the physician's interface.

[0137] The sixth module, the cross-validation and conflict warning unit, connects its first input to the output of the image analysis unit to obtain the imaging diagnostic conclusion. Its second input connects to the output of the external electronic health record inference module to obtain the text inference conclusion. The cross-validation and conflict warning unit performs a consistency comparison between the imaging diagnostic conclusion and the text inference conclusion. When the imaging diagnostic conclusion and the text inference conclusion are inconsistent, the cross-validation and conflict warning unit generates a conflict warning signal and pushes it to the human-machine collaborative interaction terminal, which then presents the conflict warning signal to the physician's interface.

[0138] The medical image structured report generation system is applied to a specific clinical scenario for large-scale population cancer screening of the liver, gallbladder, and pancreas. Subjects undergo unenhanced CT scans of the upper abdomen at health check-up centers or primary healthcare institutions. The unenhanced CT scans of the upper abdomen employ a low-dose protocol with an effective radiation dose of 2 to 3 millisieverts. The examination does not require the injection of iodine-containing contrast agents, making it suitable for patients with renal insufficiency or those allergic to contrast agents. The resulting CT images are automatically transmitted to the data acquisition and preprocessing unit via the DICOM protocol. During this process, the system utilizes routine clinical indicators to perform rule-based risk pre-screening to identify high-risk individuals. Routine clinical indicators include 13 blood biomarkers, 4 types of diagnostic records, and 1 composite lifestyle factor. The 13 blood biomarkers include glycated hemoglobin, total bilirubin, total bile acids, aspartate aminotransferase, alanine aminotransferase, gamma-glutamyl transferase, alkaline phosphatase, amylase, prothrombin time, D-dimer, creatinine, fasting blood glucose, and serum sodium. Four types of diagnostic records were included: diabetes, cirrhosis, pancreatitis, and jaundice. The composite lifestyle factor was defined as a history of both smoking and alcohol consumption. Subjects with abnormalities in any of these indicators were marked as high-risk by the system. Subsequently, the medical institution's information system pushed the subject's basic information, past medical history, blood routine test results, and biochemical test data in structured text format to the data acquisition and preprocessing unit, forming multimodal input data. The medical imaging structured report generation system generated a structured report with evidence-based chains and visual markers. The entire process of image analysis and report generation for a single subject was completed within 2 minutes. The structured report was automatically pushed to a remote imaging diagnostic center or a higher-level hospital. Radiologists reviewed the structured report, confirmed or corrected the content of each field, and signed the final report online through a human-computer collaborative interactive terminal. For cases indicated as high-risk by the system or triggering cross-validation conflict warnings, the medical imaging structured report generation system marked the corresponding cases as priority review cases to ensure timely handling of high-risk cases.

[0139] The specific implementation method of the offline training process of the self-evolutionary algorithm of the electronic health record large language model for the electronic health record inference unit in the second module mentioned above is as follows:

[0140] Labeled electronic health record data is acquired, with each data point containing the patient's longitudinal clinical timeline text and corresponding hepatobiliary and pancreatic cancer labels. All data constitute the training dataset. An LLM evaluator with three core judgment capabilities is constructed based on this training dataset. The LLM evaluator adopts the Transformer model architecture consistent with the basic large language model. The first measure of the LLM evaluator is answer correctness judgment: verifying the consistency between the model's predicted answer and the true label. The evaluation passes when the model's predicted answer matches the true label, and fails when they do not. The second measure is reasoning-answer consistency judgment: analyzing the logical chain of the reasoning process. The evaluation passes when the final conclusion of the reasoning chain effectively supports the model's predicted answer, and fails when it does not. The third measure is reasoning authenticity judgment: verifying whether the clinical evidence cited in the reasoning process is completely faithful to the input data. The evaluation passes when there is no fabricated content, and fails when fabricated content exists. Only when all three judgments pass can the generated reasoning sample be retained.

[0141] Mathematical definitions of the three criteria:

[0142] Correctness of answer:

[0143]

[0144] Reasoning-answer consistency:

[0145]

[0146] Inference accuracy:

[0147]

[0148] Only when = = When =1, the sample is retained.

[0149] The general-purpose large language model is invoked to generate initial inference samples. Training samples are input into the general-purpose large language model, which automatically generates the corresponding inference process and predicted answer. After the initial inference samples undergo three-stage filtering by the LLM evaluator, all high-quality samples that pass the validation are included in the seed dataset. Supervised fine-tuning of the basic large language model is performed based on the seed dataset, iteratively optimizing to obtain the initial inference model.

[0150]

[0151] In the formula, λ is the inference loss weight coefficient, which is used to balance the answer prediction loss and the inference generation loss;

[0152] Initiate the model self-evolution iterative process. Using the model trained in the previous iteration, generate new inference results from the training samples. These newly generated inference samples undergo quality verification according to the unified screening criteria of the LLM evaluator. High-quality samples that pass this verification are aggregated to construct the current round's dataset. Fine-tune the iterative model using the current round's dataset to obtain the updated model. Continue iterating the self-evolution process until the area under the curve (AUC) on the validation set no longer improves, or the number of iterations reaches a preset maximum value (5). At this point, terminate the iteration process and output the final electronic health record large language model.

[0153] By implementing the above self-evolutionary training mechanism, seed initialization solves the cold start problem; the self-evolutionary loop enables the electronic health record large language model to continuously learn from high-quality inference samples to improve its inference ability; three verification mechanisms ensure the addition of high-quality samples and avoid noise accumulation; and the iteration termination condition ensures that the model stops training after performance convergence, avoiding overfitting.

[0154] For the specific software and hardware implementation mechanisms of operations such as multimodal feature extraction, attention mechanism calculation, large language model fine-tuning, and autoregressive decoding, those skilled in the art can use existing conventional methods. The underlying logic of operations such as multimodal feature extraction, attention mechanism calculation, large language model fine-tuning, and autoregressive decoding is a well-known technology in this field and will not be elaborated here.

[0155] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating medical image reports using a field-constrained visual language model, characterized in that, Includes the following steps: Acquire non-contrast computed tomography (CT) images and corresponding electronic health record data of the subjects to be screened. Preprocess the non-contrast computed tomography images to convert them into a standardized three-dimensional voxel matrix and organize the electronic health record data into a clinical timeline text sequence. Multi-scale visual feature vectors of standardized three-dimensional voxel matrices and clinical semantic text feature vectors of clinical timeline text sequences are extracted separately. The multi-scale visual feature vectors and clinical semantic text feature vectors are then aligned and fused to generate joint multimodal features. The joint multimodal features are input into the visual-language model decoder. The visual-language model decoder performs autoregressive decoding according to the field order of the pre-called structured report template. The structured report template contains a preset set of fields, and each independent field is configured with a legal value range or a set of candidate values. When performing decoding operations for each independent field, the visual-language model decoder only performs probability sampling within the corresponding legal value range or set of candidate values ​​to generate a structured report. Key diagnostic features are extracted from the structured report and combined into a query statement. The query statement is encoded into a query vector and a similarity retrieval is performed in a medical literature vector database. The relevant academic literature returned by the retrieval is converted into citation numbers and automatically embedded into the corresponding fields in the structured report. Finally, a complete report with a chain of evidence-based data is output.

2. The method for generating medical image reports using a field-constrained visual language model according to claim 1, characterized in that, Aligning and fusing multi-scale visual feature vectors with clinical semantic text feature vectors to generate joint multimodal features, specifically including: A cross-attention mechanism is introduced to calculate the semantic correspondence between image regions and clinical text descriptions, obtain the attention weight matrix, and use the attention weight matrix to locate the image spatial region corresponding to the diagnostic description field to complete semantic alignment. The multi-scale visual feature vectors with semantic alignment and the clinical semantic text feature vectors are input into the gating fusion unit. The gating fusion unit dynamically adjusts the proportion of the two types of modal information through learnable gating weights, and outputs joint multimodal features after cross-modal alignment and adaptive fusion.

3. The method for generating medical image reports using a field-constrained visual language model according to claim 1, characterized in that, When decoding each individual field, the visual-language model decoder performs probability sampling only within the corresponding legal value range or candidate value set, generating a structured report, specifically including: For discrete fields, the output layer dimension of the visual-language model decoder is dynamically adjusted to the size of the set of legal candidate values ​​for the current generated field. The output values ​​are processed by the activation function to output the probability distribution of each legal candidate value. The candidate value with the highest probability is selected as the generated result of the current generated field. For numerical fields, the visual-language model decoder outputs continuous values ​​that are processed by a mapping function to a preset numerical range; For outputs with values ​​exceeding the preset range, an automatic interception operation is performed to eliminate the generation of non-predefined values; and when generating the target field value, the confidence score of the target field output is calculated simultaneously. In the final structured report, target fields with confidence scores below the preset threshold are highlighted with a color mark attribute.

4. The method for generating medical image reports using a field-constrained visual language model according to claim 1, characterized in that, The Medical Literature Vector Database only includes academic journal articles with an impact factor greater than a preset impact factor threshold, and uses an inverted file index and a product quantization structure to build a vector index in the Medical Literature Vector Database. The similarity retrieval in the medical literature vector database includes: using inner product similarity as a metric in the medical literature vector database, calculating the inner product similarity score between the query vector and each document vector; sorting the documents in descending order based on the inner product similarity score, filtering out documents with inner product similarity scores below a preset similarity threshold, and extracting relevant academic documents that meet the threshold conditions.

5. The method for generating medical image reports using a field-constrained visual language model according to claim 1, characterized in that, The method also includes performing visual-language cross-validation and conflict warning: Input the clinical timeline text sequence into the trained electronic health record big language model and output the text inference conclusion; Obtain the imaging diagnostic conclusions output by the visual-language model decoder, and perform semantic similarity calculation based on medical ontology between the imaging diagnostic conclusions and the text reasoning conclusions; When the imaging diagnosis conclusion and the text reasoning conclusion point to different disease entities, or when the imaging diagnosis conclusion and the text reasoning conclusion are in opposite directions, a conflict is determined, and a conflict warning signal containing the specific content of both conflicting parties is generated at the top of the report interface of the human-computer collaborative interaction terminal.

6. The method for generating medical image reports using a field-constrained visual language model according to claim 5, characterized in that, The trained electronic health record large language model was obtained through an offline training process using a self-evolutionary algorithm. This offline training process included: A large language model evaluator with three core judgment capabilities is constructed based on a training dataset containing patients' longitudinal clinical timeline text and corresponding labels. The three core judgment capabilities include: judging the correctness of the answer by verifying the consistency between the model's predicted answer and the actual label; judging the consistency of the reasoning-answer by analyzing the logical chain of the reasoning process to confirm whether it supports the model's predicted answer; and judging the authenticity of the reasoning by checking whether the clinical evidence cited in the reasoning process is completely faithful to the input data. The general large language model is called to infer the training samples to generate the corresponding reasoning process and predicted answer, which serve as the initial reasoning samples. After the large language model evaluator completes three judgments and screenings, all high-quality samples that pass the verification are included in the seed dataset. Based on the seed dataset, supervised fine-tuning of the basic large language model is carried out to obtain the initial inference model. The initial inference model is used as the first iteration model, and in subsequent iterations, the model obtained from the previous iteration is used as the iteration model for the current iteration. New inference results are generated for the training samples, and after the large language model evaluator completes three judgments and screening, the high-quality samples that pass the validation are summarized to construct the current iteration dataset for fine-tuning the iteration model until the area under the curve of the model on the validation set no longer improves or the iteration reaches the preset maximum value. Finally, the electronic health record large language model is output.

7. The method for generating medical image reports using a field-constrained visual language model according to claim 5, characterized in that, The method also includes performing interpretability attribution analysis based on a pre-trained large language model of electronic health records: Structured clinical feature vectors are extracted from the original clinical timeline text. The structured clinical feature vectors are used as the model input, and the risk margin output by the original large language model is used as the regression training target to construct and train a regression alternative model. By adjusting the tree depth, number of leaf nodes, and learning rate hyperparameter of the regression substitution model, we ensure that the fitting accuracy of the regression substitution model on the validation set is greater than a preset threshold. The attribution algorithm is used to quantify the local contribution of each structured feature to the prediction result. The local contribution values ​​of derived features belonging to the same clinical indicator and the same diagnostic concept are accumulated and fused to output the positive and negative contribution values ​​of various clinical concepts to the judgment of the patient's disease risk.

8. The method for generating medical image reports using a field-constrained visual language model according to claim 1, characterized in that, Before acquiring the non-contrast computed tomography (CT) images and corresponding electronic health record data of the subjects to be screened, the method also includes: Rule-based risk screening was performed using routine clinical indicators to identify high-risk populations. The routine clinical indicators included 13 blood biomarkers, 4 types of diagnostic records, and 1 composite lifestyle factor. Thirteen blood biomarkers were included, such as glycated hemoglobin, total bilirubin, total bile acids, aspartate aminotransferase, alanine aminotransferase, gamma-glutamyl transferase, alkaline phosphatase, amylase, prothrombin time, D-dimer, creatinine, fasting blood glucose, and serum sodium. Four types of diagnostic records were included, such as diabetes mellitus, cirrhosis, pancreatitis, and jaundice. The composite lifestyle factor was defined as a history of both smoking and alcohol consumption. If any indicator was abnormal, the examinee was identified as a high-risk group, and the steps of obtaining non-contrast computed tomography (CT) images and corresponding electronic health record data for the high-risk group were performed.

9. The method for generating medical image reports using a field-constrained visual language model according to claim 1, characterized in that, The structured report template covers the basic information area, clinical information area, imaging findings area, TNM staging area, and diagnostic opinion area; For the TNM staging region, the lesion measurement information output by the visual-language model, including the maximum diameter, number, vascular invasion signs, lymph node and distant metastasis, is obtained. Each component is judged according to the established cancer staging standards, and the final clinical staging conclusion is generated and filled into the corresponding staging field.

10. A medical image report generation system based on a field-constrained visual language model, characterized in that, Include: The data acquisition and preprocessing unit is used to acquire non-contrast computed tomography (CT) image data and corresponding electronic health record data of the subjects to be screened, preprocess the non-contrast computed tomography image data, convert it into a standardized three-dimensional voxel matrix, and organize the electronic health record data into a clinical timeline text sequence. The electronic health record reasoning unit is used to receive clinical timeline text sequences, predict the risk probability of the subject to be screened developing liver, gallbladder and pancreatic cancer within a preset time period, and output the diagnostic reasoning process. The image analysis unit receives non-enhanced computed tomography (CT) image data and electronic health record data corresponding to the high-risk prediction results output by the electronic health record inference unit. It extracts multi-scale visual feature vectors of the standardized three-dimensional voxel matrix and clinical semantic text feature vectors of the clinical timeline text sequence, respectively. The multi-scale visual feature vectors and clinical semantic text feature vectors are aligned and fused to generate joint multimodal features. The image analysis unit contains a vision-language model decoder, which takes the joint multimodal features as input and performs autoregressive decoding according to the field order of the pre-called structured report template. When decoding each independent field, probability sampling is performed only within the corresponding legal value range or candidate value set to generate a structured report. The evidence-based retrieval unit is used to extract key diagnostic features from the structured report, splice and combine them into a query statement, encode the query statement into a query vector and perform a similarity retrieval in the medical literature vector database, convert the relevant academic literature returned by the retrieval into citation numbers and automatically embed them into the corresponding fields in the structured report, and then output a complete report with an evidence-based evidence chain. The cross-validation and conflict warning unit is used to obtain the imaging diagnostic conclusions output by the image analysis unit and the text reasoning conclusions output by the electronic health record reasoning unit, compare the semantic consistency of the imaging diagnostic conclusions and the text reasoning conclusions, and generate a conflict warning signal when a logical contradiction occurs. The human-computer collaborative interaction terminal is used to aggregate and present structured reports, evidence-based literature citations, and conflict warning signals to the operation interface.

Images