A medical image CT report auxiliary generation method and system based on a multi-modal large model and a local intelligent agent

Abstract

The present application relates to the technical field of medical image processing, in particular to a medical image CT report auxiliary generation method and system based on a multi-modal large model and a local intelligent agent. The technical scheme of the present application relates to artificial intelligence and big data processing technology in new generation information technology, and digital medical treatment and medical image analysis technology in the biological pharmaceutical industry.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method and system for assisting in the generation of medical image CT reports based on a multimodal large model and a local intelligent agent. The technical solution of this invention relates to artificial intelligence and big data processing technologies in next-generation information technology, as well as digital healthcare and medical image analysis technologies in the biopharmaceutical industry. Background Technology

[0002] With the development of deep learning and large multimodal model (LMM) technologies, the demand for natural language generation technology in the field of medical image information management is increasing. The automatic generation of CT diagnostic reports is a typical application at the intersection of natural language processing and medical image information management. Its core objective is to transform multimodal input (CT image sequences + search examples) into structured natural language diagnostic reports, enabling semantic understanding, natural language querying, and intelligent question answering of medical image information. However, the token consumption cost of remotely calling large model APIs has become a core bottleneck restricting its large-scale application. How to minimize remote token consumption and ensure the quality of natural language generation in medical image information management scenarios has become a key technical issue.

[0003] However, existing technologies have the following drawbacks: Pain Point 1: Excessive Token Consumption by Remote Large Models. The lack of a local agent takeover mechanism leads to unnecessary consumption of remote tokens in standardized natural language post-processing steps. Lung CT image sequences are long and have many slices (typically 200-500 images). Directly inputting tokens into remote multimodal large models can easily exceed the context window limit (e.g., 8K-32K tokens), resulting in high inference costs, low efficiency, and even loss of key diagnostic information due to truncation. More seriously, existing report generation methods also delegate natural language post-processing steps such as sign verification, structured formatting, terminology standardization, and consistency verification to remote large models through multi-turn dialogues. Each multi-turn interaction generates additional token consumption. However, these steps are essentially rule-driven standardized natural language processing, which can be completely taken over by the local agent based on token standardization rules, without consuming remote tokens.

[0004] Pain Point Two: General retrieval methods struggle to adapt to the characteristics of private domain data, impacting the retrieval quality of medical image information management. Existing retrieval methods typically rely on general visual similarity (such as CLIP feature cosine similarity). However, general visual encoders cannot capture fine-grained pathological feature differences in lung CT images (such as ground-glass nodules versus solid nodules, microspiculations versus fibrous strands). This results in retrieved examples being pathologically irrelevant to the queried CT images, failing to provide effective contextual representation for multimodal large-scale models and reducing the semantic accuracy and clinical consistency of natural language generated reports. Furthermore, existing methods lack private domain data adaptation mechanisms, failing to transform local hospital image-report data into a searchable example library, thus hindering the efficient utilization of private domain data in medical image information management.

[0005] Pain Point 3: Reports generated by natural language lack completeness assurance and standardized processing, resulting in unstable quality of medical imaging information management. Existing report generation methods are mostly "one-time generation," lacking mechanisms for verifying and supplementing the completeness of lung-specific signs (such as lobulation, spiculation, and pleural traction), easily leading to the omission of key sign descriptions. Furthermore, the lack of natural language post-processing steps such as terminology standardization, consistency verification, and confidence level annotation results in unstable report quality, making it difficult to directly meet the needs of clinical review and intelligent decision-making in medical imaging information management.

[0006] Pain Point 4: Existing methods fail to achieve an end-to-end closed loop between natural language generation and medical image information management, lacking a standardized token takeover mechanism. Current lung CT image processing methods (such as those based on object detection or semantic segmentation) only focus on visual detection and segmentation, outputting lesion locations and segmentation masks, unable to directly generate doctor-readable natural language diagnostic reports, and thus do not involve natural language generation (NLG). Furthermore, these methods lack private domain data adaptation, token optimization / context window management, and a closed loop for medical image information management and review, making it difficult to meet the intelligent decision-making needs of "AI generation → confidence labeling → physician review" in actual clinical deployments. In addition, existing methods lack a standardized token takeover mechanism—there is no unified Prompt format specification and output format protocol (i.e., token standardization) between the local agent and the remote large model. This prevents the local agent from "taking over" the natural language post-processing steps (such as sign verification, structuring, terminology standardization, and consistency verification) that originally required remote large model processing, failing to achieve the token-saving closed loop of "if the local agent can solve the problem, then remote AI is not needed," thus hindering the large-scale application of natural language generation technology in medical image information management. Summary of the Invention

[0007] This invention aims to address the problems in existing technologies, such as excessive token consumption in remote multimodal large models, unnecessary consumption of remote tokens in natural language post-processing due to the lack of a local agent takeover mechanism, and ineffective local agent takeover due to the lack of a token standardization takeover mechanism. It proposes an automatic CT report generation method based on local agent takeover and token standardization mechanism. Through local agent takeover at the visual, retrieval, and language levels, as well as confidence-based hierarchical scheduling decisions, it achieves end-to-end token savings.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A method for assisting in the generation of medical image CT reports based on a multimodal large model and a local intelligent agent includes the following steps: Step S1: Construct a private lung image library and index, collect desensitized lung CT image data and corresponding historical diagnostic reports, extract joint embedding features of image-report pairs and establish vector indexes; Step S2: The local agent takes over at the visual level and performs multi-granular visual redundancy suppression on the input lung CT sequence to be diagnosed, including inter-slice redundancy suppression and intra-slice redundancy suppression, and performs token compression on the CT sequence. Step S3: The local agent takes over at the retrieval level, performs influence-oriented retrieval based on the private lung image database, predicts the contribution of candidate examples to the query image through the agent influence model, and selects the k examples with the highest influence scores as contextual demonstrations; the remote AI interface is only called when special image features are detected. Step S4: Concatenate the processed query CT image token with the retrieved Top-k example tokens according to the token standardization format to construct a Prompt. The token standardization format includes a structured template for the input sequence, output format requirements, verification rules, and post-processing rules. Input the remote multimodal large model to perform one inference to generate a preliminary report text. Then, the local agent takes over the natural language post-processing stage based on the token standardization rules. The local agent makes a decision on whether to call remote AI or manual review based on confidence level scheduling, and finally outputs the CT diagnostic report.

[0009] Further, step S2, inter-slice redundancy suppression, includes: (1) numbering the input CT sequence according to the slice order and extracting the visual feature vector of each slice using a 3D visual encoder (such as 3D-ResNet-50); (2) calculating the feature cosine similarity between adjacent slices and constructing an inter-slice similarity sequence; (3) using adaptive threshold calculation: performing kernel density estimation (KDE) on the inter-slice similarity sequence, using the valley of the similarity distribution as the redundancy judgment threshold τ_inter, and marking redundant slices when the similarity between adjacent slices exceeds τ_inter; (4) forced retention mechanism: forcibly retaining key anatomical layers containing the hilum and mediastinal region (based on the predefined layer numbering range of the anatomical prior model), even if the similarity exceeds the threshold, without suppression; (5) for redundant slice groups, using an equal-interval sampling strategy to retain representative slices and remove the remaining redundant slices.

[0010] Further, the redundancy suppression in step S2 includes: (1) using nnUNet to segment the lung parenchyma of the retained slices and generating a lung parenchyma mask; (2) multiplying the original CT slices with the lung parenchyma mask pixel by pixel to remove non-lung parenchyma regions (such as chest wall, mediastinum, spine, etc.) and retain only the lung parenchyma region; (3) performing non-zero pixel statistics on the lung parenchyma region and determining slices with a non-zero pixel ratio of less than 5% as "no diagnostic value" and removing them; (4) downsampling the lung parenchyma region of the remaining slices (such as downsampling from 512×512 to 256×256) to further reduce the number of tokens.

[0011] Further, step S3, influence-oriented retrieval, includes: (1) Query vector construction: extracting feature vectors from the CT images processed in step S2 using a 3D visual encoder, and using them as query vectors q; (2) Coarse screening: using FAISS to perform nearest neighbor retrieval on candidate examples in the private domain library, selecting the M candidate examples with the highest cosine similarity, where M≥k; (3) Fine screening (proxy influence model PIM): for the M candidate examples, using a proxy influence model to predict the contribution of each candidate example to the query image. The proxy influence model is a lightweight network, using the query vector q and the candidate examples... Example vectors are inputs, and the output is the influence score. The higher the influence score, the more valuable the candidate example is for generating a natural language report of the query image; (4) Special image processing: The remote AI interface is called for auxiliary analysis only when the local Agent detects special image features; (5) The top-k examples are selected as context demonstrations in descending order of influence score; (6) Score fusion and top-k selection: The PIM influence score and the AI ​​auxiliary analysis results (if any) are weighted and fused to obtain the final example score, and the top-k examples are selected in descending order.

[0012] Further, step S4 includes the following sub-steps: S4.1 Token Standardization Prompt Construction: Assemble the multimodal input sequence according to the token standardization template. The structure is [System Instruction] + [Example 1 Image Token | Example 1 Report Token] + [Example 2 Image Token | Example 2 Report Token] + [Example 3 Image Token | Example 3 Report Token] + [Query CT Image Token] + [Generate Instruction]. The system instruction includes role definition, output format requirements, and a list of lung-specific signs of interest. The token standardization format ensures standardized token exchange between the local agent and the remote large model, enabling the local agent to take over the natural language post-processing stage of the remote large model. S4.2 Multimodal Reasoning and Natural Language Generation: Input the standardized Token Prompt into a remote multimodal large model (such as GPT-4V, Qwen-VL, etc.) for one inference to generate a preliminary report text; this step is the main part of the whole process that calls the remote large model, and only one inference is performed. The subsequent natural language post-processing is taken over by the local agent by default. S4.3 Sign Verification: The local agent verifies whether the preliminary report includes key lung-specific signs (including lobulation, spiculation, pleural traction, vascular convergence, cavity, calcification, fibrous strands, and ground-glass opacity) based on the token-standardized verification rules. If key signs are missing, a supplementary generation instruction is triggered. This step is completed by the local agent. S4.4 Sign Supplementation: Based on the Token standardization rules, the local agent adds supplementary generation instructions for missing signs, supplementing and modifying the preliminary report. This process is repeated a maximum of two times to prevent infinite iteration. In the default mode, the local agent directly inserts standardized descriptions based on the sign template library (a predefined set of standardized description templates for lung-specific signs, including standardized text descriptions of various signs such as lobulation, spiculation, pleural traction, vascular clustering, cavitation, calcification, fibrous streaks, and ground-glass opacity, deployed locally and directly called by the local agent). In the degraded mode, a remote large model is triggered to supplement the missing signs only when they exceed the coverage of the sign template library. S4.5 Structuring and Standardizing the Format: The local agent structures the report text according to the standard diagnostic report format based on the Token standardization format specification, including four parts: examination name, imaging findings, diagnostic opinion, and recommendations; this step is completed by the local agent. S4.6 Terminology Standardization: The local agent performs terminology alignment based on the local medical imaging terminology dictionary (RadLex / SNOMED CT), replacing non-standard terms with standard terms to ensure that the diagnostic reports generated by natural language conform to the terminology standards for medical imaging information management; this step is completed by the local agent. S4.7 Consistency Verification and Correction: The local agent verifies the internal logical consistency of the report, including the correspondence between image findings and diagnostic opinions, the consistency of terminology used, and the rationality of numerical descriptions. If inconsistencies are found, they are automatically corrected. This step is completed by the local agent. S4.8 Local Agent Confidence Level Scheduling Decision - Confidence Assessment and Labeling: Label the confidence level of each diagnostic opinion in the report, calculated based on the similarity distribution between PIM search examples and query CT scans; S4.9 Output and Push: Output structured diagnostic reports and automatically push them to the physician workstation.

[0013] Further, the evaluation method of the agent influence model in step S3 includes: (1) constructing an evaluation dataset: extracting N query-report pairs from the private domain library as evaluation samples; (2) for each query, using PIM influence retrieval and random retrieval to select Top-k examples and generate a report; (3) calculating report quality indicators: including BLEU, ROUGE-L, and clinical consistency score (annotated by senior radiologists); (4) verifying the improvement of PIM retrieval in report quality indicators compared with random retrieval. If BLEU is improved by ≥5%, ROUGE-L by ≥3%, and clinical consistency score by ≥10%, then PIM is deemed effective; otherwise, the agent influence model is retrained or the parameters are tuned.

[0014] The present invention has the following beneficial effects: (1) Significantly reduces remote token consumption, with token savings of approximately 60% to 65% after local agent takeover (core achievement of this invention): Through local agent at the visual level (MRS, compressing tokens from approximately 80,000 to 100,000 to approximately 30,000 to 40,000), the retrieval level (ICR, selecting high-value examples to avoid token waste), the language level (symptom verification / structuring / terminology standardization / consistency verification, taking over the natural language post-processing link that originally required remote multi-round interaction based on token standardization rules), and confidence-based hierarchical scheduling decision (local agent decides whether to call remote AI or manual review to avoid unnecessary remote token consumption), the token savings after local agent takeover account for approximately 60% to 65%, effectively reducing the inference cost and latency of remote large models, solving the core bottleneck of excessive token consumption in remote multimodal large models, and providing an economical and feasible technical solution for the large-scale application of natural language generation technology in medical image information management; (2) Adapting private domain data to improve the accuracy of retrieval and natural language generated reports: By constructing a private domain lung image database and using the joint embedding features of image-report pairs to establish a vector index, the retrieval process can capture the pathological distribution characteristics of local hospitals, improve the pathological relevance of the retrieval, and significantly improve the retrieval accuracy compared with general retrieval methods (such as CLIP feature cosine similarity retrieval), providing high-quality private domain data support for medical image information management. (3) Improve retrieval accuracy and upgrade from "surface similarity" to "influence-oriented": Use the proxy influence model (PIM) to replace simple visual similarity retrieval. PIM directly predicts the contribution (influence) of candidate examples to the generation of natural language reports for query images, rather than just measuring visual similarity, thereby retrieving examples with guiding value at the pathological level and improving the accuracy and clinical consistency of natural language generated reports. (4) Token standardization takeover mechanism, local agent takes over natural language post-processing links to save remote tokens: Through token standardization format specification, local agent can "take over" natural language post-processing links such as sign verification, structured formatting, terminology standardization, and consistency verification that originally required multiple rounds of dialogue with a remote large model. These links are completed by local agent based on rules, avoiding the extra remote token consumption generated by multiple rounds of interaction in the original scheme; at the same time, the sign verification closed-loop mechanism (maximum 2 loops) ensures that the natural language generated report covers key lung-specific signs and avoids missing key diagnostic information; (5) Confidence-based scheduling decision-making to achieve an end-to-end closed loop between natural language generation and medical image information management: Through confidence assessment and labeling mechanism, each diagnostic opinion in the report is labeled with three levels of confidence: high, medium and low. High confidence is directly output (local agent judges that no remote AI is needed), medium confidence is labeled "recommend review" (directly output by local agent), and low confidence is labeled "requires physician manual confirmation" or triggers a second remote AI call (local agent judges that it cannot solve the problem itself and requires remote AI intervention). It is automatically pushed to the physician workstation to achieve an end-to-end closed loop between natural language generation and medical image information management of "local agent scheduling decision-making → confidence labeling → physician review". This reflects the principle of "no remote AI if local agent can solve the problem", saves remote tokens to the maximum extent, and meets the security and auditability requirements of medical image information management. (6) Proxy Influence Model Evaluation Mechanism to ensure that the retrieval quality is verifiable and iteratively optimized: By constructing an evaluation dataset, the improvement of PIM influence retrieval and random retrieval in report quality indicators (BLEU, ROUGE-L, clinical consistency score) is compared. A quantitative effective judgment threshold is set (BLEU improvement ≥5%, ROUGE-L improvement ≥3%, clinical consistency score improvement ≥10%) to ensure that the effectiveness of PIM retrieval is quantifiable and verifiable. When the evaluation result does not reach the threshold, the model is retrained or the parameters are tuned to achieve continuous iterative optimization of retrieval quality and improve the credibility and robustness of the system. (7) Achieving an end-to-end token-saving natural language generation and medical image information management closed loop to meet actual needs: Unlike existing methods that only focus on token quantity compression, this invention achieves an end-to-end token-saving closed loop of "local agent visual level takeover (MRS) → local agent retrieval level takeover (ICR) → remote large model on-demand call (only one natural language generation inference) → local agent language level takeover (token standardization takeover natural language post-processing) → local agent confidence level scheduling decision (avoiding unnecessary remote token consumption)". The effectiveness of the cross-application of natural language generation technology and medical image information management has been verified in the automatic generation scenario of CT diagnostic reports. It can directly generate complete structured CT diagnostic reports containing examination names, image findings, diagnostic opinions, and suggestions. The report format conforms to the HIS / PACS interface standard and meets the actual deployment needs of medical image information management. Attached Figure Description

[0015] Figure 1 This is an overall flowchart of a method for assisting in the generation of medical image CT reports based on a multimodal large model and a local intelligent agent; Figure 2 A detailed flowchart of multi-granularity redundancy suppression (MRS); Figure 3 A detailed flowchart for Influence-Driven Search (ICR); Figure 4 This is a flowchart illustrating the training and inference process of the agent influence model in this embodiment of the invention. Figure 5 This is a detailed flowchart of the automatic generation of CT diagnostic reports in an embodiment of the present invention; Figure 6 This is a schematic diagram of the module structure of a medical image CT report auxiliary generation system based on a multimodal large model and a local intelligent agent. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Example 1

[0017] This embodiment provides a method for assisting in the generation of medical image CT reports based on a multimodal large model and a local intelligent agent. For example... Figure 1 As shown, it includes the following steps: Step S1: Building a private lung image library and index like Figure 1 As shown, desensitized lung CT image data and corresponding historical diagnostic reports are collected to construct a private lung image database, providing a private data foundation for medical image information management. Specifically, this includes: (1) Image preprocessing: The original DICOM format CT images were adjusted by adjusting the window width and window level (lung window: window level -600HU, window width 1500HU; mediastinal window: window level 350HU, window width 50HU), resampled to a uniform resolution (1mm×1mm×1mm), and de-identified. (2) Joint embedding: Image features are extracted using a pre-trained 3D visual encoder (such as 3D-ResNet-50), and report text features are extracted using BioClinicalBERT. Image and report features are mapped to a unified embedding space through contrastive learning to obtain the joint embedding vector of the image-report pair. (3) Report structuring: Historical diagnostic reports are structured and analyzed according to a standard format (examination name, imaging findings, diagnostic opinion, and recommendations) to extract key signs and achieve structured management of medical imaging information; (4) FAISS Index Construction: The FAISS index is constructed by embedding the joint multi-granularity visual redundancy suppression (MRS) vector of all image-report pairs, which supports efficient nearest neighbor retrieval and provides index support for private domain data retrieval in medical image information management.

[0018] Step S2: The local agent takes over at the visual level. The input lung CT sequence to be diagnosed undergoes multi-granularity visual redundancy suppression, including inter-slice redundancy suppression and intra-slice redundancy suppression, compressing the token from approximately 80,000-100,000 to approximately 30,000-40,000. This process does not consume tokens from the remote large model. Figure 2 As shown, it specifically includes: (1) Interlayer redundancy suppression: a. Number the input CT sequence according to the slice order, and use a 3D visual encoder (such as 3D-ResNet-50) to extract the visual feature vector of each slice; b. Calculate the feature cosine similarity between adjacent slices and construct an inter-layer similarity sequence: sim(i, i+1) = cos(f_i, f_{i+1}) = f_i · f_{i+1} / (||f_i|| × ||f_{i+1}||) Where f_i is the visual feature vector of the i-th slice; c. Adaptive threshold calculation: Kernel density estimation (KDE) is performed on the inter-layer similarity sequence, and the valley of the similarity distribution is used as the redundancy determination threshold τ_inter. When the similarity between adjacent slices exceeds τ_inter, the redundant slice is marked. d. Forced preservation mechanism: Key anatomical layers containing the hilum and mediastinal regions (based on the predefined layer numbering range of the anatomical prior model, such as layers 40-60 for the hilum region and layers 30-45 for the mediastinal region) are forcibly preserved, even if the similarity exceeds the threshold. e. For redundant slice groups, use an equal-interval sampling strategy to retain representative slices and remove the remaining redundant slices.

[0019] (2) Intra-layer redundancy suppression: a. Use nnUNet to segment the lung parenchyma of the preserved slices and generate a lung parenchyma mask; b. Multiply the original CT slices pixel by pixel with the lung parenchyma mask to remove non-lung parenchyma areas (such as chest wall, mediastinum, spine, etc.) and retain only the lung parenchyma area; c. Perform non-zero pixel statistics on the lung parenchyma region, and remove slices with a non-zero pixel ratio of less than 5% as "no diagnostic value"; d. Downsample the lung parenchyma region of the remaining slices (from 512×512 to 256×256) to further reduce the number of tokens.

[0020] This step is executed entirely by the local agent, without calling remote large model APIs. It is a token-saving step at the visual level for the local agent, compressing the tokens from approximately 80,000-100,000 to approximately 30,000-40,000, saving approximately 60%-65% of the tokens. The adaptive threshold calculation and forced retention mechanism in inter-layer redundancy suppression, as well as the nnUNet segmentation and lung parenchyma mask generation in intra-layer redundancy suppression, are all deployed and run on the local GPU.

[0021] Step S3: Local Agent Takes Over at the Search Level – Influence-Driven Search (ICR) Based on Private Domain Database Influence-driven retrieval is performed based on a private lung image database. A proxy influence model predicts the contribution of candidate examples to the query image, selecting the k examples with the highest influence scores as contextual examples. This avoids wasting remote token budget on low-value examples, and the process does not consume remote large model tokens. The remote AI interface is only invoked when specific image features are detected. Figure 3 As shown, it specifically includes: Query vector construction: The feature vector of the CT image processed in step S2 is extracted by a 3D visual encoder and used as the query vector q; (2) Coarse screening: Use FAISS to perform nearest neighbor search on the candidate examples in the private domain library and select the M candidate examples with the highest cosine similarity (M=50, k=3). (3) Fine Screening (Proxy Influence Model PIM): For M candidate examples, the Proxy Influence Model is used to predict the contribution of each candidate example to the query image. The Proxy Influence Model is a lightweight network that takes the query vector q and the candidate example vector as input and outputs an influence score. The higher the influence score, the more valuable the candidate example is to the generation of the natural language report of the query image. The PIM model is deployed locally and does not call the remote large model API. Both coarse screening and fine screening are completed by the local Agent. (4) Special Image Processing: The local agent determines whether to call the remote AI interface through the special image feature detection module. Specifically, the special image feature detection includes the following steps: a. For the CT slices processed in step S2, use a locally deployed lightweight classification network (such as MobileNet-V3) to classify the abnormal features of each slice and output the probability distribution of whether the slice belongs to "normal" or "special"; b. Special image features include, but are not limited to: rare signs (such as rare intrapulmonary space-occupying lesions, rare diffuse lesion signs), coexistence of multiple signs (three or more signs appearing in the same lesion at the same time), and outlier cases whose similarity with the Top-k examples in the PIM retrieval library is all lower than the preset threshold (such as cosine similarity <0.3); c. When the "special" probability of any slice exceeds the preset threshold (such as 0.7), it is determined that a special image feature has been detected. The local agent determines that it cannot solve the problem itself and calls the remote AI interface for auxiliary analysis; For most normal cases, the local agent's PIM model can complete the retrieval without calling the remote AI. (5) Sort by influence score in descending order and select the top-k examples (k=3) as contextual demonstrations; (6) Score fusion and Top-k selection: The PIM influence score and the AI-assisted analysis results (if any) are weighted and fused to obtain the final example score. The Top-k examples are selected in descending order.

[0022] The training and inference process of the Proxy Influence Model (PIM) is as follows: Figure 4 As shown, it specifically includes: (1) Training data construction: N query-report pairs are extracted from the private domain library as training samples. For each query, a report is generated using different combinations of candidate examples, and the report quality index (BLEU, ROUGE-L) is calculated as the influence label. Among them, for each query's candidate examples, they are sorted in descending order according to the BLEU / ROUGE-L comprehensive score of the corresponding generated report. The candidate examples with the highest ranking are marked as high-influence positive samples, and the candidate examples with the lowest ranking are marked as low-influence negative samples. The middle part is discarded to construct positive and negative sample pairs for comparative learning training. (2) Model Architecture: PIM adopts a dual-tower architecture, with the query tower and the instance tower encoding the query vector and candidate instance vector respectively, and the influence score is calculated through a cross-attention layer: score(q, c) = σ(W · [q ⊕ c ⊕ (q ⊗ c)] + b) Where q is the query vector, c is the candidate example vector, ⊕ is the concatenation operation, ⊗ is the element-wise product, σ is the Sigmoid activation function, and W and b are learnable parameters; (3) Training objective: To use a contrastive learning loss function to make the scores of positive samples (high-influence examples) higher than those of negative samples (low-influence examples): L = -log(exp(score(q, c⁺)) / Σ_j exp(score(q, c_j))) Where c⁺ represents high-impact examples and c_j represents all candidate examples; (4) Inference: After training, PIM runs on the local GPU to refine the M candidate examples obtained from the coarse screening and output the influence score.

[0023] Step S4: Remote large model on-demand invocation + local agent language-level takeover + local agent confidence-based hierarchical scheduling decision The processed query CT image token is concatenated with the retrieved Top-k example tokens according to the token standardization format to construct a Prompt. This Prompt is then input into a remote multimodal large model for inference to generate a preliminary report text. Subsequently, the local agent, based on token standardization rules, takes over the natural language post-processing stages, including sign verification and supplementation, structure and format standardization, terminology standardization, consistency verification and correction. The local agent then decides whether to call remote AI or human review based on confidence level scheduling, ultimately outputting the CT diagnostic report. Figure 5 As shown, the specific steps include the following: S4.1 Token Standardization Prompt Construction: Assemble the multimodal input sequence according to the token standardization template. The structure is [System Instruction] + [Example 1 Image Token | Example 1 Report Token] + [Example 2 Image Token | Example 2 Report Token] + [Example 3 Image Token | Example 3 Report Token] + [Query CT Image Token] + [Generate Instruction]. The system instruction includes role definition, output format requirements, and a list of lung-specific signs of interest. The token standardization format ensures standardized token exchange between the local agent and the remote large model, enabling the local agent to take over the natural language post-processing stage of the remote large model. S4.2 Multimodal Reasoning and Natural Language Generation: Input the standardized Token Prompt into a remote multimodal large model (such as GPT-4V, Qwen-VL, etc.) for one inference to generate a preliminary report text; this step is the main part of the whole process that calls the remote large model, and only one inference is performed. The subsequent natural language post-processing is taken over by the local agent by default. S4.3 Sign Verification: The local agent uses the Token-standardized verification rules to verify whether the preliminary report includes key lung-specific signs (including lobulation, spiculation, pleural traction, vascular convergence, cavity, calcification, fibrous strands, and ground-glass opacity). If key signs are missing, a supplementary generation instruction is triggered. S4.4 Sign Supplementation: Based on the Token standardization rules, the local agent adds supplementary generation instructions for missing signs, supplementing and modifying the preliminary report. This process is repeated a maximum of two times to prevent infinite iteration. In the default mode, the local agent directly inserts standardized descriptions based on the sign template library (a predefined set of standardized description templates for lung-specific signs, including standardized text descriptions of various signs such as lobulation, spiculation, pleural traction, vascular clustering, cavitation, calcification, fibrous streaks, and ground-glass opacity, deployed locally and directly called by the local agent). In the degraded mode, a remote large model is triggered to supplement the missing signs only when they exceed the coverage of the sign template library. S4.5 Structuring and Standardizing the Format: The local agent structures the report text according to the standard diagnostic report format based on the Token standardization format specification, including four parts: examination name, imaging findings, diagnostic opinion, and recommendations; this step is completed by the local agent. S4.6 Terminology Standardization: The local agent performs terminology alignment based on the local medical imaging terminology dictionary (RadLex / SNOMED CT), replacing non-standard terms with standard terms to ensure that the diagnostic reports generated by natural language conform to the terminology standards for medical imaging information management; this step is completed by the local agent. S4.7 Consistency Verification and Correction: The local agent verifies the internal logical consistency of the report, including the correspondence between image findings and diagnostic opinions, the consistency of terminology used, and the rationality of numerical descriptions. If inconsistencies are found, they are automatically corrected. This step is completed by the local agent. S4.8 Local Agent Confidence Level Scheduling Decision – Confidence Assessment and Labeling: Each diagnostic opinion in the report is labeled with a confidence level, calculated based on the similarity distribution between the PIM search example and the query CT. The confidence level grading criteria are: high confidence (≥0.8) is output directly; medium confidence (0.5~0.8) is labeled "recommend review" and output directly by the local agent; low confidence (<0.5) is labeled "requires manual physician confirmation" or triggers a secondary remote AI call.

[0024] S4.9 Output and Push: Output structured diagnostic reports and automatically push them to the physician workstation. Example 2

[0025] This embodiment presents a medical image CT report auxiliary generation system based on a multimodal large model and a local intelligent agent, used to implement the method described in Embodiment 1. Figure 6 As shown, it includes: The input module is used to receive input data such as CT images, patient basic information, historical examination records, and clinical indications. The private domain library construction module is used to collect desensitized lung CT image data and corresponding historical diagnostic reports, extract joint embedding features of image-report pairs and build vector indexes, providing a private domain data foundation for medical image information management. It includes an image preprocessing submodule, a joint embedding submodule, a report structuring submodule and a FAISS index construction submodule. The local agent visual layer takeover module is used to perform inter-layer and intra-layer redundancy suppression to save remote tokens, including the 3D visual encoder submodule, key slice filtering submodule, token simplification submodule and feature alignment submodule; The local agent retrieval layer takeover module is used to perform example retrieval based on the agent influence model and select high-value examples; the remote AI interface is called only when special image features are detected, including query vector construction submodule, FAISS coarse screening submodule, PIM fine screening submodule, special image detection submodule, AI-assisted analysis submodule, and score fusion and Top-k selection submodule; The report generation module is used to build a Prompt and call a remote multimodal large model to generate natural language. The local Agent takes over the natural language post-processing stage based on Token standardization rules, including the Prompt construction submodule, multimodal reasoning submodule, symptom verification submodule, symptom supplementation submodule, structuring submodule, terminology standardization submodule, consistency verification submodule, and confidence assessment submodule. The output module is used to output structured diagnostic reports, confidence level annotations, and review prompts, and to connect to the HIS / PACS system to push them to the physician workstation.

Claims

1. A method for assisting in the generation of medical image CT reports based on a multimodal large model and a local intelligent agent, characterized in that, Includes the following steps: Step S1: Construct a private lung image library and index, collect desensitized lung CT image data and corresponding historical diagnostic reports, extract joint embedding features of image-report pairs and establish vector indexes; Step S2: The local agent takes over at the visual level and performs multi-granular visual redundancy suppression on the input lung CT sequence to be diagnosed, including inter-slice redundancy suppression and intra-slice redundancy suppression, and performs token compression on the CT sequence. Step S3: The local agent takes over at the retrieval level, performs influence-oriented retrieval based on the private lung image database, predicts the contribution of candidate examples to the query image through the agent influence model, and selects the k examples with the highest influence scores as contextual demonstrations; the remote AI interface is only called when special image features are detected. Step S4: Concatenate the processed query CT image token with the retrieved Top-k example tokens according to the token standardization format to construct a Prompt. The token standardization format includes a structured template for the input sequence, output format requirements, verification rules, and post-processing rules. The verification rules include verification rules for a list of signs of concern for specific examination sites. The post-processing rules include standardization rules for the natural language post-processing stage. Input the remote multimodal large model to perform one inference to generate a preliminary report text. Then, the local agent takes over the natural language post-processing stage based on the token standardization rules. The local agent makes a decision on whether to call remote AI or manual review based on confidence level scheduling, and finally outputs the CT diagnostic report.

2. The method according to claim 1, characterized in that, The specific implementation method of multi-granularity visual redundancy suppression in step S2 is as follows: Inter-slice redundancy suppression: (1) Number the input CT sequence according to the slice order and use a 3D visual encoder to extract the visual feature vector of each slice; (2) Calculate the feature cosine similarity between adjacent slices and construct the inter-slice similarity sequence; (3) Adaptive threshold calculation: Perform kernel density estimation (KDE) on the inter-slice similarity sequence and use the valley of the similarity distribution as the redundancy judgment threshold τ_inter. When the similarity between adjacent slices exceeds τ_inter, mark the redundant slice; (4) Forced retention mechanism: Force the retention of key anatomical layers containing the hilum and mediastinal region, even if the similarity exceeds the threshold, it will not be suppressed; (5) For the redundant slice group, use an equal interval sampling strategy to retain representative slices and remove the remaining redundant slices; Intra-layer redundancy suppression: (1) Use nnUNet to segment the lung parenchyma of the preserved slices and generate a lung parenchyma mask; (2) Multiply the original CT slices with the lung parenchyma mask pixel by pixel to remove non-lung parenchyma areas and retain only the lung parenchyma areas; (3) Count the non-zero pixels in the lung parenchyma areas and remove the slices with a non-zero pixel ratio of less than 5% as "no diagnostic value"; (4) Downsample the lung parenchyma areas of the remaining slices to further reduce the number of tokens.

3. The method according to claim 1, characterized in that, Step S3, influence-oriented retrieval, includes: (1) Query vector construction: The feature vector of the CT image processed in step S2 is extracted by a 3D visual encoder and used as the query vector q; (2) Coarse screening: The candidate examples in the private domain are searched for nearest neighbors using FAISS, and the M candidate examples with the highest cosine similarity are selected, where M≥k; (3) Fine screening (proxy influence model PIM): For the M candidate examples, the proxy influence model is used to predict the contribution of each candidate example to the query image. The proxy influence model is a lightweight network that takes the query vector q and the candidate example vector as input and outputs the influence score; (4) Special image processing: The remote AI interface is called for auxiliary analysis only when the local agent detects special image features; (5) The top-k examples are selected as context demonstrations in descending order of influence score; (6) Score fusion and top-k selection: The PIM influence score is weighted and fused with the AI ​​auxiliary analysis results to obtain the final example score, and the top-k examples are selected in descending order.

4. The method according to claim 1, characterized in that, Step S4 includes the following sub-steps: S4.1 Token standardization Prompt construction: Assemble the multimodal input sequence according to the token standardization template. The structure is [System Instruction] + [Example 1 Image Token | Example 1 Report Token] + [Example 2 Image Token | Example 2 Report Token] + [Example 3 Image Token | Example 3 Report Token] + [Query CT Image Token] + [Generate Instruction]. The system instruction includes role definition, output format requirements, and a list of lung-specific signs of interest. S4.2 Multimodal Reasoning and Natural Language Generation: Input the standardized Token Prompt into a remote multimodal large model for one inference to generate a preliminary report text; S4.3 Sign Verification: The local agent uses the Token-standardized verification rules to verify whether the preliminary report includes key lung-specific signs. If key signs are missing, a supplementary generation instruction is triggered. S4.4 Symptom Supplementation: Based on the Token standardization rules, the local agent adds supplementary generation instructions for missing symptoms, supplementing and modifying the preliminary report. This process is repeated a maximum of 2 times to prevent infinite iteration. In the default mode, the local agent directly inserts standardized descriptions based on the symptom template library. In degraded mode, a remote large model call is triggered to supplement and generate a new model only when the missing symptoms exceed the coverage of the symptom template library; S4.5 Structuring and Format Standardization: The local agent structures the report text according to the standard diagnostic report format based on the Token standardization format specification; S4.6 Terminology Standardization: The local agent performs terminology alignment based on the local medical imaging terminology dictionary, replacing non-standard terms with standard terms; S4.7 Consistency Verification and Correction: The local agent verifies the internal logical consistency of the report, and automatically corrects any inconsistencies found. S4.8 Confidence Assessment and Labeling: Label the confidence level of each diagnosis in the report. High confidence level is output directly, medium confidence level is labeled "recommendation for review" and output directly by the local agent, and low confidence level is labeled "requires manual confirmation by physician" or triggers a second remote AI call. S4.9 Output and Push: Output structured diagnostic reports and automatically push them to the physician workstation.

5. The method according to claim 4, characterized in that, The specific format of the Token standardization template in S4.1 is as follows: The system instruction section includes the role definition "You are a senior radiologist", the output format requirements "Output by four parts: examination name, imaging findings, diagnosis opinion, and suggestions", and the lung-specific sign attention list "lobulation sign, spiculation sign, pleural traction sign, vascular convergence sign, cavity sign, calcification sign, fibrous strands, and ground-glass opacity". S4.8 The confidence level is calculated based on the similarity distribution between the PIM search example and the queried CT. High confidence (≥0.8) is output directly, medium confidence (0.5~0.8) is marked "recommend review" and output directly by the local agent, and low confidence (<0.5) is marked "requires manual confirmation by physician" or triggers a second remote AI call.

6. The method according to claim 3, characterized in that, The agent influence model employs a dual-tower architecture, comprising a query encoding branch and an example encoding branch, used to encode the query vector and candidate example vectors, respectively. An influence score is calculated through a cross-attention layer: score(q, c) = σ(W × [q ⊕ c ⊕ (q ⊗ c)] + b), where q is the query vector, c is the candidate example vector, ⊕ is the vector concatenation operation, ⊗ is the element-wise multiplication operation, σ is the sigmoid activation function, W is the weight matrix, b is the bias vector, and × represents matrix multiplication. The training objective uses a contrastive learning loss function: L = -log(exp(score(q, c⁺)) / Σ_j exp(score(q, c_j))), where c⁺ represents high-influence examples, and c_j represents all candidate examples.

7. The method according to claim 3, characterized in that, The evaluation method of the proxy influence model includes: extracting evaluation samples from the private domain database, using PIM influence retrieval and random retrieval to select Top-k examples for each query to generate a report, calculating report quality indicators including BLEU, ROUGE-L, and clinical consistency score, verifying the improvement of PIM retrieval in report quality indicators compared to random retrieval, and determining PIM effectiveness if BLEU improves by ≥5%, ROUGE-L improves by ≥3%, and clinical consistency score improves by ≥10%; otherwise, triggering retraining or parameter tuning of the proxy influence model.

8. A medical image CT report auxiliary generation system based on a multimodal large model and a local intelligent agent, used to implement the method of any one of claims 1 to 7, characterized in that, include: The input module is used to receive CT images, patient basic information, historical examination records, and clinical indication input data. The private domain library construction module is used to collect desensitized lung CT image data and corresponding historical diagnostic reports, extract joint embedding features of image-report pairs and build vector indexes, including image preprocessing submodule, joint embedding submodule, report structuring submodule and FAISS index construction submodule; The local agent visual layer takeover module is used to perform inter-layer and intra-layer redundancy suppression to save remote tokens, including the 3D visual encoder submodule, key slice filtering submodule, token simplification submodule and feature alignment submodule; The local agent retrieval layer takeover module is used to perform example retrieval based on the agent influence model and select high-value examples; the remote AI interface is called only when special image features are detected, including query vector construction submodule, FAISS coarse screening submodule, PIM fine screening submodule, special image detection submodule, AI-assisted analysis submodule, and score fusion and Top-k selection submodule; The report generation module is used to build a Prompt and call a remote multimodal large model to generate natural language. The local Agent takes over the natural language post-processing stage based on Token standardization rules, including the Prompt construction submodule, multimodal reasoning submodule, symptom verification submodule, symptom supplementation submodule, structuring submodule, terminology standardization submodule, consistency verification submodule, and confidence assessment submodule. The output module is used to output structured diagnostic reports, confidence level annotations, and review prompts, and to connect to the HIS / PACS system to push them to the physician workstation.

Images