Medical report generation method, model training method, equipment and medium

Abstract

The invention discloses a medical report generation method, a model training method, equipment and a medium, and the model training method comprises the steps: constructing a medical report generation model framework which comprises a global semantic collaborative multi-modal enhancement module, a visual encoder, a text encoder, a medical insight analyzer and an LLM decoder; wherein the global semantic collaborative multi-modal enhancement module respectively enhances a medical image and a medical report by utilizing a selected image enhancement strategy and a text enhancement strategy, and the medical insight analyzer comprises a fine-grained structure learning device and a global context guide learning device which are connected in sequence so as to enhance the cross-modal alignment capability; and performing intelligent collaborative optimization by taking a strategy set formed by an image enhancement strategy and a text enhancement strategy and architecture configuration parameters of the medical insight analyzer as optimization targets to obtain an optimal medical report generation model. The medical report generation performance can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a medical report generation method, a model training method, equipment and a medium. BACKGROUND

[0002] Medical report generation (MRG) is a key task in medical image computing and auxiliary diagnosis, aiming to generate accurate and coherent medical descriptions based on medical images (such as X-rays, surgical images and pathological sections), thereby assisting doctors in diagnosis and improving efficiency. Early methods are mostly based on traditional image captioning techniques, such as CNN-RNN structure, using image encoders to extract image features and combining RNN or LSTM models to generate text. Such methods have limited generation capabilities and poor long text consistency when processing medical images.

[0003] With the advent of the Transformer architecture, some works have begun to introduce Transformers to improve long-distance dependency modeling. However, these models still face challenges in the medical scene, such as insufficient cross-modal alignment ability and difficulty in accurately mapping complex medical terminology and visual regions.

[0004] In recent years, general vision-language foundation models (VLFMs) such as BLIP, ALBEF, MiniGPT-4 have made significant breakthroughs in natural image fields, with stronger context modeling and semantic understanding capabilities. Some research has begun to migrate VLFMs to the MRG task, such as XrayGPT, ClinicalGPT, etc., through the fusion of visual encoders and large language models to realize image-to-text reasoning. However, existing VLFMs are mostly designed for general fields and lack deep understanding of complex semantics between medical terminology and images. In addition, their model structure is usually fixed and the enhancement strategy is static, making it difficult to adapt to changing medical image scenarios.

[0005] The closest prior art to the present application is as follows:

[0006] Technical solution one: S3-Net (S3-Net: A Self-Supervised Dual-Stream Network for Radiology Report Generation).

[0007] S3-Net is an end-to-end medical image report generation model designed to alleviate the repetitive labor of radiologists in image description. Its main goal is to fully extract visual features and improve the alignment between text and images. The implementation of this model is as follows: (1) Dual-Stream Visual Feature Extraction Module (DSVFE), S3-Net uses a dual-stream structure to encode the input medical image, where: ResNet is responsible for extracting local response features; Swin Transformer is used to model global dependencies; both processes the image in parallel, and the output features are then used for fusion. (2) Fusion Alignment Module (FAM), which fuses the above two visual features and aligns them with the text features through this module. This module aims to alleviate the modal difference between visual and language features and strengthen the consistency of cross-modal semantics. (3) Self-Supervised Learning with Mask (SSLM), during training, a visual mask self-supervised task is introduced to enhance the semantic expression ability of visual features. This mechanism helps the model pay more attention to the key areas in the medical image and improves the representation ability. (4) Report generation, the model inputs the fused visual features into the text generation module (usually based on Transformer) to gradually generate descriptive radiology reports. Although this method has made certain improvements in radiology image report generation, it still has the following technical shortcomings:

[0008] Technical shortcoming 1: fixed model structure, lack of task adaptability.

[0009] The network architecture of S3-Net is predefined before training, using a static combination of ResNet and Swin Transformer for feature extraction, which cannot dynamically adjust the model structure according to different tasks, image modalities or scenarios. This fixed structure design limits its generalization ability in cross-modal and multi-scenario scenarios.

[0010] Technical shortcoming 2: single image-text fusion method, limited cross-modal alignment ability.

[0011] The Fusion Alignment Module (FAM) used is a static structure that only performs shallow fusion without introducing task semantics or language model assisted reasoning process, so it performs unstable when dealing with complex medical terminology and regional association.

[0012] Technical shortcoming 3: lack of enhancement mechanism, lack of data diversity modeling.

[0013] S3-Net does not systematically introduce image or text enhancement strategies to simulate lesion differences or semantic variants in real medical scenarios, resulting in limited coverage of model training data and affecting inference robustness.

[0014] Technical Solution Two: Radiology Report Generation Method Based on Guided Alignment and Coarse-to-Fine Generation (Bootstrapping Large Language Models for Radiology Report Generation).

[0015] This technical solution mainly focuses on how to effectively utilize large language models (LLMs) to improve the quality of radiology image report generation (RRG). To address the low efficiency of image-text alignment and poor quality of report generation in traditional methods, a method combining "in-domain instance induction" and "coarse-to-fine generation" is proposed to bridge the knowledge gap between general LLMs and medical tasks. The implementation process of this solution is as follows: (1) In-domain Instance Induction: By constructing medical domain-specific training instances, the LLM is better aligned with radiology report text and visual features using a contrast learning strategy. This process includes selecting fragments close to the style of radiology reports from general texts, making the LLM more similar to medical expression based on the original pre-training; the contrast loss is used to enhance the image-text alignment ability, so that the medical image-related semantics can be better modeled in the LLM. (2) Coarse-to-Fine Decoding Process: After the initial generation of the report text, a candidate report ranker is used to evaluate the candidate text; combined with image features and specially designed refinement prompts, the text is generated or polished again to improve the report quality; this process forms a two-stage report generation mechanism: preliminary generation + fine enhancement, which improves the accuracy and completeness of the text. Although this method has made progress in improving the fluency and semantic rationality of the text, it still has the following technical shortcomings:

[0016] Technical Shortcoming 1: Enhancement Strategy and Structure Optimization Decoupling, Low Coordination Efficiency.

[0017] This method does not provide a systematic design of image and text enhancement strategies, nor does it optimize the linkage with the LLM invocation mechanism, resulting in limited adaptability in different tasks or scenarios. At the same time, its network structure is fixed, lacking flexibility and expansion capability.

[0018] Technical Shortcoming 2: Limited Task-Driven Capability, LLM Only Used for Text Generation.

[0019] LLM is primarily used for initial text generation and secondary polishing, but it does not participate in steps such as image-text alignment, feature modeling, or structural adjustment. It lacks the ability to guide the entire semantic chain of the task.

[0020] Technical drawback 3: The adaptation process relies on externally constructed instances, resulting in high training costs.

[0021] Domain induction requires screening and constructing training samples that conform to medical style. Contrastive learning training is complex and needs to be redesigned when extended to other modalities or tasks (such as pathological images), resulting in poor transferability.

[0022] Technical drawback 4: It requires pre-training on large-scale medical datasets, which has high resource requirements.

[0023] This approach relies on pre-training with large medical datasets such as MIMIC-CXR to bridge the knowledge gap between general LLM and the medical field. This makes the method difficult to deploy in scenarios with limited data access or low resources, thus limiting its practicality and scalability. Summary of the Invention

[0024] In view of the shortcomings of the prior art, the present invention provides a medical report generation method, model training method, device and medium. The method improves the performance of medical report generation through the following three core technical units: Global Semantic Augmentation (GSA): simulates the diverse changes in real medical scenarios and maintains key diagnostic features during the enhancement process; Medical Insight Analyzer (MIA): utilizes fine-grained and global semantic modeling to improve the cross-modal alignment capability of medical terms and image regions; Intelligent Collaborative Optimization (ICO) strategy: uses a Large Language Model (LLM) as an intelligent agent to jointly optimize the enhancement strategy and MIA network structure configuration to achieve closed-loop optimization of data-model.

[0025] Firstly, a method for training a medical report generation model is provided, including the following steps:

[0026] Acquire medical images and medical reports, standardize them, and divide them into training and validation sets;

[0027] A medical report generation model framework is constructed, comprising a global semantic collaborative multimodal enhancement module, a visual encoder, a text encoder, a medical insight analyzer, and an LLM decoder. The global semantic collaborative multimodal enhancement module enhances medical images and medical reports using selected image enhancement and text enhancement strategies, respectively. The visual encoder and text encoder extract image features and text features from the enhanced medical images and medical reports, respectively, and input them into the medical insight analyzer. The medical insight analyzer includes a fine-grained structure learner and a global context-guided learner connected in sequence to enhance the cross-modal alignment capability between medical images and medical reports. The LLM decoder generates medical reports based on the output of the medical insight analyzer.

[0028] The strategy set consisting of image enhancement strategies and text enhancement strategies, along with the architecture configuration parameters of the medical insight analyzer, are used as optimization targets for intelligent collaborative optimization. In each iteration, the medical report generation model is trained and validated using training and validation sets. The strategy set and architecture configuration parameters are updated by an LLM agent based on received task instructions, current validation set performance feedback, and historical strategy records. When the iteration termination condition is met, the optimal strategy set and architecture configuration parameters are output, resulting in the optimal medical report generation model.

[0029] Furthermore, the standardization process for medical images includes:

[0030] Size normalization of medical images;

[0031] Color standardization is performed on the normalized medical images;

[0032] For high-resolution medical images, patch segmentation is performed, and the location index is saved for global reconstruction;

[0033] The standardized processing of medical reports includes:

[0034] Segmenting medical reports into words;

[0035] The word sequence is mapped to the word vector space and padded or truncated to a uniform length.

[0036] Furthermore, the global semantic collaborative multimodal enhancement module includes an image enhancement strategy set and a text enhancement strategy set. The global semantic collaborative multimodal enhancement module uses the image enhancement strategy and text enhancement strategy selected from the strategy set by the LLM agent to enhance medical images and medical reports respectively.

[0037] Furthermore, each layer of the fine-grained structure learner is represented as follows:

[0038] MHA(Query=LN( ), Key = LN ( Value = LN ( ))+

[0039] MHA(Query=LN( ), Key = LN ( Value = LN ( ))

[0040] = + +MHA(Query=LN( ), Key = LN ( Value = LN ( ))

[0041] In the formula, The i-th layer represents the image features output by the i-th layer; MHA represents the multi-head attention layer; Query, Key, and Value represent the query vector, key vector, and value vector, respectively; LN represents the layer normalization layer. and These represent the image features and text features input to the i-th layer, respectively. Text features that incorporate visual information; The output of the i-th layer serves as the text feature input for the next layer. .

[0042] Furthermore, the global context-guided learner is represented as follows:

[0043]

[0044] = CA(Query=LN( ), Key= Value= )

[0045] = ( ))+

[0046] In the formula, This represents the output of the global context-guided learner; This represents a feedforward neural network, where LN denotes a layer normalization layer. This represents the output of the CA across the attention layer; Query, Key, and Value represent the query vector, key vector, and value vector, respectively. This represents the image features output by the last layer of the fine-grained structure learner; This represents the synthesized output of the last layer of the fine-grained structure learner.

[0047] Furthermore, the intelligent collaborative optimization process specifically includes:

[0048] Step 1: Input the task instructions and policy set into a unified search space into a task-aware LLM agent to generate the current policy set and the architecture configuration parameters of the medical insight analyzer;

[0049] Step 2: Based on the current set of strategies and the architecture configuration parameters of the medical insight analyzer, train and validate the medical report generation model using the training set and validation set, and calculate the performance feedback of the current validation set.

[0050] Step 3: Feed back the current validation set performance feedback and the current policy set and medical insight analyzer architecture configuration parameters to the LLM agent, and update the current policy set and medical insight analyzer architecture configuration parameters;

[0051] Step 4: Repeat steps 2-3 until the iteration termination condition is met, and output the optimal strategy set and the architecture configuration parameters of the medical insight analyzer.

[0052] Furthermore, constraints are introduced. ; This indicates the budget for the medical insight analyzer. This indicates the upper limit of the architecture budget.

[0053] Secondly, a method for generating medical reports is provided, including the following steps:

[0054] Acquire the medical images to be analyzed and the task description, and perform standardization processing;

[0055] The standardized data is input into the medical report generation model obtained by the training method described above, and the corresponding medical report is generated.

[0056] Thirdly, an electronic device is provided, comprising:

[0057] A memory on which computer programs are stored;

[0058] A processor is used to load and execute the computer program to implement the medical report generation model training method or medical report generation method as described above.

[0059] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the medical report generation model training method or the medical report generation method as described above.

[0060] This invention proposes a method for generating medical reports, a model training method, equipment, and media, which have the following advantages:

[0061] 1. Possesses dynamic structural adaptive capability, enhancing the model's versatility and transferability: This invention introduces an intelligent collaborative optimization (ICO) strategy, which can adaptively optimize the model structure based on task feedback, rather than being limited to a fixed two-stream structure. This allows it to maintain performance advantages in a wider range of medical image scenarios and possess stronger cross-modal and cross-task scalability.

[0062] 2. Significantly enhanced cross-modal collaborative enhancement capabilities, addressing the problem of insufficient static alignment: This invention designs a Global Semantic Augmentation (GSA) module, which achieves dynamic alignment and enhancement of visual and textual information through task-driven semantic simulation, enabling the model to better adapt to complex semantic relationships, thereby improving the consistency of image and text semantics and the accuracy of diagnostic descriptions.

[0063] 3. Capable of semantic evolution and optimization under task instruction awareness: This invention introduces a large language model (LLM) agent as a global controller, which can dynamically adjust the enhancement strategy and generation process according to task instructions and task semantics, greatly improving the applicability and flexibility of the model in complex and variable medical tasks.

[0064] 4. Collaborative optimization from data to structure to solve the problem of fragmented local improvements: This application proposes to achieve a complete optimization chain from input data augmentation and cross-modal alignment to network structure evolution through joint optimization of GSA and ICO, ensuring that the improvements of the model at different stages can work synergistically and avoid the overall performance bottleneck caused by local optima.

[0065] 5. Wider application scope and cross-modal, multi-task promotion capability: This application proposal is designed with multimodal and multi-task scenarios in mind, and can seamlessly adapt to different medical imaging modalities and tasks, with stronger cross-modal promotion and clinical application potential.

[0066] 6. The enhancement strategies are flexible and diverse, and the adaptability is better than static methods: Through the task-aware dynamic enhancement mechanism, the optimal enhancement method can be automatically selected under different tasks. This breaks through the limitation of traditional models that rely on a single self-supervised strategy. It not only improves the attention to key areas of the image, but also enhances the fluency and accuracy of generating long text diagnostic reports. It shows higher stability when dealing with changing task requirements. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart of a medical report generation model training method provided in an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the medical insight analyzer structure provided in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the FS-Learner structure provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0072] This invention proposes a medical report generation method, a model training method, an equipment, and a medium, which mainly improves the performance of model medical report generation through the following three core technologies.

[0073] (1) Global Semantic Augmentation (GSA): GSA generates augmented samples with higher realism and diversity through the collaborative design of image augmentation and text augmentation, combined with task-aware large model agent, which significantly improves the robustness of cross-modal semantic alignment, can simulate the diverse changes in real medical scenarios, and maintain key diagnostic features during the augmentation process.

[0074] (2) Medical Insight Analyzer (MIA): Utilizing fine-grained and global semantic modeling, it can extract and fuse multi-level diagnostic semantic information from medical images and report texts to form a consistent cross-modal feature representation, thereby improving the cross-modal alignment capability between medical terms and image regions.

[0075] (3) Intelligent Collaborative Optimization (ICO) Strategy: The Large Language Model (LLM) is used as an intelligent agent. Based on the dynamic instruction control and feedback loop of the large model, the data augmentation strategy and MIA network architecture configuration are jointly optimized to achieve closed-loop optimization of data-model, thereby enhancing the robustness and scalability of the medical report generation model in real applications.

[0076] The calculation scheme of the present invention will be described in detail below with reference to specific embodiments.

[0077] like Figure 1 As shown in the figure, an embodiment of the present invention discloses a method for training a medical report generation model, including the following steps:

[0078] S1: Acquire medical images and medical reports, standardize them, and divide them into training and validation sets.

[0079] Standardizing the input medical images and reports prepares the data for subsequent data augmentation and feature extraction.

[0080] In this embodiment, the standardization process for medical images includes:

[0081] Medical images are sized to ensure consistency in batch input;

[0082] Normalized medical images are color-standardized (e.g., color difference removal, brightness normalization) to reduce the impact of differences in shooting environment.

[0083] For high-resolution medical images (such as pathological slide images), perform patch segmentation (e.g., 256×256 patch) and save the location index for global reconstruction.

[0084] The standardized processing of medical reports includes:

[0085] Segmenting medical reports into words;

[0086] The word sequence is mapped to a word vector space (such as BPE encoding) and padded or truncated to a uniform length.

[0087] S2: Construct a medical report generation model framework, including a global semantic collaborative multimodal enhancement module, a visual encoder, a text encoder, a medical insight analyzer, and an LLM decoder. The global semantic collaborative multimodal enhancement module enhances medical images and medical reports using selected image enhancement and text enhancement strategies, respectively. The visual encoder and text encoder extract image features and text features from the enhanced medical images and medical reports, respectively, and input them into the medical insight analyzer. The medical insight analyzer includes a fine-grained structure learner and a global context-guided learner connected in sequence to enhance the cross-modal alignment capability between medical images and medical reports. The LLM decoder generates medical reports based on the output of the medical insight analyzer.

[0088] To enhance the ability to generate medical reports based on cross-modal medical data, this embodiment proposes a Global Semantic Collaborative Multimodal Enhancement Module (GSA). This module aims to simulate diverse changes in real-world medical environments while preserving key diagnostic features, thereby improving the robustness and generalization of the model.

[0089] The global semantic collaborative multimodal enhancement module contains a set of image enhancement strategies. and text enhancement strategy set Two parts, of which, Operations such as random resized crop, color jitter, and random rotation can be used to enhance medical images; Then, methods such as synonym replacement, back-translation, terminology substitution, and word order perturbation can be used to enhance medical reports.

[0090] The global semantic collaborative multimodal enhancement module relies on a task-aware large language model agent (LLM agent) from the image enhancement strategy set during enhancement processing. and text enhancement strategy set The selected image enhancement and text enhancement strategies are used to enhance medical images and medical reports, respectively. The processing procedure is described below:

[0091] Enhanced policy generation:

[0092] Constructing a strategy set: including an image enhancement strategy set and text enhancement strategy set ;

[0093] Strategy Selection: LLM Agent Receive task instructions and current validation set performance feedback Based on historical strategy records, new image enhancement strategies are generated. and text enhancement strategies .

[0094] Enhanced execution:

[0095] Medical images Applying image enhancement strategies Enhanced medical images were obtained. ;

[0096] Medical reports Apply text enhancement strategies The enhanced medical report was obtained. .

[0097] The process of extracting image features and text features from enhanced medical images and medical reports using a visual encoder and a text encoder, respectively, is represented as follows:

[0098]

[0099] In the formula, For visual encoder E I Extracted image features; For text encoder E T Extracted text features. These features not only preserve key lesion information but also incorporate diverse diagnostic scenario variations, providing a more robust data foundation for cross-modal alignment.

[0100] In this embodiment, the medical insight analyzer aims to enhance the cross-modal alignment capabilities between medical images and medical components, such as... Figure 2 As shown, the medical insight analyzer comprises a fine-grained structure learner and a global context-guided learner connected in sequence. The fine-grained structure learner focuses on fine-grained lesion feature modeling, and improves the model's sensitivity to abnormal tissue morphology and pathological details by introducing a residual attention mechanism in a cross-modal interaction layer. The global context-guided learner focuses on overall semantic modeling, and combines the LoRA low-rank adaptation strategy to efficiently introduce diagnostic priors and improve global feature consistency.

[0101] By utilizing a fine-grained structure learner to align key visual regions in an image with medical terminology with high sensitivity, fine-grained features such as tumor cell morphology and pathological structural boundaries can be highlighted. The output is further enhanced through an external multi-head attention mechanism (MHA) and skip connections, forming a stable representation of local details. Figure 2 As shown, the fine-grained structure learner has a multi-layer structure. Each layer consists of a first multi-head attention layer (MHA), a layer normalization layer (LN), a second multi-head attention layer, and an FS-learner unit connected in sequence.

[0102] The first multi-head attention layer updates text and image features separately through two parallel cross-attention calculations:

[0103] = MHA1 (Query= Key= Value= ) +

[0104] = MHA2(Query= Key= Value= ) +

[0105] First row formula ( (Calculation of the text features of the i-th layer of the fine-grained structure learner): As a query, it actively queries the image features of the i-th layer of the input fine-grained structure learner. The obtained information is then added to the original text features (residual connection) to obtain the updated text features. The second line of formula (Calculation of the image features input to the i-th layer of the fine-grained structure learner): As a query, it actively retrieves the text features of the i-th layer of the input fine-grained structure learner. The obtained information is then added to the original image features to obtain the updated image features. .

[0106] Among them, the FS-learner unit is as follows Figure 3 As shown, it contains the following two outputs:

[0107] Output 1: Fine-grained enhancement and updating of image features to obtain the updated image features. .

[0108] Detailed process:

[0109] Input: Image features received from the output of the first multi-head attention layer and text features ;

[0110] Feature normalization: normalizing the input image features and text features Each is fed into its respective Layer Normalization (LN) layer.

[0111] Multi-head attention fusion: Two normalized features are fed into a multi-head attention layer (MHA) for information fusion. In this attention mechanism, the model can learn and fuse key cross-modal information to improve sensitivity to pathological details;

[0112] Residual connection: Combines the output of the multi-head attention module with the original image features. Add them together;

[0113] Output: The final output is the updated image features. The corresponding formula is expressed as follows:

[0114] MHA(Query=LN( ), Key = LN ( Value = LN ( ))+

[0115] In the formula, The i-th layer represents the image features output by the i-th layer; MHA represents the multi-head attention layer; Query, Key, and Value represent the query vector, key vector, and value vector, respectively; LN represents the layer normalization layer. and Let represent the image features and text features input to the i-th layer, respectively.

[0116] Output 2: Output text features that incorporate visual information. .

[0117] The goal of this approach is to generate an enhanced text feature that incorporates the image context by performing cross-attention calculations with image features. Detailed process:

[0118] Input: Also receives image features and text features ;

[0119] Feature normalization: The two input features are passed through their respective normalization layers (LN);

[0120] Cross-attention computation: Perform a multi-head cross-attention computation; typically, normalized text features are used as the query, and normalized image features are used as the key and value to capture "visual-language internal dependencies";

[0121] Output: The direct output of the multi-head attention module is the text features that incorporate visual information. The corresponding formula is expressed as follows:

[0122] MHA(Query=LN( ), Key = LN ( Value = LN ( ))

[0123] The overall output of each layer of the fine-grained structure learner is represented as follows:

[0124] = + +MHA(Query=LN( ), Key = LN ( Value = LN ( ))

[0125] in, The output of the i-th layer serves as the text feature input for the next layer. If the fine-grained structure learner has a total of i layers, then The final integrated output of the fine-grained structure learner serves as an input to the global context-guided learner.

[0126] like Figure 2 As shown, the global context-guided learner comprises a cross-modal interactive attention layer (Cross-Attention, CA), a layer normalization layer (LN), a feedforward neural network (FFN), and a CG-Learner unit connected sequentially. By further introducing a cross-modal interactive attention layer and combining it with the feedforward neural network (FFN) and LoRA parameter efficient fine-tuning mechanism, the global diagnostic context is effectively modeled, such as the spatial relationships between lesion sites and the semantic consistency of disease classification. The modeling process is as follows:

[0127] Cross-attention captures global cross-modal dependencies, where the output of the cross-modal interaction attention layer is:

[0128] = CA(Query=LN( ), Key= Value= )

[0129] in, This represents the output of the CA across the attention layer; Query, Key, and Value represent the query vector, key vector, and value vector, respectively. This represents the image features output by the last layer of the fine-grained structure learner; This represents the synthesized output of the last layer of the fine-grained structure learner.

[0130] Utilizing CG-Learner to model the overall semantics, and efficiently adapting through LoRA parameters:

[0131]

[0132] in, = ( ))+ ; This represents the output of the global context-guided learner; denoted as feedforward neural network; LN denotes layer normalization layer. Normalization can improve the stability and convergence of the model in cross-modal alignment and deep feature extraction.

[0133] Loss function: Bidirectional contrastive loss , Used to enhance image-text alignment, ensuring its robustness and accuracy. The bidirectional contrast loss function is defined as follows:

[0134]

[0135]

[0136] Where M is the number of image-text logs in a batch, Fv is the visual embedding vector, and Ft is the text embedding vector. These are learnable parameters. In this model, Fv specifically refers to the image features output by the final MIA. Ft specifically refers to the text features of the final output. .

[0137] Through the aforementioned dual learner structure, MIA can not only capture pathological details but also globally model the semantic logic relationships of disease diagnosis, achieving end-to-end enhancement from "micro-details" to "macro-context".

[0138] The key components of the medical insight analyzer include a multi-head attention layer, a layer normalization layer, a cross-modal interactive attention layer, and a feedforward neural network. These structures enhance information flow and feature modeling capabilities within and between modalities. Furthermore, the fine-grained structure learner proposed in this embodiment is primarily responsible for modeling key diagnostic features such as abnormal regions and lesion details in images or text; the global context-guided learner focuses on global semantic consistency, achieving overall optimization of cross-modal features by integrating contextual information from multiple modalities. The medical insight analyzer constructs cross-modal interaction paths through concatenation and residual connections, ultimately using an image-text alignment loss function. , Achieve semantic alignment and representation enhancement, providing a more robust feature foundation for downstream tasks.

[0139] S3: The strategy set consisting of image enhancement strategies and text enhancement strategies, along with the architecture configuration parameters of the medical insight analyzer, are used as optimization targets for intelligent collaborative optimization. In each iteration, the medical report generation model is trained and validated using the training set and validation set. The strategy set and architecture configuration parameters are updated by the LLM agent based on the received task instructions, the current validation set performance feedback, and historical strategy records. When the iteration termination condition is met, the optimal strategy set and architecture configuration parameters are output to obtain the optimal medical report generation model.

[0140] The intelligent collaborative optimization process specifically includes:

[0141] Step 1: Input the task instructions (such as "focus on the lesion area") and the set of strategies into a task-aware LLM agent in a unified search space to generate the current set of strategies and the architecture configuration parameters of the medical insight analyzer;

[0142] Step 2: Based on the current set of strategies and the architecture configuration parameters of the medical insight analyzer, train and validate the medical report generation model using the training set and validation set, and calculate the performance feedback of the current validation set.

[0143] Step 3: Feed back the current validation set performance feedback and the current policy set and medical insight analyzer architecture configuration parameters to the LLM agent, and update the current policy set and medical insight analyzer architecture configuration parameters;

[0144] Step 4: Repeat steps 2-3 until the iteration termination condition is met, and output the optimal strategy set and the architecture configuration parameters of the medical insight analyzer.

[0145] Intelligent Collaborative Optimization (ICO) introduces a Large Language Model (LLM) agent to jointly optimize data augmentation strategies and the architectural configuration parameters (specific configurations of the neural network structure, such as the number of layers and attention heads) of the medical insight analyzer, thereby achieving optimization within a unified search space. In order to achieve efficient "data-model" co-evolution.

[0146] Objective: To jointly optimize the combination of image enhancement and text enhancement strategies within a unified search space, controlled by an LLM agent. The architecture configuration parameter 'a' of the medical insight analyzer enables the co-evolution of enhancement and structure.

[0147] This process can be viewed as a network architecture search task. The optimization objectives are as follows:

[0148]

[0149]

[0150] in Configure the weights corresponding to parameter 'a' for the optimal architecture. To train the loss function, This indicates performance on the validation set (such as accuracy, F1 score, etc.). and These represent the training set and the validation set, respectively. This represents a unified search space. These represent the optimal architecture configuration parameters and the optimal strategy combination, respectively.

[0151] To efficiently optimize the above process, a Large Language Model (LLM) is introduced as an intelligent agent. This is used to guide the search process and combine the architecture configuration parameter 'a' with the strategy. Co-evolution:

[0152]

[0153] in, and These represent the architecture configuration parameters and strategy combinations for the (i+1)th iteration, respectively. This indicates that the initial combination of architecture configuration parameters and strategies constitutes the configuration state. As iterations proceed, the configuration state... With performance indicators The process continues until it converges and the optimal solution is obtained.

[0154] After each iteration, the configuration state Performance metrics on the validation set It will be updated to:

[0155]

[0156] In the formula, This represents the best performance score on the validation set of the configuration state that has been evaluated historically after i+1 iterations; This represents the configuration state that has been evaluated historically after i iterations. In the validation set ( The cumulative best performance score; This indicates that in the current (i+1)th iteration, the newly generated architecture configuration parameters will be used. ) and strategy combination ( }) in the validation set ( The performance score obtained from ).

[0157] Finally, from the generated set of configuration states Select the optimal result ( ):

[0158]

[0159]

[0160] In the actual search process, the LLM agent acts as an intelligent controller, dynamically generating the next step's architecture configuration parameters and image enhancement and text enhancement strategies based on historical iteration results and verification performance feedback.

[0161] In some embodiments, constraints are introduced to achieve model lightweighting. ; This indicates the budget for the medical insight analyzer. This indicates the upper limit of the architecture budget.

[0162] The key differences between the solution provided in the above embodiments and the existing S3-Net are as follows:

[0163] (1) Visual encoding structure: S3-Net uses a static two-stream structure (ResNet + Swin) for image modeling. The structure is fixed and cannot adapt to the different requirements of different tasks for modeling depth or type. In contrast, this invention proposes Intelligent Cooperative Optimization (ICO), which can adaptively adjust the network structure according to task feedback, thereby improving versatility and transferability.

[0164] (2) Regarding cross-modal alignment mechanisms: FAM in S3-Net is a static module, which is weak in modeling graph-text relationships. This invention proposes a multimodal collaborative enhancement module (GSA), which enhances the graph-text collaborative modeling capability through task-driven semantic simulation.

[0165] (3) Semantic evolution capability: S3-Net relies on traditional self-supervised enhancement and cannot be adjusted according to task content or complexity. This invention introduces a Large Language Model Agent (LLM Agent) as an external "controller" to realize semantic evolution and enhancement strategy optimization under task instruction guidance, which has greater flexibility and scalability.

[0166] (4) Application scope: S3-Net mainly focuses on radiographic imaging (X-ray) tasks, while the present invention has verified its versatility in multiple sub-fields such as radiographic imaging and pathological imaging.

[0167] The differences between the solution provided in the above embodiments and existing radiological report generation methods based on guided alignment and coarse-to-fine generation are as follows:

[0168] (1) Different adaptation methods: The existing solution makes LLM closer to medical semantics by artificially constructing instance sets and combining comparative learning, while the present invention uses task instructions and diagnostic semantics to jointly guide the process, forming a multimodal collaborative enhancement and structural evolution process dominated by a large language model, which does not rely on a single training set for optimization and has stronger adaptability.

[0169] (2) Different enhancement mechanisms: The existing solution does not jointly optimize the enhancement strategies of images and text, while the present invention proposes a Global Semantic Enhancement (GSA) module, which can dynamically adjust the enhancement strategy to adapt to the task scenario and improve the modeling ability of rare cases.

[0170] (3) Lack of structural optimization capability: The existing solution adopts a static generation process and a fixed network structure, which cannot adjust the sub-model configuration according to the task. In contrast, the present invention introduces the ICO strategy, which can jointly optimize the model structure and enhancement method, and achieve full-chain adaptation from data to model;

[0171] (4) Limited multi-task expansion capability: The existing solution is mainly aimed at radiological image scenarios, while the present invention has verified its applicability in multiple medical modalities such as pathological images and radiological images, and has a wider cross-task extension capability.

[0172] This invention utilizes a collaborative mechanism led by a large language model to achieve coordinated optimization of structure and enhancement strategies through instruction guidance, enabling rapid adaptation to multimodal tasks without the need for large-scale medical pre-training. Furthermore, this invention can be designed as a lightweight, modular component framework, suitable for small to medium-sized medical data scenarios, and possesses stronger generalization capabilities and practical deployment flexibility.

[0173] This invention also provides a method for generating medical reports, comprising the following steps:

[0174] Acquire the medical images to be analyzed and the task description, and perform standardization processing;

[0175] The standardized data is input into the medical report generation model obtained by the training method of the medical report generation model as described in the previous embodiment to generate the corresponding medical report.

[0176] Furthermore, embodiments of the present invention also provide an electronic device, comprising:

[0177] A memory on which computer programs are stored;

[0178] A processor is used to load and execute the computer program to implement the medical report generation model training method or medical report generation method as described above.

[0179] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the medical report generation model training method or the medical report generation method as described above.

[0180] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0181] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0185] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A medical report generation model training method, characterized by, The method comprises the following steps: obtaining medical images and medical reports and performing standardization processing, and dividing into a training set and a validation set; building a medical report generation model framework, including a global semantic collaborative multi-modal enhancement module, a visual encoder, a text encoder, a medical insight analyzer and an LLM decoder; the global semantic collaborative multi-modal enhancement module uses selected image enhancement strategies and text enhancement strategies to enhance the medical images and the medical reports respectively; the visual encoder and the text encoder extract image features and text features of the enhanced medical images and medical reports respectively, and input the medical insight analyzer; the medical insight analyzer includes a fine-grained structure learner and a global context guided learner connected in turn to enhance the cross-modal alignment capability between the medical images and the medical reports; the LLM decoder generates a medical report based on the output of the medical insight analyzer; the strategy set composed of the image enhancement strategies and the text enhancement strategies and the architecture configuration parameters of the medical insight analyzer are used as optimization objectives for intelligent collaborative optimization, in each iteration process, the training set and the validation set are used to train and verify the medical report generation model, and the LLM agent updates the strategy set and the architecture configuration parameters according to the received task instructions, the current validation set performance feedback and the historical strategy record; when the iteration termination condition is met, the optimal strategy set and the architecture configuration parameters are output, and the optimal medical report generation model is obtained. 2.The medical report generation model training method of claim 1, wherein, The standardization processing process of the medical images comprises: normalizing the size of the medical images; color standardization is performed on the normalized medical images; for high-resolution medical images, patch segmentation is performed, and the position index is saved for global reconstruction; The standardization processing process of the medical report comprises: segmenting the medical report; map the word sequence to the word vector space, and fill or truncate to a uniform length. 3.The medical report generation model training method of claim 1, wherein, The global semantic collaborative multi-modal enhancement module includes an image enhancement strategy set and a text enhancement strategy set, and the global semantic collaborative multi-modal enhancement module uses the image enhancement strategies and the text enhancement strategies selected from the strategy set by the LLM agent to enhance the medical images and the medical reports respectively. 4.The medical report generation model training method of claim 1, wherein, In the medical insight analyzer, each layer of the fine-grained structure learner is represented as follows: MHA (Query = LN( ), Key = LN( ), Value = LN( )) + ; MHA (Query = LN( ), Key = LN( ), Value = LN( )) ; = + +MHA(Query=LN( ),Key=LN( ),Value=LN( )); wherein, represents the image feature output by the i-th layer; MHA represents a multi-head attention layer; Query, Key, Value represent query vector, key vector and value vector, respectively; LN represents a layer normalization layer; and respectively represent the image feature and the text feature input by the i-th layer; represents the text feature fused with visual information; is the comprehensive output of the i-th layer, which is the text feature input by the next layer . 5.The medical report generation model training method of claim 4, wherein, In the medical insight analyzer, the global context guided learner is represented as follows: ； = CA(Query = LN( ), Key= , Value= ); = ( ))+ ； In the formula, represents the output of the global context guidance learner; represents a feedforward neural network, and LN represents a layer normalization layer; represents the output of the cross-attention layer CA; Query, Key, and Value represent query vectors, key vectors, and value vectors, respectively; represents the image features output by the last layer of the fine-grained structure learner; represents the comprehensive output of the last layer of the fine-grained structure learner. 6.The medical report generation model training method of claim 1, wherein, The intelligent collaborative optimization process specifically comprises: Step 1: input a task-aware LLM agent with task instructions and a unified search space strategy set to generate the current strategy set and the architecture configuration parameters of the medical insight analyzer; Step 2: based on the current strategy set and the architecture configuration parameters of the medical insight analyzer, train and verify the medical report generation model using the training set and the validation set, and calculate the current validation set performance feedback; Step 3: feed back the current validation set performance feedback, the current strategy set and the architecture configuration parameters of the medical insight analyzer to the LLM agent to update the current strategy set and the architecture configuration parameters of the medical insight analyzer; Step 4: Repeat step step2-step3 until the iteration termination condition is met, and output the optimal strategy set and the architecture configuration parameters of the medical insight analyzer.

7. The medical report generation model training method of claim 1, wherein, Introducing constraints in the intelligent collaborative optimization process ; represents an architecture budget for a medical insight analyzer, represents an architecture budget upper bound.

8. A medical report generation method characterized by, The method comprises the following steps: Obtaining a medical image to be analyzed and a task description and performing standardization processing; Inputting the standardized data into a medical report generation model obtained by the medical report generation model training method according to any one of claims 1 to 7, to generate a corresponding medical report.

9. An electronic device, comprising: It comprises: A memory having a computer program stored thereon; A processor configured to load and execute the computer program to implement the medical report generation model training method according to any one of claims 1 to 7 or the medical report generation method according to claim 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the medical report generation model training method according to any one of claims 1 to 7 or the medical report generation method according to claim 8.

Images