A Report Intelligent Generation Method and System Based on Multimodal Fusion

Abstract

This invention discloses a method and system for intelligent report generation based on multimodal fusion. The method includes multimodal data input and preprocessing, single-modal feature extraction, multimodal information balancing, cross-modal interaction and fusion, report content generation, report post-processing and output, and loss function training and optimization. This invention relates to the field of natural language processing technology, and particularly to a method and system for intelligent report generation based on multimodal fusion. This scheme constructs a multimodal fusion model, employing a "pattern-semantic" dual-channel feature extraction and a two-stage hierarchical fusion method to achieve intermodal collaborative integration and fully exploit complementary information. By introducing a multimodal information balancing mechanism, this scheme dynamically evaluates the contribution of each modality and adjusts the gradient update weights, effectively preventing text dominance, improving the model's sensitivity to low-frequency but high-value visual signals, and enhancing the comprehensiveness and objectivity of the report.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for intelligent report generation based on multimodal fusion. Background Technology

[0002] With the rapid development of information technology, data is experiencing explosive growth, including text, images, videos, audio, sensor data, and tabular data. Its sources and forms are also becoming increasingly diverse. In multiple fields such as healthcare, finance, security, industrial inspection, and scientific research, professionals need to generate detailed analytical reports based on this multi-source, heterogeneous data. Traditional report generation methods mainly rely on manual work, which has the following significant drawbacks: existing technologies often can only process single-modal data (such as processing only text or only images), failing to effectively correlate and integrate information from different modalities, resulting in reports lacking comprehensiveness and in-depth insight; while multimodal learning has made progress, there is still a lack of effective solutions for how to deeply align cross-modal information, achieve semantic complementarity and conflict resolution, and ultimately generate long, structured reports that conform to professional standards and are logically rigorous. Summary of the Invention

[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a report intelligent generation method and system based on multimodal fusion. Addressing the problem that existing technologies can only process single-modal data and cannot effectively correlate and fuse information from different modalities, resulting in reports lacking comprehensiveness, this solution constructs a multimodal fusion model. It employs a "pattern-semantic" dual-channel feature extraction and a two-stage hierarchical fusion method: the first stage achieves fine-grained alignment of text and images at the local pattern and high-level semantic levels through cross-attention; the second stage first performs intra-modal weighted fusion, and then achieves inter-modal collaborative integration through a correlation matrix, fully mining complementary information. Regarding the challenge of how to deeply align cross-modal information, achieve semantic complementarity, and resolve conflicts in multimodal learning, this solution introduces a multimodal information balancing mechanism. This dynamically evaluates the contribution of each modality and adjusts the gradient update weights, effectively preventing text dominance and ensuring that low-frequency but high-value key visual signals in the image are fully expressed. This enhances the model's sensitivity to low-frequency but high-value visual signals, improving the comprehensiveness and objectivity of the report.

[0004] The present invention provides a report intelligent generation method based on multimodal fusion, which includes the following steps:

[0005] Step S1: Multimodal data input and preprocessing. Receive the original multimodal dataset to be analyzed and preprocess the data in the dataset to obtain a preprocessed dataset. The dataset contains data in two modalities: text modal data and image modal data.

[0006] Step S2: Single-modal feature extraction. Extract modal features and semantic features from the text and image modal data in the preprocessed data, resulting in four feature channels. This yields initial text modal features, initial text semantic features, initial image modal features, and initial image semantic features. The features extracted from the four feature channels are then projected onto a unified dimension, collectively referred to as modal features.

[0007] Step S3: Multimodal information balancing, calculate the activation information of each modality feature and the difference rate between modalities, and design an adaptive balancing factor to weight and regulate gradient updates accordingly;

[0008] Step S4: Cross-modal interaction and fusion. Construct a multimodal fusion model and use a two-stage feature fusion method to hierarchically fuse pattern features and semantic features to obtain multimodal fusion features.

[0009] Step S5: Report content generation. Based on the multimodal fusion features and the preset report template library, the Transformer decoder is used to generate preliminary report content.

[0010] Step S6: Post-processing and output of the report. Post-process the generated preliminary report content and output the final structured report document.

[0011] Step S7: Loss function training and optimization. Generative loss and semantic consistency loss are used to jointly train and optimize the parameters of the Transformer decoder.

[0012] Furthermore, in step S3, the multimodal information balancing specifically includes the following steps:

[0013] Step S31: Basic gradient update, for the model parameters of the feature channels of each modality feature, the gradient is updated using the normalization method;

[0014] Step S32: Calculate the information content of each modality. For each modality feature, use the logit score to obtain the information content of each modality feature.

[0015] Step S33: Quantify modal information content, convert the logit score into a comparable information content index, calculate the softmax value of each modal feature on the true class, define the quantized modality, and quantify the information content of different modal features;

[0016] Step S34: Calculate the modality difference rate, and use the difference rate to calculate the proportion of information contributed by modality features to the total information of all modality features;

[0017] Step S35: Balance Factor. Set the balance factor to adjust the gradient update intensity of each mode. The balance factor is defined as follows:

[0018] ;

[0019] In the formula, Indicates the number of iterations. Representing modal features, For hyperparameters, Indicates the difference rate. Represents the balance factor. It is the hyperbolic tangent function;

[0020] Step S36: Gradient update. Introduce the balancing factor into the original gradient update formula to obtain the improved parameter update rule. The formula used is as follows:

[0021] ;

[0022] In the formula, and They represent the first Second and third Sub-model parameters. For learning rate, Representing modes The gradient;

[0023] Furthermore, in step S4, cross-modal interaction and fusion specifically includes the following steps:

[0024] Step S41: Double-cross Transformer interaction. A double-cross Transformer model is used to perform cross-modal interaction at both the modal and semantic levels, obtaining the resulting modal and semantic features. The formulas used are as follows:

[0025] ;

[0026] In the formula, The pattern is a two-way Transformer interaction model. and These represent text pattern features and image pattern features, respectively. and These represent the text pattern features and image pattern features after interaction, respectively. The parameters of the double-cross Transformer interaction model of the pattern;

[0027] Step S42: Intramodal multi-view fusion, using attention weighting to perform text fusion and image fusion respectively, to obtain fused text features and image features, using the following formula:

[0028] ;

[0029] In the formula, This represents an attention network. For parameters in the attention network, For the fused text features, Represents the semantic features of the text after interaction;

[0030] Step S43: Intermodal correlation fusion, concatenating the fused text and image features, calculating the correlation matrix, and using an activation function to obtain the enhanced text and image features. The correlation matrix is ​​calculated as follows:

[0031] ;

[0032] In the formula, For text-image joint matrix, For transpose operation, Scaling factor Represents the image correlation matrix. Image features for fusion;

[0033] Step S44: The fusion feature output is obtained by concatenating the enhanced text features and image features to obtain the multimodal fusion feature.

[0034] The present invention provides a report intelligent generation system based on multimodal fusion, which includes a data collection and preprocessing module, a feature extraction module, a multimodal fusion module, a report generation module, and a template and knowledge base;

[0035] The data collection and preprocessing module receives raw multimodal data, obtains raw multimodal dataset, and preprocesses the raw multimodal dataset to obtain preprocessed dataset;

[0036] The feature extraction module extracts pattern features and semantic features from the text and image modal data in the preprocessed data, respectively, resulting in four feature channels: initial text pattern features, initial text semantic features, initial image pattern features, and initial image semantic features. The features extracted from the four feature channels are then projected onto a unified dimension, collectively referred to as modal features.

[0037] The multimodal fusion module is connected to the feature extraction module to construct a multimodal fusion model, perform cross-modal interaction and fusion, hierarchically fuse pattern features and semantic features to obtain multimodal fusion features, and send them to the report generation module.

[0038] The report generation module includes a structure planning unit and a content generation unit. The structure planning unit is used to generate a report outline, and the content generation unit uses a Transformer-based decoder to generate each part of the content and performs post-processing on the generated preliminary report text to finally generate a structured report document.

[0039] The template is connected to the knowledge base in the report generation module, storing various report templates, domain ontology, and standard terminology library for reference during the report generation process.

[0040] The beneficial effects achieved by adopting the above solution are as follows:

[0041] (1) In view of the problem that existing technologies can only process single-modal data and cannot effectively associate and fuse information from different modalities, resulting in a lack of comprehensiveness in the generated reports, this solution constructs a multimodal fusion model and adopts a "pattern-semantic" dual-channel feature extraction and a two-stage hierarchical fusion method: the first stage realizes fine-grained alignment of images and text at the local pattern and high-level semantic level through cross attention; the second stage first performs intramodal weighted fusion, and then realizes intermodal collaborative integration through the correlation matrix to fully explore complementary information;

[0042] (2) In response to the problem of how to deeply align cross-modal information, semantic complementarity and conflict resolution in multimodal learning, this solution introduces a multimodal information balancing mechanism to dynamically evaluate the contribution of each modality and adjust the gradient update weights. This effectively prevents text dominance, ensures that low-frequency but high-value key visual signals in the image are fully expressed, improves the model's sensitivity to low-frequency but high-value visual signals, and enhances the comprehensiveness and objectivity of the report. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the intelligent report generation method based on multimodal fusion proposed in this invention.

[0044] Figure 2 This is a schematic diagram of the intelligent report generation system based on multimodal fusion proposed in this invention.

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0046] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0047] Example 1, see Figure 1 The present invention provides a report intelligent generation method based on multimodal fusion, which includes the following steps:

[0048] Step S1: Multimodal data input and preprocessing. Receive the original multimodal dataset to be analyzed and preprocess the data in the dataset to obtain a preprocessed dataset. The dataset contains data in two modalities: text modal data and image modal data.

[0049] Step S2: Single-modal feature extraction. Extract modal features and semantic features from the text and image modal data in the preprocessed data, resulting in four feature channels. This yields initial text modal features, initial text semantic features, initial image modal features, and initial image semantic features. The features extracted from the four feature channels are then projected onto a unified dimension, collectively referred to as modal features.

[0050] Step S3: Multimodal information balancing, calculate the activation information of each modality feature and the difference rate between modalities, and design an adaptive balancing factor to weight and regulate gradient updates accordingly;

[0051] Step S4: Cross-modal interaction and fusion. Construct a multimodal fusion model and use a two-stage feature fusion method to hierarchically fuse pattern features and semantic features to obtain multimodal fusion features.

[0052] Step S5: Report content generation. Based on the multimodal fusion features and the preset report template library, the Transformer decoder is used to generate preliminary report content.

[0053] Step S6: Post-processing and output of the report. Post-process the generated preliminary report content and output the final structured report document.

[0054] Step S7: Loss function training and optimization. Generative loss and semantic consistency loss are used to jointly train and optimize the parameters of the Transformer decoder.

[0055] Example 2, based on the above examples, in step S1, the audio data and tabular data in the original multimodal dataset are converted into text data, and the video data, image data, and thermal imaging data are processed as image data. The preprocessing of each modal data includes the following:

[0056] Text data: performing word segmentation, stop word removal, named entity recognition (NER), and dependency parsing;

[0057] Image data: used for target detection, scene classification, OCR text recognition, and keyframe extraction.

[0058] Example 3, based on the above examples, includes the following steps in step S3 for multimodal information balancing:

[0059] Step S31: Basic gradient update, for the model parameters of the feature channels of each modality feature, the gradient is updated using the normalization method;

[0060] Step S32: Calculate the information content of each modality. For each modality feature, use the logit score to obtain the information content of each modality feature.

[0061] Step S33: Quantify modal information content, convert the logit score into a comparable information content index, calculate the softmax value of each modal feature on the true class, define the quantized modality, and quantify the information content of different modal features;

[0062] Step S34: Calculate the modality difference rate. Use the difference rate to calculate the proportion of information contributed by modal features to the total information of all modal features. The formula used is as follows:

[0063] ;

[0064] In the formula, Indicates the difference rate. The information content score represents the modal feature. For training batches, For training batch index, and These represent the modal information scores of the initial image pattern features and the initial image semantic features, respectively. and These represent the modal information scores of the initial text pattern features and the initial text semantic features, respectively.

[0065] Step S35: Balance Factor. Set the balance factor to adjust the gradient update intensity of each mode. The balance factor is defined as follows:

[0066] ;

[0067] In the formula, Indicates the number of iterations. Representing modal features, For hyperparameters, Represents the balance factor. It is the hyperbolic tangent function;

[0068] Step S36: Gradient update. Introduce the balancing factor into the original gradient update formula to obtain the improved parameter update rule. The formula used is as follows:

[0069] ;

[0070] In the formula, and They represent the first Second and third Sub-model parameters. For learning rate, Representing modes The gradient.

[0071] To address the challenges of deep alignment, semantic complementarity, and conflict resolution of cross-modal information in multimodal learning, this solution introduces a multimodal information balancing mechanism. This mechanism dynamically evaluates the contribution of each modality and adjusts the gradient update weights, effectively preventing text dominance and ensuring that low-frequency but high-value key visual signals in images are fully expressed. This enhances the model's sensitivity to low-frequency but high-value visual signals and improves the comprehensiveness and objectivity of the report.

[0072] Example 4, based on the above examples, includes the following steps in step S4 for cross-modal interaction and fusion:

[0073] Step S41: Double-cross Transformer interaction. A double-cross Transformer model is used to perform cross-modal interaction at both the modal and semantic levels, obtaining the resulting modal and semantic features. The formulas used are as follows:

[0074] ;

[0075] ;

[0076] In the formula, and These are dual-cross Transformer interaction models, representing both pattern and semantic aspects. and These represent text pattern features and image pattern features, respectively. and These represent text semantic features and image semantic features, respectively. and These represent the text pattern features and image pattern features after interaction, respectively. and These represent the semantic features of the text and the semantic features of the image after interaction, respectively. and These are the parameters of the dual-cross Transformer interaction model, representing both pattern and semantic aspects.

[0077] Step S42: Intramodal multi-view fusion, using attention weighting to perform text fusion and image fusion respectively, to obtain fused text features and image features, using the following formula:

[0078] ;

[0079] ;

[0080] In the formula, This represents an attention network. and For parameters in the attention network, For the fused text features, Image features for fusion;

[0081] Step S43: Intermodal correlation fusion, concatenating the fused text and image features, calculating the correlation matrix, and using an activation function to obtain the enhanced text and image features. The correlation matrix is ​​calculated as follows:

[0082] ;

[0083] In the formula, For text-image joint matrix, For transpose operation, Scaling factor Represents the image correlation matrix;

[0084] Step S44: The fusion feature output is obtained by concatenating the enhanced text features and image features to obtain the multimodal fusion feature.

[0085] By performing the aforementioned operations, this solution addresses the problem that existing technologies can only process single-modal data and cannot effectively correlate and fuse information from different modalities, resulting in reports lacking comprehensiveness. It constructs a multimodal fusion model, employing a "pattern-semantic" dual-channel feature extraction and a two-stage hierarchical fusion method: the first stage uses cross-attention to achieve fine-grained alignment of text and images at the local pattern and high-level semantic levels; the second stage first performs intra-modal weighted fusion, and then uses a correlation matrix to achieve inter-modal collaborative integration, fully mining complementary information.

[0086] Example 5, based on the above examples, in step S5, the report content is generated, including the following:

[0087] Structure planning: Select or dynamically generate a suitable report outline from the template library based on the task type (such as diagnostic report, audit report, incident briefing). The task types include diagnostic report, audit report, and incident briefing. The report outline includes summary, background, findings, analysis, conclusions, and recommendations.

[0088] Content generation: A sequence-to-sequence model based on Transformer is used as the decoder, with multimodal fusion features as the initial state and conditional input, and natural language descriptions are generated segment by segment in combination with the chapter titles of the report outline;

[0089] Controllable generation: During the generation process, keyword constraints, factual consistency verification methods, and style control tags are introduced to ensure the factual accuracy, professionalism, and style consistency of the generated content.

[0090] By performing the aforementioned operations, this approach addresses the current lack of methods to elucidate the molecular mechanisms underlying the high incidence of pCR in locally advanced rectal cancer. It utilizes Lasso regression modeling to optimize key variables, constructs a C-BLSTM network to dynamically capture changes in variables during treatment, builds nomograms to integrate dynamic features and clinical data, calculates the overall risk score for each patient, quantifies the contribution of each variable to the therapeutic effect, comprehensively analyzes the molecular mechanisms of short-course therapy, and predicts the probability of pCR.

[0091] Example 6, based on the above examples, includes the following post-processing of the generated preliminary report content in step S6:

[0092] Post-processing of the initial report text includes grammar and spell checking: corrections are made using language tools;

[0093] Terminology standardization: Replace with domain-standard terminology;

[0094] Redundancy elimination and coherence optimization: Ensure natural transitions between paragraphs and avoid repetition;

[0095] Visual element embedding: Automatically insert key images and charts into specified locations in the report as needed.

[0096] Example 7, based on the above examples, provides a multimodal fusion-based intelligent report generation system. This system includes a data collection and preprocessing module, a feature extraction module, a multimodal fusion module, a report generation module, and a template and knowledge base.

[0097] The data collection and preprocessing module receives the raw multimodal data, obtains the raw multimodal dataset, and preprocesses the raw multimodal dataset to obtain the preprocessed dataset, and then executes step S1;

[0098] The feature extraction module extracts modal features and semantic features from the text and image modal data in the preprocessed data, respectively, resulting in four feature channels: initial text modal features, initial text semantic features, initial image modal features, and initial image semantic features. The features extracted from the four feature channels are then projected onto a unified dimension, collectively referred to as modal features. Step S2 is then executed.

[0099] The multimodal fusion module is connected to the feature extraction module to construct a multimodal fusion model, perform cross-modal interaction and fusion, perform hierarchical fusion of pattern features and semantic features to obtain multimodal fusion features, and send them to the report generation module to execute steps S3 and S4.

[0100] The report generation module includes a structure planning unit and a content generation unit. The structure planning unit is used to generate a report outline, and the content generation unit uses a Transformer-based decoder to generate each part of the content. The generated preliminary report text is then post-processed to finally generate a structured report document. Steps S5 to S7 are then executed.

[0101] The template is connected to the knowledge base in the report generation module, storing various report templates, domain ontology, and standard terminology library for reference during the report generation process.

[0102] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0104] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for report intelligence generation based on multi-modal fusion, characterized in that: The method comprises the following steps: Step S1: multi-modal data input and preprocessing, receiving the original multi-modal data set to be analyzed, and preprocessing the data of the data set to obtain a preprocessed data set, the data set containing data of two modalities, text modal data and image modal data; Step S2: single-modal feature extraction, extracting mode features and semantic features from the text and image modal data in the preprocessed data, a total of four feature channels, obtaining initial text mode features, initial text semantic features, initial image mode features and initial image semantic features, and projecting the features extracted by the four feature channels to a unified dimension, collectively referred to as modal features; Step S3: multi-modal information balancing, calculating the activation information amount and the inter-modal difference rate of each modal feature, and designing an adaptive balancing factor to weight and control the gradient update; Step S4: cross-modal interaction and fusion, constructing a multi-modal fusion model, using a two-stage feature fusion method to perform hierarchical fusion on the mode features and the semantic features, and obtaining multi-modal fusion features; Step S5: report content generation, presetting a report template library, generating preliminary report content using a Transformer decoder based on the multi-modal fusion features and the preset report template library; Step S6: report post-processing and output, post-processing the generated preliminary report content, and outputting a final structured report document; Step S7: loss function training and optimization, using a generative loss and a semantic consistency loss to jointly train and optimize the parameters of the Transformer decoder; In step S3, the multi-modal information balancing specifically comprises the following steps: Step S31: basic gradient update, using a standardization method to update the gradient of the model parameters of each modal feature channel; Step S32: calculating the information amount of each modality, using the logit score to obtain the information amount of each modal feature; Step S33: quantifying the modal information amount, converting the logit score into a comparable information amount index, calculating the softmax value of each modal feature on the real class, and defining the quantified modality to quantify the information amount of different modal features; Step S34: calculating the difference rate of the modal, using the difference rate to calculate the proportion of the information amount contributed by the modal feature to the total information amount of all modal features; Step S35: balancing factor, setting a balancing factor to adjust the gradient update strength of each modality; Step S36: gradient update, introducing the balancing factor into the original gradient update formula to obtain an improved parameter update rule.

2. The report intelligent generation method based on multi-modal fusion according to claim 1, characterized in that: In step S4, cross-modal interaction and fusion, specifically comprising the following steps: Step S41: double-cross Transformer interaction, using a double-cross Transformer model to perform cross-modal interaction at the mode and semantic levels respectively, obtaining the mode features and semantic features after interaction; Step S42: multi-view fusion within the modality, using attention weighting to perform text fusion and image fusion respectively, obtaining fused text features and image features; Step S43: inter-modal correlation fusion, splicing the fused text features and image features, calculating the correlation matrix, and using an activation function to obtain enhanced text features and image features; Step S44: fusion feature output, splicing the enhanced text features and image features to obtain multi-modal fusion features.

3. The report intelligent generation system based on multi-modal fusion, for implementing the report intelligent generation method based on multi-modal fusion as claimed in any one of claims 1-2, characterized in that: The system comprises a data collection and preprocessing module, a feature extraction module, a multi-modal fusion module, a report generation module, and a template and knowledge base; The data collection and preprocessing module receives original multi-modal data, obtains an original multi-modal data set, and pre-processes the original multi-modal data set to obtain a pre-processed data set; The feature extraction module extracts mode features and semantic features from the text and image modal data in the pre-processed data, respectively, for a total of four feature channels, and projects the features extracted by the four feature channels to a unified dimension, collectively referred to as modal features; The multi-modal fusion module is connected to the feature extraction module, constructs a multi-modal fusion model, performs cross-modal interaction and fusion, hierarchically fuses the mode features and semantic features, obtains multi-modal fusion features, and sends them to the report generation module; The report generation module includes a structure planning unit and a content generation unit, the structure planning unit is used to generate a report outline, the content generation unit generates each part of the content using a decoder based on the Transformer, and the generated preliminary report text is post-processed to finally generate a structured report document; The template and knowledge base are connected to the report generation module, store various report templates, domain ontologies, and standard terminology libraries, and are used as references for the report generation process.

Images