Large model-based multi-modal data-driven general report generation method and system

Abstract

The application discloses a multi-modal data driven general report generation method and system based on a large model, and belongs to the technical field of intelligent report generation. First, text, images and sensor data related to the report theme are acquired and standardized preprocessing is performed. The report generation instruction is analyzed, and the task modality mapping library is matched according to the task demand to match the modality configuration scheme. The quality of each modality data is quantitatively evaluated, the final decision weight of each modality is dynamically calculated in combination with the basic weight, and is distributed to the corresponding processing path to form leading, supplementary and reference data. The leading data and supplementary data are input into a multi-modal model for analysis to obtain a preliminary conclusion with a confidence score. After consistency judgment, if there is no conflict, the comprehensive conclusion is formed by fusion, and if there is a conflict, the conflict is resolved in combination with the quality evaluation result and the arbitration rule. Finally, the comprehensive conclusion and the reference data are input into a large language model to generate a report text, and the complete report is output after layout and proofreading.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent report generation, in particular to a general report generation method and system driven by multi-modal data based on a large model. BACKGROUND

[0002] Currently, multi-modal report generation methods based on large language models have become an important direction of automated text production. Such methods usually adopt a post-fusion paradigm, that is, first, data of different modalities are independently encoded into feature vectors, and then the feature vectors are integrated through feature splicing or attention mechanisms, and finally, a large model is input to generate a report. However, this paradigm essentially treats data of different sources and different qualities equally, and its fusion process is static and lacks prior judgment.

[0003] For this reason, existing methods show obvious limitations when facing the inherent uncertainty of multi-modal data: first, the system cannot perceive and respond to the quality fluctuations of specific data sources before fusion, resulting in low-quality data equally interfering with the analysis results; second, when data of different sources draw conflicting conclusions, there is a lack of an explicit, rule-based arbitration mechanism to resolve the conflict, and the system can only rely on implicit calculations within the model, which may lead to a confusing report logic or ambiguous conclusions; finally, the entire fusion process is disconnected from the specific chapters and goals of the report, and cannot dynamically adjust the core data sources relied upon according to different task goals, thereby affecting the professionalism and relevance of the final report. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal data-driven general report generation method and system based on a large model to solve the problems raised in the background.

[0005] To solve the above technical problems, the present application provides the following technical solution: a multi-modal data-driven general report generation method based on a large model, comprising:

[0006] Step S100: Obtain multi-modal raw data related to the report theme, including text data, image data, and numerical sensor data, and perform standardization preprocessing to form a multi-modal data set;

[0007] Step S200: Receive a report generation instruction, analyze the report theme, target chapter, and core analysis requirements contained therein; based on the analysis results, query a pre-defined task modality mapping library to match a modality configuration scheme suitable for the current generation task; the modality configuration scheme includes core modalities, auxiliary modalities, and their corresponding basic weight priorities;

[0008] Step S300: independently quantitatively evaluate the quality of each type of modal data in the multi-modal data set; combine the quality score with the basic weight in the modal configuration scheme to dynamically calculate the final decision weight of each modal data; and according to the high and low of the final decision weight, respectively distribute each type of modal data to the leading data processing path, the supplementary data processing path and the reference data processing path, to form the leading data, the supplementary data and the reference data correspondingly;

[0009] Step S400: input the leading data and the supplementary data into the pre-trained multi-modal model for preliminary analysis to obtain a preliminary conclusion with a confidence score; after consistency checking, if the two types of conclusions are logically consistent, take the leading data conclusion as the core, and fuse the supplementary data conclusion to form a comprehensive analysis conclusion; if the checking finds that the conclusions exist logical conflicts, combine the quality evaluation results with the pre-defined arbitration rules to make conflict arbitration to generate a comprehensive analysis conclusion;

[0010] Step S500: input the comprehensive analysis conclusion and the reference data into the large language model to generate corresponding report paragraph text, and perform formatting, layout and content proofreading to finally output a report with complete structure.

[0011] Further, step S100 includes:

[0012] Step S101: obtain text data, image data and numerical sensor data related to the report theme through a data interface; the data interface includes a text interface, an image interface and a sensor interface; the text interface interfaces with a document database and a log system to obtain text data, the image interface interfaces with a camera and an image storage server to obtain image data, and the sensor interface interfaces with an Internet of Things sensor gateway to obtain numerical sensor data; extract core keywords in the report theme, match the keywords with the meta information of each source data, retain the data containing the core keywords in the meta information, eliminate the data irrelevant to the core keywords, and form a multi-modal original data set directly related to the report theme;

[0013] Step S102: correct errors and perform semantic segmentation on the text data in the multi-modal original data set, denoise and normalize the resolution of the image data, and filter outliers and align time stamps of the numerical sensor data;

[0014] Step S103: store each type of preprocessed modal data to a distributed data pool, establish a data index, and associate the data index with the data source and the preprocessing timestamp to form a structured multi-modal data set.

[0015] Further, step S200 includes:

[0016] Step S201: receiving a report generation instruction input by a user through an interactive interface, the instruction including a report topic description, a target chapter list to be generated, and core analysis requirement specifications, wherein the core analysis requirement specifications include problems to be solved or analysis goals to be achieved by the report;

[0017] Step S202: structurally analyzing the report generation instruction, extracting an application scenario category to which the report topic belongs through semantic analysis, identifying a logical hierarchical relationship of the target chapters to determine a chapter sequence, and decomposing core analysis requirement elements of each chapter; finally outputting a standardized analysis result including the application scenario category, the chapter sequence, and the requirement element of each chapter;

[0018] Step S203: taking the target chapter name and the core analysis requirement obtained by analysis as a composite query condition, querying a pre-defined task modality mapping library; the mapping library is indexed by a combination of chapter types and analysis requirements, and is mapped to a corresponding modality configuration scheme; the modality configuration scheme specifies a core modality and an auxiliary modality for the current report generation task in a structured data format, and assigns a basic weight priority value to each modality;

[0019] Step S204: for each target chapter in the report, retrieving and loading the corresponding modality configuration scheme from the mapping library to establish a mapping relationship; if there is no completely matched scheme for a chapter, a nearest neighbor matching algorithm based on semantic similarity is used to select the configuration scheme with the highest similarity from the mapping library as the default scheme.

[0020] Further, step S300 includes:

[0021] Step S301: for text data, generating a sentence-level semantic vector by sentence-by-sentence encoding of the text content through a pre-trained semantic understanding model; after converting the core analysis requirement keywords into semantic vectors, calculating the similarity between each sentence vector and the keyword vector, marking sentences with a similarity higher than a set threshold as valid information sentences, and the text information completeness is the ratio of the number of valid information sentences to the total number of sentences in the text, denoted as R txt1 ; at the same time, matching the professional terms in the text with the domain standard term table, the term consistency is the ratio of the number of matching successful terms to the total number of terms in the text, denoted as R txt2 ; according to a preset weight distribution relationship, R txt1 and R txt2 are weighted and summed to obtain a text quality score Q txt ;

[0022] For image data, a clarity evaluation model based on a deep network is used to extract features from the image, output an edge gradient value reflecting the richness of image edge details, and obtain a clarity index Rimg1 ; identifying a key region related to the core analysis requirement in the image through a target detection algorithm, calculating a ratio of a total amount of pixels of the key region to a total amount of pixels of the image to obtain a key region coverage rate R img2 ; performing weighted summation on R img1 and R img2 according to a preset weight distribution relationship to obtain an image quality score Q img ;

[0023] For numerical sensor data, a sliding window algorithm is used to segment the time series data, the ratio of the number of valid data points in each window to the total number of data points in the window is calculated, and the average value of all windows is taken as the time series integrity index R sen1 ; comparing the fluctuation range of the sensor data with the historical fluctuation interval of the numerical sensor under normal working conditions, calculating the overlap ratio of the actual fluctuation range and the historical normal interval as the fluctuation rationality index R sen2 ; performing weighted summation on R sen1 and R sen2 according to a preset weight distribution relationship to obtain a sensor quality score Q sen ;

[0024] The text quality score Q txt , the image quality score Q img and the sensor quality score Q sen are processed by the same normalization method to make all quality scores within a preset same numerical interval.

[0025] Step S302: Obtain the basic weight priority of each modality in step S200, and introduce a field adaptation coefficient and a data size coefficient; wherein the field adaptation coefficient is dynamically set according to the professional attribute of the application scene to which the report belongs, and represents the difference in relative importance of each type of modality data in different application scenes; the data size coefficient is the ratio of the effective data amount of a single modality to the total effective data amount of the multi-modal data set, wherein the text data is measured by the number of sentences, the image data is measured by the number of image frames, and the numerical sensor data is measured by the number of valid data points. When the proportion of the data amount of a certain modality exceeds a preset threshold, the data size coefficient is adjusted through a nonlinear transformation;

[0026] Then, the final decision weight W f of each modality is calculated by the following formula: W f =λ×γ×(α×P base +β×Q); wherein λ is the field adaptation coefficient, γ is the data size coefficient, and λ>0, γ>0; P base is the basic weight priority, Q is the quality score of the corresponding modality, α and β are preset weight coefficients of P base and Q respectively and α+β=1;

[0027] Step S303: Design an adaptive threshold routing mechanism. First, based on the weight distribution of all modalities of the current task, the mean μ and standard deviation σ are calculated to dynamically generate two levels of thresholds: the dominant path threshold T main based on the mean plus the standard deviation of the preset multiple; the reference path threshold T ref based on the mean minus the standard deviation of the preset multiple; then, the modal data is classified and distributed: if W f >=T main , the modal data is assigned to the dominant data processing path, marked as core analysis data, and associated with the quality score and weight calculation basis; if T ref <=W f <T main , it is assigned to the supplementary data processing path, marked as auxiliary verification data, and the features are associated with the dominant data for verification; if W f <T ref , it is assigned to the reference data processing path, marked as background support data, and only the key features are retained for report background supplementation.

[0028] Step S304: Establish data interaction rules between paths. The dominant data processing path can call the features of the supplementary data for conclusion verification, the supplementary data path needs to feedback the verification results to the dominant path, and the reference data path only responds to the background query request of the large language model in the report generation stage.

[0029] Further, step S400 includes:

[0030] Step S401: Modal adaptation preprocessing of dominant data and supplementary data: text data is converted into token sequences compatible with multi-modal models, image data is adjusted to a preset resolution and completed pixel normalization, and numerical sensor data is reconstructed into a time series feature matrix according to the time dimension; the two types of preprocessed data are input into the pre-trained multi-modal model, the model completes feature fusion and analysis through cross-modal attention mechanism, and outputs the preliminary conclusion with confidence score C: dominant conclusion Con_main generated based on dominant data, and supplementary conclusion Con_supp generated based on supplementary data; wherein the confidence score C ∈ [0, 100], which is obtained by linear mapping of the softmax probability value of the multi-modal model output layer;

[0031] The multi-modal model selects a pre-trained cross-modal base model as a backbone network, which includes a text encoder, an image encoder, and a time series data encoder, which respectively process text, image, and numerical sensor data; then a labeled dataset of the target field is collected, including text, image, sensor data triplets and corresponding labeled conclusions; based on the labeled dataset, the model is trained through fine-tuning to adapt to the application field characteristics of the current report generation task: a domain term embedding layer is added to the text encoder, a target detection head of the application field to which the current report generation task belongs is connected to the image encoder, and a sliding window attention mechanism is added to the time series encoder to capture the dynamic features of the sensor data;

[0032] Then, confidence calibration is performed, and a temperature scaling method is used to calibrate the probability distribution output by the multi-modal model. The optimal temperature parameter T is determined on the validation set through grid search, and the original softmax output pi is converted to the calibrated probability p'i = exp(zi / T) / ∑ j exp(zj / T); where i represents the target class index of the multi-modal model output conclusion, pi represents the original prediction probability of the i-th conclusion output by the multi-modal model; p'i represents the calibrated probability value of the i-th conclusion after temperature scaling, with a value range of [0, 1]; j represents the traversal index of all output conclusion classes of the multi-modal model; zi is the original prediction value of the last layer output in the multi-modal model; finally, a conflict detection branch is added to the model output layer, and the consistency of the conclusion and the labeled result is predicted through a binary classification head, and the training of the multi-modal analysis model is finally completed.

[0033] Step S402: Calculate the semantic similarity Sim between Con_main and Con_supp using a pre-trained semantic matching model, Sim ∈ [0, 100]: if Sim >= Tsim, determine that the conclusions are consistent, and start the fusion process: take the logical framework of Con_main as the core, extract the detailed information related to the core analysis requirements in Con_supp, integrate the core viewpoints and supplementary arguments into a comprehensive analysis conclusion Con_syn according to the structure, and label the modal type corresponding to the supplementary information and the quality score of the corresponding modal; where Tsim is a preset consistency threshold used to determine whether there is a conflict between the conclusions, and its value is set according to the mapping relationship between the consistency of the conclusions and the accuracy of the final report in historical data; the trained semantic matching model first encodes the text content of Con_main and Con_supp respectively to generate corresponding sentence-level semantic vectors; then, the basic matching degree of the two semantic vectors is calculated by the cosine similarity algorithm, and the basic matching degree is normalized and converted to a semantic similarity Sim with a value range of [0, 100] through linear transformation;

[0034] Step S403: If Sim < Tsim, it is determined that there is a conflict, and a conflict resolution mechanism is triggered to modify the modal quality score of step S301 and the final decision weight W of step S302 f The core resolution basis is: when the dominant data corresponds to Q main >Q supp , and W f(main) >W f(supp) , the conflict part in Con_supp is modified based on Con_main, and is integrated into Con_syn; wherein Q main is the quality score corresponding to the dominant data, Q supp is the quality score corresponding to the supplementary data; W f(main) is the final decision weight of the dominant data, and W f(supp) is the final decision weight of the supplementary data

[0035] The specific implementation of modifying the conflict part in Con_supp based on Con_main is: first, the core viewpoints, supporting arguments and judgment dimensions of the two types of conclusions are disassembled through a semantic analysis model, and the conflict part of the core viewpoints that are opposite and the arguments that cannot be compatible are located through semantic comparison; then, the core viewpoints and argument logic of Con_main are taken as a framework, and the effective arguments in Con_supp that have no conflict with them are retained; finally, the conflict part of Con_supp is processed, if the conflict argument has data deviation, the core viewpoint is directly eliminated and modified to match Con_main, if there is expression ambiguity, semantic adjustment is performed to realize logical compatibility, and finally the modified Con_supp and Con_main are integrated into Con_syn, ensuring that the integrated conclusion is consistent in core viewpoints and fuses the effective information of both parties

[0036] When Q supp -Q main >=ΔQ and W f(supp) -W f(main) >=ΔW, wherein ΔQ is a preset quality score difference threshold, and ΔW is a preset weight difference threshold, the background data related to the conflict point in the reference data processing path is called, input into the multi-modal model for secondary analysis, and the auxiliary verification conclusion Con_ref with a confidence score C_ref is generated; then a confidence score weighting screening mechanism is started to determine Con_syn: first, the confidence score C main of Con_main, the confidence score C supp of Con_supp, and the confidence score C ref of Con_ref are extracted, then the final decision weight of the data corresponding to each conclusion is taken as a weighting coefficient, and the weighted scores of the three are calculated according to the formula: weighted score = (C main ×W f(main) +C supp ×Wf(supp) +C ref ×W f(ref) ) / (W f(main) +W f(supp) +W f(ref) ), where W f(ref) This represents the final decision weight corresponding to the reference data. If a certain conclusion has the highest weighted score and the difference between its weighted scores and those of the other two types of conclusions exceeds a preset advantage threshold, then this conclusion is used as the core framework, and the non-conflicting detailed arguments from the other two types of conclusions are integrated to generate Con_syn. If the difference between the highest and second-highest weighted scores does not exceed the advantage threshold, then the matching degree between the supporting arguments of the three types of conclusions and the reference data is further compared, and the conclusion with the highest matching degree is selected as the benchmark, and the effective information from other conclusions is integrated to form Con_syn.

[0037] When |Q main -Q supp |<ΔQ and|W f(main) -W f(supp) When |<ΔW, a predefined domain arbitration rule library is loaded, and an appropriate rule is selected according to the rule priority to generate Con_syn; the predefined domain arbitration rule library is a structured set of rules built based on the professional knowledge and historical decision-making experience of the application domain to which the current report generation task belongs, and stores the priority handling strategy for the corresponding conflict by indexing the scenario type and conflict characteristics;

[0038] Step S404: Perform structured encapsulation on Con_syn to generate a standardized data packet containing the core judgment conclusion, supporting data index, and comprehensive confidence score C_syn; where the comprehensive confidence score C_syn = (C main ×W f(main) +C supp ×W f(supp) ) / (W f(main) +W f(supp) ); C main Score the confidence level of Con_main, C supp Score the confidence level of Con_supp.

[0039] Furthermore, step S500 includes:

[0040] Step 501: First, input the comprehensive analysis conclusion Con_syn from the standardized data package generated in step S404 and the background support data output from the reference data processing path into the pre-trained large language model. At the same time, load the chapter sequence and the requirement elements of each chapter as generation constraints. The large language model integrates the core viewpoints, supporting arguments, and scene information of the background support data of Con_syn according to the chapter logical hierarchy and core analysis requirements to generate chapter paragraph text that conforms to the report format specifications.

[0041] The pre-trained large language model takes a Transformer encoder-decoder as the architecture, where the encoder processes the semantic encoding of the comprehensive analysis conclusion, background data, and chapter requirements, and the decoder generates report text conforming to the format. The input layer performs unified embedding and interaction of multi-source information, and the output layer fuses domain-specific and general word tables to generate natural language text conforming to the report format specification and containing professional terms. At the same time, the text specification and coherence are ensured through the format constraint layer and the logical coherence verification layer. Model training first constructs a training set using cross-domain text, and then performs joint training through mask language modeling and report generation. Then, fine-tuning is performed on the target domain data, and domain adaptation is completed through multi-loss function optimization. Finally, task adaptation training is performed through simulated report generation of the whole process of data, and the chapter logic connection is optimized.

[0042] Step 502: Then, according to the preset report template, including font style, title level, and chart insertion rules, the chapter text is typeset, and the image data and numerical sensor data mentioned in the text are automatically associated. The image data is inserted at the corresponding resolution and the data source index is labeled, and the numerical sensor data is generated into a time trend chart or a statistical table.

[0043] Step 503: After typesetting, trigger the content proofreading mechanism, correct the text errors through the grammar correction model, check the consistency of professional terms through the domain term verification library, and combine the comprehensive confidence C_syn of step S404 to filter expressions with confidence less than the preset confidence threshold and prompt manual review. After the proofreading is passed, integrate all chapter texts, charts, and background information to generate the final report, and store the report generation log synchronously, including data source, processing timestamp, and weight calculation record, to the distributed data pool.

[0044] The multi-modal data-driven general report generation system based on a large model includes a multi-modal data acquisition module, a requirement analysis matching module, a data quality evaluation module, a multi-modal data analysis and fusion module, and a report generation module.

[0045] The multi-modal data acquisition module acquires multi-modal raw data related to the report theme, including text data, image data, and numerical sensor data, and performs standardization preprocessing to form a multi-modal data set.

[0046] The requirement analysis matching module receives the report generation instruction, analyzes the report theme, target chapter, and core analysis requirements contained therein, and based on the analysis result, queries the pre-defined task modality mapping library to match the modality configuration scheme suitable for the current generation task. The modality configuration scheme includes the core modality, auxiliary modality, and their corresponding basic weight priority.

[0047] The data quality evaluation module independently quantitatively evaluates the quality of each type of modal data in the multi-modal data set; combines the quality score with the basic weight in the modal configuration scheme to dynamically calculate the final decision weight of each modal data; and according to the high and low of the final decision weight, respectively distributes each type of modal data to a leading data processing path, a supplementary data processing path and a reference data processing path, to form leading data, supplementary data and reference data correspondingly;

[0048] The multi-modal data analysis and fusion module inputs the leading data and the supplementary data into a pre-trained multi-modal model for preliminary analysis to obtain a preliminary conclusion with a confidence score; after consistency checking, if the two types of conclusions are logically consistent, the leading data conclusion is taken as the core to fuse the supplementary data conclusion to form a comprehensive analysis conclusion; if the checking finds that the conclusions have logical conflicts, the quality evaluation result and the pre-defined arbitration rule are combined for conflict arbitration to generate a comprehensive analysis conclusion;

[0049] The report generation module inputs the comprehensive analysis conclusion and the reference data into a large language model to generate corresponding report paragraph text, and performs formatting and content proofreading to finally output a report with complete structure.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] The present application effectively improves the data usability and subsequent analysis efficiency by directional acquisition and standardized preprocessing of multi-modal data, combined with keyword matching to filter theme-related data and eliminate redundant information, while unifying data format and quality standards; by analyzing report generation instructions to match modal configuration schemes, introducing field adaptation and data scale coefficient to dynamically calculate modal decision weight, and combining adaptive threshold routing to realize hierarchical data processing, the core data dominates the analysis, the supplementary data assists the verification, and the reference data supports the background, which improves the data utilization pertinence and analysis accuracy;

[0052] In the conclusion generation link, the reliability of the preliminary conclusion is ensured by multi-modal model feature fusion and confidence calibration, and the consistency of the conclusions is checked by semantic similarity, and in the conflict scenario, quality evaluation, reference data secondary analysis and conflict arbitration are called in layers to effectively avoid conclusion conflicts and improve the accuracy of the comprehensive conclusion; finally, the large language model generates standard text according to the chapter constraints, and the automatic layout and multi-dimensional proofreading are matched to ensure that the report format is compliant, the terminology is accurate and the content is complete; the overall report generation efficiency is improved, the content quality and professionalism are guaranteed, and it is suitable for different field report requirements, solving the problems of complicated data processing, low conclusion reliability, high labor cost and poor adaptability in traditional report generation. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of the specification, illustrate embodiments of the application, and are used to explain the present application, but are not intended to limit the present application. In the drawings:

[0054] Figure 1 is a method flowchart of a large model-based multi-modal data-driven general report generation method. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0056] Please refer to Figure 1 The present application provides a technical solution: a large model-based multi-modal data-driven general report generation method, comprising:

[0057] Step S100: acquiring multi-modal original data related to a report theme, including text data, image data and numerical sensor data, and performing standardization preprocessing to form a multi-modal data set;

[0058] Step S200: receiving a report generation instruction, analyzing the report theme, target chapter and core analysis requirement contained therein; based on the analysis result, querying a pre-defined task modality mapping library to match a modality configuration scheme suitable for the current generation task; the modality configuration scheme includes a core modality, an auxiliary modality and their corresponding basic weight priorities;

[0059] Step S300: independently quantifying and evaluating the quality of each type of modality data in the multi-modal data set; combining the quality score with the basic weight in the modality configuration scheme to dynamically calculate the final decision weight of each modality data; and according to the high and low of the final decision weight, distributing each type of modality data to a leading data processing path, a supplementary data processing path and a reference data processing path, respectively, to form leading data, supplementary data and reference data correspondingly;

[0060] Step S400: inputting the leading data and supplementary data into a pre-trained multi-modal model for preliminary analysis to obtain preliminary conclusions with confidence scores; after consistency checking, if the two types of conclusions are logically consistent, taking the leading data conclusion as the core to form a comprehensive analysis conclusion by fusing the supplementary data conclusion; if the checking finds that the conclusions have logical conflicts, combining the quality evaluation result with the pre-defined arbitration rules to make conflict arbitration and generate a comprehensive analysis conclusion;

[0061] Step S500: input the comprehensive analysis conclusion and reference data into the large language model, generate the corresponding report paragraph text, and perform formatting and content proofreading, finally output the complete report structure.

[0062] Step S100 includes:

[0063] Step S101: obtain text data, image data and numerical sensor data related to the report theme through a data interface; the data interface includes a text interface, an image interface and a sensor interface; the text interface connects a document database and a log system to obtain text data, the image interface connects a camera and an image storage server to obtain image data, and the sensor interface connects an Internet of Things sensor gateway to obtain numerical sensor data; extract the core keywords in the report theme, match the keywords with the meta information of each source data, retain the data containing the core keywords in the meta information, eliminate the data irrelevant to the core keywords, and form a multi-modal original data set directly related to the report theme;

[0064] Step S102: correct errors and perform semantic segmentation on text data in the multi-modal original data set, denoise and normalize resolution on image data, and filter outliers and align timestamps on numerical sensor data;

[0065] Step S103: store each type of preprocessed modal data to a distributed data pool, establish a data index, and associate the data source and preprocessing timestamp to form a structured multi-modal data set.

[0066] Step S200 includes:

[0067] Step S201: receive the report generation instruction input by the user through the interactive interface, the instruction includes report theme description, target chapter list to be generated and core analysis requirement description, wherein the core analysis requirement description includes the problem to be solved or the analysis target to be achieved by the report;

[0068] Step S202: structurally analyze the report generation instruction, extract the application scenario category to which the report theme belongs through semantic analysis, identify the logical hierarchical relationship of the target chapters to determine the chapter sequence, and disassemble the core analysis requirement elements of each chapter; finally output the standardized analysis result including the application scenario category, chapter sequence and requirement elements of each chapter;

[0069] Step S203: The parsed target chapter name and core analysis requirement are taken as a composite query condition to query a predefined task modality mapping library; the mapping library is indexed by a combination of chapter type and analysis requirement, and is mapped to a corresponding modality configuration scheme; the modality configuration scheme specifies the core modality and auxiliary modality of the current report generation task in a structured data format, and assigns a basic weight priority value to each modality;

[0070] Step S204: For each target chapter in the report, the corresponding modality configuration scheme is retrieved and loaded from the mapping library to establish a mapping relationship; if there is no completely matched scheme for a chapter, a nearest neighbor matching algorithm based on semantic similarity is used to select the configuration scheme with the highest similarity from the mapping library as the default scheme.

[0071] Step S300 includes:

[0072] Step S301: For text data, the text content is encoded sentence by sentence by a pre-trained semantic understanding model to generate a sentence-level semantic vector; after the core analysis requirement keywords are converted into semantic vectors, the similarity of each sentence vector and the keyword vector is calculated, and the sentences with a similarity higher than a set threshold are marked as valid information sentences; the text information completeness is the ratio of the number of valid information sentences to the total number of sentences in the text, denoted as R txt1 ; meanwhile, the professional terms in the text are matched with the domain standard term table, and the term consistency is the ratio of the number of matched terms to the total number of terms in the text, denoted as R txt2 ; according to a preset weight distribution relationship, R txt1 and R txt2 are weighted and summed to obtain the text quality score Q txt ;

[0073] For image data, a clarity evaluation model based on a deep network is used to extract features from the image, output an edge gradient value reflecting the richness of image edge details, and obtain a clarity index R img1 after normalization processing; a target detection algorithm is used to identify key regions related to the core analysis requirement in the image, and the ratio of the total number of pixels in the key regions to the total number of pixels in the image is calculated to obtain a key region coverage rate R img2 ; according to a preset weight distribution relationship, R img1 and R img2 are weighted and summed to obtain an image quality score Q img ;

[0074] For numerical sensor data, a sliding window algorithm is used to segment the time series data, and the ratio of the number of valid data points in each window to the total number of data points in the window is calculated, and the mean value of all windows is taken as a time series integrity index R sen1; compare the fluctuation range of sensor data with the historical fluctuation interval of the numerical sensor under normal working conditions, calculate the overlap ratio of the actual fluctuation range and the historical normal interval as the fluctuation rationality index R sen2 ; according to the preset weight distribution relationship, the R sen1 and R sen2 are weighted and summed to obtain the sensor quality score Q sen ;

[0075] The text quality score Q txt , the image quality score Q img and the sensor quality score Q sen are processed by the same normalization method, so that all quality scores are in the same preset numerical interval.

[0076] Step S302: Obtain the basic weight priority of each modality in step S200, while introducing a field adaptation coefficient and a data size coefficient; wherein the field adaptation coefficient is dynamically set according to the professional attribute of the application scene to which the report belongs, indicating the relative importance difference of each type of modality data under different application scenes; the data size coefficient is the ratio of the effective data amount of a single modality to the total effective data amount of the multi-modal data set, wherein the text data is measured by the number of sentences, the image data is measured by the number of image frames, and the numerical sensor data is measured by the number of effective data points. When the proportion of the data amount of a certain modality exceeds a preset threshold, the data size coefficient is adjusted through a nonlinear transformation;

[0077] Then, the final decision weight W f of each modality is calculated by the following formula: W f =λ×γ×(α×P base +β×Q); wherein λ is the field adaptation coefficient, γ is the data size coefficient, and λ>0, γ>0; P base is the basic weight priority, Q is the quality score of the corresponding modality, α and β are the preset weight coefficients of P base and Q respectively and α+β=1;

[0078] Step S303: Design an adaptive threshold routing mechanism. First, based on the weight distribution of all modalities of the current task, calculate the mean μ and the standard deviation σ, and dynamically generate two thresholds: the dominant path threshold T main is generated based on the preset multiple of the mean plus standard deviation; the reference path threshold T ref is generated based on the preset multiple of the mean minus standard deviation; then classify and distribute the modality data: if W f >=T main , the modality data is distributed to the dominant data processing path and marked as core analysis data, and its quality score and weight calculation basis are associated; if T ref <=Wf <T main If W f <T ref If W

[0079] Step S304: Establishing the inter-path data interaction rule, setting the main data processing path to call the features of the supplementary data for conclusion verification, and the supplementary data path to feed back the verification result to the main path, and the reference data path to only respond to the background query request of the large language model in the report generation stage.

[0080] Step S400 includes:

[0081] Step S401: Modal adaptation preprocessing of the main data and the supplementary data: converting text data into token sequences compatible with the multi-modal model, adjusting image data to a preset resolution and completing pixel normalization, and reconstructing numerical sensor data into a time series feature matrix according to the time dimension; inputting the preprocessed two types of data into the pre-trained multi-modal model, and the model completing feature fusion and analysis through cross-modal attention mechanism, outputting the preliminary conclusion with confidence score C: the main conclusion Con_main generated based on the main data, and the supplementary conclusion Con_supp generated based on the supplementary data; wherein the confidence score C ∈ [0, 100], which is obtained by linear mapping of the softmax probability value of the output layer of the multi-modal model;

[0082] The multi-modal model selects a pre-trained cross-modal basic model as the backbone network, which contains a text encoder, an image encoder, and a time series data encoder three independent branches, respectively processing text, image, and numerical sensor data; then collecting the labeled data set of the target field, including the text, image, sensor data triplets and the corresponding labeled conclusion; based on the labeled data set, the model is trained to adapt to the features of the application field to which the current report generation task belongs: adding a domain term embedding layer to the text encoder, connecting a target detection head of the application field to which the current report generation task belongs to the image encoder, and adding a sliding window attention mechanism to the time series encoder to capture the dynamic features of the sensor data;

[0083] Then, the confidence is calibrated, the temperature scaling method is used to calibrate the probability distribution output by the multi-modal model, and the optimal temperature parameter T is determined on the validation set through grid search to convert the original softmax output pi to the calibrated probability p'i = exp(zi / T) / ∑ jexp(zj / T); wherein, i represents the target category index of the multi-modal model output conclusion, pi represents the original prediction probability of the i-th conclusion output by the multi-modal model; p'i represents the i-th conclusion probability value after temperature scaling calibration, and the value range is [0, 1]; j represents the traversal index of all output conclusion categories of the multi-modal model; zi is the original prediction value of the last layer output in the multi-modal model; and finally a conflict detection branch is added to the model output layer, the consistency of the conclusion and the labeled result is predicted through a binary classification head, and finally the training of the multi-modal analysis model is completed.

[0084] Step S402: Calculate the semantic similarity Sim of Con_main and Con_supp by using a pre-trained semantic matching model, Sim ∈ [0, 100]: if Sim >= Tsim, it is determined that there is no conflict in the conclusion, and the fusion process is started: taking the logical framework of Con_main as the core, extracting the detailed information related to the core analysis requirement in Con_supp, integrating the core view and the supplementary argument into a comprehensive analysis conclusion Con_syn according to the structure, and labeling the modal type corresponding to the supplementary information and the quality score corresponding to the modal; wherein, Tsim is a preset consistency threshold value for determining whether there is a conflict in the conclusion, and the value is set according to the mapping relationship between the conclusion consistency and the final report accuracy in historical data; the training semantic matching model first encodes the text content of Con_main and Con_supp respectively to generate corresponding sentence-level semantic vectors; then the basic matching degree of the two semantic vectors is calculated by using the cosine similarity algorithm, and after the normalization processing, the basic matching degree is converted into the semantic similarity Sim with the value range of [0, 100] through linear transformation;

[0085] Step S403: If Sim < Tsim, it is determined that there is a conflict, and the conflict resolution mechanism is triggered, and the modal quality score in step S301 and the final decision weight W f are taken as the core resolution basis: when the Q main of the dominant data is greater than the Q supp of the supplementary data, and the W f(main) of the dominant data is greater than the W f(supp) of the supplementary data, Con_main is taken as the basis to modify the conflict part in Con_supp and integrate it into Con_syn; wherein, Q main is the quality score corresponding to the dominant data, Q supp is the quality score corresponding to the supplementary data; W f(main) is the final decision weight of the dominant data, and W f(supp) is the final decision weight of the supplementary data.

[0086] The specific implementation of correcting the conflicting part in Con_supp based on Con_main is as follows: first, the core viewpoints, supporting arguments and judgment dimensions of the two types of conclusions are disassembled through a semantic analysis model, and the conflicting part of the core viewpoints that are opposite and the arguments that cannot be compatible are located through semantic comparison; then, the effective arguments in Con_supp that are not in conflict with the core viewpoints and argument logic of Con_main are retained; finally, the conflicting part of Con_supp is processed, if the conflicting arguments have data bias, the core viewpoints are directly eliminated and corrected to match Con_main, if there is ambiguity in expression, semantic adjustment is performed to realize logical compatibility, and finally the corrected Con_supp and Con_main are integrated into Con_syn, ensuring that the comprehensive conclusion is consistent in core viewpoints and fuses the effective information of both parties;

[0087] when Q supp -Q main >=ΔQ and W f(supp) -W f(main) >=ΔW, wherein ΔQ is a preset mass fraction difference threshold, ΔW is a preset weight difference threshold, the background data related to the conflict point in the reference data processing path is called, input into the multi-modal model for secondary analysis, and the auxiliary verification conclusion Con_ref with a confidence score C_ref is generated; then a confidence score weighting screening mechanism is started to determine Con_syn: first, the confidence score C main of Con_main, the confidence score C supp of Con_supp, and the confidence score C ref of Con_ref are extracted, and then the final decision weight of the data corresponding to each conclusion is used as a weighting coefficient, and the weighted scores of the three are calculated according to the formula: weighted score=(C main ×W f(main) +C supp ×W f(supp) +C ref ×W f(ref) ) / (W f(main) +W f(supp) +W f(ref) ), wherein W f(ref) represents the final decision weight corresponding to the reference data; if the weighted score of a conclusion is the highest and the difference between the weighted scores of the other two types of conclusions exceeds a preset advantage threshold, the conclusion is taken as the core framework, and the non-conflicting detailed arguments in the other two types of conclusions are integrated to generate Con_syn; if the difference between the highest and the second highest weighted scores does not exceed the advantage threshold, the matching degrees of the supporting arguments of the three types of conclusions and the reference data are further compared, the conclusion with the highest matching degree is taken as the reference, and the effective information of the other conclusions is fused to form Con_syn;

[0088] when |Q main -Q supp| < AQ and | < W f(main) - W f(supp) | < AW, load a predefined domain arbitration rule library, select an adaptive rule according to rule priority to generate Con_syn; the predefined domain arbitration rule library is a structured rule set constructed based on professional knowledge and historical decision experience of the application domain to which the current report generation task belongs, indexed by scene type and conflict characteristics, and stores priority processing strategies corresponding to conflicts;

[0089] Step S404: structuring and packaging Con_syn to generate a standardized data package containing core judgment conclusions, supporting data indexes, and comprehensive confidence C_syn; wherein comprehensive confidence C_syn=(C main × W f(main) + C supp × W f(supp) ) / (W f(main) + W f(supp) ); C main is the confidence score of Con_main, and C supp is the confidence score of Con_supp.

[0090] Step S500 includes:

[0091] Step 501: first input the comprehensive analysis conclusion Con_syn in the standardized data package generated in step S404 and the background supporting data output by the reference data processing path into the pre-trained large language model, simultaneously load the chapter sequence and each chapter requirement element determined in step S202 as a generation constraint condition, and the large language model fuses the core viewpoints of Con_syn, supporting arguments, and scene information of background supporting data according to chapter logic levels and core analysis requirements to generate chapter paragraph text conforming to report format specifications;

[0092] The pre-trained large language model takes a Transformer encoder-decoder as an architecture, wherein the encoder processes semantic encoding of the comprehensive analysis conclusion, background data, and chapter requirements, the decoder generates report text conforming to the format, the input layer performs unified embedding and interaction of multi-source information, the output layer fuses domain-specific and general word tables, natural language text conforming to report format specifications and containing professional terms; meanwhile, the text is standardized and coherent through a format constraint layer and a logical coherence verification layer; model training first constructs a training set using cross-domain text, and performs joint training through mask language modeling and report generation double tasks; then fine-tunes the target domain data, and completes domain adaptation through multi-loss function optimization; finally, task adaptation training is performed through simulation report generation full-process data, and chapter logic connection is optimized;

[0093] Step 502: Then, according to the preset report template, including font style, title level, chart insertion rules, the text of each chapter is typeset, and the image data and numerical sensor data mentioned in the text are automatically associated; wherein the image data is inserted with corresponding resolution and the data source index is labeled, and the numerical sensor data is generated into a time trend chart or a statistical table;

[0094] Step 503: After the typesetting is completed, the content proofreading mechanism is triggered, the text errors are corrected by the grammar correction model, the professional term consistency is checked by the domain term verification library, and the expressions with confidence less than the preset confidence threshold are screened and manually reviewed in combination with the comprehensive confidence C_syn of step S404. After the proofreading is passed, all chapter texts, charts and background information are integrated to generate a final report, and report generation logs including data sources, processing timestamps and weight calculation records are stored in a distributed data pool.

[0095] Embodiment 1: Taking the generation of a regional substation monthly operation and maintenance report as an example: first, the device inspection records and fault handling text data are obtained by connecting the substation operation and maintenance log database and the device account system through a text interface, the image data of the transformer oil level, circuit breaker contact and other key components are obtained by connecting the on-site high-definition camera and image storage server through an image interface, and the numerical sensor data of the transformer vibration, bus temperature, current and voltage are obtained by connecting the Internet of Things sensor gateway through a sensor interface; the core keywords of the substation monthly operation and maintenance are extracted, the data meta information is matched, the irrelevant data of other substations is excluded, the text data is corrected for errors and the power professional semantic segmentation is performed, the image data is denoised and the resolution is normalized, the sensor data is filtered for abnormal values such as transient voltage surge and aligned with the timestamp, stored in a distributed data pool to establish an index of associated data sources, and form a multi-modal data set;

[0096] The user inputs the report generation instructions through the interactive interface: "substation monthly operation and maintenance" theme, "device running state-fault analysis-maintenance suggestion" chapter list and "evaluate core device health degree and potential risk" requirement; the power operation and maintenance scene, chapter sequence and each chapter requirement element are parsed, the "device running state chapter and core device health degree evaluation" is taken as a condition to query the task modal mapping library, and the configuration scheme of "core modal: numerical sensor data, auxiliary modal: image data, basic weight 1:0.7" is matched; the three types of data are quantitatively evaluated, the text data calculates the effective information sentence proportion and the power terminology matching degree to obtain Q txt =90, the image data calculates the sharpness and key component coverage to obtain Q img =85, and the sensor data calculates the time sequence integrity and fluctuation reasonableness to obtain Q sen=95, the introduction of the field adaptation coefficient (sensor data λ=1.3) and the data size coefficient (sensor data γ=1.2), the sensor data W is calculated according to the formula f =0.94, image data W f =0.72, text data W f =0.60, according to the adaptive threshold, the sensor data is assigned as the dominant data, the image data is assigned as the supplementary data, and the text data is assigned as the reference data;

[0097] The dominant and supplementary data are preprocessed and input into the multi-modal model, and the conclusion Con_main with confidence is output: the vibration value of the No. 3 transformer is slightly above the threshold, C=96; Con_supp: the oil level of the No. 3 transformer is normal without appearance defects, C=92; the semantic similarity Sim=88 is calculated ≥Tsim=82, without the need for conflict resolution; taking Con_main as the core, the effective argument "oil level normal" in Con_supp is fused, and the comprehensive conclusion Con_syn is integrated and the standardized data package is packaged.

[0098] The Con_syn and the text reference data are input into the large language model, the chapter sequence and the demand elements are loaded to generate chapter text; the power operation and maintenance report template is arranged, the No. 3 transformer vibration trend graph and the oil level image marked with the source are inserted, the text is corrected through grammar and power terminology verification, the expressions with a confidence lower than 90% are selected, and the complete operation and maintenance report is directly integrated and generated, and the data source, timestamp and other logs are stored in the distributed data pool.

[0099] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and range of equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be considered as limiting the claims involved.

Claims

1. A general report generation method based on multimodal data driven by large models, characterized by: The method includes: Step S100: Obtain multimodal raw data related to the report topic, including text data, image data, and numerical sensor data, and perform standardized preprocessing to form a multimodal dataset; Step S200: Receive the report generation instruction, parse the report topic, target chapter and core analysis requirements contained therein; based on the parsing results, query the predefined task modality mapping library, and match to obtain a modality configuration scheme suitable for the current generation task; the modality configuration scheme includes core modality, auxiliary modality and their corresponding basic weight priority; Step S300: Independently quantify and evaluate the quality of each type of modality data in the multimodal dataset; combine the quality score with the basic weight in the modality configuration scheme to dynamically calculate the final decision weight of each modality data; and according to the level of the final decision weight, allocate each type of modality data to the dominant data processing path, the supplementary data processing path and the reference data processing path respectively to form dominant data, supplementary data and reference data. Step S300 includes: Step S301: For the text data, the text content is encoded sentence by sentence using a pre-trained semantic understanding model to generate sentence-level semantic vectors. After converting the core analysis requirement keywords into semantic vectors, the similarity between each sentence vector and the keyword vector is calculated. Sentences with similarity higher than a set threshold are marked as valid information sentences. The text information completeness is the ratio of the number of valid information sentences to the total number of sentences in the text, denoted as R. txt1 Simultaneously, the technical terms in the text are matched against a domain-standard thesaurus. Terminology consistency is defined as the ratio of the number of successfully matched terms to the total number of terms in the text, denoted as R. txt2 According to the preset weight allocation relationship, R is... txt1 With R txt2 We perform a weighted summation to obtain the text quality score Q. txt ; For image data, a sharpness evaluation model based on deep networks is used to extract features from the images, outputting edge gradient values ​​that reflect the richness of edge details. After normalization, the sharpness index R is obtained. img1 The algorithm identifies key regions in the image that are relevant to the core analysis requirements. The ratio of the total number of pixels in these key regions to the total number of pixels in the image is calculated to obtain the key region coverage R. img2 According to the preset weight allocation relationship, R is... img1 With R img2 We perform weighted summation to obtain the image quality score Q. img ; For numerical sensor data, a sliding window algorithm is used to segment the time-series data. The ratio of the number of valid data points in each window to the total number of data points in the window is calculated, and the average of all windows is taken as the time-series integrity index R. sen1 The fluctuation range of sensor data is compared with the historical fluctuation range of numerical sensors under normal operating conditions. The overlap ratio between the actual fluctuation range and the historical normal range is calculated and used as the fluctuation rationality index R. sen2 According to the preset weight allocation relationship, R is... sen1 With R sen2 We perform a weighted summation to obtain the sensor quality score Q. sen ; Text quality score Q txt Image quality score Q img and sensor quality Q sen The same normalization method is used to process all mass scores within the same preset numerical range. Step S400: Input the dominant data and supplementary data into the pre-trained multimodal model for preliminary analysis to obtain preliminary conclusions with confidence scores; after consistency verification, if the two types of conclusions are logically consistent, the conclusion of the dominant data is taken as the core and the conclusion of the supplementary data is integrated to form a comprehensive analysis conclusion; if the verification finds that there is a logical conflict in the conclusions, the conflict is adjudicated by combining the quality assessment results with the predefined arbitration rules to generate a comprehensive analysis conclusion. Step S500: Input the comprehensive analysis conclusions and reference data into the large language model to generate the corresponding report paragraph text, and perform formatting and content proofreading to finally output a structurally complete report.

2. The method for generating general reports based on large-model multimodal data according to claim 1, characterized in that: Step S100 includes: Step S101: Obtain text data, image data, and numerical sensor data related to the report topic through data interfaces; the data interfaces include text interfaces, image interfaces, and sensor interfaces; the text interface connects to a document database and log system to obtain text data, the image interface connects to a camera and image storage server to obtain image data, and the sensor interface connects to an IoT sensor gateway to obtain numerical sensor data; extract core keywords from the report topic, match the keywords with the metadata of each source data, retain data containing core keywords in the metadata, and remove data unrelated to the core keywords to form a multimodal raw dataset directly related to the report topic; Step S102: Correct typos and perform semantic segmentation on the text data in the multimodal raw dataset, denoise and normalize the resolution of the image data, and filter outliers and align timestamps on the numerical sensor data. Step S103: Store the preprocessed modal data in a distributed data pool, establish a data index, associate the data source with the preprocessing timestamp, and form a structured multimodal data set.

3. The method for generating general reports based on large-model multimodal data according to claim 2, characterized in that: Step S200 includes: Step S201: Receive the report generation instruction input by the user through the interactive interface. The instruction includes a description of the report topic, a list of target chapters to be generated, and a description of the core analysis requirements. The description of the core analysis requirements includes the key problems to be solved or the analysis goals to be achieved in the report. Step S202: Perform structured parsing on the report generation instructions, extract the application scenario category to which the report topic belongs through semantic analysis, identify the logical hierarchy of the target chapters to determine the chapter sequence, and decompose the core analysis requirement elements of each chapter; finally, output a standardized parsing result containing the application scenario category, chapter sequence, and requirement elements of each chapter. Step S203: Use the parsed target chapter name and core analysis requirements as composite query conditions to query the predefined task modality mapping library; the mapping library uses the combination of chapter type and analysis requirements as index to map to the corresponding modality configuration scheme; the modality configuration scheme specifies the core modality and auxiliary modality for the current report generation task in a structured data format, and assigns a basic weight priority value to each modality; Step S204: For each target chapter in the report, retrieve and load the corresponding modal configuration scheme from the mapping library to establish a mapping relationship; if there is no completely matching scheme for a certain chapter, the nearest neighbor matching algorithm based on semantic similarity is used to select the configuration scheme with the highest similarity from the mapping library as the default scheme.

4. The method for generating general reports based on large-model multimodal data according to claim 3, characterized in that: Step S300 further includes: Step S302: Obtain the basic weight priority of each modality in step S200, and introduce the domain adaptation coefficient and data scale coefficient. The domain adaptation coefficient is dynamically set according to the professional attributes of the application scenario to which the report belongs, representing the difference in the relative importance of various modal data under different application scenarios. The data scale coefficient is the ratio of the effective data volume of a single modality to the total effective data volume of the multimodal data set. For text data, the number of sentences is used as the unit of measurement; for image data, the number of image frames is used as the unit of measurement; and for numerical sensor data, the number of effective data points is used as the unit of measurement. When the proportion of data volume of a certain modality exceeds the preset threshold, the data scale coefficient is adjusted through nonlinear transformation. Next, the final decision weight W for each mode is calculated using the following formula. f :W f =λ×γ×(α×P base +β×Q); where λ is the domain fit coefficient, γ is the data size coefficient, and λ>0, γ>0; P base Based on the basic weight priority, Q is the quality score of the corresponding mode, and α and β are P base The preset weighting coefficients for Q and α+β=1; Step S303: Design an adaptive threshold routing mechanism. First, based on the modal weight distribution of the current task, calculate the weight mean μ and standard deviation σ, and dynamically generate two levels of thresholds: the dominant path threshold T. main Generated based on a preset multiple of the mean plus standard deviation; reference path threshold T ref Data is generated based on a preset multiple of the mean minus the standard deviation; subsequently, the modal data are classified and allocated: if W f >=T main If so, the modal data is assigned to the dominant data processing path, marked as core analysis data, and its quality score is associated with the basis for weight calculation; if T ref <=W f <T main If W is not found, it will be assigned to the supplementary data processing path, marked as auxiliary verification data, and its features will be verified against the primary data; f <T ref If so, it will be assigned to the reference data processing path and marked as background support data, with only key features retained for supplementing the report background; Step S304: Establish data interaction rules between paths, set the dominant data processing path to call the features of supplementary data for conclusion verification, the supplementary data path needs to feed back the verification results to the dominant path, and the reference data path only responds to the background query request of the large language model during the report generation stage.

5. The method for generating general reports based on large-model multimodal data according to claim 4, characterized in that: Step S400 includes: Step S401: Modality adaptation preprocessing of dominant and supplementary data: text data is converted into a multimodal model-compatible token sequence, image data is adjusted to a preset resolution and pixel normalization is completed, and numerical sensor data is reconstructed into a temporal feature matrix according to the time dimension; the two types of preprocessed data are input into the pre-trained multimodal model respectively, and the model completes feature fusion and analysis through a cross-modal attention mechanism, and outputs preliminary conclusions with confidence scores C: dominant conclusion Con_main generated based on dominant data, and supplementary conclusion Con_supp generated based on supplementary data; where the confidence score C∈[0,100] is obtained by linear mapping from the softmax probability value of the output layer of the multimodal model; The multimodal model uses a pre-trained cross-modal base model as its backbone network, which includes three independent branches: a text encoder, an image encoder, and a temporal data encoder, to process text, image, and numerical sensor data, respectively. Subsequently, a labeled dataset of the target domain is collected, including text, image, and sensor data triples and corresponding labeling conclusions. Based on the labeled dataset, the model is fine-tuned to adapt to the application domain features of the current report generation task: a domain term embedding layer is added to the text encoder, a target detection head of the application domain of the current report generation task is connected to the image encoder, and a sliding window attention mechanism is added to the temporal encoder to capture the dynamic features of the sensor data. Next, confidence calibration is performed. The probability distribution of the multimodal model output is calibrated using a temperature scaling method. The optimal temperature parameter T is determined through grid search on the validation set. The original softmax output pi is then converted into the calibrated probability p'i=exp(zi / T) / ∑ j exp(zj / T); where i represents the target category index of the multimodal model output conclusion, pi represents the original predicted probability of the i-th category conclusion output by the multimodal model; p'i represents the probability value of the i-th category conclusion after temperature scaling calibration, with a value range of [0,1]; j represents the traversal index of all output conclusion categories of the multimodal model; zi is the original predicted value of the last layer output in the multimodal model; finally, a conflict detection branch is added to the model output layer, and the training of the multimodal analysis model is completed by checking the consistency between the predicted conclusion and the labeled result through the binary classification head.

6. The method for generating general reports based on large-model multimodal data according to claim 5, characterized in that: Step S400 further includes: Step S402: Calculate the semantic similarity Sim between Con_main and Con_supp using a pre-trained semantic matching model, where Sim ∈ [0, 100]. If Sim >= Tsim, the conclusion is considered to be conflict-free, and the fusion process is initiated: Taking the logical framework of Con_main as the core, extract detailed information related to the core analysis requirements from Con_supp, integrate it into a comprehensive analysis conclusion Con_syn according to the structure of core viewpoints and supplementary arguments, and label the modality type corresponding to the supplementary information and the quality score corresponding to that modality; where Tsim is a preset consistency threshold used to determine whether there is a conflict in the conclusion, and its value is set according to the mapping relationship between the consistency of the conclusion and the accuracy of the final report in historical data; the trained semantic matching model first performs semantic encoding on the text content of Con_main and Con_supp respectively to generate corresponding sentence-level semantic vectors; then calculates the basic matching degree of the two semantic vectors through the cosine similarity algorithm, and after normalization, converts the basic matching degree into a semantic similarity Sim with a value range of [0, 100] through linear transformation; Step S403: If Sim < Tsim, it is determined that there is a conflict, and then the conflict resolution mechanism is triggered. The modal quality score in step S301 and the final decision weight W in step S302 are used as the core decision-making basis: when Q corresponding to the dominant data f > Q main and W supp > W f(main) > W f(supp) in step S302 are used as the core decision-making basis: when Q corresponding to the dominant data main is the quality score corresponding to the dominant data, Q supp is the quality score corresponding to the supplementary data; W f(main) is the final decision weight of the dominant data, W f(supp) is the final decision weight of the supplementary data; When Q supp -Q main >=ΔQ and W f(supp) -W f(main) When ΔQ >= ΔW, where ΔQ is the preset quality score difference threshold and ΔW is the preset weight difference threshold, background data related to the conflict point in the reference data processing path is called and input into the multimodal model for secondary analysis to generate an auxiliary verification conclusion Con_ref with a confidence score C_ref; then, the confidence score weighted screening mechanism is started to determine Con_syn: first, the confidence score C of Con_main is extracted. main The confidence score C of Con_supp supp The confidence score C of Con_ref ref Then, using the final decision weights of the data corresponding to each conclusion as weighting coefficients, the weighted score of the three is calculated according to the formula: Weighted Score = (C main ×W f(main) +C supp ×W f(supp) +C ref ×W f(ref) ) / (W f(main) +W f(supp) +W f(ref) ), where W f(ref) This represents the final decision weight corresponding to the reference data. If a certain conclusion has the highest weighted score and the difference between its weighted scores and those of the other two types of conclusions exceeds a preset advantage threshold, then this conclusion is used as the core framework, and the non-conflicting detailed arguments from the other two types of conclusions are integrated to generate Con_syn. If the difference between the highest and second-highest weighted scores does not exceed the advantage threshold, then the matching degree between the supporting arguments of the three types of conclusions and the reference data is further compared, and the conclusion with the highest matching degree is selected as the benchmark, and the effective information from other conclusions is integrated to form Con_syn. When |Q main -Q supp |<ΔQ and|W f(main) -W f(supp) When |<ΔW, a predefined domain arbitration rule library is loaded, and an appropriate rule is selected according to the rule priority to generate Con_syn; the predefined domain arbitration rule library is a structured set of rules built based on the professional knowledge and historical decision-making experience of the application domain to which the current report generation task belongs, and stores the priority handling strategy for the corresponding conflict by using scenario type and conflict characteristics as indexes; Step S404: Perform structured encapsulation on Con_syn to generate a standardized data packet containing the core judgment conclusion, supporting data index, and comprehensive confidence score C_syn; where the comprehensive confidence score C_syn = (C main ×W f(main) +C supp ×W f(supp) ) / (W f(main) +W f(supp) );C main Score the confidence level of Con_main, C supp Score the confidence level of Con_supp.

7. The method for generating general reports based on large-model multimodal data according to claim 6, characterized in that: Step S500 includes: Step 501: First, input the comprehensive analysis conclusion Con_syn from the standardized data package generated in step S404 and the background support data output from the reference data processing path into the pre-trained large language model. At the same time, load the chapter sequence and the requirement elements of each chapter as generation constraints. The large language model integrates the core viewpoints, supporting arguments, and scene information of the background support data of Con_syn according to the chapter logical hierarchy and core analysis requirements to generate chapter paragraph text that conforms to the report format specifications. The pre-trained large language model adopts a Transformer encoder-decoder architecture, in which the encoder performs semantic encoding on the comprehensive analysis conclusion, background data, and chapter requirements, and the decoder generates report text that conforms to the format. The input layer performs unified embedding and interaction of multi-source information, and the output layer integrates domain-specific and general vocabularies to output natural language text that conforms to the report format specifications and contains professional terminology. Step 502: Then, according to the preset report template, including font style, title level, and chart insertion rules, the text of each chapter is formatted. Based on the data index identifiers embedded in the text, the corresponding original image data and numerical sensor data are automatically associated and inserted. Among them, the image data is inserted with the corresponding resolution and the data source index is marked. The numerical sensor data is generated with time series trend charts or statistical tables. Step 503: After typesetting is completed, the content proofreading mechanism is triggered. The text is corrected by the grammar correction model, and the consistency of professional terms is checked by the domain terminology verification library. Combined with the comprehensive confidence C_syn from step S404, expressions with confidence scores lower than the preset confidence threshold are filtered and prompted for manual review. After the proofreading is passed, all chapter texts, charts and background information are integrated to generate the final report. The report generation log, including data source, processing timestamp, and weight calculation records, is stored synchronously in the distributed data pool.

8. A general report generation system based on large-model multimodal data driven by a general report generation method based on large-model multimodal data driven by any one of claims 1-7, characterized in that: The system includes a multimodal data acquisition module, a demand analysis and matching module, a data quality assessment module, a multimodal data analysis and fusion module, and a report generation module; The multimodal data acquisition module acquires multimodal raw data related to the report topic, including text data, image data, and numerical sensor data, and performs standardized preprocessing to form a multimodal data set; The requirement parsing and matching module receives the report generation instruction and parses the report topic, target chapter and core analysis requirements contained therein. Based on the parsing results, a predefined task modality mapping library is queried to match and obtain a modality configuration scheme suitable for the currently generated task; the modality configuration scheme includes core modalities, auxiliary modalities and their corresponding basic weight priorities; The data quality assessment module independently quantifies and assesses the quality of various modal data in the multimodal dataset; it combines the quality score with the basic weight in the modality configuration scheme to dynamically calculate the final decision weight of each modality data. Based on the weight of the final decision, the various modal data are assigned to the dominant data processing path, the supplementary data processing path, and the reference data processing path, respectively, thus forming dominant data, supplementary data, and reference data. The multimodal data analysis and fusion module inputs the dominant data and supplementary data into the pre-trained multimodal model for preliminary analysis to obtain preliminary conclusions with confidence scores. After consistency verification, if the two types of conclusions are logically consistent, the conclusions of the dominant data are taken as the core, and the conclusions of the supplementary data are fused to form a comprehensive analysis conclusion. If the verification finds a logical conflict in the conclusion, the conflict resolution will be made by combining the quality assessment results with the predefined arbitration rules, and a comprehensive analysis conclusion will be generated. The report generation module inputs the comprehensive analysis conclusions and reference data into the large language model, generates corresponding report paragraph text, performs formatting and content proofreading, and finally outputs a structurally complete report.

Images