An intelligent processing method, an optimization training framework and an application system thereof for cross-modal information fusion

Abstract

The application relates to the technical field of artificial intelligence, and provides an intelligent processing method for cross-modal information fusion, an optimization training framework and an application system thereof.The method sequentially performs multi-modal recognition, perception, reasoning and efficiency evaluation on multi-modal input data, and obtains a comprehensive evaluation report through multi-dimensional indexes such as accuracy, F1 score, intersection over union, average precision, BLEU and ROUGE.The framework covers three types of test modules, namely text-to-image, image-to-text and visual question answering, and realizes quantitative evaluation of the performance of each cross-modal task by using indexes such as InceptionScore, R-precision, BLEUScore and METEORScore.The system realizes full-process functional testing of a multi-modal large model through five modules, namely data reading, model loading, test case generation, index calculation and result output, and provides a systematic evaluation means for researchers and developers.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent processing method, an optimized training framework, and an application system for cross-modal information fusion. Background Technology

[0002] With the continuous evolution of deep learning technology and the rapid growth of multi-source heterogeneous data, large multimodal models have become an important research direction in the field of artificial intelligence. These models utilize deep neural networks to extract features from data of different modalities, such as text, images, audio, and video, and achieve information interaction and fusion between different modalities. This results in significant performance advantages and generalization capabilities in tasks such as image description generation, visual question answering, cross-modal retrieval, and multimodal sentiment analysis. As the amount of multimodal data increases and the demand for multimodal information processing deepens, systematic evaluation of large multimodal models becomes increasingly crucial. However, with the increase in model size and task complexity, how to accurately and comprehensively evaluate the performance and inference efficiency of large multimodal models on different tasks has become a key issue restricting their engineering implementation and large-scale application.

[0003] Existing multimodal large model evaluation methods generally suffer from limitations such as single-dimensionality and fragmented systems. For recognition tasks, only overall accuracy is typically used for coarse-grained evaluation, failing to comprehensively characterize the model's cross-modal recognition capabilities from multiple perspectives, including precision, recall, F1 score, and Top-K accuracy. For perception tasks, the calculation of intersection-union ratio (IU / U) for object detection and image segmentation lacks comprehensive consideration of pixel-level segmentation accuracy across multiple categories. For inference and generation tasks, the applicable scenarios for metrics such as BLEU, ROUGE, CIDEr, SPICE, and METEOR are vaguely defined, and various metrics lack systematic integration, making it difficult to form a unified multi-task evaluation system. In terms of inference efficiency, systematic measurement methods for key metrics such as first-to-toe latency, P95 latency, P99 latency, throughput, and batch processing efficiency are still incomplete, failing to effectively support the differentiated evaluation needs of different application scenarios such as real-time inference, low-latency inference, and high-throughput inference. In addition, there is still a lack of dedicated functional testing tools for core cross-modal tasks such as text-to-image, image-to-text, and visual question answering. The automated evaluation process for key indicators such as image quality, text generation similarity, and visual question answering accuracy is not yet perfect, which restricts the standardization level and engineering implementation efficiency of multimodal large model evaluation. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent processing method for cross-modal information fusion, an optimized training framework and its application system, in order to solve the problems in the existing technology of single evaluation dimensions for multimodal large models, scattered evaluation index systems, and lack of dedicated functional testing tools for cross-modal tasks such as text-to-image, image-to-text, and visual question answering.

[0005] To achieve the above objectives, on the one hand, the present invention provides an intelligent processing method for cross-modal information fusion, comprising: Step S1: Obtain multimodal input data, which includes at least two modalities among text, image, audio, video, and 3D data.

[0006] Step S2: Evaluate the multimodal recognition task on the multimodal input data. Obtain the recognition performance evaluation results by calculating accuracy, precision, recall, F1 score and Top-K accuracy.

[0007] Step S3: Perform multimodal perception task evaluation on the multimodal input data. Obtain the perception performance evaluation results by calculating the cross-union ratio, average cross-union ratio, Dice coefficient, and average accuracy.

[0008] Step S4: Perform multimodal reasoning task evaluation on the multimodal input data. Obtain the reasoning performance evaluation results by calculating the question-answering accuracy, exact matching rate, fragment-level F1 score, BLEU score, and ROUGE score.

[0009] Step S5: For the image description generation task, the description generation quality is evaluated by calculating the BLEU index, METEOR index, CIDEr index and SPICE index, and the description generation quality evaluation result is obtained.

[0010] Step S6: Efficiency evaluation of the cross-modal information fusion processing process is performed. The efficiency evaluation results are obtained by calculating inference latency, throughput and batch processing efficiency.

[0011] Step S7: The recognition performance evaluation results, perception performance evaluation results, reasoning performance evaluation results, description generation quality evaluation results, and efficiency evaluation results are summarized and integrated to obtain a comprehensive evaluation report on cross-modal information fusion intelligent processing.

[0012] Furthermore, in step S2, the formula for calculating the accuracy is: ; in, The total number of samples, For the model to predict the category, For the true category; the stated The formula for calculating fractions is: ; in, For accuracy, Recall; the formula for calculating TOP-K precision is: ; in, For the model to sample The output class probabilities are sorted in the first... A collection of categories.

[0013] Furthermore, in step S3, the intersection-union ratio is obtained by dividing the area of ​​the intersection of the predicted region and the actual region by the area of ​​their union; the formula for calculating the average intersection-union ratio is: ; in, The total number of categories, For the first The intersection-union ratio of the classes; the The formula for calculating the coefficient is: ; in, The number of true positives. The number of false positives. The number of false negatives; the formula for calculating the average precision is: ; in, For the first The area under the precision-recall curve for class detection results.

[0014] Furthermore, in step S4, the formula for calculating the BLEU score is: ; in, For a brief penalty factor, ; for Tuple weights for Tuple precision; the ROUGE-L score is calculated by predicting text. With reference text The longest common subsequence length L between them is used to obtain the accuracy. and recall rate Then, it can be obtained through the following formula: .

[0015] Furthermore, in step S6, the formula for calculating the throughput is: ; in, The total number of requests processed. The total processing time; the formula for calculating the batch processing efficiency is: ; The inference latency includes the first token latency, the P95 latency, and the P99 latency. The first token latency is the time from when the client initiates a request to when the model generates the first token. The P95 latency is the 95th percentile value of all request latencies, and the P99 latency is the 99th percentile value of all request latencies.

[0016] Based on the same inventive concept, this invention also provides an optimized training framework for cross-modal information fusion, including a text-to-image testing module, an image-to-text testing module, and a visual question-answering testing module.

[0017] The text-to-image testing module is used to read the test dataset, load the multimodal large model to be tested, generate evaluation text description test cases, perform inference on the multimodal large model to generate images, and evaluate the quality of the generated images and their relevance to the text by calculating the InceptionScore and R-precision metrics, thereby obtaining the text-to-image evaluation results.

[0018] The image-to-text testing module is used to read the test dataset, load the multimodal large model to be tested, generate evaluation image test cases and their corresponding reference texts, perform inference on the multimodal large model to generate candidate texts, and evaluate the similarity between the generated text and the reference text by calculating the BLEU score and METEOR score, thereby obtaining the image-to-text evaluation results.

[0019] The visual question answering testing module is used to read the test dataset, load the multimodal large model to be tested, generate visual question answering test cases, perform inference on the multimodal large model to generate answers, and evaluate the performance of the visual question answering task and the quality of the generated answers by calculating Top-1 accuracy, Top-5 accuracy, question type accuracy and BLEUScore, and obtain the visual question answering evaluation results.

[0020] Furthermore, the formula for calculating the InceptionScore by the text-to-image testing module is as follows: ; in, To generate the conditional class probability distribution of image x under the pre-trained classification model, To generate the edge category probability distribution of the image set, for divergence, This represents the expectation of all generated images.

[0021] The InceptionScore is obtained through the following steps: inputting the generated image into a pre-trained image classification model to obtain the class probability distribution of each image; calculating the KL divergence between the conditional probability distribution and the marginal probability distribution of each image; and taking the expectation and exponent of the KL divergence of all images to obtain the InceptionScore.

[0022] The R-precision is obtained through the following steps: using the text description as a query; generating a set of images using a multimodal large model; performing similarity matching between each generated image and the query text description; and obtaining the R-precision by calculating the number of matched relevant images.

[0023] Furthermore, the formula for calculating the BLEUS score by the image-to-text testing module is as follows: ; in, For a brief penalty factor, For n-gram precision, For n-gram weights.

[0024] The image-to-text testing module calculates METEORScore using word alignment and matching methods, including exact matching, synonym matching, stemming, and word order changes. The word alignment and matching results are weighted and combined to obtain the METEORScore.

[0025] Based on the same inventive concept, the present invention also provides an application system for cross-modal information fusion, characterized in that the system includes a data reading module, a model loading module, a test case generation module, an indicator calculation module, and a result output module.

[0026] The data reading module is used to read the dataset to be tested.

[0027] The model loading module is used to load the large multimodal model to be tested.

[0028] The test case generation module is used to generate evaluation test cases covering text-to-image, image-to-text, and visual question-answering tasks.

[0029] The indicator calculation module is used to perform inference prediction on the multimodal large model based on the test cases, and to calculate the InceptionScore, R-precision, BLEUScore, METEORScore, Top-k accuracy and question type accuracy indicators to obtain the evaluation indicator data for each task.

[0030] The result output module is used to summarize and integrate the evaluation index data of each task, obtain the comprehensive functional test evaluation results of the multimodal large model, and output them.

[0031] Furthermore, the indicator calculation module includes an identification evaluation unit, a perception evaluation unit, a reasoning evaluation unit, and an efficiency evaluation unit.

[0032] The recognition and evaluation unit is used to evaluate the classification task by calculating the accuracy, F1 score and Top-K accuracy, and obtain recognition and evaluation data.

[0033] The perception evaluation unit is used to evaluate spatial understanding tasks by calculating the intersection-union ratio, average intersection-union ratio, and average accuracy, and to obtain perception evaluation data.

[0034] The reasoning evaluation unit is used to evaluate reasoning tasks by calculating the exact match rate and ROUGE score, and obtain reasoning evaluation data.

[0035] The efficiency evaluation unit is used to evaluate the processing efficiency by calculating throughput, inference latency, and batch processing efficiency, and obtain efficiency evaluation data.

[0036] (3) Beneficial effects Compared with existing technologies, the beneficial effects of this invention lie in constructing a complete evaluation benchmark system covering multiple dimensions such as multimodal recognition, perception, reasoning, and reasoning efficiency. It achieves a refined quantitative evaluation of the comprehensive capabilities of multimodal large models through a systematic combination of indicators such as accuracy, F1 score, Top-K accuracy, intersection-over-union ratio, average precision, BLEU, ROUGE, CIDEr, and SPICE. Simultaneously, it provides three types of functional testing tools: text-to-image, image-to-text, and visual question answering. It employs multiple evaluation indicators such as Inception Score, R-precision, BLEU Score, and METEOR Score to comprehensively test various cross-modal tasks, effectively solving the problems of single evaluation dimensions and lack of testing tools. This provides researchers and developers with a systematic and standardized means of evaluating the functionality of multimodal large models, contributing to the standardized evaluation and engineering development of multimodal large models. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is an overall flowchart of the multimodal large model functional testing provided in the embodiments of the present invention; Figure 2 This is a functional architecture diagram of the optimized training framework provided in the embodiments of the present invention; Figure 3 This is a functional block diagram of an application system provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Throughout this specification, references to "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "an embodiment," "an example," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. Moreover, those skilled in the art will understand that the illustrations provided herein are for illustrative purposes and are not necessarily drawn to scale. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0041] This embodiment provides an intelligent processing method for cross-modal information fusion, an optimized training framework, and its application system. The overall process of multimodal large model functional testing sequentially goes through four stages: data reading, model loading, test execution, and index calculation. It covers three types of tasks: text-to-image testing, image-to-text testing, and visual question answering testing. Finally, a comprehensive evaluation result is obtained. Figure 1 The image shows an intelligent processing method for cross-modal information fusion, comprising: Step S1: Obtain multimodal input data, which includes at least two modalities among text, image, audio, video and 3D data, as the input source for various subsequent evaluation tasks.

[0042] Step S2: Evaluate the multimodal recognition task on the multimodal input data. Multimodal recognition focuses on recognizing and classifying objects, actions, or attributes in multimodal inputs such as images and text. Typical tasks include concept recognition (object / scene classification), attribute recognition, action recognition, and OCR text recognition.

[0043] The given model is based on multimodal input (e.g., images). and text Output prediction category The accuracy rate metric yields the overall percentage of correct predictions, calculated using the following formula: ; in, The total number of samples, For the model to predict the category, This is the true category.

[0044] Precision (Pc) is defined as the proportion of results predicted as class c that actually belong to class c; recall (Rc) is defined as the proportion of actual samples of class c that were successfully identified by the model; the F1 score is the harmonic mean of precision and recall, and its calculation formula is: ; Top-K accuracy represents the proportion of correct classes among the model's top K most likely predicted classes, and its calculation formula is: ; in, For the model to sample The output is a set of the top K categories in the category probability ranking. Combining these metrics, the recognition performance evaluation result is obtained.

[0045] Step S3: Evaluate the multimodal perception task on the multimodal input data. Multimodal perception focuses on understanding spatial and relational information in inputs such as images and videos, including tasks such as target localization, detection, segmentation, and cross-modal indicator localization.

[0046] The intersection-union ratio (IU) is used to evaluate the degree of overlap between predicted and ground regions in object detection and image segmentation. It is obtained by dividing the area of ​​the intersection of the predicted and ground regions by the area of ​​their union. ; Mean Intersection over Union (mIoU) is often used for image segmentation tasks. It is obtained by taking the mean of the mIoU values ​​for all categories. ; Where C is the total number of categories, and IoU i Let be the intersection-union ratio (CUNR) of the i-th class. The Dice coefficient is used to evaluate the similarity between the predicted mask and the true mask, and its calculation formula is: ; Where TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives. Mean accuracy (mAP) is obtained by averaging the AP values ​​across all categories. ; Where APc is the area under the precision-recall curve for the c-th type of detection result. Combining the above metrics, the perception performance evaluation results are obtained.

[0047] In one optional implementation, the multimodal perception task evaluation may further include the evaluation of the visual pointing localization task. The visual pointing localization task, also known as referring expression comprehension, refers to the task of finding the corresponding region in an image based on natural language descriptions, requiring the model to simultaneously understand linguistic referential relationships and image spatial information. The evaluation of the visual pointing localization task mainly includes two metrics: First, localization accuracy. If the intersection-union ratio (IUU) between the model's output bounding box and the ground truth bounding box exceeds a preset threshold (e.g., 0.5), it is considered correctly located. The localization accuracy is the proportion of correctly located samples out of the total number of samples. Second, pointing game accuracy. The model needs to provide the approximate center coordinates of the target. If these coordinates fall within the real target area, it is considered a hit. The pointing game accuracy is the proportion of hit samples out of the total number of samples. For example, when evaluating on the commonly used visual pointing localization benchmark dataset RefCOCO, the model's localization accuracy on multiple subsets can be reported separately to measure the strength of the model's multimodal alignment and localization capabilities. The evaluation results of the visual pointing positioning task are obtained by calculating the positioning accuracy and the pointing game accuracy, and then incorporated into the perception performance evaluation results.

[0048] Step S4: Evaluate the multimodal input data using a multimodal reasoning task. The multimodal reasoning evaluation model is capable of performing advanced reasoning and question answering in cross-modal scenarios, including tasks such as visual question answering (VQA), multimodal dialogue, and visual commonsense reasoning.

[0049] For question-answering tasks requiring discrete answers, the accuracy rate is used to evaluate the proportion of correct answers provided by the model; the exact match rate is used to evaluate the proportion of model outputs that are completely identical to the reference answer text; and the fragment-level F1 score allows for partial matching, comprehensively considering both bag-of-words precision and recall. For tasks requiring natural language output, the BLEU score is obtained by calculating the overlap between the generated text and the reference text on the n-gram, and its calculation formula is as follows: ; Wherein, BP is the brief penalty factor. ; for Tuple weights for Tuple precision. ROUGE-L score is calculated by predicting text. With reference text The length L of the longest common subsequence between them gives the definition of precision. and recall rate Then, it can be obtained through the following formula: ; Based on the above indicators, the inference performance evaluation results are obtained.

[0050] In an optional implementation, the evaluation of multimodal inference tasks may also include the evaluation of cross-modal retrieval tasks. Cross-modal retrieval refers to using data from one modality as a query to retrieve data related to the query from a dataset composed of data from another modality. Common forms include image-to-text and text-to-image searches. The evaluation of cross-modal retrieval tasks mainly includes two metrics: First, Recall@K, which means that for each query, if the top K items in the search results list contain data related to the query, it is considered a hit. Recall@K is the proportion of hit queries to the total number of queries. Commonly used metrics include Recall@1, Recall@5, and Recall@10. Second, median rank, which is the median of the position of the correctly matched items in the search results list. The smaller the median rank, the stronger the overall ranking ability of the model. For example, in the image-to-text retrieval scenario, for each image, relevant text is retrieved, and the presence of relevant text in the top K search results is counted to calculate Recall@K; text-to-image retrieval can be calculated in a symmetrical manner. The evaluation results of the cross-modal retrieval task are obtained by calculating Recall@K and the median ranking metric, and then incorporated into the inference performance evaluation results.

[0051] Step S5: Evaluate the image captioning generation task. The image captioning generation task requires the model to generate descriptive text from images. A comprehensive evaluation is conducted using metrics such as BLEU, METEOR, ROUGE-L, CIDEr, and SPICE. BLEU measures the overlap between the generated description and the human description in terms of n-grams; METEOR evaluates lexical precision and recall while considering synonyms; CIDEr is a metric specifically designed for image captioning tasks, calculating n-tuple matching using TF-IDF weighting, and is commonly used in benchmarks such as COCO Caption; SPICE parses the description into a scene graph and compares the consistency between the predicted and reference semantic content, paying particular attention to the matching of objects, attributes, and relationships. The overall quality evaluation result of the description generation is obtained through the comprehensive calculation of these multiple metrics.

[0052] Step S6: Efficiency evaluation of the cross-modal information fusion processing process is performed, assessing it from three dimensions: real-time inference, low-latency inference, and high-throughput inference. Inference latency includes first-to-toe latency, P95 latency, and P99 latency. First-to-toe latency is the time from when the client initiates a request to when the model generates the first token; P95 latency is the 95th percentile of all request latencies, indicating that 95% of request latencies are less than or equal to this value; P99 latency is the 99th percentile of all request latencies, indicating that 99% of request latencies are less than or equal to this value. Throughput represents the number of requests the model can process per second, and its calculation formula is: ; Where N is the total number of requests processed, and T is the total processing time. Batch processing efficiency represents the percentage increase in throughput under batch processing mode compared to single-processing, and its calculation formula is: ; Based on the above metrics of latency, throughput, and batch processing efficiency, the efficiency evaluation results are obtained.

[0053] In one optional implementation, efficiency evaluation can be further refined from dimensions such as end-to-end latency, cold start latency, latency jitter, and resource utilization: end-to-end latency is the total time from input data entering the model to outputting the final inference result, reflecting the overall response speed of the model in practical applications; cold start latency is the initial latency required for the system to go from an unloaded model state to completing model loading and having inference capabilities, mainly used to evaluate the availability performance of the model in service first-time startup or scaling scenarios; latency jitter is used to characterize the fluctuation range between multiple request latency values ​​when the same model processes different modalities and different task requests, the smaller the fluctuation range, the higher the stability of model inference; resource utilization is used to evaluate the ratio of the actual hardware resources such as processor, memory, or video memory occupied during model inference to the total available resources. Resource utilization can be used to measure the resource efficiency of model deployment, and can also serve as a reference for load balancing and scaling decisions in high-throughput scenarios. Furthermore, in batch, high-concurrency scenarios, load latency can be further calculated, which is the average processing latency of a single request under high-concurrency request load, as well as the consistency between cross-software environments and inference response time balance, i.e., the degree to which the model maintains consistency between inference results and response time across different software frameworks, versions, or different single-modal and cross-modal task requests. By comprehensively calculating the above indicators, a more comprehensive efficiency evaluation result is obtained and incorporated into the overall efficiency evaluation result.

[0054] Step S7: Summarize and integrate the recognition performance evaluation results, perception performance evaluation results, reasoning performance evaluation results, description generation quality evaluation results, and efficiency evaluation results to obtain a comprehensive evaluation report on cross-modal information fusion intelligent processing.

[0055] like Figure 2 As shown, this is an optimized training framework for cross-modal information fusion, which includes a text-to-image testing module, an image-to-text testing module, and a visual question-answering testing module.

[0056] (I) Text-to-Image Test Module The text-to-image task aims to transform natural language text descriptions into corresponding visual content. Currently, the multimodal large-scale models used for this task mainly include three categories: diffusion models, autoregressive models, and generative adversarial network models. This module primarily evaluates the quality of the generated image and the relevance of the generated image to the text.

[0057] Regarding image quality, the quality and diversity of the generated images are evaluated by calculating the Inception Score, the formula of which is as follows: ; Where p(y|x) is the conditional class probability distribution of the generated image x under the pre-trained classification model, p(y) is the edge class probability distribution of the generated image set, and KL(·‖·) is the KL divergence. The Inception Score is obtained by taking the expectation of all generated images. The Inception Score is obtained through the following steps: inputting the generated images into a pre-trained image classification model to obtain the class probability distribution of each image; calculating the KL divergence between the conditional class probability distribution and the marginal class probability distribution for each image; and taking the expectation and exponent of the KL divergences for all images to obtain the Inception Score.

[0058] Regarding the relevance between images and text, R-precision is used for evaluation. R-precision is obtained through the following steps: using the text description as the query; generating a set of images using a multimodal large model; performing similarity matching between each generated image and the query text description; and calculating the number of relevant matching images to obtain the R-precision.

[0059] The testing process for the module is as follows: read the test dataset; load the multimodal large model to be tested; generate evaluation text description test cases; perform inference on the multimodal large model to generate images; evaluate the model by calculating the Inception Score and R-precision index to obtain the text-to-image evaluation results.

[0060] (II) Image-to-Text Test Module The image-to-text task aims to generate corresponding text descriptions from given images. It requires the model to extract key features and semantic information from the image and convert them into accurate, fluent, and meaningful text. The module primarily evaluates the similarity between the generated text and the reference text, as well as the matching quality in terms of vocabulary, grammar, and semantics.

[0061] Regarding text similarity, the BLEU Score is used to measure the n-gram overlap between the generated text and the reference text. The formula for calculating this score is as follows: ; Wherein, BP is the brief penalty factor. For n-gram precision, For n-gram weights.

[0062] In terms of vocabulary, grammar, and semantic matching, the METEOR Score is calculated for evaluation. The module calculates the METEOR Score through word alignment and matching methods, including exact matching, synonym matching, stemming, and word order changes; the word alignment and matching results are weighted and combined to obtain the METEOR Score.

[0063] The testing process for the module is as follows: read the test dataset; load the multimodal large model to be tested; generate evaluation image test cases and their corresponding reference texts; perform inference on the multimodal large model to generate candidate texts; evaluate the similarity between the generated text and the reference text by calculating the BLEU Score and METEOR Score, and obtain the image-generated text evaluation results.

[0064] In training and evaluation, commonly used image-to-text datasets include MSCOCO, Flickr30k, and VisualGenome. MSCOCO contains over 82,000 images and over 400,000 corresponding text descriptions; Flickr30k contains over 31,000 images and corresponding text descriptions, mainly covering people, objects, and activities in various real-world scenes; VisualGenome contains over 100,000 images and corresponding text descriptions with detailed object annotations, and can be used for training and evaluation of more complex image-to-text tasks.

[0065] (III) Visual Question Answering Test Module Visual question answering is a task that requires models to simultaneously understand images and natural language questions and generate accurate text answers, demanding that the model possess both text understanding and image processing capabilities. This module primarily evaluates the performance of the visual question answering task and the quality of the generated answers, with evaluation metrics including Top-1 accuracy, Top-5 accuracy, question type accuracy, and BLEU score; it can also calculate human evaluation metrics such as relevance, completeness, and coherence scores.

[0066] The testing process for the module is as follows: read the test dataset; load the multimodal large model to be tested; generate visual question answering test cases; perform inference on the multimodal large model to generate answers; evaluate the performance of the visual question answering task and the quality of the generated answers by calculating Top-1 accuracy, Top-5 accuracy, question type accuracy and BLEU Score, and obtain the visual question answering evaluation results.

[0067] Commonly used visual question answering datasets include VQA, COCO-QA, and Visual7W. VQA contains approximately 265,000 images, each accompanied by three questions and corresponding text answers; COCO-QA contains 123,287 images, each accompanied by five questions and corresponding text answers; and Visual7W contains approximately 47,000 images, each accompanied by seven questions and corresponding text answers.

[0068] (iv) Cross-modal retrieval test module Cross-modal retrieval tasks aim to retrieve semantically relevant data from a large dataset composed of data in one modality, using data from another modality as the query. Common forms include image-to-text retrieval (finding text from images) and text-to-image retrieval (finding images from text). It is a crucial task for evaluating the cross-modal semantic alignment capability of large multimodal models. This module primarily evaluates large multimodal models based on two aspects: the hit rate and ranking quality of the retrieval results.

[0069] In terms of hit rate, Recall@K is used to measure the proportion of the top K items in the sorted list of search results that contain data related to the query: for each query, if the top K items in the sorted list of search results contain data related to the query, it is considered a hit; the proportion of hits in all queries is counted to obtain Recall@K, and commonly used values ​​include Recall@1, Recall@5 and Recall@10.

[0070] In terms of ranking quality, the median rank is used to reflect the model's overall ability to rank relevant data: for each query, find the position of the first relevant data in the search results ranking list; calculate the median of the corresponding positions for all queries, which is the median rank. The smaller the median rank, the stronger the model's ability to rank relevant data in a higher position.

[0071] The testing process for the module is as follows: 1) Read the test dataset; 2) Load the multimodal large model to be tested; 3) Generate cross-modal retrieval test cases, which include one type of data used as a query and another type of data set to be retrieved; 4) Use the multimodal large model to infer the test cases and generate ranked retrieval results; 5) Calculate Recall@K and median ranking metrics to evaluate the cross-modal retrieval performance of the model and obtain the cross-modal retrieval evaluation results.

[0072] (v) Visual pointing and positioning test module Visual pointing localization, also known as referential understanding, refers to the task of finding the corresponding region in an image based on a natural language description. It requires the model to possess both language understanding and image spatial localization capabilities. This module primarily evaluates the degree of agreement between the model's localization results and the actual target region.

[0073] The module calculates the localization accuracy as follows: natural language description and image are used as input, and a multimodal large model is used to output the corresponding rectangular region; the intersection-union ratio (IU) between the model output region and the real labeled region is compared with a preset threshold (e.g., 0.5). If the IU exceeds the threshold, the localization is determined to be correct; the localization accuracy is obtained by counting the proportion of the number of correctly localized samples to the total number of samples.

[0074] The module calculates the accuracy of the pointing game by: using a multimodal large model to output the center point coordinates of the target; determining whether the coordinates fall within the actual target area, and if so, determining that it is a hit; and calculating the proportion of hit samples to the total number of samples to obtain the pointing game accuracy.

[0075] The testing process for the module is as follows: 1) Read the test dataset; 2) Load the multimodal large model to be tested; 3) Generate visual pointing localization test cases, which include images, natural language descriptions, and corresponding ground truth labeled areas; 4) Use the multimodal large model to infer the test cases and generate localization results; 5) Calculate the localization accuracy and pointing game accuracy metrics to evaluate the visual pointing localization performance of the model and obtain the visual pointing localization evaluation results.

[0076] like Figure 3 As shown, this is an application system for cross-modal information fusion. The application system includes a data reading module, a model loading module, a test case generation module, an indicator calculation module, and a result output module.

[0077] The data reading module is used to read the dataset to be tested, which covers test samples of various cross-modal tasks such as text-to-image, image-to-text, and visual question answering.

[0078] The model loading module is used to load the large multimodal model to be tested, enabling it to perform inference and prediction on various cross-modal inputs.

[0079] The test case generation module is used to generate evaluation test cases covering text-to-image, image-to-text, and visual question-answering tasks according to the evaluation requirements of each task. Specifically, it includes text description test cases, image test cases and reference text, and visual question-answering test cases.

[0080] The indicator calculation module includes an identification evaluation unit, a perception evaluation unit, an inference evaluation unit, and an efficiency evaluation unit. The identification evaluation unit obtains identification evaluation data by calculating accuracy, F1 score, and Top-K accuracy; the perception evaluation unit obtains perception evaluation data by calculating intersection-over-union ratio, average intersection-over-union ratio, and average precision; the inference evaluation unit obtains inference evaluation data by calculating exact match rate and ROUGE score; and the efficiency evaluation unit obtains efficiency evaluation data by calculating throughput, inference latency, and batch processing efficiency. Simultaneously, the indicator calculation module also calculates indicators such as InceptionScore, R-precision, BLEU Score, METEOR Score, Top-k accuracy, and question type accuracy to obtain evaluation indicator data for each task.

[0081] The result output module is used to summarize and integrate the evaluation index data of each task, obtain the comprehensive functional test evaluation results of the multimodal large model, and output them.

[0082] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent processing method for cross-modal information fusion, characterized in that, The method includes the following steps: Step S1: Obtain multimodal input data, wherein the multimodal input data includes at least two modalities among text, image, audio, video and three-dimensional data; Step S2: Evaluate the multimodal recognition task on the multimodal input data. Obtain the recognition performance evaluation results by calculating accuracy, precision, recall, F1 score and Top-K accuracy. Step S3: Perform multimodal perception task evaluation on the multimodal input data. Obtain the perception performance evaluation results by calculating the cross-union ratio, average cross-union ratio, Dice coefficient, and average accuracy. Step S4: Evaluate the multimodal reasoning task on the multimodal input data. The reasoning performance evaluation results are obtained by calculating the question-answering accuracy, exact matching rate, fragment-level F1 score, BLEU score and ROUGE score. Step S5: For the image description generation task, the description generation quality is evaluated by calculating the BLEU index, METEOR index, CIDEr index and SPICE index, and the description generation quality evaluation result is obtained. Step S6: Efficiency evaluation of the cross-modal information fusion processing process is performed. The efficiency evaluation results are obtained by calculating inference latency, throughput and batch processing efficiency. Step S7: The recognition performance evaluation results, perception performance evaluation results, reasoning performance evaluation results, description generation quality evaluation results, and efficiency evaluation results are summarized and integrated to obtain a comprehensive evaluation report on cross-modal information fusion intelligent processing.

2. The intelligent processing method for cross-modal information fusion according to claim 1, characterized in that, In step S2, the formula for calculating the accuracy is: ； in, The total number of samples, For the model to predict the category, For the true category; the stated The formula for calculating fractions is: ； in, For accuracy, Recall; the formula for calculating TOP-K precision is: ； in, For the model to sample The output class probabilities are sorted in the first... A collection of categories.

3. The intelligent processing method for cross-modal information fusion according to claim 1, characterized in that, In step S3, the intersection-union ratio (IUGR) is obtained by dividing the area of ​​the intersection of the predicted region and the actual region by the area of ​​their union; the formula for calculating the average IUGR is: ； in, The total number of categories, For the first The intersection-union ratio of the classes; the The formula for calculating the coefficient is: ； in, The number of true positives. The number of false positives. The number of false negatives; the formula for calculating the average precision is: ； in, For the first The area under the precision-recall curve for class detection results.

4. The intelligent processing method for cross-modal information fusion according to claim 1, characterized in that, In step S4, the formula for calculating the BLEU score is: ； in, For a brief penalty factor, ; for Tuple weights for Tuple precision; the ROUGE-L score is calculated by predicting text. With reference text The longest common subsequence length L between them is used to obtain the accuracy. and recall rate Then, it can be obtained through the following formula: .

5. The intelligent processing method for cross-modal information fusion according to claim 1, characterized in that, In step S6, the formula for calculating the throughput is: ； in, The total number of requests processed. The total processing time; the formula for calculating the batch processing efficiency is: ； The inference latency includes the first token latency, the P95 latency, and the P99 latency. The first token latency is the time from when the client initiates a request to when the model generates the first token. The P95 latency is the 95th percentile value of all request latencies, and the P99 latency is the 99th percentile value of all request latencies.

6. An optimized training framework for cross-modal information fusion, characterized in that, The framework includes a text-to-image testing module, an image-to-text testing module, and a visual question-answering testing module. The text-to-image testing module is used to read the test dataset, load the multimodal large model to be tested, generate evaluation text description test cases, perform inference on the multimodal large model to generate images, and evaluate the quality of the generated images and their relevance to the text by calculating the InceptionScore and R-precision metrics, thereby obtaining the text-to-image evaluation results. The image-to-text testing module is used to read the test dataset, load the multimodal large model to be tested, generate evaluation image test cases and their corresponding reference texts, perform inference on the multimodal large model to generate candidate texts, and evaluate the similarity between the generated text and the reference text by calculating the BLEU score and METEOR score to obtain the image-to-text evaluation results. The visual question answering testing module is used to read the test dataset, load the multimodal large model to be tested, generate visual question answering test cases, perform inference on the multimodal large model to generate answers, and evaluate the performance of the visual question answering task and the quality of the generated answers by calculating Top-1 accuracy, Top-5 accuracy, question type accuracy and BLEU score, and obtain the visual question answering evaluation results.

7. The optimized training framework for cross-modal information fusion according to claim 6, characterized in that, The formula for calculating the InceptionScore by the text-to-image testing module is as follows: ； in, To generate the conditional class probability distribution of image x under the pre-trained classification model, To generate the edge category probability distribution of the image set, for divergence, This represents the expectation of all generated images; The InceptionScore is obtained through the following steps: inputting the generated image into a pre-trained image classification model to obtain the class probability distribution of each image; calculating the KL divergence between the conditional probability distribution and the marginal probability distribution of each image; taking the expectation and then the exponent of the KL divergence of all images to obtain the InceptionScore. The R-precision is obtained through the following steps: using the text description as a query; generating a set of images using a multimodal large model; performing similarity matching between each generated image and the query text description; and obtaining the R-precision by calculating the number of matched relevant images.

8. The optimized training framework for cross-modal information fusion according to claim 6, characterized in that, The formula for calculating the BLEUS score by the image-to-text testing module is as follows: ； in, For a brief penalty factor, For n-gram precision, For n-gram weights; The image-to-text testing module calculates METEORScore using word alignment and matching methods, including exact matching, synonym matching, stemming, and word order changes. The word alignment and matching results are weighted and combined to obtain the METEORScore.

9. An application system for cross-modal information fusion, characterized in that, The system includes a data reading module, a model loading module, a test case generation module, an indicator calculation module, and a result output module; The data reading module is used to read the dataset to be tested; The model loading module is used to load the large multimodal model to be tested; The test case generation module is used to generate evaluation test cases covering text-to-image, image-to-text, and visual question-answering tasks. The indicator calculation module is used to perform inference prediction on the multimodal large model based on the test cases, and to calculate the InceptionScore, R-precision, BLEUScore, METEORScore, Top-k accuracy and question type accuracy indicators to obtain the evaluation indicator data for each task. The result output module is used to summarize and integrate the evaluation index data of each task, obtain the comprehensive functional test evaluation results of the multimodal large model, and output them.

10. The application system for cross-modal information fusion according to claim 9, characterized in that, The indicator calculation module includes an identification evaluation unit, a perception evaluation unit, a reasoning evaluation unit, and an efficiency evaluation unit; The recognition and evaluation unit is used to evaluate the classification task by calculating the accuracy, F1 score and Top-K accuracy, and obtain recognition and evaluation data. The perception evaluation unit is used to evaluate spatial understanding tasks by calculating the intersection-union ratio, average intersection-union ratio, and average accuracy, and to obtain perception evaluation data. The reasoning evaluation unit is used to evaluate reasoning tasks by calculating the exact match rate and ROUGE score, and obtain reasoning evaluation data. The efficiency evaluation unit is used to evaluate the processing efficiency by calculating throughput, inference latency, and batch processing efficiency, and obtain efficiency evaluation data.

Images