Multi-modal post-fusion method and device based on consistency auxiliary channel, equipment, medium and product

Abstract

This application discloses a multimodal post-fusion method, apparatus, device, medium, and product based on a consistency auxiliary channel, relating to the field of artificial intelligence technology. The method first acquires multimodal samples; these samples are then input into expert models of each modality for parallel processing, yielding a first prediction confidence level for the main decision path and an evidence package from the consistency auxiliary channel's bypass, containing evidence summaries, quality scores, and uncertainty scores. Based on the evidence summaries, a single-modal consistency score for each modality is determined; combined with the quality score and uncertainty score, a fusion weight is determined, and then the first prediction confidence level is weighted and fused to obtain a second prediction confidence level. Furthermore, a divergence index can be determined based on the three types of scores; if it exceeds a threshold, a divergence correction coefficient for each modality is obtained, correcting the second prediction confidence level to obtain a third prediction confidence level. Finally, a credibility index containing the highest probability category, probability entropy, energy score, and average consistency for each modality is output. This method improves the accuracy of multimodal decision-making.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal post-fusion method, apparatus, device, medium and product based on a consistency-assisted channel. Background Technology

[0002] Multimodal post-fusion technology is a key technology in the field of artificial intelligence for integrating cross-modal information, and it is widely used in scenarios such as question answering, content understanding, medical detection, remote sensing, and multi-sensor perception. Multimodal post-fusion technology can maintain the independence of each modality expert model and support modular reuse and engineering expansion: each modality can be trained, updated, or replaced independently without reconstructing the overall fusion architecture, thus reducing the development and maintenance costs of industrial-grade multimodal systems.

[0003] Typical multimodal post-fusion schemes can be divided into four categories: rule-based schemes summarize the decision results of each modality through pre-set logic such as weighted averaging, voting, and thresholding; learning-based schemes rely on meta-classifiers, linear or nonlinear fusion machines, and use additional data to train the fusion model to achieve integration; probabilistic schemes are based on probability theories such as PoE, Dempster-Shafer, and Copula to handle modal uncertainty and complete fusion; and dynamic gating schemes adjust the fusion ratio according to modal quality or scene context through adaptive weights or MoE routing.

[0004] Currently, the integration of modal output results based on independent reasoning of each modality and the aggregation of decision-level scores does not delve into the cross-modal correlation information, thus leading to a higher probability of misjudgment. Summary of the Invention

[0005] This application provides a multimodal post-fusion method, apparatus, device, medium, and product based on a consistency auxiliary channel, which can reduce the probability of misjudgment.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides a multimodal post-fusion method based on a consistency-assisted channel, including:

[0008] Acquire multimodal samples;

[0009] The multimodal samples are input into each modal expert model for parallel processing to obtain the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes evidence summary, quality score and uncertainty score.

[0010] Based on the evidence summaries in the evidence packages output by each modality expert model, determine the monomodal consistency score for each modality;

[0011] The fusion weights for each modality are determined based on the single-modal consistency score, quality score, and uncertainty score of each modality.

[0012] The calibration domains are divided according to the multimodal sample types and task categories. Adaptive temperature parameters are configured for different domains. The probability distribution of the first prediction confidence of each mode is adjusted by adjusting the temperature parameters to obtain the calibrated first prediction confidence.

[0013] The calibrated first prediction confidence is fused based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

[0014] Optionally, a divergence index can be determined based on the single-modal consistency score, quality score, and uncertainty score of each modality;

[0015] If the divergence index is greater than the divergence index threshold, then the divergence correction coefficient for each mode is obtained.

[0016] By using the divergence correction coefficients of each modality, the second prediction confidence for each task category is corrected, resulting in the third prediction confidence for each task category.

[0017] Optionally, the divergence correction coefficient for the i-th mode is obtained in the following way:

[0018] Obtain the k-th type of conflict identifier for the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold;

[0019] The divergence correction coefficient of the i-th mode is determined based on the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold.

[0020] Optionally, the method further includes:

[0021] Output a confidence index corresponding to the third prediction confidence level. The confidence index includes the category, probability entropy, energy score, and average consistency corresponding to the highest probability of the i-th modality output.

[0022] Optionally, the multimodal samples include image samples, text samples, or audio samples.

[0023] Secondly, this application provides a multimodal post-fusion device based on a consistency auxiliary channel, comprising:

[0024] The acquisition module is used to acquire multimodal samples;

[0025] The data processing module is used to input the multimodal samples into various modal expert models for parallel processing, obtaining the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes an evidence summary, a quality score, and an uncertainty score. Based on the evidence summary in the evidence package output by each modal expert model, the single-modal consistency score of each modality is determined. Based on the single-modal consistency score, quality score, and uncertainty score of each modality, the fusion weight of each modality is determined. The calibration domain is divided according to the multimodal sample type and task category, and suitable temperature parameters are configured for different domains. The probability distribution of the first prediction confidence of each modality is adjusted by the temperature parameters to obtain the calibrated first prediction confidence.

[0026] The fusion module is used to fuse the calibrated first prediction confidence based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

[0027] Thirdly, this application provides a computing device, including a memory and a processor;

[0028] The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.

[0029] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.

[0030] As can be seen from the above technical solution, this application has at least the following beneficial effects:

[0031] From the perspective of cross-modal information utilization, this method overcomes the limitation of traditional schemes that only summarize decision-level scores. It extracts evidence summaries for each modality through a consistency auxiliary channel bypass, including text entities, image objects, and audio keywords, and calculates a single-modal consistency score based on these evidence summaries. This method can deeply capture fine-grained cross-modal associations such as the matching degree between text entities and image objects, and the conflict between verbal negation and textual affirmation, avoiding decision bias caused by semantic fragmentation between modalities and reducing the probability of misjudgment caused by neglecting cross-modal associations in traditional schemes.

[0032] From the perspective of ensuring decision reliability, this method jointly determines the fusion weights using three dimensions: single-modal consistency score, quality score, and uncertainty score. Compared to the fixed weights of traditional rule-based schemes and dynamic gating schemes that rely solely on a single dimension of quality or context, this method can more comprehensively evaluate the decision value of each modality: the quality score ensures that high-quality inputs such as high-resolution images and high signal-to-noise ratio audio receive reasonable weights; the uncertainty score suppresses the interference of overconfident modalities; and the consistency score strengthens the influence of modalities whose conclusions are consistent with those of other modalities. The three work together to achieve a weight bias towards high-quality, reliable, and consistent modalities, avoiding the problems of erroneous resonance of homologous modalities and the dominance of overconfident modalities in decision-making in traditional schemes. At the same time, a divergence correction mechanism is triggered by the divergence index. For modal conclusion conflict scenarios, correction coefficients are generated based on parameters such as conflict identifiers and consistency deviations to slightly calibrate the fusion results, further reducing the risk of serious misjudgment. The output of credibility indicators such as probability entropy and energy score also provides an interpretable basis for downstream business such as triggering manual review and formulating rejection strategies.

[0033] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0034] Figure 1 A flowchart illustrating a multimodal post-fusion method based on a consistency auxiliary channel, provided in this application embodiment;

[0035] Figure 2 A schematic diagram of a multimodal post-fusion device based on a consistency auxiliary channel provided in this application embodiment;

[0036] Figure 3 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0037] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0038] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0040] Multimodal post-fusion technology refers to a technique in artificial intelligence systems where expert models (single-modal processing models) of different modalities, such as images, text, and audio, first complete reasoning independently, and then the results of each modality are integrated at the decision-making layer. Its advantages lie in maintaining the independence of each modal expert model, supporting modular reuse, allowing each modality to be trained, updated, or replaced independently without reconstructing the overall architecture, significantly reducing the development and maintenance costs of industrial-grade multimodal systems, and being widely applied in scenarios such as question answering, medical detection, and remote sensing.

[0041] This application focuses on the problems of traditional multimodal post-fusion schemes in practical applications. The key technical problems are mainly manifested in the high probability of misjudgment and insufficient decision reliability: First, the fragmentation of cross-modal associations leads to semantic misjudgment. Traditional schemes only summarize the scores of each modality decision layer, which cannot capture fine-grained associations such as the matching degree between text entities and image objects, and the conflict between voice negation and text affirmation. It is easy to produce wrong conclusions due to semantic contradictions between modalities. Second, the modal weight allocation is unreasonable. The fixed weights of rule-based schemes ignore the differences in modal quality, and dynamic gating schemes rely on only a single dimension for evaluation, resulting in problems such as overconfident low-quality modal dominating fusion and erroneous resonance of homologous modalities. Third, there is a lack of effective correction for conflict scenarios. When the multimodal conclusions are obviously conflicting, there is no targeted adjustment mechanism. The original logic can only be used, which amplifies the risk of serious misjudgment. Fourth, the decision interpretability is poor. Only the final result is output, and there is a lack of evidence chain and credibility indicators. It is difficult for downstream businesses to trace the basis and locate the cause of misjudgment.

[0042] The root cause of the above problems lies in three shortcomings of traditional solutions: First, the information utilization dimension is singular, focusing only on the predictive confidence of the main decision-making pathway and failing to mine semantic evidence in modal outputs, resulting in the complete neglect of cross-modal correlation information; Second, the weight evaluation dimension is one-sided, failing to form a multi-dimensional collaborative mechanism. Rule-based, learning-based, and probabilistic solutions have limitations such as ignoring dynamic changes, relying on additional data, and failing to correlate consistency, making it impossible to comprehensively screen high-quality, credible, and consistent modalities; Third, there is a lack of conflict handling and interpretability systems. There are neither quantitative standards for judging disagreements nor evidence-based correction logic, nor is key decision information recorded, leading to passive acceptance of results when faced with contradictory conclusions and an inability to trace the decision-making process.

[0043] In view of this, embodiments of this application provide a multimodal post-fusion method based on a consistency auxiliary channel, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0044] To address the issues of high misjudgment rates and insufficient reliability caused by the lack of cross-modal correlation, one-sided weight evaluation, and gaps in conflict handling in traditional multimodal post-fusion schemes, this application fills the gaps in cross-modal correlation analysis by mining semantic evidence and quality and uncertainty information of each modality through auxiliary channels. Then, a weight calculation and disagreement correction mechanism is constructed using three-dimensional indicators of consistency, quality, and uncertainty to achieve evidence-based fusion and corrective decision-making. At the same time, a credibility indicator is output to ensure interpretability. Ultimately, without reconstructing the existing modal expert model, the accuracy, reliability, and engineering adaptability of multimodal fusion are significantly improved.

[0045] To make the technical solution of this application clearer and easier to understand, the following description, in conjunction with the accompanying drawings, introduces a multimodal post-fusion method based on a consistency auxiliary channel provided by an embodiment of this application. Figure 1 As shown, this figure is a flowchart of a multimodal post-fusion method based on a consistency auxiliary channel provided in an embodiment of this application. The method includes:

[0046] S201. The processing equipment acquires multimodal samples.

[0047] Multimodal samples include image samples, text samples, or audio samples. A multimodal sample is a data set containing two or more different types of information. Image samples are data that carry visual information; text samples are data that carry textual information; and audio samples are data that carry sound information. These three types of samples can be used individually or in combination to form multimodal data, providing a foundation for subsequent cross-modal fusion analysis.

[0048] The processing device uses pre-defined data interfaces to acquire and store multimodal data. These interfaces include receiving cloud data via a network interface, receiving local device data via USB or HDMI interfaces, and receiving real-time acquired data via sensor interfaces. The processing device collects multimodal data such as images, text, and audio from different data sources and stores it in the local computing unit.

[0049] The processing equipment needs to verify whether the data from each modality meets the format requirements, such as whether the image resolution, text encoding, and audio sampling rate meet the standards, while also removing damaged or invalid data. This step lays the foundation for subsequent input of samples into the expert models of each modality for parallel inference, and is a key bridge connecting the raw data and model processing.

[0050] S202. The processing device inputs multimodal samples into each modal expert model for parallel processing, and obtains the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes evidence summary, quality score and uncertainty score.

[0051] Modal expert models are specialized processing models designed for single-modal data. Each modality corresponds to one or a group of expert models, which focus only on feature extraction and inference of their own modality and do not depend on other modal data.

[0052] Parallel processing refers to the process where the processing device simultaneously inputs different types of multimodal samples into the corresponding modal expert models. Each model runs independently and does not interfere with the others. There is no need to wait for one model to complete inference before starting another model, thereby improving the overall data processing efficiency.

[0053] The main decision path is the link in the modal expert model that directly outputs the task prediction results. Its first prediction confidence score is the model's initial judgment basis for each task category, presented in the form of probability value or logits score, reflecting the model's confidence in the prediction results.

[0054] The consistency auxiliary channel is a bypass information link that runs parallel to the main decision-making path. It is an innovative design of this application and is specifically used to extract key auxiliary information in the modal reasoning process. It does not directly participate in the initial prediction but only provides support for subsequent cross-modal consistency analysis and weight calculation.

[0055] The evidence package is a collection of information output from the consistency auxiliary channel, containing three categories: first, evidence summaries, which are information such as entities in text, objects in images, and keywords in audio, representing a condensed presentation of the semantics of each modality; second, quality scores, which serve as indicators of the quality of modal input data, specifically including image clarity and audio signal-to-noise ratio; and third, uncertainty scores, which reflect the degree of hesitation in the predictions of the modal expert model, commonly manifested in forms such as probability entropy and MC dropout variance.

[0056] The first prediction confidence is the initial prediction result output by the main decision path for each task category. For example, in a medical detection task, the prediction probability of the image expert model for the two categories of lesion and normal is the first prediction confidence of the model.

[0057] After acquiring multimodal samples, the processing device allocates the samples to the corresponding modal expert models according to their type. For example, image samples are input into the image expert model, and text samples are input into the text expert model. Subsequently, all models simultaneously start inference operations. During the inference process, each modal expert model synchronously generates two types of outputs: on the one hand, the main decision path outputs a first prediction confidence score that reflects the reliability of the prediction for each category in the task; on the other hand, the consistency auxiliary path synchronously generates an evidence package, which includes semantic key information (evidence summary), data quality assessment (quality score), and model prediction risk (uncertainty score).

[0058] This step retains the advantages of modal independent reasoning in traditional post-fusion techniques, and fills the data gap in cross-modal association analysis by outputting evidence packages, providing a basis for subsequent consistency score calculation and fusion weight determination.

[0059] S203. The processing device determines the single-modal consistency score of each modality based on the evidence summary in the evidence package output by each modal expert model.

[0060] Evidence summaries are semantic information from evidence packages of various modal expert models. They are a condensed and refined summary of the key content of modal data. For example, evidence summaries for text modalities include named entities, relations, or negation polarity; for image modalities, they include detected objects, attributes, or spatial attention features; and for audio modalities, they include identified keywords, emotional events, etc. Their role is to provide a comparable common language for cross-modal semantic matching.

[0061] The monomodal consistency score is a quantitative indicator that measures the degree of semantic matching between a certain modality and all other modalities in the system. The score ranges from 0 to 1. The higher the score, the more semantically consistent the evidence summary of that modality is with the evidence summaries of other modalities. Conversely, the lower the score, the more semantically inconsistent there is between modalities.

[0062] After acquiring the evidence packages output by all modal expert models, the processing device first extracts the evidence summary from each evidence package, focusing on the core semantic information of each modality. Then, it calculates the semantic overlap between any two modal evidence summaries using a specific algorithm, i.e., the pairwise consistency score. Finally, it robustly aggregates the pairwise consistency scores of a certain modality with all other modalities to obtain the single modality consistency score of that modality.

[0063] The formula for calculating pairwise consistency score is:

[0064]

[0065] in, Indicates pairwise consistency score. Representing modes The Confidence level of a concept Representing modes The Confidence level of a concept Indicates the number of concepts. This represents the time step decay factor.

[0066] The simplified formula for calculating the single-modal consistency score is as follows:

[0067]

[0068] in, Indicates the single-modal consistency score. This represents the total number of modes participating in the fusion.

[0069] S204. The processing device determines the fusion weight of each mode based on the single-mode consistency score, quality score and uncertainty score of each mode.

[0070] Quality score is an indicator for measuring the quality of modal input data. It ranges from 0 to 1 and is calculated based on the physical or semantic characteristics of the data itself, such as image clarity, audio signal-to-noise ratio, and text integrity. A higher score indicates that the input data is more reliable and the model inference foundation is more solid.

[0071] Uncertainty score is an indicator that reflects the degree of hesitation in the prediction of a modal expert model. It has a value of 0-1 and is calculated by the probability distribution characteristics of the model output, such as probability entropy (the more dispersed the distribution, the higher the score) and MC dropout variance (the greater the fluctuation, the higher the score). The higher the score, the lower the model's confidence in the prediction results.

[0072] Fusion weights are coefficients that determine the influence of each modality's prediction results in the final fusion. The sum of all modality weights is 1. The higher the weight, the greater the impact of the modality's prediction confidence on the final fusion result.

[0073] The processing device first aggregates the single-modal consistency score, quality score, and uncertainty score for each modality. Then, it performs a comprehensive calculation on the three types of scores using a gated mapping function with monotonic constraints. Specifically, the three types of scores are first weighted and summed using non-negative coefficients to obtain the comprehensive score for each modality. The consistency score and quality score are positive contributors, directly increasing the comprehensive score; the uncertainty score is a negative penalty, decreasing the comprehensive score. The expression for calculating the comprehensive score is:

[0074]

[0075] in, Representing modes The overall score, Indicates the non-negative first weight coefficient. This represents the non-negative second weight coefficient. Indicates the non-negative third weight coefficient. Represents the prior preference coefficient. Indicates quality score. This indicates an uncertainty score.

[0076] The combined scores of all modalities are then normalized to ensure that the sum of the weights of all modalities is 1. The initial fusion weights are calculated as follows:

[0077]

[0078] in, Representing modes The initial fusion weights, Representing modes The overall score, Representing modes The overall score.

[0079] Meanwhile, to avoid any single modality monopolizing the weights, such as preventing overconfident, low-quality modalities from dominating decision-making, steady-state constraints are imposed. These constraints include setting upper and lower limits for the weights of each modality and ensuring the minimum entropy value of the weight distribution. The resulting fusion weights are essentially a quantitative assessment of the decision value of each modality, providing a basis for the subsequent weighted fusion of modal prediction confidence. This effectively solves the unreasonable problem caused by traditional schemes that rely solely on a single dimension (such as fixed rules or a single quality indicator) to allocate weights. The final expression for the fusion weights is:

[0080]

[0081] in, This represents the final fusion weight. Represents the smoothing coefficient. Representing modes The initial fusion weights, Representing modes The initial fusion weights.

[0082] S205. The processing device divides the calibration domains according to the multimodal sample types and task categories, configures appropriate temperature parameters for different domains, adjusts the probability distribution of the first prediction confidence of each mode through the temperature parameters, and obtains the calibrated first prediction confidence. Based on the fusion weight of each mode, the calibrated first prediction confidence is fused to obtain the second prediction confidence for each task category.

[0083] Multimodal sample types are independent data categories that participate in multimodal fusion, such as images, text, audio, and sensor data. Each type corresponds to a dedicated modality expert model that processes its own data format and features.

[0084] The task category is the prediction target category that multimodal fusion needs to complete, such as normal, minor fault, and severe fault categories in fault detection, which are the evaluation objects of prediction confidence.

[0085] The first prediction confidence score is the initial prediction result for each task category output by each modal expert model through the main decision path, reflecting the reliability of the judgment of single-modal independent reasoning.

[0086] The first prediction confidence level after calibration is converted into a probability value in the 0-1 range after temperature parameter adjustment, which solves the problem of incomparability of predictions in different domains and is closer to the actual prediction performance.

[0087] The second prediction confidence score is a cross-modal unified prediction result obtained by weighting and integrating the first prediction confidence scores of each modality according to the fusion weight. It is a preliminary decision conclusion after multimodal information collaboration and is more reliable than single-modal prediction.

[0088] The processing device splits the scene according to the multimodal sample type and task category. For example, the image mode corresponds to the fault diagnosis task as a subdomain, ensuring that the predicted scene of each subdomain is consistent, laying the foundation for accurate calibration.

[0089] The temperature parameters are configured for each domain. Based on the validation set data, the optimal temperature parameters that minimize the calibration error (such as ECE) are selected through grid search or gradient descent optimization to adapt to the prediction characteristics of that domain.

[0090] First, the first prediction confidence of each modality output, which is the original score of the model output layer before activation function processing, is scaled according to the temperature parameter corresponding to the calibration domain to which the modality belongs, to obtain the calibrated score after temperature adjustment; then, this calibrated score is substituted into the Softmax function for calculation, and converted into a probability value between 0 and 1, which is the calibrated first prediction confidence.

[0091]

[0092] in, Representing modes Calibration domain For the The calibrated logits score for each task category. Representing modes Calibration domain For the The first prediction confidence for each task category Representing modes Calibration domain The corresponding temperature parameters.

[0093]

[0094] in, Representing modes Calibration domain For the The first calibrated prediction confidence for each task category. Representing modes Calibration domain For the The calibrated logits score for each task category. This indicates the total number of task categories.

[0095] After this processing, the prediction confidence under different types of modalities (such as images, text, and audio) and different application scenarios (such as fault detection and medical diagnosis) will be unified into the same probability evaluation standard, which provides a basis for direct comparison and fusion. At the same time, it can effectively compress the excessively amplified class differences in the original predictions and avoid the problem of high confidence misjudgment caused by model overconfidence.

[0096] The formula for calculating the second prediction confidence level is:

[0097]

[0098] in, This indicates the second prediction confidence level. This represents the final fusion weight. Representing modes Calibration domain For the The calibrated first prediction confidence for each task category, This represents the total number of modes participating in the fusion.

[0099] The method also includes optimization of the second prediction confidence, with the following specific steps:

[0100] First, the processing device determines the divergence index based on the single-modal consistency score, quality score, and uncertainty score of each modality.

[0101] The divergence index is an indicator that combines three types of scores to quantify the degree of conflict between conclusions in multimodal studies. It has a value of 0-1 and serves as the basis for determining whether there is a contradiction between modalities. The higher the score, the greater the differences between multimodal studies in terms of semantic consistency, data quality reliability, and model prediction confidence, and the more obvious the conflict between conclusions.

[0102] When the processing equipment calculates the divergence index, it will operate step by step according to the following steps:

[0103] The first step is to collect three types of key scores for all modalities: the single-modal consistency score, the quality score, and the uncertainty score for each modality, in order to prepare data for subsequent calculations.

[0104] The second step is to analyze the degree of difference, or dispersion, between different modalities for each type of rating. For example, when calculating the dispersion of quality ratings, the coefficient of variation is used, which is measured by dividing the standard deviation of quality ratings across all modalities by the mean. If some modal quality ratings are as high as 0.9, while others are as low as 0.3, the large standard deviation and small mean indicate a high coefficient of variation, suggesting significant quality differences between different modalities. The same coefficient of variation method can be used to calculate the dispersion of consistency ratings and uncertainty ratings between modalities, thereby determining the magnitude of the differences between these two types of ratings.

[0105] The third step involves assigning different non-negative weights to the three types of dispersion based on the specific business scenario. For example, in scenarios like medical testing where semantic matching is crucial, a higher weight is assigned to the consistency score dispersion; while in sensor monitoring scenarios, where data reliability is a primary concern, the weight of the quality score dispersion might be increased. Next, a weighted value is calculated using each type of dispersion and its corresponding weight, and then the three weighted values ​​are summed to obtain the divergence index. The expression for calculating the divergence index is:

[0106]

[0107] in, Indicating a divergence index, The weights representing the dispersion of quality scores. This represents the dispersion of the quality score. The weights representing the dispersion of the consistency score. This represents the dispersion of the consistency score. The weights representing the dispersion of the uncertainty score. This represents the dispersion of the uncertainty score.

[0108] If the divergence index is greater than the divergence index threshold, then obtain the divergence correction coefficient for each mode.

[0109] The processing device first compares the calculated divergence index with a preset divergence index threshold. If the divergence index is greater than the divergence index threshold, for example, if the calculated divergence index is 0.75 and the divergence index threshold is set to 0.6, it indicates that the current multimodal conclusions are significantly conflicting, and directly using the second prediction confidence level may have a high risk of misjudgment. Further correction is needed through a divergence correction coefficient.

[0110] The bifurcation correction coefficient for the i-th mode is obtained in the following way:

[0111] First, the processing device obtains the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold.

[0112] The type k conflict identifier for the i-th modality is used to mark whether the i-th modality has a conflicting conclusion with other modalities in the k-th task category. For example, in a fault detection task, if the first image modality predicts yes for the device fault (k=1) category, but most modalities predict no, then the type 1 conflict identifier for this modality is conflicted.

[0113] The consistency deviation of the i-th modality is an indicator that quantifies the difference in consistency level between the i-th modality and the overall consistency level of all modalities. It is calculated by subtracting the average consistency score of all modalities from the single-modality consistency score of the i-th modality. A negative result indicates that the consistency of this modality is lower than the overall level and its semantic matching degree with other modalities is worse; a positive result indicates the opposite.

[0114] The average consistency score for all modalities is the average of the sum of the individual modal consistency scores of all participating modalities and the result of dividing by the total number of modalities. It reflects the overall level of semantic consistency of all modalities and serves as the benchmark for judging whether the consistency of a single modality meets the standard.

[0115] The correction amplitude threshold is an upper limit for coefficient adjustment set to avoid overcorrection of divergence and reasonable prediction of distorted modes. It limits the maximum range of variation of the divergence correction coefficient and ensures that the correction process is smooth and controllable.

[0116] Then, the processing device determines the divergence correction coefficient for the i-th mode based on the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold. The expression for calculating the divergence correction coefficient is as follows:

[0117]

[0118] in, This represents the bifurcation correction coefficient for the i-th mode. Indicates the correction amplitude threshold. This represents the consistency deviation of the i-th mode. This represents the k-th type of conflict identifier for the i-th modality. This represents the average score of consistency across all modes.

[0119] Finally, the processing device uses the divergence correction coefficients of each modality to correct the second prediction confidence for each task category, thus obtaining the third prediction confidence for each task category.

[0120] The third prediction confidence score is the final fusion result obtained after adjusting for the divergence correction coefficient. It resolves the modal conflict problem present in the second prediction confidence score and is more accurate and reliable than the former. It is the final decision output of the multimodal post-fusion process. The formula for calculating the third prediction confidence score is:

[0121]

[0122] in, This indicates the confidence level of the third prediction.

[0123] The confidence index corresponding to the output of the processing device and the third prediction confidence level includes the category, probability entropy, energy score and average consistency corresponding to the highest probability of the output of the i-th mode.

[0124] The credibility index is a set of supplementary information that is output simultaneously with the third prediction confidence level and is used to assess the reliability of the final multimodal decision. Its role is to provide decision-making basis and risk assessment reference for downstream businesses (such as manual review and compliance audit), and to avoid blind decision-making caused by relying on a single prediction result.

[0125] The category with the highest probability output of the i-th modality is the category with the highest probability value after each modality expert model predicts all task categories.

[0126] Probability entropy is an indicator that measures the degree of concentration of the probability distribution of multimodal prediction results. It is calculated based on the probability distribution of the third prediction confidence level. The lower the entropy value, the higher the confidence level of a certain category is compared with other categories, and the clearer the decision direction is. The higher the entropy value, the closer the confidence levels of each category are, and the higher the decision uncertainty is.

[0127] The energy score is an indicator that reflects the confidence level of multimodal prediction. It is calculated by exponentially normalizing the third prediction confidence score. The higher the energy score, the stronger the model's confidence in the final prediction category and the lower the possibility of the prediction result being affected by noise. Conversely, it indicates that the model is still hesitant about the conclusion.

[0128] Average consistency is the arithmetic mean of the single-modal consistency scores of all modalities, reflecting the overall semantic matching level of multimodalities. The higher the average consistency, the more consensus most modalities have reached on the core semantics, and the more solid the semantic foundation of the fusion conclusion is; conversely, it indicates that there are still many semantic differences between modalities, and the decision-making risk should be guarded against.

[0129] After generating the third prediction confidence for each task category, the processing device will simultaneously compile and output the corresponding confidence metrics:

[0130] First, extract the category with the highest probability when reasoning independently in each modality, and clearly present the correspondence between the single-modal conclusion and the multimodal fusion conclusion. For example, the highest probability category in most modalities is lesion, which is consistent with the fusion conclusion, and can enhance the confidence of decision-making.

[0131] Next, the probability entropy and energy score based on the third prediction confidence are calculated. From the two dimensions of probability distribution concentration and model confidence, the uncertainty of the final decision is quantified. For example, low entropy and high energy score represent a clear and credible decision.

[0132] Finally, the single-modal consistency scores of all modalities are summarized, and the average consistency is calculated to reflect the overall semantic consensus level of the multimodal system. For example, an average consistency of 0.85 indicates a high degree of semantic matching between modalities and a reliable basis for the fusion conclusion.

[0133] These credibility indicators, combined with the third prediction confidence level, constitute a complete output system for decision-making results and supporting evidence.

[0134] Based on the above description, this application has the following beneficial effects:

[0135] From the perspective of cross-modal information utilization, this method overcomes the limitation of traditional schemes that only summarize decision-level scores. It extracts evidence summaries for each modality through a consistency auxiliary channel bypass, including text entities, image objects, and audio keywords, and calculates a single-modal consistency score based on these evidence summaries. This method can deeply capture fine-grained cross-modal associations such as the matching degree between text entities and image objects, and the conflict between verbal negation and textual affirmation, avoiding decision bias caused by semantic fragmentation between modalities and reducing the probability of misjudgment caused by neglecting cross-modal associations in traditional schemes.

[0136] From the perspective of ensuring decision reliability, this method uses three-dimensional indicators—monomodal consistency score, quality score, and uncertainty score—to jointly determine the fusion weights. Compared to the fixed weights of traditional rule-based schemes and dynamic gating schemes that rely solely on a single dimension of quality or context, this method can more comprehensively evaluate the decision value of each modality: the quality score ensures that high-quality inputs such as high-resolution images and high signal-to-noise ratio audio receive reasonable weights; the uncertainty score suppresses the interference of overconfident modalities; and the consistency score strengthens the influence of modalities whose conclusions are consistent with those of other modalities. The three work together to achieve a weight bias towards high-quality, reliable, and consistent modalities, avoiding the problems of erroneous resonance of homologous modalities and the dominance of overconfident modalities in decision-making in traditional schemes.

[0137] Meanwhile, this method triggers a divergence correction mechanism through a divergence index. For modal conclusion conflict scenarios, it generates correction coefficients based on parameters such as conflict identifiers and consistency deviations to slightly calibrate the fusion results, further reducing the risk of serious misjudgment. The output of credibility indicators such as probability entropy and energy score also provides an interpretable basis for downstream business such as manual review triggering and rejection strategy formulation.

[0138] The above text combined Figure 1 The multimodal post-fusion method based on the consistency auxiliary channel provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0139] like Figure 2 As shown in the figure, this is a schematic diagram of a multimodal post-fusion device based on a consistency auxiliary channel provided in an embodiment of this application. The device includes:

[0140] Acquisition module 301 is used to acquire multimodal samples;

[0141] The data processing module 302 is used to input the multimodal samples into each modality expert model for parallel processing, to obtain the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modality expert model. The evidence package includes an evidence summary, a quality score, and an uncertainty score. Based on the evidence summary in the evidence package output by each modality expert model, the single-modality consistency score of each modality is determined. Based on the single-modality consistency score, quality score, and uncertainty score of each modality, the fusion weight of each modality is determined.

[0142] The calibration domains are divided according to the multimodal sample types and task categories. Adaptive temperature parameters are configured for different domains. The probability distribution of the first prediction confidence of each mode is adjusted by adjusting the temperature parameters to obtain the calibrated first prediction confidence.

[0143] The fusion module 303 is used to fuse the calibrated first prediction confidence based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

[0144] Optionally, the data processing module 302 is further configured to determine a divergence index based on the single-modal consistency score, quality score, and uncertainty score of each modality; if the divergence index is greater than the divergence index threshold, then obtain the divergence correction coefficient of each modality; and use the divergence correction coefficient of each modality to correct the second prediction confidence of each task category to obtain the third prediction confidence of each task category.

[0145] Optionally, the acquisition module 301 is specifically used to acquire the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold.

[0146] The divergence correction coefficient of the i-th mode is determined based on the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold.

[0147] Optionally, the data processing module 302 is also used to output a confidence index corresponding to the third prediction confidence level, wherein the confidence index includes the category, probability entropy, energy score and average consistency corresponding to the highest probability of the i-th modality output.

[0148] The multimodal post-fusion apparatus based on a consistency auxiliary channel according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the multimodal post-fusion apparatus based on a consistency auxiliary channel are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0149] This application also provides a computing device. For example... Figure 3 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate with each other via the bus 701.

[0150] The 701 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0151] The processor 702 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0152] The communication interface 703 is used for communication with external devices.

[0153] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0154] The memory 704 stores executable code, and the processor 702 executes the executable code to perform the aforementioned multimodal post-fusion method based on the consistency auxiliary channel.

[0155] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the multimodal post-fusion device based on the consistency auxiliary channel described in the embodiments are implemented by software, the execution... Figure 2 The software or program code required for the functions of each module / unit can be partially or entirely stored in memory 704. Processor 702 executes the program code corresponding to each unit stored in memory 704, and executes the aforementioned multimodal post-fusion method based on consistency auxiliary channel.

[0156] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned multimodal post-fusion method based on a consistency-assisted channel.

[0157] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0158] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0159] When the computer program product is executed by a computer, the computer executes any of the aforementioned methods of the multimodal post-fusion method based on the consistency auxiliary channel. The computer program product can be a software installation package; when any of the aforementioned methods of the multimodal post-fusion method based on the consistency auxiliary channel needs to be used, the computer program product can be downloaded and executed on the computer.

[0160] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0161] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A multimodal post-fusion method based on a consistency auxiliary channel, characterized in that, The method includes: Acquire multimodal samples; The multimodal samples are input into each modal expert model for parallel processing to obtain the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes evidence summary, quality score and uncertainty score. The main decision path is the link in the modal expert model that directly outputs the task prediction results. Its first prediction confidence score is the initial judgment basis of the model for each task category. The consistency auxiliary channel bypass is a bypass information link that runs parallel to the main decision path. It is specifically used to extract key auxiliary information in the modal reasoning process and does not directly participate in the initial prediction. The evidence package is the information set output by the consistency auxiliary channel, which includes three types of content: first, the evidence summary includes at least one of entities in text, objects in images, and keywords in audio; second, the quality score includes at least one of image sharpness and audio signal-to-noise ratio; and third, the uncertainty score includes at least one of probability entropy and MC dropout variance. Based on the evidence summaries in the evidence packages output by each modality expert model, determine the monomodal consistency score for each modality; The fusion weights for each modality are determined based on the single-modal consistency score, quality score, and uncertainty score of each modality. The calibration domains are divided according to the multimodal sample types and task categories. Adaptive temperature parameters are configured for different domains. The probability distribution of the first prediction confidence of each mode is adjusted by adjusting the temperature parameters to obtain the calibrated first prediction confidence. The calibrated first prediction confidence is fused based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

2. The method according to claim 1, characterized in that, The method further includes: The divergence index is determined based on the single-modal consistency score, quality score, and uncertainty score for each modality. If the divergence index is greater than the divergence index threshold, then the divergence correction coefficient for each mode is obtained. By using the divergence correction coefficients of each modality, the second prediction confidence for each task category is corrected, resulting in the third prediction confidence for each task category.

3. The method according to claim 2, characterized in that, The bifurcation correction coefficient for the i-th mode is obtained in the following way: Obtain the k-th type of conflict identifier for the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold; The divergence correction coefficient of the i-th mode is determined based on the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold.

4. The method according to claim 3, characterized in that, The method further includes: Output a confidence index corresponding to the third prediction confidence level. The confidence index includes the category, probability entropy, energy score, and average consistency corresponding to the highest probability of the i-th modality output.

5. The method according to any one of claims 1-4, characterized in that, The multimodal samples include image samples, text samples, or audio samples.

6. A multimodal post-fusion device based on a consistency auxiliary channel, characterized in that, The device includes: The acquisition module is used to acquire multimodal samples; The data processing module is used to input the multimodal samples into various modal expert models for parallel processing, obtaining the first prediction confidence of the main decision path for each task category in each modal expert model and the evidence package of the consistency auxiliary channel bypass. The evidence package includes an evidence summary, a quality score, and an uncertainty score. The main decision path is the link in the modal expert model that directly outputs the task prediction result, and its output first prediction confidence is the initial judgment basis of the model for each task category. The consistency auxiliary channel bypass is a bypass information link parallel to the main decision path, specifically used to extract key auxiliary information in the modal inference process, and does not directly participate in the initial prediction. The evidence package is the information set output by the consistency auxiliary channel, containing three types of content: first, the evidence summary includes at least one of entities in text, objects in images, and keywords in audio; second, the quality score includes at least one of image clarity and audio signal-to-noise ratio; and third, the uncertainty score includes probability entropy and MC. At least one of the dropout variances; based on the evidence summary in the evidence package output by each modality expert model, determine the single-modality consistency score of each modality; based on the single-modality consistency score, quality score and uncertainty score of each modality, determine the fusion weight of each modality; divide the calibration domain according to the multimodal sample type and task category, configure the appropriate temperature parameters for different domains, adjust the probability distribution of the first prediction confidence of each modality through the temperature parameters, and obtain the calibrated first prediction confidence; The fusion module is used to fuse the calibrated first prediction confidence based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

7. The apparatus according to claim 6, characterized in that, The data processing module is further configured to determine a divergence index based on the single-modal consistency score, quality score, and uncertainty score of each modality; if the divergence index is greater than the divergence index threshold, then obtain the divergence correction coefficient for each modality; and use the divergence correction coefficient for each modality to correct the second prediction confidence for each task category, thereby obtaining the third prediction confidence for each task category.

8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, The computer program product includes one or more computer instructions, which, when executed by a computer, perform the method as described in any one of claims 1 to 5.

Images