Multi-modal post-fusion method, device and equipment based on consistent auxiliary channel, medium and product

Abstract

The invention discloses a multi-modal post-fusion method and device based on a consistency auxiliary channel, equipment, a medium and a product, and relates to the technical field of artificial intelligence, and the method comprises the steps: firstly obtaining a multi-modal sample; inputting the main decision path into each modal expert model for parallel processing to obtain a first prediction confidence coefficient of the main decision path and an evidence packet containing an evidence abstract, a quality score and an uncertainty score of a consistency auxiliary channel bypass; and determining a single-modal consistency score of each modal based on the evidence abstract, determining a fusion weight in combination with the quality score and the uncertainty score, and further performing weighted fusion on the first prediction confidence to obtain a second prediction confidence. And a divergence index can also be determined based on the three types of scores, if the divergence index is greater than a threshold value, a divergence correction coefficient of each modal is obtained, the second prediction confidence coefficient is corrected to obtain a third prediction confidence coefficient, and a credibility index including the highest probability category, the probability entropy, the energy score and the average consistency of each modal is output, so that the multi-modal decision accuracy is improved by the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal post-fusion method and device based on a consistency auxiliary channel, equipment, medium and product. BACKGROUND

[0002] Multi-modal post-fusion technology is a key technology for integrating cross-modal information in the field of artificial intelligence, and is widely used in retrieval question answering, content understanding, medical detection, remote sensing and multi-sensor perception scenes. Multi-modal post-fusion technology can maintain the independence of each modal expert model, support modular reuse and engineering expansion: each modal can be trained, updated or replaced independently, without the need to reconfigure the overall fusion architecture, thereby reducing the development and maintenance cost of industrial multi-modal systems.

[0003] Typical multi-modal post-fusion schemes are mainly divided into four categories: rule-based schemes that aggregate modal decision results through weighted averaging, voting, thresholding and other preset logic; learning-based schemes that rely on meta-classifiers, linear or nonlinear fusioners, and train fusion models using additional data to achieve integration; probabilistic schemes based on PoE, Dempster-Shafer, Copula and other probability theories to handle modal uncertainty and complete fusion; and dynamic gating schemes that adjust the fusion proportion through adaptive weights or MoE routing according to modal quality or scene context.

[0004] Currently, the integration of modal output results by the scheme of independent reasoning and decision layer score aggregation of each modal does not deeply mine the correlation information between cross-modals, and therefore has a problem of high probability of misjudgment. SUMMARY

[0005] The present application provides a multi-modal post-fusion method, device, equipment, medium and product based on a consistency auxiliary channel, which can reduce the probability of misjudgment.

[0006] To achieve the above purpose, the present application adopts the following technical scheme: In a first aspect, the present application provides a multi-modal post-fusion method based on a consistency auxiliary channel, comprising: obtaining a multi-modal sample; inputting the multi-modal sample into each modal expert model for parallel processing to obtain a first prediction confidence of each task category of a main decision path in each modal expert model and an evidence package bypassed by a consistency auxiliary channel, the evidence package comprising an evidence summary, a quality score and an uncertainty score; determining a single-modal consistency score of each modal based on the evidence summary in the evidence package output by each modal expert model; determining a fusion weight of each modal based on the single-modal consistency score, the quality score and the uncertainty score of each modal; According to the multi-modal sample type and the task category division calibration sub-domain, the temperature parameters suitable for different sub-domains are configured, the probability distribution of the first prediction confidence of each modality is adjusted through the temperature parameters, and the calibrated first prediction confidence is obtained; The calibrated first prediction confidence is fused based on the fusion weight of each modality, and the second prediction confidence for each task category is obtained.

[0007] Optionally, the divergence index is determined based on the single-modal consistency score, the quality score and the uncertainty score of each modality. If the divergence index is greater than the divergence index threshold value, the divergence correction coefficient of each modality is obtained. The second prediction confidence for each task category is corrected by using the divergence correction coefficient of each modality, and the third prediction confidence for each task category is obtained.

[0008] Optionally, the divergence correction coefficient of the i-th modality is obtained by the following method: The k-th conflict identifier of the i-th modality, the consistency deviation of the i-th modality, the consistency average score of all modalities and the correction amplitude threshold value are obtained. According to the k-th conflict identifier of the i-th modality, the consistency deviation of the i-th modality, the consistency average score of all modalities and the correction amplitude threshold value, the divergence correction coefficient of the i-th modality is determined.

[0009] Optionally, the method further comprises: Output the confidence index corresponding to the third prediction confidence, and the confidence index includes the category corresponding to the highest probability output by the i-th modality, the probability entropy, the energy score and the average consistency.

[0010] Optionally, the multi-modal sample includes an image sample, a text sample or an audio sample.

[0011] In a second aspect, the present application provides a multi-modal post-fusion device based on a consistency auxiliary channel, comprising: An acquisition module is configured to acquire a multi-modal sample. a data processing module, configured to input the multi-modal sample into respective modal expert models for parallel processing, to obtain first prediction confidence of a main decision path of each modal expert model for each task category and an evidence package of a consistency auxiliary channel bypass, the evidence package comprising an evidence summary, a quality score and an uncertainty score; determine a single-modal consistency score of each modal based on the evidence summary in the evidence package output by each modal expert model; determine a fusion weight of each modal based on the single-modal consistency score, the quality score and the uncertainty score of each modal; divide calibration sub-domains according to the multi-modal sample type and the task category, configure adaptive temperature parameters for different sub-domains, adjust the probability distribution of the first prediction confidence of each modal through the temperature parameters, and obtain calibrated first prediction confidence; a fusion module, configured to fuse the calibrated first prediction confidence based on the fusion weight of each modal to obtain second prediction confidence for each task category.

[0012] In a third aspect, the present application provides a computing device, comprising a memory and a processor; wherein one or more computer programs are stored in the memory, the one or more computer programs comprising instructions; when the instructions are executed by the processor, the computing device performs the method of any one of the first aspect.

[0013] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program for executing the method of any one of the first aspect.

[0014] From the above technical solutions, the present application has at least the following beneficial effects: From the perspective of cross-modal information utilization, the method breaks through the limitation of traditional schemes that only summarize decision layer scores, extracts evidence summaries of each modal including text entities, image objects, audio keywords, etc. through a consistency auxiliary channel bypass, and calculates a single-modal consistency score based on the evidence summaries. The method can deeply capture fine-grained cross-modal associations such as matching degree of text entities and image objects, conflictiveness of speech negation and text affirmation, avoid decision bias caused by semantic fragmentation between modalities, and reduce the probability of misjudgment caused by traditional schemes ignoring cross-modal associations.

[0015] From the perspective of decision reliability guarantee, the method determines the fusion weight jointly by three-dimensional indexes of single-modal consistency score, quality score and uncertainty score. Compared with the fixed weight of traditional rule-based scheme and the single dimension of quality or context of dynamic gating scheme, the method can more comprehensively evaluate the decision value of each modality: the quality score ensures that high-quality inputs such as high-definition images and high signal-to-noise ratio audio obtain reasonable weight, the uncertainty score suppresses the interference of overconfident modalities, and the consistency score strengthens the influence of modalities consistent with the conclusions of other modalities, so as to realize the weight inclination of high-quality, reliable and consistent modalities, and avoid the problems of error resonance of homologous modalities and overconfident modalities dominating the decision in traditional schemes. At the same time, through the divergence index to trigger the divergence correction mechanism, aiming at the scene of conflicting conclusions of modalities, the correction coefficient is generated based on the conflict identifier and consistency deviation, and the fusion result is slightly calibrated, so as to further reduce the risk of serious misjudgment; and the output of the reliability indexes such as probability entropy and energy distribution also provides an interpretable basis for the triggering of downstream business such as manual review and the formulation of rejection strategy.

[0016] It should be understood that the description of technical features, technical solutions, advantages or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and advantages described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a particular embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a multi-modal post-fusion method based on a consistency auxiliary channel provided by an embodiment of the application; Figure 2 A schematic diagram of a multi-modal post-fusion device based on a consistency auxiliary channel provided by an embodiment of the application; Figure 3 A schematic diagram of a computing device provided by an embodiment of the application. DETAILED DESCRIPTION

[0018] The terms "first", "second" and "third" and the like in the specification and drawings of this application are used to distinguish different objects, not to limit a specific order.

[0019] In the embodiments of the present application, the words such as “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application should not be construed as preferred or advantageous over other embodiments or design solutions. Rather, the use of the words such as “exemplary” or “for example” is intended to present relevant concepts in a specific manner.

[0020] For the sake of clear and concise description of each of the following embodiments, a brief introduction to the related art is first given: The multi-modal post-fusion technology refers to a technology in an artificial intelligence system, in which different modal expert models (single modal processing models) of images, texts, audios, etc. independently complete reasoning, and then the results output by each modal are integrated in the decision layer. The advantage thereof lies in maintaining the independence of each modal expert model, supporting modular reuse, each modal can be trained, updated or replaced independently, without the need to reconfigure the overall architecture, greatly reducing the development and maintenance cost of an industrial multi-modal system, and being widely applied to scenarios such as retrieval and question answering, medical detection, remote sensing perception, etc.

[0021] The present application focuses on the problems of the traditional multi-modal post-fusion scheme in practical application, and the key technical problems are mainly manifested as high misjudgment probability and insufficient decision reliability: first, cross-modal correlation is fragmented, leading to semantic misjudgment. The traditional scheme only summarizes the scores of the decision layer of each modal, and cannot capture the matching degree of text entities and image objects, the conflict between voice negation and text affirmation, and other fine-grained correlations, which is easy to produce false conclusions due to semantic contradiction between modes; second, the modal weight distribution is unreasonable. The fixed weight of the rule-based scheme ignores the quality difference of the modes, and the dynamic gating scheme only relies on a single dimension evaluation, leading to problems such as overconfidence of low-quality modes dominating the fusion, error resonance of homologous modes, etc.; third, there is no effective correction for conflict scenarios. When the multi-modal conclusion is obviously in conflict, there is no targeted adjustment mechanism, and only the original logic can be followed, which amplifies the risk of serious misjudgment; fourth, the decision explainability is poor. Only the final result is output, and there is a lack of evidence chain and credibility index, making it difficult for downstream businesses to trace the basis and locate the misjudgment causes.

[0022] The root cause of the above problems lies in the three defects of the traditional scheme: first, the information utilization dimension is single, only the prediction confidence of the main decision path is concerned, and the semantic evidence in the modal output is not mined, leading to the complete neglect of cross-modal correlation information; second, the weight evaluation dimension is one-sided, and there is no multi-dimensional collaborative mechanism. The rule-based, learning-based and probabilistic schemes respectively have limitations such as ignoring dynamic changes, relying on additional data, and not associating consistency, which cannot comprehensively screen high-quality, reliable and consistent modes; third, the conflict processing and explainable system are blank. There is no quantitative divergence judgment standard and evidence-based correction logic, and key decision information is not recorded, leading to passive acceptance of the results when facing contradictory conclusions, and the decision process cannot be traced.

[0023] In view of this, embodiments of this application provide a multimodal post-fusion method based on a consistency auxiliary channel, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0024] To address the issues of high misjudgment rates and insufficient reliability caused by the lack of cross-modal correlation, one-sided weight evaluation, and gaps in conflict handling in traditional multimodal post-fusion schemes, this application fills the gaps in cross-modal correlation analysis by mining semantic evidence and quality and uncertainty information of each modality through auxiliary channels. Then, a weight calculation and disagreement correction mechanism is constructed using three-dimensional indicators of consistency, quality, and uncertainty to achieve evidence-based fusion and corrective decision-making. At the same time, a credibility indicator is output to ensure interpretability. Ultimately, without reconstructing the existing modal expert model, the accuracy, reliability, and engineering adaptability of multimodal fusion are significantly improved.

[0025] To make the technical solution of this application clearer and easier to understand, the following description, in conjunction with the accompanying drawings, introduces a multimodal post-fusion method based on a consistency auxiliary channel provided by an embodiment of this application. Figure 1 As shown, this figure is a flowchart of a multimodal post-fusion method based on a consistency auxiliary channel provided in an embodiment of this application. The method includes: S201. The processing equipment acquires multimodal samples.

[0026] Multimodal samples include image samples, text samples, or audio samples. A multimodal sample is a data set containing two or more different types of information. Image samples are data that carry visual information; text samples are data that carry textual information; and audio samples are data that carry sound information. These three types of samples can be used individually or in combination to form multimodal data, providing a foundation for subsequent cross-modal fusion analysis.

[0027] The processing device uses pre-defined data interfaces to acquire and store multimodal data. These interfaces include receiving cloud data via a network interface, receiving local device data via USB or HDMI interfaces, and receiving real-time acquired data via sensor interfaces. The processing device collects multimodal data such as images, text, and audio from different data sources and stores it in the local computing unit.

[0028] The processing equipment needs to verify whether the data from each modality meets the format requirements, such as whether the image resolution, text encoding, and audio sampling rate meet the standards, while also removing damaged or invalid data. This step lays the foundation for subsequent input of samples into the expert models of each modality for parallel inference, and is a key bridge connecting the raw data and model processing.

[0029] S202. The processing device inputs multimodal samples into each modal expert model for parallel processing, and obtains the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes evidence summary, quality score and uncertainty score.

[0030] Modal expert models are specialized processing models designed for single-modal data. Each modality corresponds to one or a group of expert models, which focus only on feature extraction and inference of their own modality and do not depend on other modal data.

[0031] Parallel processing refers to the process where the processing device simultaneously inputs different types of multimodal samples into the corresponding modal expert models. Each model runs independently and does not interfere with the others. There is no need to wait for one model to complete inference before starting another model, thereby improving the overall data processing efficiency.

[0032] The main decision path is the link in the modal expert model that directly outputs the task prediction results. Its first prediction confidence score is the model's initial judgment basis for each task category, presented in the form of probability value or logits score, reflecting the model's confidence in the prediction results.

[0033] The consistency auxiliary channel is a bypass information link that runs parallel to the main decision-making path. It is an innovative design of this application and is specifically used to extract key auxiliary information in the modal reasoning process. It does not directly participate in the initial prediction but only provides support for subsequent cross-modal consistency analysis and weight calculation.

[0034] The evidence package is a collection of information output from the consistency auxiliary channel, containing three categories: first, evidence summaries, which are information such as entities in text, objects in images, and keywords in audio, representing a condensed presentation of the semantics of each modality; second, quality scores, which serve as indicators of the quality of modal input data, specifically including image clarity and audio signal-to-noise ratio; and third, uncertainty scores, which reflect the degree of hesitation in the predictions of the modal expert model, commonly manifested in forms such as probability entropy and MC dropout variance.

[0035] The first prediction confidence is the initial prediction result output by the main decision path for each task category. For example, in a medical detection task, the prediction probability of the image expert model for the two categories of lesion and normal is the first prediction confidence of the model.

[0036] After acquiring multimodal samples, the processing device allocates the samples to the corresponding modal expert models according to their type. For example, image samples are input into the image expert model, and text samples are input into the text expert model. Subsequently, all models simultaneously start inference operations. During the inference process, each modal expert model synchronously generates two types of outputs: on the one hand, the main decision path outputs a first prediction confidence score that reflects the reliability of the prediction for each category in the task; on the other hand, the consistency auxiliary path synchronously generates an evidence package, which includes semantic key information (evidence summary), data quality assessment (quality score), and model prediction risk (uncertainty score).

[0037] This step retains the advantages of modal independent reasoning in traditional post-fusion techniques, and fills the data gap in cross-modal association analysis by outputting evidence packages, providing a basis for subsequent consistency score calculation and fusion weight determination.

[0038] S203. The processing device determines the single-modal consistency score of each modality based on the evidence summary in the evidence package output by each modal expert model.

[0039] Evidence summaries are semantic information from evidence packages of various modal expert models. They are a condensed and refined summary of the key content of modal data. For example, evidence summaries for text modalities include named entities, relations, or negation polarity; for image modalities, they include detected objects, attributes, or spatial attention features; and for audio modalities, they include identified keywords, emotional events, etc. Their role is to provide a comparable common language for cross-modal semantic matching.

[0040] The monomodal consistency score is a quantitative indicator that measures the degree of semantic matching between a certain modality and all other modalities in the system. The score ranges from 0 to 1. The higher the score, the more semantically consistent the evidence summary of that modality is with the evidence summaries of other modalities. Conversely, the lower the score, the more semantically inconsistent there is between modalities.

[0041] After acquiring the evidence packages output by all modal expert models, the processing device first extracts the evidence summary from each evidence package, focusing on the core semantic information of each modality. Then, it calculates the semantic overlap between any two modal evidence summaries using a specific algorithm, i.e., the pairwise consistency score. Finally, it robustly aggregates the pairwise consistency scores of a certain modality with all other modalities to obtain the single modality consistency score of that modality.

[0042] The formula for calculating pairwise consistency score is:

[0043] in, Indicates pairwise consistency score. Representing modes The Confidence level of a concept confidence of the i-th concept of the modality confidence of the i-th concept of the modality confidence of the i-th concept of the modality number of concepts time step decay factor

[0044] The single-modality consistency score is slightly calculated as follows:

[0045] wherein, comprehensive score of the i-th modality total number of modalities participating in fusion

[0046] S204, the processing device determines the fusion weight of each modality based on the single-modality consistency score, the quality score and the uncertainty score of each modality.

[0047] The quality score is an index for measuring the quality of the modality input data, with a value of 0-1, which is calculated based on the physical or semantic characteristics of the data itself, such as the sharpness of an image, the signal-to-noise ratio of audio, the integrity of text, etc. The higher the score, the more reliable the input data, and the more solid the model inference foundation.

[0048] The uncertainty score is an index reflecting the degree of hesitation of the modality expert model in prediction, with a value of 0-1, which is calculated through the probability distribution characteristics of the model output, such as probability entropy (the more dispersed the distribution, the higher the score), MC dropout variance (the greater the fluctuation, the higher the score), and the higher the score, the lower the model's confidence in the prediction result.

[0049] The fusion weight is a coefficient that determines the speaking power of each modality's prediction result in the final fusion. The sum of all modality weights is 1, and the higher the weight, the greater the influence of the modality's prediction confidence on the final fusion result.

[0050] The processing device first summarizes the single-modality consistency score, the quality score and the uncertainty score of each modality, and then performs comprehensive calculation on the three types of scores through a monotone-constrained gating mapping function. In the specific calculation, the three types of scores are first weighted and summed with non-negative coefficients to obtain the comprehensive score of each modality, wherein the consistency score and the quality score are positive contribution terms that directly improve the comprehensive score, and the uncertainty score is a negative penalty term that reduces the comprehensive score. The calculation expression of the comprehensive score is as follows:

[0051] wherein, comprehensive score of the i-th modality comprehensive score of the i-th modality non-negative first weight coefficient non-negative second weight coefficient denotes a non-negative third weight coefficient, denotes a prior preference coefficient, denotes a quality score, denotes an uncertainty score.

[0052] The comprehensive scores of all modalities are then normalized to ensure that the sum of the weights of all modalities is 1. The calculation expression of the initial fusion weight is:

[0053] wherein, denotes the initial fusion weight of the modality denotes the comprehensive score of the modality denotes the comprehensive score of the modality denotes the comprehensive score of the modality denotes the comprehensive score of the modality denotes the comprehensive score of the modality

[0054] At the same time, to avoid a certain modality monopolizing the weight, such as preventing an overconfident low-quality modality from dominating the decision, a steady-state constraint is also imposed, for example, setting upper and lower limits for the weight of each modality, ensuring a minimum entropy value of the weight distribution. The final fusion weight is essentially a quantitative determination of the decision value of each modality, providing a basis for subsequent weighted fusion of the prediction confidence of each modality, effectively solving the unreasonable problem caused by the traditional scheme of relying solely on a single dimension (such as fixed rules or a single quality indicator) to allocate weights. The expression of the final fusion weight is:

[0055] wherein, denotes the final fusion weight, denotes a smoothing coefficient, denotes the initial fusion weight of the modality denotes the initial fusion weight of the modality denotes the initial fusion weight of the modality denotes the initial fusion weight of the modality

[0056] S205, the processing device divides the calibration sub-domain according to the multi-modal sample type and the task category, configures an adaptive temperature parameter for different sub-domains, adjusts the probability distribution of the first prediction confidence of each modality through the temperature parameter, obtains the calibrated first prediction confidence, and fuses the calibrated first prediction confidence based on the fusion weight of each modality to obtain the second prediction confidence for each task category.

[0057] The multi-modal sample type is an independent data category participating in multi-modal fusion, such as images, texts, audio, sensor data, etc. Each type corresponds to a dedicated modality expert model that processes its own data format and features.

[0058] Task category is the prediction target category that needs to be completed in multi-modal fusion, such as normal, mild fault and severe fault categories in fault detection, and is the evaluation object of prediction confidence.

[0059] The first prediction confidence is the initial prediction result of each task category output by the main decision path of each modal expert model, reflecting the reliability of single-modal independent reasoning.

[0060] The calibrated first prediction confidence is the probability value converted to the interval of 0-1 after temperature parameter adjustment, solving the problem of incomparable predictions in different domains and better fitting the real prediction performance.

[0061] The second prediction confidence is the cross-modal unified prediction result obtained by weighting and integrating the first prediction confidence of each modal according to the fusion weight, which is the preliminary decision conclusion after multi-modal information coordination, and is more reliable than single-modal prediction.

[0062] The processing device splits the scene according to multi-modal sample types and task categories, for example, an image modal corresponds to a fault diagnosis task as a domain, ensuring that the prediction scene of each domain has consistency and laying a foundation for accurate calibration.

[0063] The temperature parameter is configured for each domain, and the optimal temperature parameter that minimizes the calibration error (such as ECE) is selected based on the validation set data through grid search or gradient descent optimization, which adapts to the prediction characteristics of the domain.

[0064] First, the first prediction confidence output by each modal, that is, the original score of the model output layer without activation function processing, is scaled according to the temperature parameter corresponding to the calibration domain of the modal, to obtain the calibrated score after temperature adjustment; then the calibrated score is substituted into the Softmax function for calculation, converting it into a probability value between 0 and 1, which is the calibrated first prediction confidence.

[0065]

[0066] wherein, represents the modal , calibration domain , the calibrated logits score of the first task category, represents the modal , calibration domain , the first prediction confidence of the first task category, represents the temperature parameter corresponding to the modal , calibration domain .

[0067]

[0068] wherein, denotes the modality , calibration sub-domain the first prediction confidence after calibration of the first task category. denotes the modality , calibration sub-domain the calibrated logits score of the first task category, denotes the total number of task categories.

[0069] After such processing, the prediction confidence under different types of modalities (such as images, text, and audio) and different application scenarios (such as fault detection and medical diagnosis) is unified into the same probability evaluation standard, providing a basis for direct comparison and fusion, while effectively compressing the excessively amplified category gap in the original prediction and avoiding high-confidence misjudgment caused by excessive confidence of the model.

[0070] The calculation expression of the second prediction confidence is:

[0071] wherein, denotes the second prediction confidence, denotes the final obtained fusion weight, denotes the modality , calibration sub-domain the first prediction confidence after calibration of the first task category, denotes the total number of modalities participating in fusion.

[0072] The method further includes optimization of the second prediction confidence, and the specific steps are as follows: First, the processing device determines the divergence index based on the single-modality consistency score, quality score, and uncertainty score of each modality.

[0073] The divergence index is an index that quantifies the degree of conclusion conflict among multiple modalities by comprehensively considering the three types of scores, and its value is 0-1. It is the basis for determining whether there is a contradiction among the modalities. The higher the score, the greater the difference among the multiple modalities in terms of semantic consistency, data quality reliability, and model prediction confidence, and the more obvious the conclusion conflict.

[0074] When calculating the divergence index, the processing device will operate step by step as follows: First, collect the three types of key scores of all modalities, i.e., the single-modality consistency score, quality score, and uncertainty score of each modality, to prepare data for subsequent calculation.

[0075] ​​​Second, the difference degree of each type of score between different modalities, i.e., dispersion, is analyzed respectively. For example, when calculating the dispersion of the quality score, the coefficient of variation is used to measure the standard deviation divided by the mean of the quality scores of all modalities. If the quality score of some modalities is as high as 0.9 and that of some modalities is as low as 0.3, the standard deviation is large and the mean is small, and the coefficient of variation is high, indicating that the quality difference between different modalities is obvious. The dispersion of the consistency score and the uncertainty score between modalities can also be calculated by the same coefficient of variation method to determine the difference between the two types of scores.

[0076] Third, different non-negative weights are assigned to the three types of dispersion according to specific business scenarios. For example, in medical detection scenarios that require high semantic matching, a higher weight is assigned to the consistency score dispersion; and in sensor monitoring scenarios, if more attention is paid to the reliability of the data itself, the weight of the quality score dispersion may be increased. Then, the weighted value is calculated by each type of dispersion and the corresponding weight, and the three weighted values are added to obtain the divergence index. The calculation expression of the divergence index is:

[0077] wherein, represents the divergence index, represents the weight of the dispersion of the quality score, represents the dispersion of the quality score, represents the weight of the dispersion of the consistency score, represents the dispersion of the consistency score, represents the weight of the dispersion of the uncertainty score, represents the dispersion of the uncertainty score.

[0078] If the divergence index is greater than the divergence index threshold, the divergence correction coefficient of each modality is obtained.

[0079] The processing device compares the calculated divergence index with the preset divergence index threshold. If the divergence index is greater than the divergence index threshold, for example, the divergence index is calculated as 0.75 and the divergence index threshold is set as 0.6, it indicates that the current multi-modality conclusion conflict is obvious, and the use of the second prediction confidence may have a high risk of misjudgment, which needs to be further corrected by the divergence correction coefficient.

[0080] The divergence correction coefficient of the i-th modality is obtained by the following method: First, the processing device obtains the k-th type of conflict identifier of the i-th modality, the consistency deviation of the i-th modality, the consistency average of all modalities, and the correction amplitude threshold.

[0081] The kth conflict identifier of the ith modality is identification information for marking whether the ith modality has a conclusion conflict with other modalities on the kth task category. For example, in a fault detection task, the first image modality predicts that the device fault (k = 1) category is yes, but most modalities predict that it is no, and therefore the first conflict identifier of the modality is conflict.

[0082] The consistency deviation of the ith modality is an index for quantifying the difference between the consistency level of the ith modality and the overall consistency level of all modalities, and is calculated by subtracting the average consistency score of all modalities from the single-modality consistency score of the ith modality. If the result is negative, it means that the consistency of the modality is lower than the overall level and the semantic matching degree with other modalities is worse; if the result is positive, it is the opposite.

[0083] The average consistency score of all modalities is an average value obtained by adding the single-modality consistency scores of all modalities participating in fusion and then dividing by the total number of modalities, which reflects the overall level of all modalities in semantic consistency and is a benchmark for judging whether the single-modality consistency meets the standard.

[0084] The correction amplitude threshold is a coefficient adjustment upper limit set to avoid excessive divergence correction and distort the reasonable prediction of the modality, limit the maximum variation range of the divergence correction coefficient, and ensure that the correction process is stable and controllable.

[0085] Then, the processing device determines the divergence correction coefficient of the ith modality according to the kth conflict identifier of the ith modality, the consistency deviation of the ith modality, the average consistency score of all modalities, and the correction amplitude threshold. The calculation expression of the divergence correction coefficient is:

[0086] wherein, represents the divergence correction coefficient of the ith modality, represents the correction amplitude threshold, represents the consistency deviation of the ith modality, represents the kth conflict identifier of the ith modality, represents the average consistency score of all modalities.

[0087] Finally, the processing device corrects the second prediction confidence for each task category using the divergence correction coefficient of each modality to obtain the third prediction confidence for each task category.

[0088] The third prediction confidence is the final fusion result obtained after adjustment by the divergence correction coefficient, which solves the modality conflict problem existing in the second prediction confidence and is more accurate and reliable than the former, and is the final decision output of the multi-modality post-fusion process. The calculation expression of the third prediction confidence is:

[0089] wherein, represents a third prediction confidence.

[0090] The processing device outputs a confidence index corresponding to the third prediction confidence, and the confidence index includes a class corresponding to a highest probability of an i-th modality output, a probability entropy, an energy fraction, and an average consistency.

[0091] The confidence index is a set of supplementary information output synchronously with the third prediction confidence, for evaluating reliability of a multi-modal final decision, and functions to provide a decision basis traceability and a risk judgment reference for downstream business (such as artificial review and compliance audit), so as to avoid blind decision caused by relying on a single prediction result.

[0092] The class corresponding to the highest probability of the i-th modality output is a class with a highest probability value after each modality expert model predicts all task classes.

[0093] The probability entropy is an index for measuring concentration of a probability distribution of a multi-modal prediction result, and is calculated based on the probability distribution of the third prediction confidence. The lower the entropy value is, the more the confidence of a certain class is higher than those of other classes, and the more the decision direction is clear. The higher the entropy value is, the closer the confidence of each class is, and the higher the decision uncertainty is.

[0094] The energy fraction is an index for reflecting a self-confidence degree of a multi-modal prediction, and is calculated by exponential normalization of the third prediction confidence. The higher the energy fraction is, the stronger the confidence of the model to the final prediction class is, and the lower the possibility of the prediction result being interfered by noise is. Conversely, it indicates that the model still hesitates about the conclusion.

[0095] The average consistency is an arithmetic mean of single-modal consistency scores of all modalities, and reflects a semantic matching level of the multi-modal whole. The higher the average consistency is, the more the consensus on the core semantics is reached by most modalities, and the more solid the semantic basis of the fusion conclusion is. Conversely, it indicates that there are still more semantic differences among the modalities, and the decision risk needs to be alerted.

[0096] After the processing device generates the third prediction confidence of each task class, the corresponding confidence index is synchronized and output: First, the class corresponding to the highest probability during independent inference of each modality is extracted, so as to clearly present the corresponding relationship between the single-modal conclusion and the multi-modal fusion conclusion, for example, the highest probability class of most modalities is a lesion, which is consistent with the fusion conclusion, so as to enhance the decision trustworthiness. Then, the probability entropy and the energy fraction based on the third prediction confidence are calculated, so as to quantify the uncertainty of the final decision from two dimensions of probability distribution concentration and model self-confidence degree, for example, low entropy and high energy fraction represent clear and reliable decision. Finally, the single-modal consistency scores of all modalities are aggregated to calculate the average consistency, reflecting the overall semantic consensus level of the multi-modal, for example, an average consistency of 0.85 represents a high degree of semantic matching between modalities, and the basis for the fusion conclusion is reliable.

[0097] These credibility indicators, combined with the third prediction confidence, constitute a complete output system of decision results and supporting evidence.

[0098] Based on the above content description, the present application has the following beneficial effects: From the perspective of cross-modal information utilization, this method breaks through the limitations of traditional schemes that only aggregate decision layer scores. It extracts evidence summaries of each modality, including text entities, image objects, audio keywords, etc., through a consistency auxiliary channel bypass, and calculates single-modal consistency scores based on the evidence summaries. This method can deeply capture fine-grained cross-modal associations such as the matching degree of text entities and image objects, and the conflict between speech negation and text affirmation, avoiding decision bias caused by semantic fragmentation between modalities and reducing the probability of misjudgment caused by traditional schemes that ignore cross-modal associations.

[0099] From the perspective of decision reliability guarantee, this method determines the fusion weight based on the three-dimensional indicators of single-modal consistency score, quality score, and uncertainty score. Compared with the fixed weight of traditional rule-based schemes and the single dimension of quality or context in dynamic gating schemes, this method can more comprehensively evaluate the decision value of each modality: the quality score ensures that high-quality inputs such as high-definition images and high signal-to-noise ratio audio receive reasonable weights, the uncertainty score suppresses the interference of overly confident modalities, and the consistency score strengthens the influence of modalities consistent with other modalities' conclusions. The three work together to tilt the weights of high-quality, reliable, and consistent modalities, avoiding the problems of error resonance of homologous modalities and the dominance of overly confident modalities in traditional schemes.

[0100] Meanwhile, this method triggers a divergence correction mechanism through a divergence index, which generates a correction coefficient based on conflict identifiers and consistency deviations for small-scale calibration of the fusion results in the presence of conflicting modality conclusions, further reducing the risk of serious misjudgment. The output of credibility indicators such as probability entropy and energy distribution also provides interpretable evidence for downstream business such as artificial review triggering and rejection strategy formulation.

[0101] The above Figure 1 The multi-modal post-fusion method based on a consistency auxiliary channel provided by the embodiments of the present application is described in detail, and the device and equipment provided by the embodiments of the present application will be introduced below with reference to the accompanying drawings.

[0102] As Figure 2 shown, the figure is a schematic diagram of a multi-modal post-fusion device based on a consistency auxiliary channel provided by an embodiment of the present application, which includes: The acquisition module 301 is configured to acquire a multi-modal sample. The data processing module 302 is configured to input the multi-modal sample into respective modal expert models for parallel processing to obtain a first prediction confidence of a main decision channel of each modal expert model for each task category and an evidence package of a consistency auxiliary channel bypass, the evidence package including an evidence summary, a quality score and an uncertainty score; determine a single-modal consistency score of each modal based on the evidence summary in the evidence package output by each modal expert model; and determine a fusion weight of each modal based on the single-modal consistency score, the quality score and the uncertainty score of each modal. According to the multi-modal sample type and the task category, a calibration sub-domain is divided, and adaptive temperature parameters are configured for different sub-domains; the probability distribution of the first prediction confidence of each modal is adjusted through the temperature parameters to obtain calibrated first prediction confidence. The fusion module 303 is configured to fuse the calibrated first prediction confidence based on the fusion weight of each modal to obtain second prediction confidence for each task category.

[0103] Optionally, the data processing module 302 is further configured to determine a disagreement index based on the single-modal consistency score, the quality score and the uncertainty score of each modal; if the disagreement index is greater than a disagreement index threshold, obtain a disagreement correction coefficient of each modal; and correct the second prediction confidence for each task category by using the disagreement correction coefficient of each modal to obtain third prediction confidence for each task category.

[0104] Optionally, the acquisition module 301 is specifically configured to acquire a kth conflict identifier of an ith modal, a consistency deviation of the ith modal, a consistency average of all modalities and a correction amplitude threshold. According to the kth conflict identifier of the ith modal, the consistency deviation of the ith modal, the consistency average of all modalities and the correction amplitude threshold, the disagreement correction coefficient of the ith modal is determined.

[0105] Optionally, the data processing module 302 is further configured to output a confidence index corresponding to the third prediction confidence, the confidence index including a category corresponding to the highest probability output by the ith modal, a probability entropy, an energy score and an average consistency.

[0106] The multi-modal post-fusion device based on the consistency auxiliary channel according to the embodiments of the present application can correspond to the method described in the embodiments of the present application, and the above-mentioned other operations and / or functions of each module / unit of the multi-modal post-fusion device based on the consistency auxiliary channel are respectively implemented to realize Figure 1 The corresponding flow of each method in the illustrated embodiments, for the sake of brevity, will not be described here.

[0107] The embodiment of the present application also provides a computing device. As shown in the figure, the figure is a schematic diagram of a computing device provided by the embodiment of the present application, and the computing device 700 comprises a bus 701, a processor 702, a communication interface 703 and a memory 704. The processor 702, the memory 704 and the communication interface 703 communicate through the bus 701. Figure 3 As shown in the figure, the figure is a schematic diagram of a computing device provided by the embodiment of the present application, and the computing device 700 comprises a bus 701, a processor 702, a communication interface 703 and a memory 704. The processor 702, the memory 704 and the communication interface 703 communicate through the bus 701.

[0108] The bus 701 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 3 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0109] The processor 702 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP) processor.

[0110] The communication interface 703 is used for external communication.

[0111] The memory 704 can include a volatile memory, such as a random access memory (RAM). The memory 704 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0112] The memory 704 stores executable code, and the processor 702 executes the executable code to perform the aforementioned multi-modal post-fusion method based on a consistency auxiliary channel.

[0113] Specifically, in the case of implementing the embodiment shown in the figure, and Figure 2 In the embodiment, in the case of implementing the multi-modal post-fusion device based on the consistency auxiliary channel described in the embodiment by software, the executable code is executed to perform Figure 2 Figure 2 ​The software or program code required for the functions of the various modules / units in the foregoing embodiments can be partially or entirely stored in the memory 704. The processor 702 executes the program code corresponding to each unit stored in the memory 704 to perform the foregoing method for multi-modal post-fusion based on a consistency auxiliary channel.

[0114] The computer readable storage medium can be any available medium or data storage device that can be accessed by the computing device, such as a data center, etc. The available medium can be a magnetic medium (e.g., floppy diskette, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the foregoing method for multi-modal post-fusion based on a consistency auxiliary channel.

[0115] The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computing device, the foregoing method for multi-modal post-fusion based on a consistency auxiliary channel is generated in whole or in part.

[0116] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0117] The computer program product is executed by a computer, and the computer executes any of the foregoing methods for multi-modal post-fusion based on a consistency auxiliary channel. The computer program product can be a software installation package, and when any of the foregoing methods for multi-modal post-fusion based on a consistency auxiliary channel is needed, the computer program product can be downloaded and executed on the computer.

[0118] The foregoing description of the flow or structure corresponding to each figure has its own emphasis, and the parts not described in detail in a certain flow or structure can be referred to the related description of other flows or structures.

[0119] The foregoing merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

Claims

1. A multimodal post-fusion method based on a consistency auxiliary channel, characterized in that, The method includes: Acquire multimodal samples; The multimodal samples are input into each modal expert model for parallel processing to obtain the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes evidence summary, quality score and uncertainty score. Based on the evidence summaries in the evidence packages output by each modality expert model, determine the monomodal consistency score for each modality; The fusion weights for each modality are determined based on the single-modal consistency score, quality score, and uncertainty score of each modality. The calibration domains are divided according to the multimodal sample types and task categories. Adaptive temperature parameters are configured for different domains. The probability distribution of the first prediction confidence of each mode is adjusted by adjusting the temperature parameters to obtain the calibrated first prediction confidence. The calibrated first prediction confidence is fused based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

2. The method according to claim 1, characterized in that, The method further includes: The divergence index is determined based on the single-modal consistency score, quality score, and uncertainty score for each modality. If the divergence index is greater than the divergence index threshold, then the divergence correction coefficient for each mode is obtained. By using the divergence correction coefficients of each modality, the second prediction confidence for each task category is corrected, resulting in the third prediction confidence for each task category.

3. The method according to claim 2, characterized in that, The bifurcation correction coefficient for the i-th mode is obtained in the following way: Obtain the k-th type of conflict identifier for the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold; The divergence correction coefficient of the i-th mode is determined based on the k-th type of conflict identifier of the i-th mode, the consistency deviation of the i-th mode, the average consistency score of all modes, and the correction magnitude threshold.

4. The method according to claim 3, characterized in that, The method further includes: Output a confidence index corresponding to the third prediction confidence level. The confidence index includes the category, probability entropy, energy score, and average consistency corresponding to the highest probability of the i-th modality output.

5. The method according to any one of claims 1-4, characterized in that, The multimodal samples include image samples, text samples, or audio samples.

6. A multimodal post-fusion device based on a consistency auxiliary channel, characterized in that, The device includes: The acquisition module is used to acquire multimodal samples; The data processing module is used to input the multimodal samples into various modal expert models for parallel processing, obtaining the first prediction confidence of the main decision path for each task category and the evidence package of the consistency auxiliary path bypass in each modal expert model. The evidence package includes an evidence summary, a quality score, and an uncertainty score. Based on the evidence summary in the evidence package output by each modal expert model, the single-modal consistency score of each modality is determined. Based on the single-modal consistency score, quality score, and uncertainty score of each modality, the fusion weight of each modality is determined. The calibration domain is divided according to the multimodal sample type and task category, and suitable temperature parameters are configured for different domains. The probability distribution of the first prediction confidence of each modality is adjusted by the temperature parameters to obtain the calibrated first prediction confidence. The fusion module is used to fuse the calibrated first prediction confidence based on the fusion weights of each modality to obtain the second prediction confidence for each task category.

7. The apparatus according to claim 6, characterized in that, The data processing module is further configured to determine a divergence index based on the single-modal consistency score, quality score, and uncertainty score of each modality; if the divergence index is greater than the divergence index threshold, then obtain the divergence correction coefficient for each modality; and use the divergence correction coefficient for each modality to correct the second prediction confidence for each task category, thereby obtaining the third prediction confidence for each task category.

8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, The computer program product includes one or more computer instructions, which, when executed by a computer, perform the method as described in any one of claims 1 to 5.

Images