Method and system for bias correction of multi-modal data based on causal reasoning

Abstract

The application provides a multi-modal data bias correction method and system based on causal reasoning, and belongs to the technical field of data bias correction. The method comprises the following steps: collecting a multi-modal original data set according to a deployed data acquisition terminal, and constructing a multi-modal data set; performing bias label annotation to generate a structured feature set with bias labels; using a bias tracing mechanism, constructing a basic cross-modal causal graph, embedding a cross-modal attention layer, generating a dynamic cross-modal causal graph, and combining dynamic edge weights to generate a bias attribution heat map; generating counterfactual samples of an unbiased hypothesis scenario, constructing a dynamic correction model, and training the dynamic correction model using reinforcement learning; performing bias correction on the structured feature set with bias labels to obtain a corrected feature set, and generating a correction result evaluation report. The application improves bias positioning accuracy, enhances bias traceability interpretability, quickly adapts to new bias, and deeply adapts the correction result to the target task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data bias correction, in particular to a multi-modal data bias correction method and system based on causal reasoning. BACKGROUND

[0002] With the penetration of artificial intelligence technology into complex scenarios, multi-modal data has become the core support for improving the performance of target tasks because it can provide complementary semantic representation and environmental perception dimensions. In recent years, multi-modal data processing technology has evolved from simple feature splicing to deep modal fusion. However, multi-modal data is prone to introduce bias in the whole process of collection, labeling, transmission and preprocessing due to complex factors, which has become a key bottleneck restricting the reliability of the technology.

[0003] In the prior art, when correcting multi-modal data bias, the multi-modal data bias traceability is not accurate and does not have dynamic adaptation, resulting in ambiguous bias positioning, waste of correction resources, and low correction efficiency; multi-modal data is often in a dynamic environment, and multi-modal bias drifts or is newly added in a dynamic environment, and a static correction model cannot be self-adaptive, resulting in decay of correction effect; the target of traditional multi-modal bias correction is single, multi-modal bias correction is disconnected from the target task, only the bias residual rate is concerned, and the actual support effect of the corrected data on the target task is ignored. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal data bias correction method and system based on causal reasoning to solve the problems of inaccurate data bias traceability, no dynamic adaptation, and static correction model unable to adapt to dynamic environment in existing multi-modal data bias correction.

[0005] In order to achieve the above-mentioned purpose, the present application provides a multi-modal data bias correction method based on causal reasoning, which comprises: collecting a multi-modal original data set according to a deployed data collection terminal, and constructing a multi-modal data set; standardizing the multi-modal data set and labeling bias labels to generate a structured feature set with bias labels; based on the structured feature set with bias labels, using a bias traceability mechanism, constructing a basic cross-modal causal graph, embedding a cross-modal attention layer to generate a dynamic cross-modal causal graph, and combining the dynamic edge weight of the dynamic cross-modal causal graph to generate a bias attribution heat map; based on the bias attribution heat map, generating counterfactual samples of bias-free hypothesis scenarios, constructing a dynamic correction model, and training the dynamic correction model using reinforcement learning; using the trained dynamic correction model to correct the bias of the structured feature set with bias labels to obtain a corrected feature set, and performing multi-dimensional evaluation on the corrected feature set to generate a correction result evaluation report.

[0006] Optionally, the collecting the multi-modal raw data set according to the deployed data collection terminal comprises: determining the type of modality that needs to be collected according to the application scene of the target task, and deploying a multi-modal data collection terminal; recording modality meta information, environment information and annotation information during the process of collecting the multi-modal raw data, and generating a multi-modal raw data set; constructing a preset preliminary screening rule according to the characteristics of the multi-modal, and using the preset preliminary screening rule to eliminate invalid data in the multi-modal raw data set.

[0007] Optionally, the standardizing the multi-modal data set and performing bias label annotation comprises: constructing a differential feature extraction logic according to the characteristics of the multi-modal, converting the unstructured multi-modal valid data set into a structured vector feature; unifying the structured vector features of the multi-modal into the same dimension to generate a structured feature set; defining a multi-modal bias factor set according to a modality meta information library, and using a hybrid annotation strategy to annotate the structured feature set with a bias label.

[0008] Optionally, the constructing a basic cross-modal causal graph using a bias traceability mechanism comprises: constructing nodes and edges of a basic cross-modal causal graph according to the structured feature set with a bias label and the multi-modal bias factor set, and setting the weight of the initial edge; generating a basic cross-modal causal graph based on the nodes, edges and weight of the initial edge, and storing it in a causal graph database.

[0009] Optionally, the embedding a cross-modal attention layer, generating a dynamic cross-modal causal graph, and combining the dynamic edge weight of the dynamic cross-modal causal graph to generate a bias attribution heat map comprises: embedding a cross-modal attention layer based on the basic cross-modal causal graph, updating the weight of the initial edge according to the attention weight, and generating a dynamic cross-modal causal graph; using an attribution algorithm to calculate the total contribution degree of bias factors and the modal feature bias contribution degree based on the dynamic edge weight of the dynamic cross-modal causal graph, and generating a bias attribution heat map.

[0010] Optionally, the generating an unbiased counterfactual sample set based on the bias attribution heat map comprises: screening a high-contribution bias factor set based on the total contribution degree of bias factors in the bias attribution heat map; generating a candidate counterfactual sample using a causal generative adversarial network according to the high-contribution bias factor set; performing causal consistency verification on the candidate counterfactual sample based on the dynamic cross-modal causal graph, screening out valid counterfactual samples, storing all valid counterfactual samples, and generating an unbiased counterfactual sample set.

[0011] Optionally, the multi-dimensional evaluation on the corrected feature set is performed to generate a correction result evaluation report, including: calculating an average feature difference of a corresponding modality in the unbiased counterfactual sample set based on all feature vectors of each modality in the corrected feature set; comparing the average feature difference with a pre-correction average feature difference to obtain a modality-level bias residual rate; integrating the modality-level bias residual rates of each modality, and performing weighted calculation according to a modality feature bias contribution degree to obtain an overall bias residual rate, so as to realize bias residual degree evaluation; determining a plurality of performance indicators according to a target task type, calculating corresponding performance indicators based on the corrected feature set and the structured feature set with biased labels, to obtain pre-correction indicator values and post-correction indicator values; calculating a performance improvement amplitude based on the pre-correction indicator values and the post-correction indicator values, so as to realize target task performance evaluation; and integrating the bias residual degree evaluation result and the target task performance evaluation result to generate the correction result evaluation report.

[0012] Optionally, the correction method further includes: based on the result of the correction result evaluation report, optimizing the bias tracing mechanism and the dynamic correction model, and triggering a closed-loop iteration mechanism to make a closed-loop response to bias changes.

[0013] Optionally, based on the result of the correction result evaluation report, optimizing the bias tracing mechanism and the dynamic correction model includes: based on the bias residual degree evaluation result in the correction result evaluation report, identifying potential problems of the multi-modality, and determining whether the bias tracing mechanism needs to be optimized; using the optimized bias tracing mechanism to perform bias attribution on the structured feature set with labels to generate a new bias attribution heat map; comparing the new bias attribution heat map with the bias attribution heat map to perform attribution verification; based on the bias residual degree evaluation result and the target task performance evaluation result, identifying optimization needs of the dynamic correction model; retraining the optimized dynamic correction model, and performing correction effect verification after the training is completed.

[0014] In another aspect, the present application provides a multi-modal data bias correction system based on causal reasoning, which includes a control module, the control module includes a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the multi-modal data bias correction method according to any one of the above.

[0015] By the technical scheme, the application constructs a dynamic cross-modal causal graph and generates a bias attribution heat map, thereby significantly improving bias positioning accuracy, realizing efficient allocation of correction resources, and enhancing bias traceability interpretability; through reinforcement learning and closed-loop optimization of dynamically generated counterfactual sample sets and dynamic correction models, the correction effect stability in a dynamic environment is improved, new bias is quickly adapted, and the cost of manual intervention is reduced; through two-dimensional design of the reward function of the dynamic correction model and two-dimensional verification of the correction effect evaluation, deep adaptation of the correction result and the target task can be realized, over-correction or under-correction is avoided, and the development efficiency of the target task is improved.

[0016] Other features and advantages of the present application will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of embodiments of the application, and constitute a part of the specification, and are used together with the following detailed description to explain the embodiments of the application, but do not constitute a limitation on the embodiments of the application. In the drawings:

[0018] Figure 1 is a flowchart of the bias correction method for multi-modal data based on causal reasoning of the application Figure 1 ;

[0019] Figure 2 is a flowchart of constructing a multi-modal data set in the application;

[0020] Figure 3 is a flowchart of generating a structured feature set with bias labels in the application;

[0021] Figure 4 is a flowchart of constructing a basic cross-modal causal graph in the application;

[0022] Figure 5 is a flowchart of generating a bias attribution heat map in the application;

[0023] Figure 6 is a flowchart of generating a bias-free counterfactual sample set in the application;

[0024] Figure 7 is a flowchart of generating a correction result evaluation report in the application;

[0025] Figure 8 is a flowchart of optimizing the bias traceability mechanism and the dynamic correction model in the application;

[0026] Figure 9 is a flowchart of the bias correction method for multi-modal data based on causal reasoning of the application Figure 2 . DETAILED DESCRIPTION

[0027] The specific implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiments of the present application, and is not used to limit the embodiments of the present application.

[0028] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of national laws and regulations. In the embodiments of the present application, some existing industry solutions may be mentioned, such as software, components, models, etc. They should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0029] Please refer to Figure 1 The embodiments of the present application provide a multi-modal data bias correction method based on causal reasoning, which can include:

[0030] Step S110: According to the deployed data acquisition terminal, a multi-modal original data set is collected, and a multi-modal data set is constructed.

[0031] Please refer to Figure 2 In the embodiments of the present application, collecting a multi-modal original data set according to a deployed data acquisition terminal can include:

[0032] Step S1101: According to the application scenario of the target task (for example, medical, industrial, autonomous driving, etc.), determine the type of modality that needs to be collected (for example, medical scene requires image, text, sensor modality, industrial scene requires image, sensor, log text modality, etc.), and deploy a multi-modal data acquisition terminal.

[0033] In combination with Figure 9 In the preferred embodiments of the present application, the multi-modal data acquisition terminal needs to have multi-modal data synchronous acquisition capability, for example, connecting each modality acquisition device (for example, camera, sensor, text input terminal) through a hardware interface (for example, USB, Ethernet), and ensuring the timestamp consistency of different modal data through the built-in time synchronization module (avoiding modal misplacement caused by acquisition delay).

[0034] Step S1102: In the process of collecting multi-modal original data, record the modal meta information, environment information and annotation information, and generate a multi-modal original data set.

[0035] In the preferred embodiment of the present application, the model of the acquisition device, the current working parameters (for example, camera resolution, sensor sampling frequency), the last maintenance time (to determine whether there is a deviation caused by aging of the device) can be recorded for the acquisition device; the temperature, humidity, and light intensity (for example, the dust concentration in the workshop of an industrial scene, the radiation dose in the scanning room of a medical scene) can be recorded for the acquisition environment; the name of the annotator, the annotation time, and the annotation guide version (to trace the source of the annotation deviation) can be recorded for the manual annotation link. All the meta-information is stored in the modal meta-information library in a structured format of "data ID-meta-information item-value", ensuring that each piece of original data is traceable. After the acquisition is completed, the original multi-modal data set is formed, and the data set is indexed with "data ID" as the core, and is associated with each modal original file (for example,.jpg image file,.txt text file,.csv sensor time series file).

[0036] Step S1103: According to the characteristics of the multi-modal, a preset preliminary screening rule is constructed, and the invalid data in the multi-modal original data set is removed by using the preset preliminary screening rule.

[0037] In the preferred embodiment of the present application, the construction of the preset preliminary screening rule needs to combine the field common sense and the physical logic boundary of the modal data: for example, the rules of the image modal include "no pixel missing area (for example, no black block at the edge of the image)" and "no obvious blur (for example, the texture details are identifiable)"; the rules of the text modal include "no random code character" and "no semantic irrelevant content (for example, only containing punctuation or meaningless letters)"; the rules of the sensor modal include "data value within the physical range (for example, the temperature sensor does not appear-100℃ or 200℃ abnormal value)" and "time series data without continuous same value (to avoid invalid data caused by sensor jam)".

[0038] In the preferred embodiment of the present application, the original multi-modal data set can be screened according to the logic of checking each data: the corresponding rule is applied to each modal file of each data, if any modal does not conform to the rule (for example, there is a large area of missing in the image of a certain data), it is determined that the data is invalid data, which is removed from the original multi-modal data set; if all the modal conform to the rule, it is retained as valid data, and finally an initial valid data set is formed. At the same time, the invalid data information (for example, data ID, belonging modal, non-conforming rule, and removal time) is recorded to the invalid data log, and the log needs to support the classification query according to the invalid type (for example, image missing, sensor abnormality), so as to facilitate the subsequent tracing of potential deviation in the acquisition link (for example, a batch of data is removed due to sensor jam, which can be determined that the sensor needs to be maintained).

[0039] Step S120: The multi-modal data set is standardized and labeled with a deviation label to generate a structured feature set with a deviation label.

[0040] Reference is made to Figure 3 In the embodiments of the present application, the multi-modal data set is standardized and labeled with bias labels, which can include:

[0041] Step S1201: According to the characteristics of multi-modal, construct differential feature extraction logic, and convert the unstructured multi-modal effective data set into a structured vector feature.

[0042] In the preferred embodiments of the present application, when processing the text modality, first, the text data is preprocessed (for example, removing punctuation, stop words, converting uppercase letters to lowercase), and then a pre-trained language model (for example, BERT, RoBERTa) is used for semantic encoding, the model input is the processed text, and the output is a fixed length semantic vector; if the text length exceeds the maximum input length of the model, the "truncation + splicing" strategy (retaining the core semantic part of the text) is used to ensure the consistency of the output vector;

[0043] In the preferred embodiments of the present application, when processing the image modality, first, the image is preprocessed (for example, resized to a uniform size, normalized pixel values to the [0, 1] interval), and then a visual feature extraction model (for example, ResNet, MobileNet) is used to extract key visual features, the model input is the preprocessed image, and the output is a fixed length visual feature vector; for images with slight noise (for example, a small number of pixel abnormalities), light filtering (for example, Gaussian filtering) is performed before feature extraction to avoid noise interference with the features.

[0044] In the preferred embodiments of the present application, when processing the sensor modality, first, the time series data is segmented (for example, segmented according to a fixed time window to ensure that each segment of data contains a complete time series trend), and then the statistical features (for example, mean, variance, maximum, minimum, peak value number) of each segment of data are extracted to form a statistical feature vector; if the time series data has missing values, linear interpolation is used to complete them to avoid missing dimensions of the feature vector.

[0045] Step S1202: Unify the structured vector features of the multi-modal to the same dimension to generate a structured feature set.

[0046] In the preferred embodiment of the present application, dimension unification can be achieved by a fully connected neural network. The original feature vectors of each modality are input into the fully connected layer, and the dimension is mapped to the target dimension through linear transformation, while an activation function is introduced to enhance the feature expression capability. After mapping, the numerical distribution (e.g., mean, standard deviation) of the features of each modality needs to be verified to ensure that there is no numerical overflow or distribution distortion caused by dimension mapping. Finally, a multi-modal structured feature set is formed, and the data set format can be "data ID-text feature vector-image feature vector-sensor feature vector".

[0047] Step S1203: According to the modal meta-information library, define a set of multi-modal bias factors, and use a hybrid labeling strategy to label the structured feature set with bias labels.

[0048] In the preferred embodiment of the present application, the multi-modal bias factor set can be defined in combination with the modal meta-information library and domain expertise. The bias factor classification needs to cover the potential bias of the multi-modal data from the whole process of acquisition to preprocessing. It can include bias caused by acquisition equipment or environment, such as camera lens stains (image modality), sensor hardware offset (sensor modality), and text entry errors (text modality); bias caused by manual labeling, such as text label mislabeling (e.g., "pneumonia" labeled as "cold"), and image lesion region labeling offset (e.g., tumor region labeling too large / small); and bias caused by multi-modal data heterogeneity, such as different modal time stamps not aligned (e.g., CT image acquisition time and medical record entry time difference more than 30 minutes), and different modal feature scale differences (e.g., text vector numerical range [-1, 1], image vector numerical range [0, 100]).

[0049] In the preferred embodiment of the present application, the mixed labeling strategy can include automatic labeling and manual sampling verification. In automatic labeling, the deviation factor can be associated based on the abnormal record in the modal meta-information library, for example, if the "CT device maintenance time is more than 6 months" of a data in the modal meta-information library, it is labeled that the image feature of the data has acquisition deviation; if the "labeling personnel is a new employee" of a data in the modal meta-information library, it is labeled that the text feature of the data has labeling deviation; if the "modal timestamp difference is more than 10 minutes" of a data in the modal meta-information library, it is labeled that the data has modal heterogeneous deviation. In manual sampling verification, 10%-20% of samples (ensuring covering all deviation types) can be randomly selected from the automatically labeled data set, and 2-3 field experts (for example, respiratory department doctors in medical scenarios, device engineers in industrial scenarios) can be invited to independently label; the expert labeling results are compared with the automatic labeling results, if the consistency is more than 90%, the automatic labeling result is recognized; if the consistency is less than 90%, the automatic labeling rule is adjusted (for example, the association logic of the modal meta-information library abnormal record and the deviation factor is optimized), and the automatic labeling is performed again until the consistency is up to standard, and finally the structured feature set with deviation label is formed, the format is "data ID-each modal feature vector-deviation factor label", and each feature data has clear deviation type identification.

[0050] Step S130: based on the structured feature set with deviation label, using the deviation tracing mechanism, constructing a basic cross-modal causal graph, embedding a cross-modal attention layer, generating a dynamic cross-modal causal graph, and combining the dynamic edge weight of the dynamic cross-modal causal graph, generating a deviation attribution heat map.

[0051] Please refer to Figure 4 In the embodiment of the present application, using the deviation tracing mechanism to construct a basic cross-modal causal graph can include:

[0052] Step S1301: constructing the nodes and edges of the basic cross-modal causal graph according to the structured feature set with deviation label and the multi-modal deviation factor set, and setting the weight of the initial edge.

[0053] In the preferred embodiment of the present application, the modal feature nodes can be defined first, each node corresponding to a standardized feature of a modal, then the deviation factor nodes are defined, each node corresponding to a deviation factor, and all nodes need to be labeled with type (for example, feature node, deviation node) and associated data dimension, to ensure that the node meaning is clear.

[0054] In the preferred embodiment of the present application, the edges represent the causal dependence between nodes, and the construction needs to combine "domain knowledge + statistical correlation verification". First, the potential causal relationship is preliminarily determined based on domain knowledge, and then the statistical correlation strength between nodes is calculated based on the biased label structured feature set (for example, the correlation between the acquisition bias node label and the image feature node feature is calculated, and if the correlation is higher than the set threshold, it is confirmed that the edge exists). If the domain knowledge and the statistical correlation result conflict (for example, the domain knowledge believes that the modal heterogeneous bias node affects the sensor feature node, but the statistical correlation is lower than the threshold), the domain expert needs to be invited to review, and finally determine the existence of the edge.

[0055] In the preferred embodiment of the present application, the weight of the initial edge represents the strength of the causal effect, which can be calculated based on the causal effect value. For example, for the edge acquisition bias node→image feature node, the feature difference between the "data labeled with acquisition bias" and the "unbiased data" in the image feature node in the biased label structured feature set is calculated. The greater the difference, the higher the edge weight, and the weight value is normalized to the [0, 1] interval to ensure that the weights of different edges are comparable. All edges are recorded in the format of "start node-end node-weight value".

[0056] Step S1302: Based on the nodes, edges and weights of the initial edges, a basic cross-modal causal graph is generated and stored in the causal graph database (the database needs to support the addition, deletion, modification and query of nodes and edges, and reserve interfaces for subsequent dynamic adjustment).

[0057] Please refer to Figure 5 In the embodiment of the present application, the cross-modal attention layer is embedded to generate a dynamic cross-modal causal graph, and a bias attribution heat map is generated combined with the dynamic edge weight of the dynamic cross-modal causal graph, which can include:

[0058] Step S1311: Based on the basic cross-modal causal graph, the cross-modal attention layer is embedded, and the weight of the initial edge is updated according to the attention weight to generate a dynamic cross-modal causal graph.

[0059] In the preferred embodiment of the present application, the core function of the cross-modal attention layer is to dynamically adjust the attention degree between the bias factor node and the modal feature node according to the specific data of the current input, making up for the defect that the fixed weight of the basic cross-modal causal graph cannot adapt to the data difference. The input of the cross-modal attention layer includes the current feature data to be processed in the structured feature set with bias label and the feature representation of the node in the basic cross-modal causal graph; the output of the cross-modal attention layer is the attention weight between nodes, which can be calculated by the cross-modal similarity measurement method (for example, calculating the cosine similarity between the feature vector of the image feature node and the feature vector of the collection bias node, the higher the similarity, the greater the potential impact of the current image feature on the collection bias, and the higher the corresponding attention weight), and then updating the edge weight of the basic cross-modal causal graph based on the attention weight to generate a dynamic cross-modal causal graph.

[0060] Step S1312: Based on the dynamic edge weight of the dynamic cross-modal causal graph, the attribution algorithm is used to calculate the bias factor total contribution degree and the modal feature bias contribution degree, and generate a bias attribution heat map.

[0061] In the preferred embodiment of the present application, the bias factor total contribution degree is calculated for each bias factor node, which is the sum of the dynamic edge weights of all associated modal feature nodes, and this value reflects the total influence degree of the bias factor on the overall multi-modal data bias, and the higher the value, the more the bias factor is the main bias source; the modal feature bias contribution degree is calculated for each modal feature node, which is the sum of the dynamic edge weights of all associated bias factor nodes, and this value reflects the total degree of the bias impact on the modal feature, and the higher the value, the more serious the bias of the modal feature.

[0062] In the preferred embodiment of the present application, the horizontal axis of the bias attribution heat map is the modal feature node, and the vertical axis is the bias factor node, forming a two-dimensional matrix of bias factor-modal feature, and each cell in the matrix corresponds to the dynamic edge weight of an edge, and the weight value is directly mapped to the color depth of the cell, and the deeper the color (for example, dark red), the higher the dynamic edge weight, that is, the higher the contribution degree of the bias factor to the modal feature, and the lighter the color (for example, light blue), the lower the contribution degree; and a numerical column of bias factor total contribution degree is added on the right side of the heat map, and a numerical row of modal feature bias contribution degree is added below, which directly displays the two types of contribution degrees, and a color scale bar (for example, 0-0.3 is light blue, 0.3-0.7 is light red, and 0.7-1.0 is dark red) is labeled, which facilitates quick judgment of the contribution degree level.

[0063] Step S140: Based on the bias attribution heat map, generate counterfactual samples of the bias-free hypothesis scenario, and construct a dynamic correction model, and train the dynamic correction model by reinforcement learning.

[0064] Please refer to Figure 6In the embodiment of the present application, based on the bias attribution heat map, generating the unbiased counterfactual sample set can include:

[0065] Step S1401: Based on the total contribution degree of the bias factor in the bias attribution heat map, a high-contribution bias factor set is screened.

[0066] In the preferred embodiment of the present application, the screening rule can be "total contribution degree of bias factor ≥ 0.5". If the total contribution degree of a bias factor exceeds the threshold value, it means that it is the main source of bias of the current data and needs to be eliminated in the counterfactual scenario.

[0067] Step S1402: According to the high-contribution bias factor set, a candidate counterfactual sample is generated using the causal generative adversarial network.

[0068] In the preferred embodiment of the present application, the input of the generator of the causal generative adversarial network is the feature vector of the current data and the counterfactual scenario condition, and the output is the candidate counterfactual sample; the input of the discriminator of the causal generative adversarial network is the candidate counterfactual sample and the unbiased sample in the biased labeled structured feature set, and the core task is to distinguish the candidate counterfactual sample from the real unbiased sample; through multiple rounds of adversarial training, the generator gradually generates candidate samples consistent with the distribution of real unbiased samples, ensuring the authenticity of the samples.

[0069] Step S1403: Based on the dynamic cross-modal causal graph, the causal consistency of the candidate counterfactual sample is verified, the effective counterfactual sample is screened out, and all effective counterfactual samples are stored to generate the unbiased counterfactual sample set.

[0070] In the preferred embodiment of the present application, the verification logic of the causal consistency verification can be: if the counterfactual scenario is to eliminate the collection bias node, then verify whether the bias degree of the image feature node in the candidate sample is significantly reduced, and whether other modal features are affected; if the verification condition is met, it is determined as an effective counterfactual sample; if not, feedback to the generator, adjust the training parameters and regenerate until an effective sample is obtained.

[0071] In the embodiment of the present application, training the dynamic correction model using reinforcement learning can include:

[0072] Step S1411: A dynamic correction model is constructed using an Actor-Critic reinforcement learning framework, which includes a policy network (Actor) and a value network (Critic).

[0073] In the preferred embodiment of the present application, when defining the state space, the state needs to include the current feature data to be corrected and the bias attribution information, ensuring that the dynamic correction model can perceive the bias condition of the data. The state is input into the dynamic correction model in the form of a vector, and the dimension matches the dimension of the feature vector and the number of contribution degrees. When defining the action space, the action needs to cover the targeted correction operations of all modalities and support the adjustment of correction strength, and different actions are designed for different modalities. When designing the reward function, the reward function needs to comprehensively evaluate the correction effect, guide the dynamic correction model to select the action of “eliminating bias completely and achieving optimal target task performance”, and adopt double-dimension (distribution difference reward and target task reward) weighted calculation, and then calculate the total reward (based on the distribution difference reward, the target task reward, and the weight coefficient). The higher the total reward is, the better the correction action is.

[0074] Step S1412: initialization, iterative training, and convergence judgment of the dynamic correction model.

[0075] In the preferred embodiment of the present application, the labeled structured feature set and the unbiased counterfactual sample set can be loaded, the parameters of the Actor network and the Critic network are initialized, the training rounds (for example, 100-200 rounds) and the batch size of each round (for example, 32 data) are set; each round randomly selects a batch of data from the labeled structured features, and performs iterative training on each data: the Actor outputs the action based on the current state, performs the action to correct the feature vector of the data, obtains the corrected feature, calculates the reward, the Critic updates the value function based on the current state and the reward, and the Actor updates the policy parameters through gradient descent based on the value evaluation of the Critic; after each round of training, the average reward value of the batch data is calculated, if the fluctuation range of the average reward value of the last 10 rounds is <0.05 (indicating that the reward is stable), the training is stopped, otherwise the iteration is continued until the convergence condition is met.

[0076] Step S1413: store the trained dynamic correction model to the model library (the dynamic correction model needs to support end-to-end inference of input feature data to output corrected feature data, and can automatically adjust the action selection logic according to the bias attribution heat map of the input data).

[0077] Step S150: using the trained dynamic correction model, performing bias correction on the structured feature set with bias label to obtain a corrected feature set, and performing multi-dimensional evaluation on the corrected feature set to generate a correction result evaluation report.

[0078] Please refer to Figure 7 In the embodiment of the present application, the multi-dimensional evaluation of the corrected feature set and the generation of the correction result evaluation report can include:

[0079] Step S1501: Based on all feature vectors of each modality in the corrected feature set, the average feature difference (e.g., average Euclidean distance) of the corresponding modality in the unbiased counterfactual sample set is calculated.

[0080] Step S1502: The average feature difference is compared with the average feature difference before correction (the average difference between the modality features in the labeled structured feature set and the unbiased counterfactual sample set), and the modality-level bias residual rate is obtained.

[0081] In the preferred embodiment of the present application, the modality-level bias residual rate can be represented by the following formula:

[0082]

[0083] Wherein, the lower the residual rate, the more complete the bias elimination of the modality, and if the residual rate is < 20%, it is determined that the modality bias residual meets the standard.

[0084] Step S1503: The modality-level bias residual rate of each modality is integrated, and the overall bias residual rate (reflecting the overall bias residual situation of the multi-modal data, if the overall residual rate is < 15%, it is determined that the overall bias residual meets the standard) is calculated by weighting according to the contribution of the modality feature bias, so as to realize bias residual degree evaluation.

[0085] Step S1504: According to the target task type, a plurality of performance indicators are determined, and based on the corrected feature set and the structured feature set with biased labels, the corresponding performance indicators are calculated to obtain the pre-correction indicator value and the post-correction indicator value.

[0086] In the preferred embodiment of the present application, the plurality of performance indicators can include classification task (e.g., disease diagnosis, fault classification) selection accuracy, recall rate, text feature score; regression task (e.g., temperature prediction, concentration estimation) selection root mean square error, mean absolute error; to ensure that the indicators can directly reflect the support effect of the corrected data on the target task.

[0087] Step S1505: Based on the pre-correction indicator value and the post-correction indicator value, the performance improvement amplitude is calculated to realize target task performance evaluation.

[0088] In the preferred embodiment of the present application, the corrected feature set and the structured feature set with biased labels are respectively input into the target multi-modal task model, and the task is executed on the same test set to obtain the pre-correction indicator value and the post-correction indicator value.

[0089] In the preferred embodiment of the present application, the performance improvement amplitude can be represented by the following formula:

[0090]

[0091] In the classification task, the accuracy improvement amplitude is positive, indicating performance optimization. In the regression task, the root mean square error reduction amplitude is positive, indicating performance optimization. If the performance index improvement amplitude is greater than or equal to 5%, it is determined that the downstream task performance meets the standard.

[0092] Step S1506: Integrate the bias residual degree evaluation result and the target task performance evaluation result to generate a correction result evaluation report.

[0093] In the preferred embodiment of the present application, the correction result evaluation report can include: evaluation profile (evaluation time, evaluation data range, target task type, evaluation module version), bias residual degree result (list the modal level bias residual rate by mode, mark the meeting standard; give the overall bias residual rate, mark the overall meeting standard; if the residual rate of a certain mode does not meet the standard, the possible reasons need to be analyzed), downstream task performance result (list the core performance indicators before and after correction, calculate the performance improvement amplitude; if the task includes multiple sub-tasks, the indicators need to be listed separately according to the sub-tasks; mark the performance meeting standard; if the performance does not meet the standard, the reasons need to be analyzed), and problem summary and suggestion (summarize the non-standard items and make preliminary optimization suggestions).

[0094] Step S160: Based on the results of the correction result evaluation report, optimize the bias traceability mechanism and the dynamic correction model, and use the closed-loop iteration triggering mechanism to make the closed-loop response bias change.

[0095] Please refer to Figure 8 In the embodiment of the present application, based on the results of the correction result evaluation report, the bias traceability mechanism and the dynamic correction model can be optimized, which can include:

[0096] Step S1601: Based on the bias residual degree evaluation result in the correction result evaluation report, identify potential problems of the multi-modal, and determine whether the bias traceability mechanism needs to be optimized.

[0097] In the preferred embodiment of the present application, if the residual rate of a certain mode does not meet the standard, and the correction result evaluation report analyzes the reason that the hidden bias factor of the mode is not identified, a hidden bias factor node is added in the dynamic cross-modal causal graph, and based on the invalid data log and the domain expert knowledge, the causal edges of the node and the corresponding modal feature node are supplemented, the initial weight of the new edge is calculated to ensure the rationality of the new node and the edge, and the updated dynamic cross-modal causal graph is stored in the causal graph database to replace the original version. Then retrain the cross-modal attention layer, increase the data samples containing hidden bias in the training data, adjust the calculation logic of the attention weight, replace the original module with the trained cross-modal attention layer to ensure adaptation to the updated dynamic cross-modal causal graph.

[0098] In a preferred embodiment of the present invention, if the residual rate of a certain modality meets the standard but is close to the threshold, and the analysis of the correction result evaluation report is that the bias attribution accuracy of the modality is insufficient, then only the cross-modal attention layer is adjusted, without adding new nodes, only the training data and parameters of the cross-modal attention layer are optimized, the cross-modal attention layer is retrained and the original module is replaced to improve the attribution accuracy.

[0099] Step S1602: Using the optimized deviation attribution mechanism, perform deviation attribution on the labeled structured feature set and generate a new deviation attribution heatmap.

[0100] Step S1603: Compare the new deviation attribution heatmap with the deviation attribution heatmap to verify the attribution.

[0101] In a preferred embodiment of the present invention, it is verified whether the contribution of the newly added deviation factor is reasonable (e.g., whether the contribution of sensor noise deviation to sensor feature nodes is >0.3) and whether the attribution accuracy is improved (e.g., whether the calculation error of the contribution of labeled deviation nodes to text feature nodes is reduced). If the verification passes, the optimization is confirmed to be effective; if it fails, the reasons are re-analyzed and the optimization strategy is adjusted.

[0102] Step S1604: Based on the deviation residual evaluation results and the target task performance evaluation results, identify the optimization requirements of the dynamic correction model.

[0103] In a preferred embodiment of the present invention, if the target task performance improvement does not meet the standard (e.g., accuracy improvement of 3% < 5%), the correction result evaluation report analyzes the reason as the task performance weight in the reward function being too low. Then, the second weight coefficient of the target task reward in the reward function is increased, and the first weight coefficient is decreased to ensure that the model training pays more attention to the downstream task performance. At the same time, the calculation logic of the target task reward is optimized (e.g., task recall rate is introduced as a supplementary evaluation indicator to avoid the limitations of a single indicator).

[0104] In a preferred embodiment of the present invention, if the correction action of a certain modality is poor (for example, the residual rate of image feature nodes is still high after correction), the correction result evaluation report analyzes that the reason is that the types of correction actions for that modality are insufficient. Then, a targeted correction action is added for that modality (for example, an image contrast adjustment action is added for image feature nodes), and the correction intensity parameters of the new action are defined (for example, weak, medium, strong). At the same time, the output layer of the Actor network is updated (for example, the output dimension of the new action is increased) to ensure that the model can select the new action.

[0105] In a preferred embodiment of the present invention, if the correction effect fluctuates in a dynamic environment (e.g., the residual rate of different batches of data varies greatly), the reason is that the dynamic correction model is not adaptable enough to changes in data distribution. Therefore, a data distribution monitoring module is added to the dynamic correction model to calculate the distribution difference between the input data and the training data in real time. If the difference exceeds a threshold (e.g., 0.2), the model is automatically fine-tuned (e.g., the dynamic correction model is fine-tuned 10-20 times with a small amount of data from a new batch) to ensure that the model adapts to the new data distribution.

[0106] Step S1605: Retrain the optimized dynamic correction model and verify the correction effect after training is completed.

[0107] In a preferred embodiment of the present invention, the reward function, action space and adaptive module in the updated dynamic correction model are loaded, and the model is retrained with a structured feature set with bias labels and an unbiased counterfactual sample set (e.g., iterating 50-100 times to reduce training costs). After training, the correction effect is verified (e.g., using validation set data to test the residual rate and task performance after correction). If the target is met, the original dynamic correction model in the model library is replaced; otherwise, the optimization strategy is adjusted and optimized.

[0108] In a preferred embodiment of the present invention, the closed-loop iteration triggering mechanism can be designed as a dual-condition triggering mechanism, which may include: fixed-cycle triggering, setting a closed-loop iteration cycle (e.g., every 1000 new data entries processed or every 7 days), and automatically reading the latest correction result evaluation report after the cycle ends to determine whether optimization is needed. This mechanism is suitable for slowly changing deviations (e.g., deviation drift caused by equipment aging); and abnormal triggering, setting an abnormal threshold (e.g., overall deviation residual rate > 20%, target task performance improvement < 3%, a certain modality residual rate > 25%), monitoring the evaluation results in real time, and immediately triggering the closed loop if a certain indicator exceeds the abnormal threshold, without waiting for a fixed cycle. This mechanism is suitable for sudden deviations (e.g., a sudden increase in deviation caused by acquisition equipment failure). In addition, after each closed-loop trigger, a closed-loop update log is generated, recording the update time, triggering conditions, optimized modules, and comparison of key indicators before and after optimization, and stored in the system log library for easy subsequent system maintenance and version traceability.

[0109] This invention also provides a multimodal data bias correction system based on causal reasoning. The multimodal data bias correction system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the multimodal data bias correction method according to any one of the above.

[0110] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0118] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multimodal data bias correction method based on causal reasoning, characterized in that, The multimodal data deviation correction method includes: Based on the deployed data acquisition terminals, collect the original multimodal dataset and construct the multimodal dataset; The multimodal dataset is standardized and biased labels are added to generate a structured feature set with biased labels. Based on the structured feature set with bias labels, a basic cross-modal causal graph is constructed using a bias tracing mechanism, and a cross-modal attention layer is embedded to generate a dynamic cross-modal causal graph. Combined with the dynamic edge weights of the dynamic cross-modal causal graph, a bias attribution heatmap is generated. Based on the bias attribution heatmap, a counterfactual sample set of unbiased assumption scenarios is generated, and a dynamic correction model is constructed. The dynamic correction model is then trained using reinforcement learning. Using the trained dynamic correction model, the biased structured feature set with bias labels is corrected to obtain the corrected feature set. The corrected feature set is then evaluated in multiple dimensions to generate a correction result evaluation report. The method of constructing a basic cross-modal causal graph using the deviation tracing mechanism includes: Based on the structured feature set with bias labels and the multimodal bias factor set, the nodes and edges of the basic cross-modal causal graph are constructed, and the weights of the initial edges are set. Based on the weights of the nodes, edges, and initial edges, a basic cross-modal causal graph is generated and stored in the causal graph database; The embedded cross-modal attention layer generates a dynamic cross-modal causal graph, and combines the dynamic edge weights of the dynamic cross-modal causal graph to generate a biased attribution heatmap, including: Based on the aforementioned basic cross-modal causal graph, a cross-modal attention layer is embedded, and the weights of the initial edges are updated according to the attention weights to generate a dynamic cross-modal causal graph. Based on the dynamic edge weights of the dynamic cross-modal causal graph, the total contribution of the deviation factor and the contribution of the modal feature deviation are calculated using an attribution algorithm, and a deviation attribution heatmap is generated.

2. The multimodal data deviation correction method according to claim 1, characterized in that, The collection of multimodal raw datasets, based on the deployed data acquisition terminals, includes: Based on the application scenario of the target task, determine the types of modalities that need to be collected and deploy multimodal data acquisition terminals; During the collection of multimodal raw data, modal metadata, environmental information, and annotation information are recorded to generate multimodal raw datasets; Based on the characteristics of multimodality, a preset initial screening rule is constructed, and invalid data in the original multimodal dataset is removed using the preset initial screening rule.

3. The multimodal data deviation correction method according to claim 1, characterized in that, The standardization and bias labeling of the multimodal dataset includes: Based on the characteristics of multimodal processing, a differentiated feature extraction logic is constructed to transform unstructured multimodal effective datasets into structured vector features. The structured vector features of multimodal approaches are unified into a single dimension to generate a structured feature set; Based on the modal metadata database, a set of multimodal deviation factors is defined, and a hybrid annotation strategy is used to label the structured feature set with deviation tags.

4. The multimodal data deviation correction method according to claim 1, characterized in that, The generation of an unbiased counterfactual sample set based on the biased attribution heatmap includes: Based on the total contribution of the deviation factors in the deviation attribution heatmap, a set of high-contribution deviation factors is selected. Based on the set of high-contribution bias factors, candidate counterfactual samples are generated using a causal generative adversarial network. Based on the dynamic cross-modal causal graph, the causal consistency of the candidate counterfactual samples is verified, valid counterfactual samples are selected, and all valid counterfactual samples are stored to generate an unbiased counterfactual sample set.

5. The multimodal data deviation correction method according to claim 1, characterized in that, The step of performing a multi-dimensional evaluation of the corrected feature set and generating a correction result evaluation report includes: Based on all feature vectors of each modality in the corrected feature set, calculate the average feature difference with the corresponding modality in the unbiased counterfactual sample set; The modal-level deviation residual rate is obtained by comparing the average feature difference with the average feature difference before correction. The modal-level deviation residual rate of each mode is integrated and weighted according to the contribution of modal characteristic deviations to obtain the overall deviation residual rate, so as to realize the deviation residual rate assessment. Multiple sets of performance indicators are determined based on the target task type. Based on the corrected feature set and the structured feature set with bias labels, the corresponding performance indicators are calculated to obtain the indicator values ​​before and after correction. Based on the pre-correction and post-correction index values, the performance improvement is calculated to achieve target task performance evaluation. The results of the deviation residual assessment are integrated with the results of the target task performance assessment to generate a correction result assessment report.

6. The multimodal data deviation correction method according to claim 1, characterized in that, The correction method further includes: Based on the results of the correction result evaluation report, the deviation tracing mechanism and the dynamic correction model are optimized, and the closed-loop iterative triggering mechanism is used to change the closed-loop response deviation.

7. The multimodal data deviation correction method according to claim 6, characterized in that, The optimization of the deviation tracing mechanism and the dynamic correction model based on the results of the correction result evaluation report includes: Based on the deviation residual assessment results in the correction result evaluation report, identify potential problems in the multimodal model and determine whether the deviation tracing mechanism needs to be optimized. Using the optimized deviation attribution mechanism, deviations are attributed to the labeled structured feature set, generating a new deviation attribution heatmap. The new deviation attribution heatmap is compared with the existing deviation attribution heatmap to verify the attribution. Based on the deviation residual evaluation results and the target task performance evaluation results, the optimization requirements of the dynamic correction model are identified; The optimized dynamic correction model was retrained, and the correction effect was verified after training was completed.

8. A multimodal data bias correction system based on causal reasoning, characterized in that, The multimodal data deviation correction system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the multimodal data deviation correction method according to any one of claims 1-7.

Images