A method for constructing high-quality pseudo-annotated datasets based on complex tasks

Abstract

The present invention discloses a method for constructing a high-quality pseudo-annotated dataset based on complex tasks, which relates to the field of multimodal learning technology, including: constructing a cross-modal causal graph based on multimodal original data, loading a domain knowledge graph, identifying inter-modal confounding variables, and generating initial pseudo-labels; generating counterfactual samples by forcibly cutting off the non-causal path in the cross-modal causal graph, and comparing the pseudo-label differences between the original samples and the counterfactual samples to generate cross-modal debiased pseudo-labels; combining the cross-modal debiased pseudo-labels with the semantic consistency pseudo-labels to fuse and generate a standardized pseudo-annotated dataset with multimodal alignment, clear entity relationships, and semantic consistency. The present invention adopts a counterfactual intervention framework to identify and eliminate the influence of non-causal paths in cross-modal interactions by analyzing the differences in probability distributions, effectively suppressing false associations.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning technology, and in particular to a method for constructing a high-quality pseudo-annotated dataset based on complex tasks. Background Art

[0002] Semi-supervised learning on pseudo-annotated data has become an important approach to addressing the high cost of labeling complex tasks. Existing technologies primarily employ self-training frameworks to generate pseudo-labels through cross-modal feature alignment and confidence threshold filtering. However, with the introduction of cross-modal association modeling methods based on graph neural networks (such as CM-GNN) and knowledge graph enhancement methods (such as KG-BERT), these approaches optimize the label generation process by introducing inter-modal attention mechanisms and entity embedding alignment. However, significant deficiencies remain in eliminating interference from multimodal heterogeneous data, integrating structured domain knowledge constraints, and dynamically optimizing causal pathways.

[0003] Current technical bottlenecks lie in the accumulation of confounding biases and semantic drift during cross-modal pseudo-label generation. Traditional methods rely on static thresholds to filter low-confidence samples, but fail to effectively distinguish causally relevant features from cross-modal co-occurrence noise, resulting in pseudo-labels contaminated by spurious paths. While knowledge graph fusion methods can incorporate domain priors, entity alignment uses a fixed similarity threshold, making it incapable of adapting to dynamic semantic shifts. Furthermore, they lack an interpretable causal decoupling mechanism for counterfactual interventions. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for constructing a high-quality pseudo-annotated dataset based on complex tasks to solve the problems of noise accumulation, causal confusion and semantic drift caused by cross-modal interference and lack of domain knowledge in existing pseudo-annotated datasets.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, the present invention provides a method for constructing a high-quality pseudo-annotated dataset based on complex tasks, which includes: constructing a cross-modal causal graph based on multimodal original data, loading a domain knowledge graph, identifying inter-modal confounding variables, and generating initial pseudo-labels; calculating the distance between the semantic embedding of the initial pseudo-label and the domain knowledge graph node, generating semantic correction weights to semantically correct the initial pseudo-labels, and outputting semantically consistent pseudo-labels; generating counterfactual samples by forcibly cutting off non-causal paths in the cross-modal causal graph, and comparing the pseudo-label differences between the original samples and the counterfactual samples to generate cross-modal debiased pseudo-labels; constructing a dynamic optimization closed loop to monitor the conflict frequency between the gradient direction on the cross-modal debiased pseudo-label and the domain knowledge graph in real time, and dynamically adjusting the cross-modal causal graph structure, semantic correction weights and intervention intensity in the counterfactual sample generation process; combining the cross-modal debiased pseudo-labels and the semantically consistent pseudo-labels to generate a standardized pseudo-annotated dataset with multimodal alignment, clear entity relationships and consistent semantics.

[0008] As a preferred solution of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks described in the present invention, the construction of a cross-modal causal graph includes extracting causal relationships using a PC algorithm combined with a conditional independence test, and generating a directed acyclic causal graph of a set of annotated confounding variables after optimization using a causal structural equation model and adversarial training.

[0009] As a preferred solution of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks described in the present invention, the generation of initial pseudo-labels includes weighted fusion of spatiotemporally aligned multimodal features through a cross-modal attention network, dynamic filtering thresholds of confidence levels of entity relationships in the domain knowledge graph as prior constraints, and backdoor path correction of modal features with confounding variables in combination with conditional independence tests;

[0010] The confidence dynamic filtering threshold is obtained by dynamically calculating the confidence scores of the pseudo-labels of the samples in the current iteration batch.

[0011] As a preferred solution for the method of constructing a high-quality pseudo-annotated dataset based on complex tasks described in the present invention, the semantic correction refers to using the knowledge graph node embedding method to map the pseudo labels and domain knowledge graph entity nodes to a unified semantic space, calculating the cosine similarity between the two, and dynamically adjusting the semantic correction weight based on piecewise functions and hierarchical relationship constraints.

[0012] As a preferred solution of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks described in the present invention, the generating of counterfactual samples includes the following steps:

[0013] Traverse the non-causal paths that meet the d-separation criterion in the cross-modal causal graph and select the paths to be intervened based on the causal effect strength formula;

[0014] Through cross-modal variational autoencoders, we perform intervention operations on non-causal paths, cut off the dependencies of confounding variables, and reconstruct the counterfactual feature distribution;

[0015] Jensen-Shannon divergence is used to quantify the difference between the pseudo labels of the original sample and the counterfactual sample, and a cross-modal bias weight based on an exponential decay function is generated to perform probability normalization debiasing on the pseudo labels.

[0016] As a preferred solution of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks described in the present invention, the construction of a dynamic optimization closed loop includes the following steps:

[0017] Extract the gradient direction features of the cross-modal debiased pseudo-labels and calculate the conflict frequency of the cosine similarity between the gradient direction features and the knowledge graph node embeddings;

[0018] According to the dynamic adjustment strategy of the conflict threshold, the high-frequency conflict paths in the causal graph are deleted, and the causal edges verified by the knowledge graph relationship paths are added;

[0019] Based on the sliding window statistics of the cosine similarity conflict frequency, an exponential decay is applied to the semantic modification weight, and the intervention intensity coefficient in the adversarial loss function of CM-VAE is adaptively adjusted.

[0020] As an optimal solution for the method of constructing a high-quality pseudo-annotated dataset based on complex tasks described in the present invention, the standardized pseudo-annotated dataset includes probabilistic fusion of debiased pseudo-labels and semantically consistent pseudo-labels through a weighted averaging strategy, entity alignment verification is performed based on knowledge graph relationship path traversal, and labels are screened using a confidence dynamic filtering threshold.

[0021] As a preferred solution of the method for constructing a high-quality pseudo-labeled dataset based on complex tasks described in the present invention, the pseudo-label confidence of the samples in the current iteration batch refers to the probability score generated by the initial pseudo-label after semantic alignment with the knowledge graph and dynamic weight correction;

[0022] Each sample in the batch is a set of multimodal data units that have completed spatiotemporal alignment processing.

[0023] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks as described in the first aspect of the present invention.

[0024] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks as described in the first aspect of the present invention.

[0025] The beneficial effects of the present invention are: improving the quality of pseudo-annotated data through the construction of cross-modal causal graphs and the fusion mechanism of dynamic knowledge graphs. The detection of confounding variables based on the backdoor criterion combined with the adversarial training strategy effectively suppresses the interference of cross-modal co-occurrence noise on the causal path and enhances the robustness of causal associations between multimodal features. The dynamic weight correction mechanism embedded in the knowledge graph relies on semantic space projection and hierarchical relationship constraints to achieve precise control of label semantic drift and improve the semantic consistency of domain entity alignment. The counterfactual intervention framework drives the debiasing of non-causal paths through probability distribution differences, eliminating the influence of false correlations in cross-modal interactions. The dynamic optimization closed loop realizes online adaptive adjustment of the causal graph structure based on gradient conflict monitoring, ensuring the stability and reliability of the pseudo-label generation process in complex reasoning scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 This is a flowchart of the method for constructing a high-quality pseudo-annotated dataset based on complex tasks in Example 1;

[0028] Figure 2 Flowchart generated for cross-modal causal annotation in Example 1;

[0029] Figure 3 This is a flowchart of the knowledge graph semantic correction in Example 1;

[0030] Figure 4 This is a flowchart of the counterfactual intervention and debiasing processing in Example 1. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0033] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0034] Example 1, reference Figures 1 to 4 This embodiment provides a method for constructing a high-quality pseudo-annotated dataset based on a complex task, comprising the following steps:

[0035] S1. Construct a cross-modal causal graph based on multimodal raw data, load the domain knowledge graph, identify inter-modal confounding variables, and generate initial pseudo-labels.

[0036] Specifically, the following steps are included:

[0037] It receives input multimodal raw data (text, images, audio, etc.) and performs spatiotemporal alignment, compensates for time offsets through dynamic time warping, uses a cross-modal attention network to align semantic regions, and outputs a multimodal feature representation with a unified spatiotemporal basis.

[0038] Based on the aligned multimodal features, the PC algorithm combined with the conditional independence test is used to learn the causal relationship between modalities. The edge weights are optimized through the causal structural equation model, a directed acyclic causal graph is constructed, and adversarial training is used to enhance the robustness of the causal relationship.

[0039] It should be noted that the directed acyclic causal graph (DAG) is a specific manifestation of the cross-modal causal graph, which is a causal relationship graph constructed in the context of multimodal data.

[0040] Load a predefined domain knowledge graph (such as Wikidata), use the TransE algorithm to embed graph entities and relations into vectors, and use contrastive learning to map multimodal features and knowledge graph entities into the same semantic space, forcing alignment of high-confidence entities.

[0041] In a directed acyclic causal graph, we detect cross-modal common cause paths based on the backdoor criterion. We then combine the domain relationships of the knowledge graph to verify confounding variables (e.g., "lighting affects images and text"). We then eliminate spurious correlation paths through conditional independence tests and output a set of confounding variables.

[0042] The multimodal feature confidence is propagated along the causal path in the directed acyclic causal graph. Combined with the entity relationship constraint reasoning label of the knowledge graph, the confidence dynamic filtering threshold of the entity relationship in the domain knowledge graph is used as a priori constraint. Low-confidence samples are filtered through the confidence dynamic filtering threshold, and noisy labels are cleaned based on entity consistency conflict detection, and the denoised initial pseudo-labels are output.

[0043] It should be noted that the confidence dynamic filtering threshold is dynamically calculated based on the pseudo-label confidence scores of the current batch of samples. Specifically, the confidence scores of all samples are collected and their mean and standard deviation are calculated. The confidence dynamic filtering threshold is calculated by subtracting a certain number of standard deviations from the mean, where the adjustable parameter is usually 2 or 3 to control the strictness of the filtering. The confidence score of each sample is compared with the confidence dynamic filtering threshold. Samples below the confidence dynamic filtering threshold are marked as low confidence and filtered out, thereby ensuring the quality of the pseudo-labels. Batch samples are a collection of multimodal data units divided according to a preset scale and that have completed spatiotemporal alignment.

[0044] S2. Calculate the distance between the semantic embedding of the initial pseudo-label and the domain knowledge graph node, generate semantic correction weights to perform semantic correction on the initial pseudo-label, and output semantically consistent pseudo-labels.

[0045] Specifically, the following steps are included:

[0046] The initial pseudo-labels are encoded into low-dimensional semantic vectors through a pre-trained semantic model, and the embedding representations of the corresponding nodes in the domain knowledge graph are extracted to ensure that the labels and graph entities are represented in a unified semantic space.

[0047] It should be noted that the pre-training of the semantic model adopts a two-stage optimization strategy. Basic semantic representation learning is carried out based on a large-scale general corpus, and the deep Transformer architecture is trained through the masked language modeling task. The semantic model masters the semantic representation capabilities at the lexical and syntactic levels, and the contrastive learning goal enhances fine-grained semantic discrimination. Adaptive fine-tuning is performed using professional corpus in the target domain, and the entity-relationship classification task is constructed. The knowledge distillation technology integrates the structural information of the domain knowledge graph into the semantic model representation. The training process adopts a 12-24-layer Transformer network, the hierarchical normalization strategy ensures training stability, and mixed precision training accelerates convergence. The semantic model introduces orthogonal constraints to avoid dimensionality collapse, adversarial training enhances cross-domain generalization capabilities, and the prototype contrast loss and dynamic negative sampling strategy strengthen intra-class aggregation and semantic boundary clarity.

[0048] Knowledge graph node embedding representations are extracted using translation-based models (such as TransE / RotatE). Structured representations are obtained through triple fact and negative sampling training. For entities with textual descriptions, the text embeddings generated by the language model are fused with the graph structure embeddings and dynamically weighted using a gating mechanism. The training process incorporates an adaptive margin loss and curriculum learning strategy to hierarchically capture direct relationships and multi-hop semantics. The resulting embeddings are validated through tasks such as link prediction, preserving the graph topology and semantic relationships, providing a stable knowledge representation foundation for downstream tasks.

[0049] Calculate the cosine similarity between the pseudo-label semantic embedding and the knowledge graph node embedding, filter out matching nodes that are higher than the preset similarity threshold, and record label-node pairs with low similarity as semantic conflict candidate sets;

[0050] Among them, the preset similarity threshold refers to a dynamically optimized boundary value used to determine the semantic matching quality between pseudo-labels and knowledge graph nodes. Specifically, the similarity threshold is determined in the following way: During the verification phase, the similarity distribution of labeled entity pairs in the knowledge graph is statistically analyzed, and the critical value that can achieve the best balance between precision and recall is selected (usually the 85th percentile of the similarity distribution). In actual applications, a hierarchical strategy is adopted: a similarity threshold of 0.7-0.8 is set for core entities, and a similarity threshold of 0.6-0.7 is used for edge entities. The final similarity threshold must ensure that the semantic matching accuracy on the verification set is not less than 90%. Modification cases triggered by the similarity threshold will be recorded and used regularly to optimize the similarity threshold parameters.

[0051] Dynamically assign weights based on the similarity between tags and nodes. High-similarity tags are assigned higher weights, while low-similarity tags are assigned lower weights by introducing penalty factors. At the same time, the hierarchical relationship constraints of the knowledge graph are combined to correct the tag semantics.

[0052] Specifically, in the weight allocation process, "high similarity labels" are defined as label-node pairs whose cosine similarity values ​​are greater than the preset similarity threshold (such as 0.8), indicating that their semantics are highly consistent with the knowledge graph nodes; "low similarity labels" refer to label-node pairs whose cosine similarity values ​​are lower than the threshold (such as 0.6), indicating that there is a significant semantic deviation.

[0053] It should be noted that weight assignment is implemented using a piecewise function: for high-similarity tags, the weight increases linearly, with a maximum value of 1; for low-similarity tags, the weight is reduced by a penalty factor (e.g., 0.5), with a minimum value of 0.2. At the same time, the weights are modified based on the hierarchical relationships of the knowledge graph (e.g., is-a or part-of): if there is a direct hierarchical relationship between the tag and the node, the weight is increased by an additional 0.1; if there is an indirect hierarchical relationship, the weight is increased by 0.05, ensuring that the semantic correction process fully utilizes the structured information of the graph.

[0054] The probability distribution of the initial pseudo-labels is re-weighted to increase the probability of high-weight labels and suppress low-weight labels. For the semantic conflict candidate set, the semantic definition of the conflicting labels is corrected through entity relationship path reasoning of the knowledge graph;

[0055] Specifically, when reweighting the probability distribution of the initial pseudo-labels, the probability value of each label should be multiplied by its corresponding weight. For high-weight labels (weight > 0.8), an exponential amplification strategy is used to increase the probability value to 1.2 times the original value; for low-weight labels (weight < 0.4), a linear suppression strategy is used to reduce the probability value to 0.8 times the original value.

[0056] Furthermore, for the semantic conflict candidate set, corrections are made through entity relationship path reasoning of the knowledge graph: the k-hop neighbor nodes (k=2) of the conflicting label in the knowledge graph are retrieved, their semantic similarity with the label is calculated, and the node with the highest similarity is selected as the correction target; then the correction signal is backpropagated along the graph path, the semantic definition of the label is updated to make it consistent with the semantics of the correction target node, and the weight distribution of related entities is adjusted at the same time.

[0057] The corrected label probability distribution is integrated, high-quality labels are retained through dynamic statistical threshold filtering, and a semantically consistent pseudo-label set that is strictly aligned with the knowledge graph entity relationships is output.

[0058] Among them, when fusing the corrected label probability distribution, a weighted average strategy is adopted: for each label, its original probability value and the corrected probability value are weighted and summed in a ratio of 0.6:0.4 to ensure a smooth transition in the correction process.

[0059] It should be noted that the dynamic statistical threshold is set based on the standard deviation of the label probability distribution: specifically, the mean and standard deviation of all label probability values ​​are calculated, and the dynamic statistical threshold is initially set to the mean plus 1.5 times the standard deviation. If the recall rate is lower than 90%, the dynamic statistical threshold is lowered by 0.1 times the standard deviation; if the recall rate is higher than 95%, the dynamic statistical threshold is raised by 0.1 times the standard deviation. Ultimately, this dynamic statistical threshold is used to filter out low-probability labels, retaining labels with probability values ​​above the threshold, and outputting a semantically consistent pseudo-label set that is strictly aligned with the knowledge graph entity relationships, ensuring that each label has a clear semantic correspondence with at least one knowledge graph node.

[0060] S3. Generate counterfactual samples by forcibly cutting off non-causal paths in the cross-modal causal graph, and compare the pseudo-label differences between the original samples and the counterfactual samples to generate cross-modal debiased pseudo-labels.

[0061] Specifically, the following steps are included:

[0062] Based on a directed acyclic causal graph, the d-separation criterion is used to identify all non-causal paths that meet the backdoor criterion (such as the "image background color → text emotion label" path), and the paths containing confounding variables (such as lighting conditions and cross-modal co-occurrence noise) in the non-causal paths are screened; among them, the paths may be true causal or false non-causal paths, and need to be further verified by the strength of the causal effect.

[0063] Specifically, the path to be intervened is selected based on the causal effect strength formula, which is expressed as:

[0064] ,

[0065] Where, Y represents the original sample pseudo label (such as classification probability distribution or regression value), Indicates the path P right Y The causal effect strength value is used to quantify the effect of the path (whether causal or non-causal) on Y The net impact, X represents the confounding variable being intervened, located on the non-causal path, and Respectively express X The two different intervention values ​​applied must be comparable. Express X Apply intervention (force to value )hour, Y The expected value of Indicates the expected value, To show causal intervention (rather than conditional probability), that is, to force the cutoff X All incoming edges of (eliminating the influence of confounding), Express X Apply another intervention value hour Y expected value;

[0066] It should be noted that the causal path P Refers to the indirect causal path connecting two modalities in a cross-modal causal graph (e.g., “image background → text label”).

[0067] reserve Generate an intervention strategy table, define uniform sampling intervention for discrete variables (such as replacing the image background with a random color), and perform mean intervention for continuous variables (such as fixing the audio spectrum energy to the training set mean);

[0068] Furthermore, by cross-modal decoupling and intervention, counterfactual samples are generated to eliminate the influence of non-causal paths. Specifically, a cross-modal variational autoencoder (CM-VAE) is used to decouple multimodal features into causal factors. and irrelevant factors , reconstructing counterfactual characteristics through structural equation modeling , expressed as:

[0069] ,

[0070] Where Decoder represents a cross-modal decoder, which is responsible for reconstructing latent variables (causal factors and non-causal factors) into multimodal features. Indicates causal interference equation, used to cut off All input dependencies of (eliminating contamination), Represents non-causal factors Intervene and force it to take a specific value ;

[0071] Among them, the specific value The assignment methods include: continuous variables: set to the mean of the training set or random sampling (such as ; Discrete variables: Forced to a certain category (such as setting the background color to a solid color).

[0072] The best approach uses a cross-modal variational autoencoder (CM-VAE) to decompose data features into causal and non-causal components. Adversarial training ensures that interventions only alter the non-causal components, leaving the true causal relationships intact. This allows for precise elimination of spurious correlations, such as modifying an image's background while preserving the underlying content. It also maintains semantic consistency across modalities and allows for intuitive identification of the features that have been manipulated, resulting in greater accuracy and reliability. This approach is particularly well-suited for processing complex multimodal data.

[0073] It should be noted that the causal interference model , need to pass the adversarial loss function Constrained Causal Factors Consistent with the original distribution, the adversarial loss function is expressed as:

[0074] ;

[0075] ;

[0076] Where, D The Discriminator is a two-class neural network used to distinguish between original causal factors and counterfactual causal factors. represents the causal factor of the counterfactual sample, Represents the causal factor encoder (Causal Factor Encoder);

[0077] Furthermore, the pseudo labels of the original samples Y and counterfactual sample pseudo labels , using Jensen-Shannon divergence to quantify the difference value , expressed as:

[0078] ,

[0079] Where JSD represents Jensen-Shannon divergence, represents the Kullback-Leibler divergence, Represents the middle distribution (mean distribution) of the two distributions, used for symmetric JSD calculation, Represents the normalization coefficient, ensuring that the JSD value is within the scope;

[0080] According to the difference value Dynamically allocate debiased weights, and the weight function is defined as:

[0081] ;

[0082] Where, w is the debiasing weight, which indicates the strength of label correction. , is the attenuation coefficient (default value is 5), exp represents the exponential function, which is used to ensure that the weight increases smoothly as the JSD increases. Indicates inverted output, making JSD larger w The closer it is to 1 (perfect correction);

[0083] The original pseudo-label and the counterfactual difference are fused to generate the debiased pseudo-label and then normalized by probability, which is expressed as:

[0084] ,

[0085] ;

[0086] Where, is the pseudo label after debiasing, represents the difference vector between the original and counterfactual labels, reflecting the interference direction of the non-causal path, Represents the weighted correction term of the difference vector, weight w Dynamically determined by JSD, Indicates the sum of all category scores to achieve normalization. Represents the Softmax function.

[0087] S4. Build a dynamic optimization closed loop to monitor in real time the conflict frequency between the gradient direction on the cross-modal debiased pseudo-labels and the domain knowledge graph, and dynamically adjust the cross-modal causal graph structure, semantic correction weights, and intervention intensity during the counterfactual sample generation process.

[0088] Specifically, the following steps are included:

[0089] Calculate the difference between the gradient direction of the cross-modal debiased pseudo-label and the semantic direction of the knowledge graph node.

[0090] Specifically, let the semantic direction of the entity node in the domain knowledge graph be the embedding vector , the pseudo-label gradient direction is (Generated by the back-propagation process of the cross-modal joint training framework, reflecting the optimization direction of the current pseudo-label);

[0091] Conflict frequency C is the ratio of the pseudo-label gradient direction to the semantic direction of the knowledge graph, expressed as:

[0092] ;

[0093] Where, N Represents the total number of entity nodes involved in the calculation in the domain knowledge graph, i Represents the entity node index in the domain knowledge graph, j represents the sample index of the pseudo label in the current batch, M represents the number of pseudo labels to be evaluated in the current batch, Indicates the calculation of pseudo-label gradient and knowledge graph nodes The cosine similarity of is the conflict threshold, which is the critical value used to determine whether there is a conflict;

[0094] It should be noted that the pseudo-label gradient Semantic direction projected to knowledge graph nodes , if the projection value is lower than , it is determined to be a conflict; the initial Set to 0.5, if the conflict frequency of 3 consecutive batches C Rise, then fall (step size 0.05); otherwise increase .

[0095] The conflict threshold The acquisition is achieved through multi-stage dynamic optimization: based on the validation set pre-calibration, the gradient cosine similarity distribution between the knowledge graph nodes and the real labels is statistically analyzed, and the 85th percentile value is taken as the initial threshold (typical range 0.5-0.7); the conflict frequency is monitored in real time during the operation phase. C The sliding window mean (window size 10 batches) is calculated when 3 consecutive batches are C When rising, decrease by 0.05 steps To relax the judgment criteria, otherwise increase the threshold and strengthen the alignment requirements, the whole process is constrained Prevent extreme adjustments; after each adjustment, the validation set must be reviewed. If the accuracy drops by more than 2%, it will be rolled back to the previous effective threshold. The final output The standard deviation of the conflict frequency C in the last 20 batches must be less than 0.1 to ensure the conflict threshold stability.

[0096] If the conflict frequency corresponding to a causal path (such as "image → text") is continuously higher than the mean, delete the causal path and recalculate the causal effect (i.e., recalculate the CausalEffect formula);

[0097] For frequently conflicting pseudo-labels, retrieve unconnected entity relationships (such as "drug-disease") in the domain knowledge graph. If the path causal effect strength , then add a new edge to the causal graph;

[0098] For tags with high conflict frequency, reduce their semantic correction weight , forcing the model to rely more on data features rather than knowledge graphs, expressed as:

[0099] ;

[0100] Where, represents the new weight after adjusting the conflict frequency, Indicates the t The semantic correction weight at the iteration, t represents the number of iteration steps, represents the hyperparameter that controls the speed at which the weight decreases, , C represents the conflict indicator, , is the attenuation factor, the value range is ;

[0101] If the conflict frequency decreases continuously, then Recover linearly (e.g. +0.05 / batch)

[0102] The initial value is set to 1.0 (full intervention), and the intervention intensity coefficient is dynamically adjusted according to the conflict frequency. The expression is:

[0103] ,

[0104] Where, is the intervention intensity coefficient, which is used to control the intervention intensity when generating counterfactual samples;

[0105] It should be noted that in CM-VAE training, if C Too high, it increases the causal factor Irrelevant factors The adversarial loss weight forces a more thorough decoupling;

[0106] The pseudo-label error rate is preset (e.g., the classification error rate is <5%), and the error rate is calculated for each batch. If the error rate meets the standard for 10 consecutive batches, the process is terminated. If the error rate fluctuation (standard deviation) is <1% for 5 consecutive batches, the process is forced to terminate and the current optimal parameter set is output.

[0107] It should be noted that the optimal parameters output at the termination of the dynamic optimization closed loop include four core components: first, the optimized cross-modal causal graph structure, including the causal edges that are ultimately retained and their strength values, as well as the deleted obfuscated paths and their corresponding conflict frequencies; second, the weight parameters for dynamically adjusting convergence, covering semantic correction weights, adversarial loss coefficients, and intervention strength; third, the key parameters of the feature decoupling model, including encoder-decoder weights, discriminator parameters, and the distribution characteristics of causal and non-causal factors; and finally, the output verification metrics, including pseudo-label error rate, stable conflict frequency, and alignment scores for each modality. Together, these parameters form the basic configuration for generating standardized datasets.

[0108] The best approach is to establish an adaptive conflict threshold mechanism through real-time comparison of pseudo-label gradient directions with knowledge graph semantic vectors, enabling intelligent correction of multimodal data. This automatically identifies high-frequency conflict paths and performs causal graph reconstruction, dynamically balancing the weight of semantic corrections with the intensity of counterfactual interventions. Furthermore, the optimization process continuously refines initial causal hypotheses, precisely aligning data features with domain knowledge. The output is a core parameter set encompassing an optimized causal structure, stable parameter configurations, and verification metrics, resulting in an adaptive and interpretable multimodal annotation solution.

[0109] S5. Combine cross-modal debiased pseudo labels with semantically consistent pseudo labels to generate a standardized pseudo-annotated dataset with multimodal alignment, clear entity relationships, and semantic consistency.

[0110] Specifically, the following steps are included:

[0111] Cross-modal debiased pseudo labels (causality) and semantically consistent pseudo labels (knowledge constraints) are fused with a weight of 0.6:0.4, and conflicting labels are forced to be unified through synonym relationships in the knowledge graph;

[0112] Verify the existence of entity relationship paths and remove labels that are not supported by the knowledge graph;

[0113] A pre-trained semantic model is used to align the multimodal embedding space (cosine similarity < 0.2), and misaligned samples trigger counterfactual intervention regeneration;

[0114] Eliminate invalid labels based on knowledge graph node mapping, and eliminate logical conflict labels based on causal graph pruning status, completing double filtering;

[0115] Output standardized datasets in the formats of COCO (images), BIO (text), and TIMIT (audio), with additional knowledge graph entity IDs and causal path metadata.

[0116] This embodiment also provides a computer device, which is suitable for the method of constructing a high-quality pseudo-annotated dataset based on complex tasks, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the method of constructing a high-quality pseudo-annotated dataset based on complex tasks proposed in the above embodiment.

[0117] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0118] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a high-quality pseudo-annotated dataset based on a complex task as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0119] In summary, the present invention improves the quality of pseudo-annotated data through the construction of cross-modal causal graphs and the fusion mechanism of dynamic knowledge graphs. The detection of confounding variables based on the backdoor criterion is combined with the adversarial training strategy to effectively suppress the interference of cross-modal co-occurrence noise on the causal path and enhance the robustness of causal associations between multimodal features. The dynamic weight correction mechanism embedded in the knowledge graph relies on semantic space projection and hierarchical relationship constraints to achieve precise control of label semantic drift and improve the semantic consistency of domain entity alignment. The counterfactual intervention framework drives the debiasing of non-causal paths through probability distribution differences, eliminating the influence of false correlations in cross-modal interactions. The dynamic optimization closed loop realizes online adaptive adjustment of the causal graph structure based on gradient conflict monitoring, ensuring the stability and reliability of the pseudo-label generation process in complex reasoning scenarios.

[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for constructing a high-quality pseudo-annotated dataset based on complex tasks, characterized by: include, Build a cross-modal causal graph based on multimodal raw data, load the domain knowledge graph, identify inter-modal confounding variables, and generate initial pseudo-labels; The cross-modal causal graph construction includes extracting causal relationships using a PC algorithm combined with a conditional independence test, and generating a directed acyclic causal graph with a set of labeled confounding variables after optimization using a causal structural equation model and adversarial training; Calculate the distance between the semantic embedding of the initial pseudo-label and the domain knowledge graph node, generate semantic correction weights to perform semantic correction on the initial pseudo-label, and output semantically consistent pseudo-labels; Counterfactual samples are generated by forcibly cutting off non-causal paths in the cross-modal causal graph, and the pseudo-label differences between the original samples and the counterfactual samples are compared to generate cross-modal debiased pseudo-labels. Build a dynamic optimization closed loop to monitor the conflict frequency between the gradient direction of cross-modal debiased pseudo-labels and the domain knowledge graph in real time, and dynamically adjust the cross-modal causal graph structure, semantic correction weights, and intervention intensity during counterfactual sample generation. Combining cross-modal debiased pseudo labels with semantically consistent pseudo labels, we generate a standardized pseudo-annotated dataset with multimodal alignment, clear entity relationships, and semantic consistency.

2. The method for constructing a high-quality pseudo-annotated dataset based on a complex task according to claim 1, characterized in that: Generating the initial pseudo-labels includes weighted fusion of spatiotemporally aligned multimodal features through a cross-modal attention network, using a dynamic confidence filtering threshold of entity relationships in the domain knowledge graph as a priori constraints, and performing backdoor path correction on modal features with confounding variables in combination with a conditional independence test; The confidence dynamic filtering threshold is obtained by dynamically calculating the confidence scores of the pseudo-labels of the samples in the current iteration batch.

3. The method for constructing a high-quality pseudo-annotated dataset based on a complex task according to claim 2, wherein: The semantic correction refers to using the domain knowledge graph node embedding method to map the initial pseudo-labels and the domain knowledge graph entity nodes to a unified semantic space, calculating the cosine similarity between the two, and dynamically adjusting the semantic correction weight based on the piecewise function and hierarchical relationship constraints.

4. The method for constructing a high-quality pseudo-annotated dataset based on a complex task according to claim 1, wherein: Generating counterfactual samples includes the following steps: Traverse the non-causal paths that meet the d-separation criterion in the cross-modal causal graph and select the paths to be intervened based on the causal effect strength formula; Through cross-modal variational autoencoders, we perform intervention operations on non-causal paths, cut off the dependencies of confounding variables, and reconstruct the counterfactual feature distribution; Jensen-Shannon divergence is used to quantify the difference between the pseudo labels of the original sample and the counterfactual sample, and a cross-modal bias weight based on an exponential decay function is generated to perform probability normalization debiasing on the pseudo labels.

5. The method for constructing a high-quality pseudo-annotated dataset based on a complex task according to claim 4, characterized in that: The construction of the dynamic optimization closed loop includes the following steps: Extract the gradient direction features of the cross-modal debiased pseudo-labels and calculate the conflict frequency of the cosine similarity between the gradient direction features and the domain knowledge graph node embeddings; Based on the dynamic adjustment strategy of the conflict threshold, high-frequency conflict paths in the cross-modal causal graph are deleted, and causal edges verified by the knowledge graph relationship path are added; Based on the sliding window statistics of cosine similarity conflict frequency, exponential decay is applied to the semantic modification weight, and the intervention intensity coefficient in the adversarial loss function of the cross-modal variational autoencoder is adaptively adjusted.

6. The method for constructing a high-quality pseudo-annotated dataset based on a complex task according to claim 1, wherein: The standardized pseudo-annotation dataset includes probabilistic fusion of cross-modal debiased pseudo labels and semantically consistent pseudo labels through a weighted averaging strategy, entity alignment verification based on domain knowledge graph relationship path traversal, and label screening using a confidence dynamic filtering threshold.

7. The method for constructing a high-quality pseudo-annotated dataset based on a complex task according to claim 2, wherein: The sample pseudo-label confidence of the current iteration batch refers to the probability score generated by the initial pseudo-label after semantic alignment of the knowledge graph and dynamic weight correction; Each sample in the batch is a set of multimodal data units that have completed spatiotemporal alignment processing.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for constructing a high-quality pseudo-annotated dataset based on a complex task according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a high-quality pseudo-annotated dataset based on a complex task according to any one of claims 1 to 7 are implemented.

Images