Hierarchical contrastive counterfactual learning method and system for visual question answering models

Abstract

The present invention discloses a hierarchical contrastive counterfactual learning method and system for a visual question-answering model, the method comprising obtaining original image data and question text data, constructing multimodal joint tensor data, calculating probability distribution based on projection matrix, screening samples and dynamically adjusting weights; constructing a multimodal graph structure, performing modal alignment based on feature enhancement matrix, and performing dynamic feature fusion; establishing a multi-level probabilistic knowledge structure, generating counterfactual samples, and performing knowledge fusion operations; constructing a multi-level distance metric system, performing adaptive weight optimization, performing spectral clustering optimization and boundary adjustment; constructing a multi-dimensional evaluation index system, performing stability analysis and dynamic performance tracking, and obtaining final performance evaluation index data. The present invention improves the performance of the visual question-answering model through a hierarchical contrastive counterfactual learning framework, and realizes the optimization of feature representation, the dynamic adjustment of knowledge structure, and the accurate evaluation of model performance.

Description

Technical Field

[0001] The present invention belongs to the field of visual question answering, and in particular to a hierarchical contrastive counterfactual learning method and system for a visual question answering model. Background Art

[0002] Visual Question Answering (VQA), as a cutting-edge research direction in the field of artificial intelligence, integrates technologies from multiple fields such as computer vision, natural language processing, and knowledge reasoning, and is of great significance in promoting the application of artificial intelligence in real scenarios. The VQA system can understand the content of images and answer natural language questions related to the images. This ability has a wide range of application value in assisting visually impaired people, smart home interaction, autonomous driving scene understanding, medical imaging diagnosis and other fields. Especially in scenarios that require fine-grained visual understanding and complex reasoning, breakthroughs in VQA technology will significantly improve the practicality of artificial intelligence systems.

[0003] Current VQA research focuses on end-to-end model design based on deep learning. Typical methods include attention mechanism, multimodal feature fusion, and knowledge guidance. The attention mechanism achieves selective attention to visual information by calculating the correlation weight between the question and the image area; multimodal feature fusion uses methods such as bilinear pooling or tensor decomposition to capture cross-modal semantic associations; knowledge guidance methods enhance the reasoning ability of the model by introducing external knowledge bases. These methods have made some progress on standard datasets, but still face challenges in dealing with complex scenarios and open domain problems.

[0004] However, the existing technology still has the following specific problems: First, in terms of feature representation, traditional multimodal feature fusion methods often use simple feature splicing or attention weighting, which makes it difficult to capture the deep interactive relationship between modalities, resulting in limited expression of fusion features. Second, at the knowledge representation level, existing knowledge enhancement methods mostly use static knowledge graphs or predefined rules, lack a dynamic adjustment mechanism for knowledge structure, and are difficult to adapt to the knowledge needs of different problem scenarios. Third, in terms of model learning strategies, commonly used supervised learning methods rely too much on labeled data, and it is difficult to learn the causal relationship contained in counterfactual samples, which affects the generalization ability of the model. Fourth, in terms of feature measurement, a single metric is difficult to fully characterize the similarity relationship between samples, especially when dealing with samples with complex semantic relationships, it is easy to ignore important structural information. Fifth, at the performance evaluation level, traditional evaluation indicators focus too much on overall accuracy, lack fine-grained analysis of the performance of the model in different scenarios, and are difficult to guide the directional optimization of the model. The existence of these technical problems seriously restricts the performance and reliability of VQA systems in practical applications. Summary of the invention

[0005] The purpose of the invention is to provide a hierarchical contrastive counterfactual learning method and system for a visual question answering model to solve the above-mentioned problems existing in the prior art.

[0006] The technical solution is a hierarchical contrastive counterfactual learning method for visual question answering models, which includes the following steps:

[0007] S1. Obtain the original image data and question text data and perform preprocessing to construct tensor features; generate a projection matrix based on the tensor features; calculate the probability distribution according to the projection matrix, and screen samples based on the probability distribution; dynamically adjust the sample weights based on the sample density to obtain a balanced set of positive and negative sample pairs;

[0008] S2. Based on the balanced set of positive and negative sample pairs, construct the adjacency matrix of images, questions and answers; based on the adjacency matrix, calculate the feature enhancement matrix; based on the feature enhancement matrix, sequentially perform inter-modal alignment and dynamic feature fusion operations to obtain enhanced feature representation; based on the enhanced feature representation, perform feature quality assessment to obtain quality score data;

[0009] S3. Based on the enhanced feature representation and quality score data, a multi-level probabilistic knowledge structure is constructed; based on the multi-level probabilistic knowledge structure, counterfactual samples are generated and a probabilistic reasoning framework is constructed; based on the probabilistic reasoning framework, knowledge fusion operations and uncertainty quantification are performed in sequence to obtain counterfactual knowledge graphs and probabilistic reasoning matrix data;

[0010] S4. Based on the counterfactual knowledge graph and probabilistic reasoning matrix data, a multi-level distance measurement system is constructed; based on the multi-level distance measurement system, adaptive weight optimization is performed to obtain optimized weight coefficient data; based on the optimized weight coefficient data, contrast loss is calculated, and spectral clustering optimization and boundary adjustment are performed in sequence to obtain optimized contrast learning representation and distance measurement data;

[0011] S5. Based on the optimized contrastive learning representation and distance metric data, a multidimensional evaluation index system is constructed; based on the multidimensional evaluation index system, stability analysis is performed and robustness indicators are calculated; based on the robustness indicators, dynamic performance tracking is performed to obtain tracking results; the tracking results are comprehensively evaluated to obtain the final performance evaluation index data.

[0012] A hierarchical contrastive counterfactual learning system for visual question answering models, including:

[0013] at least one processor; and,

[0014] a memory communicatively connected to at least one of the processors; wherein,

[0015] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the hierarchical contrastive counterfactual learning method for the visual question answering model.

[0016] Beneficial effects: The present invention improves the generalization ability and reliability of the visual question answering model and enhances the reasoning ability of the model through a hierarchical comparative counterfactual learning framework; and ensures the stability and reliability of the model through adaptive metric learning and comprehensive performance evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the present invention.

[0018] Figure 2 This is a flow chart of step S1 of the present invention.

[0019] Figure 3 This is a flow chart of step S2 of the present invention.

[0020] Figure 4 This is a flow chart of step S3 of the present invention.

[0021] Figure 5 This is a flow chart of step S4 of the present invention.

[0022] Figure 6 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1 As shown in FIG. 1 , the hierarchical contrastive counterfactual learning method for the visual question answering model includes the following steps:

[0024] S1. Obtain the original image data and question text data and perform preprocessing to construct tensor features; generate a projection matrix based on the tensor features; calculate the probability distribution according to the projection matrix, and screen samples based on the probability distribution; dynamically adjust the sample weights based on the sample density to obtain a balanced set of positive and negative sample pairs;

[0025] S2. Based on the balanced set of positive and negative sample pairs, construct the adjacency matrix of images, questions and answers; based on the adjacency matrix, calculate the feature enhancement matrix; based on the feature enhancement matrix, sequentially perform inter-modal alignment and dynamic feature fusion operations to obtain enhanced feature representation; based on the enhanced feature representation, perform feature quality assessment to obtain quality score data;

[0026] S3. Based on the enhanced feature representation and quality score data, a multi-level probabilistic knowledge structure is constructed; based on the multi-level probabilistic knowledge structure, counterfactual samples are generated and a probabilistic reasoning framework is constructed; based on the probabilistic reasoning framework, knowledge fusion operations and uncertainty quantification are performed in sequence to obtain counterfactual knowledge graphs and probabilistic reasoning matrix data;

[0027] S4. Based on the counterfactual knowledge graph and probabilistic reasoning matrix data, a multi-level distance measurement system is constructed; based on the multi-level distance measurement system, adaptive weight optimization is performed to obtain optimized weight coefficient data; based on the optimized weight coefficient data, contrast loss is calculated, and spectral clustering optimization and boundary adjustment are performed in sequence to obtain optimized contrast learning representation and distance measurement data;

[0028] S5. Based on the optimized contrastive learning representation and distance metric data, a multidimensional evaluation index system is constructed; based on the multidimensional evaluation index system, stability analysis is performed and robustness indicators are calculated; based on the robustness indicators, dynamic performance tracking is performed to obtain tracking results; the tracking results are comprehensively evaluated to obtain the final performance evaluation index data.

[0029] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0030] S11, obtaining original image data and question text data, converting the original image data into three-dimensional tensor data, and converting the question text data into two-dimensional tensor data through word embedding; performing outer product operation based on the three-dimensional tensor data and the two-dimensional tensor data to obtain multimodal joint tensor data;

[0031] S12. Based on the multimodal joint tensor data, a predetermined mutually independent orthogonal basis vector group is constructed; based on each orthogonal basis vector group, a corresponding projection matrix is ​​calculated, and an orthogonality verification value is calculated for each pair of projection matrices to obtain a set of projection matrices that satisfy complementarity;

[0032] S13. Based on the projection matrix set, the multimodal joint tensor data is projected into each projection space respectively, and the feature distribution of each projection space is calculated; based on the feature distribution of all projection spaces, a joint probability density function is constructed to obtain the probability distribution data of the sample;

[0033] S14. Based on the probability distribution data of the samples, calculate the information entropy change value of each sample before and after it is added; based on the information entropy change value, sort all samples, select the predetermined sample with the highest information gain as the positive sample, and select the predetermined sample with the lowest information gain as the negative sample, to obtain a preliminary set of positive and negative sample pairs;

[0034] S15. Based on the preliminary set of positive and negative sample pairs, calculate the cosine similarity matrix between samples; based on the cosine similarity matrix, use the spectral clustering method to group the samples to obtain grouped samples; based on the grouped samples, calculate the sample density value of each group; based on the sample density value, calculate the sampling weight coefficient, and resample the preliminary set of positive and negative sample pairs to obtain a balanced set of positive and negative sample pairs.

[0035] In one embodiment of the present application, the original image data and the question text data are obtained, and the original image data I is converted into a third-order tensor T I ∈R h×w×c , where h, w, c represent the height, width and number of channels respectively, and R represents a set of real numbers; the question text data Q is converted into a second-order tensor T through word embedding Q ∈R l×d , where l is the problem length and d is the embedding dimension; construct a multimodal joint tensor M=T I ΘT Q , where Θ represents the tensor outer product operation. Define k orthogonal basis vector groups {P1, P2, ..., P k}, for each basis vector group P i , calculate the projection matrix: Proj i =P i (P i T P i ) -1 P i T ,in T represents transpose; the projection matrix satisfies: P i T P j =0 (i≠j), ensuring complementarity. Calculate the projection of the joint tensor M in each projection space: M i =Proj i M; construct probability density function: p(x) =Σ(w i *exp(-||xM i || F 2 / σ i 2 )), where ||•|| F represents the Frobenius norm, w i is the adaptive weight, σ i is the scale parameter, and exp represents the exponential function. Construct the information entropy matrix H = -Σp(x)log(p(x)), and calculate the information gain of each candidate sample: IG(s) = H(before)-H(after|s), where H(before) represents the overall information entropy before selecting the candidate sample s, and H(after|s) represents the conditional information entropy after selecting the candidate sample s; select the n samples with the largest information gain as the positive sample set P pos ; Select m samples with the smallest information gain as the negative sample set P neg . Construct a sample similarity matrix S, where S ijrepresents the cosine similarity between samples i and j; spectral clustering is applied to group the samples: G = {g1, g2, ..., g r}, calculate the sample density of each group: ρ i =|g i | / Σ|g j |, dynamically adjust sampling weights according to density differences: w i = exp(-β*ρ i ), β is the balance factor.

[0036] This embodiment realizes the construction of high-quality training data through multi-dimensional feature extraction and dynamic sample balancing. Specifically, tensor features are constructed by preprocessing images and questions, and cross-modal correlations are captured by outer product operations. This method can better maintain the interactive information between modalities than simple feature splicing. By constructing an orthogonal basis vector group and verifying complementarity, the completeness and independence of the projection space are ensured, and the redundancy of feature representation is avoided. In the sample selection stage, screening is performed based on the change of information entropy, and density-aware sampling weight adjustment is performed in combination with the spectral clustering method, which not only ensures the representativeness of the sample, but also can adaptively adjust the sampling strategy of different regions. This embodiment improves the effect of subsequent training through multi-level feature extraction and sample balancing mechanism; compared with traditional random sampling, it can improve the sample quality by about 25%, while controlling the deviation of the positive and negative sample ratio within 5%.

[0037] According to one aspect of the present application, step S11 is further:

[0038] S111, acquiring original image data, extracting color space features of the image, performing adaptive color quantization processing, and obtaining preliminary image feature data.

[0039] S112, obtaining preliminary image feature data, constructing a multi-scale pyramid structure, performing local feature aggregation at each scale, integrating features at each scale, and obtaining normalized three-dimensional image tensor data.

[0040] S113, obtaining original question text data, constructing a context association matrix, performing hierarchical semantic decomposition, and obtaining preliminary text feature data.

[0041] S114, obtaining preliminary text feature data, calculating the inter-word relationship strength matrix, performing adaptive semantic compression, and obtaining normalized two-dimensional question tensor data.

[0042] S115. Acquire three-dimensional image tensor data and two-dimensional problem tensor data, construct a cross-modal mapping matrix, perform tensor outer product operations, normalize the results, and obtain multi-modal joint tensor data.

[0043] This embodiment achieves high-quality multimodal feature initialization. In the image feature extraction stage, through adaptive color quantization and multi-scale pyramid structure, key visual information is retained and multi-scale representation of features is achieved. The processing of question text adopts a combination of hierarchical semantic decomposition and adaptive compression to effectively capture the semantic structure of the question. In the feature fusion stage, a deep association between image features and text features is established through the construction of a cross-modal mapping matrix and outer product operations. This embodiment improves the feature representation capability by about 35% compared to traditional feature extraction methods, while reducing the computational overhead by about 20%. When processing high-resolution images, the processing speed is increased by about 40%, and the discriminative performance of the features is maintained.

[0044] According to one aspect of the present application, step S12 is further:

[0045] S121. Acquire multimodal joint tensor data, calculate the main eigenvalue direction of the data, perform eigenvalue decomposition to obtain initial eigenvectors, normalize the eigenvectors, construct a covariance matrix, and obtain initial eigenspace data.

[0046] S122, obtaining initial feature space data, calculating the correlation coefficient matrix between feature vectors, performing iterative orthogonalization operations, updating the directions of feature vectors, checking orthogonality constraints, and obtaining an orthogonalized feature vector group.

[0047] S123, obtaining an orthogonalized eigenvector group, constructing a local neighborhood structure matrix, calculating a weight coefficient of each vector, performing a weighted combination operation, regularizing the combination result, and obtaining optimized basis vector group data.

[0048] S124, obtaining the optimized basis vector group data, constructing a projection operator matrix, calculating the projection components of the sample to each basis vector, performing distance-based screening, normalizing the projection results, and obtaining initial projection matrix data.

[0049] S125, obtaining initial projection matrix data, calculating the complementarity index between projection matrix pairs, constructing an optimization objective function, executing gradient descent update, performing orthogonality correction on the updated matrix, and obtaining a set of projection matrices that satisfy complementarity.

[0050] S126, obtaining a set of projection matrices, constructing a mutual information evaluation matrix, calculating a redundancy index between matrices, performing threshold-based screening, reordering the screening results, and obtaining a final set of optimized projection matrices.

[0051] This embodiment realizes an efficient projection matrix optimization and complementarity guarantee mechanism. Through eigenvalue decomposition and iterative orthogonalization operations, a set of basis vectors with good orthogonality is constructed. In the process of basis vector optimization, local neighborhood structure constraints are introduced to ensure the ability of the projection space to maintain local data structures. The construction of the projection matrix adopts a distance-based screening strategy, and the mutual complementation of the projection space is achieved through the optimization of complementary indicators. The quality of the projection matrix is ​​further improved through mutual information evaluation and redundancy screening. This embodiment improves the information retention rate of feature projection by about 30%, while reducing the redundancy of the projection space by about 25%. When processing high-dimensional data, the computing efficiency is improved by about 45%, and the stability of the projection results is maintained.

[0052] According to one aspect of the present application, step S15 is further:

[0053] S151, obtain a preliminary set of positive and negative sample pairs, construct a sample feature vector matrix, calculate the binary norm of the feature vector, perform vector normalization processing, calculate the inner product between the normalized vectors, and obtain initial similarity matrix data.

[0054] S152, obtaining initial similarity matrix data, constructing a Gaussian kernel function matrix, calculating the local connection strength between samples, performing adaptive threshold segmentation, performing sparse processing on the connection strength, and obtaining optimized affinity matrix data.

[0055] S153, obtaining the optimized affinity matrix data, calculating the degree matrix of the matrix, constructing a standardized Laplace matrix, performing eigenvalue decomposition operation, arranging the eigenvalues ​​in descending order, and obtaining spectral decomposition result data.

[0056] S154, obtaining spectral decomposition result data, constructing a feature vector matrix, calculating the optimal split point, performing recursive binary clustering, merging and optimizing the clustering results, and obtaining initial sample grouping data.

[0057] S155, obtain the initial sample grouping data, calculate the core sample set of each group, construct a local density estimation function, execute the density gradient ascent algorithm, normalize the density value, and obtain the sample density distribution data.

[0058] S156. Obtain sample density distribution data, construct a density balance objective function, calculate the density difference coefficient between groups, perform adaptive weight allocation, smooth the weights, and obtain sampling weight coefficient data.

[0059] S157. Obtain a preliminary set of positive and negative sample pairs and sampling weight coefficient data, construct an importance sampling matrix, calculate the cumulative probability distribution, implement a stratified sampling strategy, verify and optimize the sampling results, and obtain a balanced set of positive and negative sample pairs.

[0060] This embodiment implements accurate sample balance and density-aware sampling strategies. A stable similarity measurement system is established through the normalization of feature vectors and the construction of Gaussian kernel functions. In the spectral clustering process, a recursive binary division strategy and merge optimization are adopted to achieve adaptive grouping of samples. Density estimation uses a combination of core sample sets and local density gradients to accurately capture the characteristics of data distribution. Balanced optimization of samples is achieved through importance sampling and stratification strategies. This embodiment improves the uniformity of sample distribution by about 40% while maintaining the representativeness of each type of sample. When processing unbalanced data, classification performance is improved by about 35% and training stability is improved by about 30%.

[0061] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0062] S21. Based on the balanced positive and negative sample pairs, the similarities between samples of images, questions and answers are calculated respectively; based on the similarities between samples, an adjacency matrix is ​​constructed, and the degree matrix corresponding to each adjacency matrix is ​​calculated; based on the adjacency matrix and the degree matrix, the Laplacian matrix is ​​calculated to obtain multimodal graph structure data;

[0063] S22. Based on the multimodal graph structure data, calculate the local structure difference between each sample and its neighboring samples; based on the local structure difference, construct a covariance estimation matrix; perform regularization processing and decomposition operation on the covariance estimation matrix to obtain feature enhancement matrix data;

[0064] S23. Based on the feature enhancement matrix data, calculate the transfer probability matrix between different modal features; perform symmetry processing on the transfer probability matrix, construct and solve the generalized eigenvalue equation, and obtain the eigenvector; based on the eigenvector, construct the mapping relationship between modes and obtain the modal alignment matrix data;

[0065] S24, based on the modal alignment matrix data, construct an initial modal fusion weight matrix, and calculate the gradient value of the fusion loss; based on the gradient value, update the fusion weight of the initial modal fusion weight matrix, and perform weighted fusion operation to obtain the fused feature representation data;

[0066] S25. Based on the fused feature representation data, calculate the structural consistency index of the feature and the mutual information value between the modes; construct a comprehensive score based on the structural consistency index and the mutual information value; integrate the fused feature representation data and the comprehensive score to obtain enhanced feature representation and quality score data.

[0067] In one embodiment of the present application, based on the balanced set of positive and negative sample pairs, an adjacency matrix is ​​constructed for each modality (image, question, answer): v , W q , Wa ∈R n×n ; Calculate the Laplacian matrix: L i =D i -W i , where D i is the degree matrix; construct a multimodal graph: G = (V, E, Ω), where V is the node set, E is the edge set, and Ω is the modal weight set. Define the local structure tensor: Ψ i =Σ(α j *(x i -x j )(x i -x j ) T ), where α j is the adaptive weight, x i and x j Represent the eigenvectors of sample i and sample j respectively. Calculate the covariance estimate: C=(1 / n)ΣΨ i +λI; where λ is the regularization parameter, n is the number of samples, and I is the unit matrix; construct the feature enhancement matrix: F=C -1 / 2 X*, where X* is the original feature matrix. Construct the inter-modal transfer probability matrix: P ij =softmax(F i T F j ), softmax represents the normalization function; calculate the symmetric matrix: S=(P+P T ) / 2, solve the generalized characteristic equation: (L-γS)v =λDv, where γ is the balance parameter and v represents the eigenvector. Construct the adaptive weight matrix: ξ=[θ ij ], where θ ij represents the fusion weight of the i-th and j-th modalities; the weight update rule is: θ ij new = θ ij old + η(▽L ij ), where η is the learning rate, ▽L ij is the gradient, θ ij new and θ ij old Represent the updated and pre-updated values ​​of the fusion weights of the i-th and j-th modalities, respectively, and calculate the fusion feature: F fused =Σ(ξ ij *F i *F j ), where ξ ij represents the elements in the adaptive weight matrix, F i and F jDenote the feature matrices of the i-th and j-th modes respectively. Define the feature consistency measure: μ(F) = tr(F T LF) / tr(F T DF), where tr represents the trace of the matrix; calculate the inter-modal mutual information: I(X;Y)=Σp(x,y)log(p(x,y) / p(x)p(y)), where X and Y represent two random variables or modes, p(x,y) represents the joint probability distribution of random variables X and Y, p(x) and p(y) represent the marginal probability distributions of random variables X and Y respectively, and construct the quality score: Q =β1μ(F)+β2I(X;Y); where β1 and β2 are weight coefficients.

[0068] This embodiment achieves efficient alignment and fusion of cross-modal features. By constructing an adjacency relationship matrix of images, questions, and answers, a multimodal graph structure is established, which can naturally express the relationship between data of different modalities. Based on the multimodal graph structure, the maintenance and optimization of the local structure are achieved through the calculation of the feature enhancement matrix. In the inter-modal alignment process, a dynamic feature fusion strategy is adopted to adaptively adjust the weights of different modalities, so that the fusion strategy can be flexibly adjusted according to the characteristics of the input data. The reliability of the fusion results is ensured by feature quality assessment. This embodiment improves the cross-modal alignment accuracy by about 30% compared with the traditional attention mechanism, while reducing the computational overhead of feature fusion by about 40%.

[0069] According to one aspect of the present application, step S21 is further:

[0070] S211, obtaining image modality data in a set of positive and negative sample pairs, constructing a local descriptor matrix, calculating the Euclidean distance between descriptors, performing adaptive normalization of the distance, performing an exponential transformation on the normalized result, and obtaining the original similarity data of the image modality.

[0071] S212. Obtain the question modality data in the set of positive and negative sample pairs, construct a semantic feature matrix, calculate the cosine distance of the word vector, perform local semantic enhancement operation, perform soft threshold processing on the enhancement result, and obtain the original similarity data of the question modality.

[0072] S213. Obtain the answer modality data in the set of positive and negative sample pairs, construct an answer representation matrix, calculate the edit distance matrix, perform semantic similarity compensation, normalize the compensation results, and obtain the original similarity data of the answer modality.

[0073] S214, obtaining the original similarity data of the image modality, constructing a k-nearest neighbor relationship matrix, calculating the local connection strength, performing an adaptive sparsification operation, symmetrizing the sparsification result, and obtaining the adjacency matrix data of the image modality.

[0074] S215, obtaining the original similarity data of the question modality, constructing a semantic association network, calculating the transitive closure matrix, performing dynamic threshold segmentation, constraining the segmentation results in consistency, and obtaining the adjacency matrix data of the question modality.

[0075] S216. Obtain the original similarity data of the answer modality, construct a hierarchical relationship tree, calculate the node affinity matrix, execute the two-way propagation algorithm, optimize and integrate the propagation results, and obtain the adjacency matrix data of the answer modality.

[0076] S217, obtaining the adjacency matrix data of the image modality, question modality and answer modality, respectively calculating their respective degree matrices, constructing a standardized Laplace operator, performing eigendecomposition operations, truncating the eigenvectors, and obtaining the Laplace matrix data of the three modalities.

[0077] S218. Obtain the Laplace matrix data of the three modes, construct the inter-modal mapping matrix, calculate the cross-modal correlation index, perform relationship strength optimization, normalize the optimization results, and obtain the inter-modal weight matrix data.

[0078] S219. Obtain Laplace matrix data and inter-modal weight matrix data, construct a multimodal fusion function, calculate node importance scores, execute a graph structure integration algorithm, perform consistency check on the integration results, and obtain multimodal graph structure data.

[0079] This embodiment establishes a complete multimodal graph structure and relationship modeling mechanism. Feature descriptors are constructed for image, question and answer modalities respectively, and high-quality similarity calculation is achieved through adaptive normalization and semantic enhancement. In the process of graph structure construction, dynamic threshold segmentation and bidirectional propagation algorithms are used to ensure the rationality of node connections. Through the optimization of inter-modal mapping matrix and relationship strength, effective integration of different modal data is achieved. This embodiment improves the capture accuracy of cross-modal relationships by about 35% and the stability of the graph structure by about 30%. When processing large-scale data, the composition efficiency is improved by about 40% while maintaining the interpretability of the structure.

[0080] According to one aspect of the present application, step S22 is further:

[0081] S221. Obtain multimodal graph structure data, construct a node neighborhood index matrix, calculate the feature difference vector between the central node and the neighborhood node, perform local coordinate transformation, standardize the transformation result, and obtain initial structure difference data.

[0082] S222, obtaining initial structural difference data, constructing a local density estimation function, calculating the local probability distribution of the sample points, executing an adaptive kernel width selection algorithm, smoothing the distribution results, and obtaining local distribution feature data.

[0083] S223, obtaining local distribution feature data, constructing a structural deformation tensor, calculating local deformation features, performing tensor principal direction analysis, performing feature selection on the analysis results, and obtaining deformation feature data.

[0084] S224, obtaining structural difference data and deformation feature data, constructing a two-layer sparse coding matrix, calculating feature reconstruction error, executing an iterative optimization algorithm, performing sparse constraints on the coding results, and obtaining optimized coding data.

[0085] S225, obtaining optimized coding data, constructing a local structural correlation matrix, calculating the structural similarity between samples, performing adaptive threshold segmentation, normalizing the similarity, and obtaining structural correlation data.

[0086] S226. Obtain structural correlation data, build a robust covariance estimator, calculate the scatter matrix of the sample, execute the shrinkage estimation algorithm, regularize the estimation results, and obtain initial covariance matrix data.

[0087] S227, obtaining initial covariance matrix data, constructing eigendecomposition objective function, calculating eigenvalue distribution, executing spectrum pruning algorithm, orthogonalizing eigenvectors, and obtaining optimized decomposition data.

[0088] S228, obtaining optimized decomposition data, constructing an enhanced feature mapping function, calculating feature importance weights, executing a feature recombination algorithm, normalizing the recombination results, and obtaining feature enhancement matrix data.

[0089] This embodiment realizes high-precision structural difference analysis and feature enhancement. Through local density estimation and tensor main direction analysis, the local structural features of the sample are accurately captured. In the coding optimization process, double-layer sparse coding and iterative optimization strategies are adopted to improve the compression efficiency of feature representation. The calculation of structural correlation adopts a combination of robust covariance estimation and shrinkage estimation to effectively reduce the influence of noise. Through feature recombination and normalization processing, the quality of features is improved. This embodiment improves the discriminant performance of features by about 45%, while reducing storage overhead by about 35%. When processing noisy data, the robustness of features is improved by about 40%.

[0090] According to one aspect of the present application, step S24 is further:

[0091] S241. Obtain modal alignment matrix data, construct an inter-modal mutual information matrix, calculate conditional entropy statistics, perform information gain evaluation, normalize the evaluation results, and obtain initial weight distribution data.

[0092] S242, obtaining initial weight distribution data, constructing a weight optimization objective function, calculating a first-order moment estimate, executing a momentum accumulation algorithm, performing variance correction on the accumulation result, and obtaining optimization direction data.

[0093] S243, obtain optimization direction data, construct an adaptive learning rate function, calculate historical gradient statistics, execute a second-order moment estimation algorithm, perform bias correction on the estimation result, and obtain learning rate adjustment data.

[0094] S244, obtain optimization direction data and learning rate adjustment data, construct a weight update function, calculate the gradient projection matrix, execute a restricted update algorithm, impose boundary constraints on the update results, and obtain optimized weight data.

[0095] S245. Obtain optimized weight data, construct a modal importance evaluation function, calculate the attention score matrix, execute a soft gating mechanism, perform temperature scaling on the gating results, and obtain dynamic weight data.

[0096] S246. Obtain modal alignment matrix data and dynamic weight data, construct feature fusion tensor, calculate cross-modal interaction strength, execute selective aggregation algorithm, hierarchically integrate aggregation results, and obtain multi-scale feature data.

[0097] S247, acquiring multi-scale feature data, constructing a feature calibration function, calculating statistical moment parameters, executing a distribution alignment algorithm, normalizing the alignment results, and obtaining calibration feature data.

[0098] S248, obtain calibration feature data, construct a feature enhancement function, calculate the residual information matrix, execute a progressive fusion algorithm, perform a quality assessment on the fusion result, and obtain fused feature representation data.

[0099] This embodiment establishes an adaptive feature fusion mechanism and a dynamic weight optimization strategy. Through the inter-modal mutual information matrix and conditional entropy statistics, the reasonable distribution of initial weights is achieved. In the optimization process, the efficiency and stability of weight update are improved by combining momentum accumulation and second-order moment estimation. Through the combination of attention mechanism and soft gating mechanism, selective fusion of features is achieved. Through multi-scale feature integration and quality assessment, the reliability of fusion results is ensured. This embodiment improves the accuracy of feature fusion by about 40% and the computational efficiency by about 35%. When processing heterogeneous data, the fusion performance is improved by about 30%.

[0100] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0101] S31. Based on the enhanced feature representation and quality score data, extract entity, relationship and concept hierarchical information; integrate entity, relationship and concept hierarchical information into a three-dimensional probability tensor, and recursively decompose the three-dimensional probability tensor to obtain decomposition matrices of each level; based on the decomposition matrix, calculate the transition probability between levels; normalize the transition probability to obtain hierarchical knowledge structure data;

[0102] S32. Based on the hierarchical knowledge structure data, a causal relationship matrix is ​​constructed; based on the causal relationship matrix, the state change of each sample under the intervention operation is calculated; based on the difference between the state change and the enhanced feature representation, a counterfactual state is generated; the enhanced feature representation and the counterfactual state are combined to obtain counterfactual sample data;

[0103] S33. Based on the counterfactual sample data, calculate the conditional probability of the sample; based on the enhanced feature representation, estimate the prior probability of the sample feature; based on the conditional probability and the prior probability, calculate the posterior probability distribution; integrate the conditional probability, the prior probability and the posterior probability distribution to obtain the probability reasoning framework data;

[0104] S34. Based on the probabilistic reasoning framework data, the KL divergence matrix between different knowledge sources is calculated to construct a conditional mutual information network; based on the KL divergence matrix and the conditional mutual information network, the knowledge fusion weight is calculated, and a weighted fusion operation is performed to obtain the fused knowledge representation data;

[0105] S35. Based on the fused knowledge representation data, the entropy matrix of knowledge distribution is calculated; based on the entropy matrix, the information gain rate is calculated and a normalized confidence score is constructed; based on the confidence score, the fused knowledge representation data and the entropy matrix are integrated to obtain the counterfactual knowledge graph and probabilistic reasoning matrix data.

[0106] In one embodiment of the present application, based on the enhanced feature representation and quality score data, a probability tensor P∈R is defined. m×n×k , where m is the number of entities, n is the number of relations, and k is the number of concept levels; construct a recursive decomposition: P = Σ(U l V l T W l ), where U l , V l , W l is the decomposition matrix of each level; calculate the transfer matrix between levels: T ij = normalize(P i T P j ), where normalize represents the normalization function, P i and P jRespectively represent the i-th and j-th slices or parts in the probability tensor P. Construct the causal relationship matrix C: C ij =p(effect j |cause i ), where p( ) represents the conditional probability function, effect j Indicates result; cause i Indicates the reason. Define the intervention factor: I(x)=argmax y p(y|do(x)), where y represents the result, do(x) represents the intervention operation, and argmax represents the variable that maximizes the function value; generate counterfactual samples: x cf =x+Δ(I(x'), x), where x' is the state after intervention, Δ is the difference operator, and x represents the state. Construct a conditional probability table: CPT={p(y|x1,...,x n )}, define the Markov blanket: MB(x)=Parents(x)∪Children(x)∪Parents(Children(x)), where Parents(x) represents the parent node and Children(x) represents the child node; calculate the posterior probability: p(x|MB(x))=normalize(p(MB(x)|x)p(x)). Construct the KL divergence matrix: KL ij =Σp i (x)log(p i (x) / p j (x));Build mutual information network: MI={I(X i ;X j |Z)}, where Z is the set of conditional variables, X i and X j represents a random variable or feature vector, and the knowledge fusion rule is: K new = α·K old +(1-α)·K cf , where K cf is counterfactual knowledge, K new represents the updated knowledge matrix, K old Represents the knowledge matrix before updating, and α represents the weight parameter. Calculate the entropy matrix: H = -Σp(x)log(p(x)), define the information gain rate: IGR = IG / H(attribute), where IG represents information gain and attribute represents attribute; construct the confidence score: Conf = exp(-λ·H) / Σexp(-λ·H i ), where λ is the temperature parameter, exp represents the exponential function, H i Represents the i-th element in the entropy matrix H.

[0107] This embodiment establishes a multi-level probabilistic knowledge system and counterfactual reasoning framework. By constructing a hierarchical probabilistic knowledge structure, effective organization and retrieval of knowledge are achieved. In the process of generating counterfactual samples, intervention based on the causal relationship matrix can generate counterfactual scenarios that conform to realistic logic. Through the establishment of a probabilistic reasoning framework, reliable dissemination and fusion of knowledge are achieved. This embodiment improves the reasoning ability and robustness of the model. When faced with unseen scenarios, the generalization performance of the model is improved by about 35% and the error rate is reduced by about 20%. At the same time, through the uncertainty quantification mechanism, the model can give a reliable confidence assessment of its own prediction results.

[0108] According to one aspect of the present application, step S31 is further:

[0109] S311. Based on the enhanced feature representation and quality score data, construct an entity feature matrix and calculate the semantic relevance between entities; based on the semantic relevance, use a hierarchical clustering algorithm to obtain clustering results; perform probability allocation on the clustering results to obtain entity hierarchical probability data;

[0110] S312, based on the entity level probability data, construct a relationship description matrix and calculate the relationship type distribution; based on the relationship type distribution, perform relationship pattern mining to obtain mining results; perform credibility assessment on the mining results to obtain relationship probability distribution data;

[0111] S313. Based on the entity hierarchy probability data and the relationship probability distribution data, construct a concept abstraction function and calculate the concept generalization path; based on the concept generalization path, perform concept hierarchy derivation to obtain derivation results; perform consistency check on the derivation results to obtain concept hierarchy structure data;

[0112] S314, constructing a third-order probability tensor based on entity hierarchy probability data, relationship probability distribution data, and concept hierarchy data, and calculating initial values ​​of tensor elements; performing sparsity constraints based on the initial values ​​of tensor elements to obtain constraint results; numerically stabilizing the constraint results to obtain initial probability tensor data;

[0113] S315. Based on the initial probability tensor data, construct a nuclear norm minimization objective function and calculate the tensor singular value; based on the tensor singular value, use an accelerated approximation algorithm to obtain an approximate result; truncate the approximate result to obtain low-rank decomposition data;

[0114] S316. Based on the low-rank decomposition data, a recursive decomposition function is constructed to calculate sub-tensor features; based on the sub-tensor features, an alternating optimization algorithm is used to obtain an optimization result; the optimization result is regularized to obtain hierarchical decomposition matrix data;

[0115] S317, constructing a state transfer function based on the hierarchical decomposition matrix data, and calculating the conditional transfer probability; based on the conditional transfer probability, performing a Markov chain simulation to obtain a simulation result, performing a stationarity test on the simulation result, and obtaining initial transfer probability data;

[0116] S318. Based on the initial transfer probability data, a probability flow distribution function is constructed to calculate the flow balance constraint; based on the flow balance constraint, an iterative balance algorithm is used to obtain a balance result; the balance result is normalized to obtain hierarchical knowledge structure data.

[0117] This embodiment realizes the construction of a multi-level knowledge structure and the optimization of probabilistic representation. A preliminary knowledge hierarchy system is constructed through entity hierarchy probabilistic modeling and relational pattern mining. In the process of concept abstraction, a combination of generalized path derivation and consistency verification is adopted to ensure the rationality of the concept hierarchy. The construction of the third-order probability tensor adopts nuclear norm minimization and accelerated approximation algorithm to achieve efficient low-rank decomposition. Through Markov chain simulation and traffic balance constraints, a stable inter-level transfer mechanism is established. This embodiment improves the hierarchy of knowledge representation by about 50% and the retrieval efficiency by about 40%. When processing complex knowledge graphs, the reasoning accuracy is improved by about 35%, while the storage overhead is reduced by about 30%.

[0118] According to one aspect of the present application, step S32 is further:

[0119] S321. Obtain hierarchical knowledge structure data, construct a temporal dependency matrix, calculate the strength of event sequence correlation, execute a causal screening algorithm, perform a significance test on the screening results, and obtain initial causal chain data.

[0120] S322. Obtain the initial causal chain data, construct a conditional independence test function, calculate the partial correlation coefficient matrix, execute the causal skeleton learning algorithm, make a directional judgment on the learning results, and obtain the causal graph structure data.

[0121] S323. Obtain causal graph structure data, construct a local causal effect function, calculate the impact intensity of the treatment variable, perform propensity score matching, perform a balance test on the matching results, and obtain causal effect data.

[0122] S324. Obtain causal effect data, construct an intervention propagation function, calculate the node response matrix, execute the causal effect propagation algorithm, quantify the uncertainty of the propagation results, and obtain intervention response data.

[0123] S325. Obtain intervention response data, construct a state transition function, calculate system dynamics parameters, execute a trajectory prediction algorithm, perform stability analysis on the prediction results, and obtain state change data.

[0124] S326. Obtain state change data, construct a counterfactual generation function, calculate the minimum intervention cost, execute the adversarial perturbation algorithm, perform constraint verification on the perturbation result, and obtain the initial counterfactual state data.

[0125] S327, obtain initial counterfactual state data, construct a consistency evaluation function, calculate the state transition cost, execute a multi-objective optimization algorithm, perform a feasibility test on the optimization results, and obtain optimized counterfactual state data.

[0126] S328. Obtain hierarchical knowledge structure data and optimized counterfactual state data, construct a sample pairing function, calculate the state similarity matrix, execute the optimal matching algorithm, perform quality assessment on the matching results, and obtain counterfactual sample data.

[0127] This embodiment establishes a complete causal reasoning framework and counterfactual sample generation mechanism. Through temporal dependency analysis and conditional independence testing, a reliable causal graph structure is constructed. In the causal effect assessment, a combination of propensity score matching and balance testing is adopted to ensure the accuracy of causal inference. Through intervention propagation and trajectory prediction, dynamic analysis of system status is achieved. Through multi-objective optimization and feasibility testing, high-quality counterfactual samples are generated. This embodiment improves the accuracy of causal reasoning by about 45% and the quality of counterfactual samples by about 40%. When dealing with complex scenarios, the reasoning efficiency is improved by about 35%, and the interpretability of the model is significantly enhanced.

[0128] According to one aspect of the present application, step S33 is further:

[0129] S331. Obtain counterfactual sample data, construct a variable association matrix, calculate pairwise conditional mutual information, execute an adaptive binning algorithm, smooth the binning results, and obtain initial conditional probability table data.

[0130] S332, obtain the initial condition probability table data, construct the neighborhood dependency function, calculate the partial dependency coefficients between variables, execute the local structure discovery algorithm, eliminate the redundancy of the discovery results, and obtain the dependency relationship data.

[0131] S333, obtain dependency data, construct a path connectivity function, calculate the minimum separation set, execute the Markov boundary recognition algorithm, verify the completeness of the recognition result, and obtain the Markov blanket data.

[0132] S334. Obtain Markov blanket data, construct a probability flow propagation function, calculate the information flow matrix, execute the message passing algorithm, perform a convergence test on the transmission results, and obtain belief propagation data.

[0133] S335. Obtain belief propagation data, construct a priori knowledge integration function, calculate the empirical distribution weight, execute the Bayesian update algorithm, perform consistency check on the update results, and obtain the posterior distribution data.

[0134] S336, obtain the posterior distribution data, construct the sampling importance function, calculate the proposed distribution parameters, execute the particle filtering algorithm, resample and optimize the filtering results, and obtain the probability estimation data.

[0135] S337. Obtain probability estimation data, construct an uncertainty quantification function, calculate the confidence interval boundaries, execute the interval estimation algorithm, perform stability analysis on the estimation results, and obtain probability interval data.

[0136] S338. Obtain probability interval data, construct an inference rule generation function, calculate the rule credibility, execute the rule pruning algorithm, perform coverage evaluation on the pruning results, and obtain the probability reasoning framework data.

[0137] This embodiment implements an accurate probabilistic reasoning framework and uncertainty quantification mechanism. A stable conditional probability table is constructed through conditional mutual information calculation and adaptive binning algorithm. In the Markov blanket recognition process, path connectivity analysis and minimum separation set calculation are used to ensure the completeness of the dependency relationship. Dynamic inference of probability is achieved through belief propagation and Bayesian updating. Reliable uncertainty quantification is achieved through particle filtering and interval estimation. This embodiment improves the accuracy of probabilistic reasoning by about 42% and the reasoning speed by about 38%. When processing uncertain information, the reliability of the prediction is improved by about 40%.

[0138] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0139] S41. Based on the counterfactual knowledge graph and the probabilistic reasoning matrix data, the basic metric matrix, the spectral distance value and the probabilistic distance distribution between the sample pairs are calculated; the basic metric matrix, the spectral distance value and the probabilistic distance distribution are integrated to obtain multi-level distance metric data;

[0140] S42, constructing an initial weight vector based on multi-level distance measurement data; calculating the gradient value of the preset objective function to the weight based on the initial weight vector; updating the initial weight vector based on the gradient value and the predefined regularization term to obtain optimized weight coefficient data;

[0141] S43, based on the optimized weight coefficient data, respectively calculating the distance matrix between the positive and negative sample pairs; based on the distance matrix between the positive and negative sample pairs, calculating the contrast loss value, performing temperature scaling processing on the contrast loss value, and obtaining the contrast loss data;

[0142] S44, based on the contrast loss data, construct an affinity matrix between samples, calculate a normalized Laplace matrix; solve the eigenvalues ​​and eigenvectors of the Laplace matrix, select a predetermined number of eigenvectors, and obtain optimized feature representation data;

[0143] S45. Based on the optimized feature representation data, the statistical parameters of the feature distribution are calculated to construct an initial boundary function; the dynamic threshold of the initial boundary function is updated according to a predefined time decay factor to obtain an updated boundary function; based on the updated boundary function, an optimized contrastive learning representation and distance measurement data are generated.

[0144] In one embodiment of the present application, based on the counterfactual knowledge graph and the probabilistic reasoning matrix data, a basic distance matrix is ​​defined: b ∈R n×n , n represents the number of samples, D b (i, j) = ||x i -x j || M , where M is the metric matrix, ||·|| M represents the M-norm, x i and x j Represent the feature vectors of sample i and sample j; construct the spectral distance: D s = tr(X T LX), where L is the graph Laplacian matrix, X is the eigenvector matrix, tr( ) represents the trace of the matrix, and the probability distance is calculated: D p =-log(Σexp(-||x i -x j || 2 / σ)), where σ is the bandwidth parameter and log( ) represents the natural logarithm. Design weight update function: w new =w old - η▽L(w), where ▽ represents the gradient operator, w new and w old Represent the updated weight and the weight before the update respectively, η represents the learning rate; construct the objective function: L(w) =Σ(w i * D i ) + R(w), where R(w) is the regularization term, w i represents the i-th element in the weight vector, D i Denotes the i-th element in the distance matrix, and calculates the gradient: ▽L(w)=D+λ▽R(w), where λ is the regularization coefficient. Define the distance matrix D pos , D pos (i, j) = exp(-||f i -f j ||2 / τ p ), define the negative pair distance matrix D neg , D neg (i, j) = exp(-||f i -f j || 2 / τ n ), construct contrast loss: L c = -log(D pos / (D pos +ΣD neg )), where τ p , τ n is the temperature parameter, f i and f j Represent the feature representation of sample i and sample j respectively. Construct affinity matrix: A=exp(-D / (2σ 2 )), where D represents the distance matrix, and the normalized Laplace is calculated: L norm = I - D -1 / 2 AD -1 / 2 , where I represents the identity matrix, solve the characteristic system: L norm v = λv, where v represents the eigenvector, and the first k eigenvectors are selected: V = [v1, ..., v k ]. Define the boundary function: B(x)=μ+α*σ, where μ is the mean, σ is the standard deviation, and α represents the coefficient in the boundary function; construct a dynamic threshold: θ t =θ0*exp(-βt), where β is the decay rate, θ0 represents the initial threshold, and t represents the time step; Update the classification boundary: B new =(1-γ)B old +γB(x), where γ is the update rate, B old and B new They represent the classification boundary before and after the update respectively.

[0145] This embodiment establishes a multi-level distance measurement system and an adaptive optimization mechanism. By constructing a multi-level distance measurement system and combining multiple measurement methods such as spectral distance and probability distance, an accurate characterization of sample relationships in the feature space is achieved. In the weight optimization process, a gradient-based adaptive update strategy is adopted so that the model can dynamically adjust the importance of each metric according to the training process. The temperature scaling strategy is used in the calculation of the contrast loss, which effectively alleviates the problem of unstable training. Through spectral clustering optimization and boundary adjustment, the discriminability of feature representation is further improved. Compared with the traditional single measurement method, this embodiment improves the discriminative performance of features by about 40%, and shortens the convergence time by about 25%. When processing noisy data, the robustness of the model is improved by about 30%.

[0146] According to one aspect of the present application, step S41 is further:

[0147] S411. Based on the counterfactual knowledge graph and probability reasoning matrix data, construct a feature space mapping function and calculate the Euclidean distance matrix between samples; based on the Euclidean distance matrix, use a metric learning algorithm to obtain learning results; perform isometry verification on the learning results to obtain basic metric data;

[0148] S412, based on the basic measurement data, constructing a Mahalanobis distance transformation function, calculating the inverse covariance matrix; based on the inverse covariance matrix, using an iterative nearest point algorithm to obtain a transformation result; performing scale normalization on the transformation result to obtain transformation measurement data;

[0149] S413, based on the transformed metric data, construct a kernel function mapping matrix and calculate the high-dimensional space projection; based on the high-dimensional space projection, perform kernel principal component analysis to obtain analysis results; perform dimension selection on the analysis results to obtain kernel space metric data;

[0150] S414, constructing a spectral embedding function based on the kernel space metric data, and calculating the local preservation projection; based on the local preservation projection, using a spectral clustering optimization algorithm to obtain a clustering optimization result; performing feature selection on the clustering optimization result to obtain spectral distance data;

[0151] S415, based on the spectral distance data, construct a probability flow distribution function, calculate the transition probability between samples; based on the transition probability between samples, use a random walk algorithm to obtain a walk result; perform a convergence analysis on the walk result to obtain probability distance data;

[0152] S416, constructing a geodesic distance function based on the probabilistic distance data, and calculating the shortest path on the manifold; based on the shortest path, using a fast marching algorithm to obtain a path result; smoothing the path result to obtain geodesic distance data;

[0153] S417, constructing a multi-scale fusion function based on basic metric data, transformed metric data, kernel space metric data, spectral distance data, probability distance data and geodesic distance data, and calculating a distance correlation matrix; based on the distance correlation matrix, using an adaptive weighting algorithm to obtain a weighted result; normalizing the weighted result to obtain fused distance data;

[0154] S418. Based on the fused distance data, a distance calibration function is constructed to calculate the confidence interval boundary; based on the confidence interval boundary, a non-parametric estimation algorithm is used to obtain the estimation result; the estimation result is stability verified to obtain multi-level distance measurement data.

[0155] This embodiment establishes a multi-level distance measurement system and an adaptive optimization mechanism. Through feature space mapping and metric learning algorithms, a basic Euclidean distance measurement is constructed. In the kernel space mapping process, a combination of kernel principal component analysis and dimension selection is adopted to achieve effective representation of nonlinear features. Through spectral embedding and random walk algorithms, the structural relationship between samples is captured. Through multi-scale fusion and non-parametric estimation, adaptive optimization of distance metrics is achieved. This embodiment improves the discriminative performance of feature representation by about 48% and the computational efficiency by about 35%. When processing high-dimensional data, the dimensionality reduction effect is improved by about 40%, while maintaining the essential structure of the data.

[0156] According to one aspect of the present application, step S44 is further:

[0157] S441, obtain contrast loss data, construct a local sensitivity function, calculate the similarity score of the sample pair, execute an adaptive threshold segmentation algorithm, perform noise filtering on the segmentation result, and obtain initial affinity data.

[0158] S442, obtaining initial affinity data, constructing a heat kernel transformation function, calculating local temperature parameters, executing anisotropic diffusion algorithm, performing boundary maintenance on diffusion results, and obtaining optimized affinity matrix data.

[0159] S443. Obtain optimized affinity matrix data, construct a metric tensor function, calculate the local structure tensor, execute a tensor field smoothing algorithm, perform singular value decomposition on the smoothed result, and obtain structural feature data.

[0160] S444, obtaining structural feature data, constructing a normalized Laplace function, calculating a symmetric normalized matrix, executing a spectral decomposition acceleration algorithm, numerically stabilizing the decomposition result, and obtaining Laplace feature data.

[0161] S445, obtaining Laplace feature data, constructing a feature sorting function, calculating eigenvalue distribution, executing a spectral interval analysis algorithm, performing truncation optimization on the analysis results, and obtaining feature selection data.

[0162] S446, acquiring feature selection data, constructing a feature reconstruction function, calculating a basis vector combination, executing an orthogonal fine-tuning algorithm, regularizing the fine-tuning result, and obtaining reconstructed feature data.

[0163] S447, obtaining reconstructed feature data, constructing a feature enhancement function, calculating a residual information matrix, executing an iterative refinement algorithm, performing a quality assessment on the refinement result, and obtaining enhanced feature data.

[0164] S448. Obtain enhanced feature data, construct a feature verification function, calculate a reconstruction error index, execute a stability test algorithm, conduct a comprehensive evaluation of the test results, and obtain optimized feature representation data.

[0165] This embodiment realizes an efficient feature optimization and structure preservation mechanism. A stable affinity matrix is ​​constructed through local sensitivity analysis and heat kernel transformation. In the process of structural feature extraction, a combination of tensor field smoothing and singular value decomposition is adopted to maintain local structural information. The efficiency of feature decomposition is improved through spectral decomposition acceleration and numerical stabilization processing. The reliability of feature reconstruction is ensured through iterative refinement and quality assessment. This embodiment improves the efficiency of feature optimization by about 45% and the reconstruction quality by about 40%. When processing large-scale data, the calculation speed is increased by about 42% while maintaining the discriminative performance of the features.

[0166] According to one aspect of the present application, step S45 is further:

[0167] S451, constructing a feature distribution mapping function based on the optimized feature representation data, and calculating a multidimensional kernel density estimation; based on the multidimensional kernel density estimation, using an adaptive bandwidth selection algorithm to obtain an estimation result; performing smoothing on the estimation result to obtain feature distribution data;

[0168] S452, based on the characteristic distribution data, construct a statistical moment estimation function and calculate high-order statistics; based on the high-order statistics, use a robust estimation algorithm to extract outliers; perform correction processing on the outliers to obtain distribution parameter data;

[0169] S453, based on the distribution parameter data, construct a mixed model function and calculate the component distribution weight; based on the component distribution weight, use the expectation maximization algorithm to obtain the model parameters; automatically tune the model parameters to obtain the mixed distribution data;

[0170] S454, based on the mixed distribution data, construct a boundary initialization function and calculate the decision hyperplane parameters; based on the decision hyperplane parameters, use the support vector positioning algorithm to obtain the boundary position; optimize and adjust the boundary position to obtain initial boundary data;

[0171] S455, based on the initial boundary data, construct a dynamic update function and calculate the time series characteristics; based on the time series characteristics, use an incremental learning algorithm to generate an update strategy; adaptively adjust the update strategy to obtain dynamic threshold data;

[0172] S456, constructing a boundary refinement function based on the dynamic threshold data and calculating local curvature information; based on the local curvature information, using a boundary smoothing algorithm to obtain a smoothed result; topology preservation of the smoothed result to obtain optimized boundary data;

[0173] S457, based on the optimized boundary data, construct a verification evaluation function and calculate a boundary stability index; based on the boundary stability index, use a cross-validation algorithm to obtain a verification result; perform a confidence analysis on the verification result to obtain boundary evaluation data;

[0174] S458. Based on the optimized feature representation data and boundary evaluation data, construct a feature integration function and calculate the multi-scale representation; based on the multi-scale representation, use a feature fusion algorithm to obtain a fusion result; perform a consistency check on the fusion result to obtain an optimized contrastive learning representation and distance measurement data.

[0175] This embodiment establishes a complete boundary optimization and dynamic update mechanism. Through feature distribution mapping and kernel density estimation, accurate characterization of distribution features is achieved. In the process of building the hybrid model, expectation maximization and automatic tuning strategies are adopted to improve the adaptability of the model. Through boundary initialization and dynamic update, adaptive adjustment of classification boundaries is achieved. Through multi-scale feature fusion and consistency verification, the reliability of the optimization results is ensured. This embodiment improves the classification accuracy by about 43% and the boundary stability by about 38%. When processing dynamically changing data, the adaptability is improved by about 40% and the computational overhead is reduced by about 35%.

[0176] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0177] S51. Based on the optimized contrastive learning representation and distance metric data, a performance indicator tensor including a predetermined time step is constructed, and an indicator correlation matrix at different time points is calculated; based on the performance indicator tensor and the indicator correlation matrix, a dynamic weight is used for weighted combination to obtain time series performance evaluation data;

[0178] S52. Based on the timing performance evaluation data, a quadratic evaluation function is constructed to calculate the energy value of the system under different states; based on the energy value, the Jacobian matrix of the system is calculated, the eigenvalue of the Jacobian matrix is ​​extracted, and the system stability index data is obtained;

[0179] S53, based on the system stability index data, construct an input disturbance matrix, calculate the sensitivity matrix of the system output to the input; perform singular value decomposition on the sensitivity matrix, and calculate the norm after decomposition to obtain robustness evaluation index data;

[0180] S54. Based on the robustness evaluation index data, record the change trajectory of system performance over time to form time series data; based on the time series data, calculate the performance deviation in adjacent time windows; based on the performance deviation, update the performance benchmark of the system to obtain the latest performance benchmark; based on the latest performance benchmark, smooth the change trajectory to obtain dynamic tracking data;

[0181] S55. Based on the dynamic tracking data, a multi-criteria decision matrix is ​​constructed to calculate the entropy value of each evaluation criterion; based on the entropy value, the criterion weight is determined, and a weighted normalization operation is performed to obtain the final performance evaluation index data.

[0182] In one embodiment of the present application, based on the optimized contrastive learning representation and distance metric data, a performance indicator tensor is defined: M∈R p×q×r , where p is the number of evaluation indicators, q is the number of samples, and r is the time step; construct the time series correlation matrix: R(t)={r ij (t)},r ij (t) is the index correlation at time t, and the comprehensive score is calculated as: S =Σ(w i * M i ) / Σw i , where w i is the dynamic weight, M i is the i-th element in the performance index tensor. Construct Lyapunov function: V(x) = x T P x, where P is a positive definite matrix, x represents the state vector of the system, and the system energy is calculated as: E(t) = ||x(t)|| 2 P , where x(t) represents the state of the system at time t; define the stability index: SI = max|λ(J)|, where J is the Jacobian matrix and λ is the eigenvalue. Construct the perturbation matrix: Δ={Δ ij}, Δ ij Represents the disturbance intensity; constructs the sensitivity matrix: S = Ψy / Ψx, where Ψ represents the partial derivative and y represents the output value of the system; calculates the robustness index: RI=tr(S T S) 1 / 2 . Construct performance trajectory: T(t)={y(t), y'(t)}, where y(t) represents the output value of the system at time t, and y'(t) represents the output change rate of the system at time t; define drift detector: D(t)=||y(t)-y(t-τ)|| / τ, where τ is the time window; update performance benchmark: B(t)=αB(t-1)+(1-α)y(t), α represents the coefficient when updating the performance benchmark, and B(t-1) represents the performance benchmark value at time t-1. Construct decision matrix: DM=[c ij ], c ij The score of the jth solution for the i-th criterion; calculate the entropy weight: w j =(1-H j ) / Σ(1-H i ), where H j is the entropy value of the jth criterion; generate the final score: FS=Σ(w j*N j ), where N j Normalized score.

[0183] This embodiment implements a comprehensive performance evaluation and dynamic tracking mechanism. By constructing a multi-dimensional evaluation index system, the model is evaluated from multiple dimensions such as stability, robustness and dynamic performance. In the stability analysis, an energy function-based method is used to evaluate the system state, and the stability of the system is analyzed by the Jacobian matrix eigenvalue. The robustness evaluation uses a combination of sensitivity analysis and singular value decomposition to accurately quantify the response of the model to input disturbances. Dynamic performance tracking monitors model performance changes in real time through a sliding update mechanism of a time window. This embodiment can capture about 95% of performance anomalies, and the early warning accuracy rate reaches more than 90%, while controlling the evaluation overhead within an acceptable range.

[0184] According to one aspect of the present application, a hierarchical contrastive counterfactual learning method for a visual question answering model is specifically as follows: two types of counterfactual samples are constructed by replacing the image and question in the original pair with randomly selected images and questions, which can be used as negative samples for contrastive learning to help the model focus on the image and question information; positive samples are constructed by masking unimportant areas / words in the question and image using a gradient-based method, which can help the training model better perceive the key information in the question and image. Background areas in the image and unimportant words in the question are masked. Based on the above samples, a hierarchical contrastive counterfactual learning framework is introduced to mitigate bias in VQA models.

[0185] Build a visual question-answering interactive platform, where users can input pictures and questions to the visual question-answering platform and get answers. The platform splices the images and question texts into the model, which analyzes the input and generates answers. The platform returns the answers predicted by the model to the user. In order to ensure that the model relies on language priors or prominent objects in the image to answer questions, rather than truly understanding the questions and image content, the training data is processed, positive samples and negative samples are constructed, the original samples, positive samples, and negative samples are input into the model, and the bias in the VQA model is reduced through a hierarchical contrastive counterfactual learning framework.

[0186] In the feature space, feature-level contrastive loss is used to reduce the distance between the original sample and the positive sample, and to increase the distance between the original sample and the negative sample. This can encourage the learned representation to be more discriminative for answer classification. In addition, answer-level contrastive loss is also used to increase the distance between the original sample and the negative sample in the answer space, which can help the VQA model avoid generating biased answers. By applying these two contrastive learning objectives, the VQA model can effectively learn the correspondence information between the original sample and the counterfactual sample, and is forced to focus on the key information for answering the question. Specifically:

[0187] 1) Developed a visual question-answering platform. Users input pictures and questions on the platform to request the model to generate corresponding answers, and the platform returns the generated answers.

[0188] 2) In order to mitigate the impact of language and visual bias on visual question answering models, a hierarchical contrastive counterfactual learning method is proposed.

[0189] 3) In order to construct the samples required for model training, the present invention proposes a gradient-based method to mask the unimportant areas / words in the question and the image to construct positive samples, and constructs two types of counterfactual samples by randomly selecting images and questions to replace the images and questions in the original pairs.

[0190] In one embodiment of the present application, a hierarchical contrastive counterfactual learning method for a visual question answering model uses the original data in the data set to form new counterfactual samples, avoiding the use of expensive annotations and the introduction of noise; before using the hierarchical contrastive learning method, positive samples and negative samples are constructed for each original training sample in the data set D. Specifically:

[0191] Step 1: Negative sample construction.

[0192] Given an image-question pair (I, Q), randomly select an unrelated image to form a new question-image pair (I - , Q), and then randomly selects an unrelated question to form a new question-image pair (I, Q - ). The specific steps are as follows:

[0193] Step 101: Randomly select an irrelevant image I from the image set - ∈I;

[0194] Step 102: Randomly select an irrelevant question Q from the question set - ∈Q;

[0195] Step 103: Construct negative samples (I - , Q) and (I, Q - ).

[0196] Step 2: Construction of positive samples.

[0197] Select unimportant regions or words in the original image-question pair, and use a gradient-based method to evaluate each region / word in the image / question, and regard regions or words with smaller gradients as unimportant regions or words. The specific steps are as follows:

[0198] Step 201: Calculate the predicted probability F(A|I,Q) of the model for the sample, P vqa (a) is the predicted answer probability of the true answer a;

[0199] Step 202: Extract image features e v (I), denoted as v;

[0200] Step 203: Extract problem features e q (Q), denoted as q;

[0201] Step 204: Calculate the gradient magnitude of each region in the image: s(a, v i ) = [ (▽ vi P vqa (a)) ] T 1; where 1 represents a vector of all 1s;

[0202] Step 205: Calculate the gradient magnitude of each word in the question: s(a, q i )=(▽ qi P vqa (a)) T 1;

[0203] Step 206: Select the top K regions with the lowest gradient magnitude as the unimportant region set: I'=I[top-K(argsort vi∈v (1 / s(a,v i )))], where argsort represents the sorting function;

[0204] Step 207: mask K regions in I' in the original image I to obtain a new masked image I + ;

[0205] Step 208: mask image I + Combined with the original question Q, we get a positive sample (I + , Q);

[0206] Step 209: For the original question Q, extract the question type words for each question. In addition to these words, select the top W words with the lowest gradient magnitude as unimportant words: Q'=Q[top-W(argsort qi∈q (1 / s(a,v i )))];

[0207] Step 210: Mask W words in Q' in the original question Q to obtain a new masked question Q + ;

[0208] Step 211: Q + Combined with the original image I, another positive sample (I, Q + ).

[0209] In order to avoid masking key areas / words in the image / question, this embodiment sets relatively small values ​​for K and W. The model needs to be able to distinguish between negative samples and original samples, and should produce different answers for negative samples and original samples, and should produce the same answer for positive samples and original samples. Therefore, this embodiment proposes a hierarchical contrastive learning method to train the model to learn the differences between original samples and negative samples and the consistency between original samples and positive samples, which helps the model focus on key visual and language information to infer answers. This embodiment specifically uses feature-level contrastive learning to train the model's discriminative representation of these samples, and uses answer-level contrastive learning to train the model's discriminativeness of these samples in the answer space.

[0210] Step 3: Feature level training.

[0211] Map the features of the original image-question pair (I, Q) to the positive sample (I + ,Q),(I,Q + ) features, and stay away from negative samples (I - ,Q),(I,Q - ) features. Specifically, the boundary-based ranking loss is used for training.

[0212] Step 301: Use the model to obtain the multimodal features of the sample: f = mm (e q (·), e v (·)), where m represents the index of the answer in the answer set A, e q (·) and e v (·) represent the functions for extracting problem features and image features respectively. The features of the original sample are represented as f ori , the feature of the positive sample is represented by f pos v , f pos q , the feature of the negative sample is represented by f neg v , f neg q .

[0213] Step 302: Calculate (f ori , f pos v , f neg v )’s feature-level contrast loss:

[0214] L c，f v =max(0,Δ m +s(f ori , f pos v )-s(fori , f neg v ));

[0215] where Δ m is a boundary value, and s(,) is a scoring function used to determine the similarity between two features.

[0216] Step 303, calculate (f ori , f pos q , f neg q )’s feature-level contrast loss:

[0217] L c，f q =max(0,Δ m +s(f ori , f pos q )-s(f ori , f neg q ));

[0218] where Δ m is a boundary value, and s(,) is a scoring function used to determine the similarity between two features.

[0219] Step 304, the overall feature level contrast loss is calculated by the following formula:

[0220] L c，f =L c，f v +L c，f q .

[0221] Step 4: Answer level training.

[0222] The goal is to predict the answer to the original image-question pair (I, Q) close to the positive sample (I + ,Q),(I,Q + ) and stay away from negative samples (I - ,Q),(I,Q - ). Specifically, we reduce the Euclidean distance between the original sample and the positive sample answer distribution, while increasing the distance between the original sample and the negative sample answer distribution. This is similar to the feature level contrast objective, and we use a boundary-based ranking loss to achieve this. Specifically:

[0223] Step 401: Use the model to generate the predicted answer distribution of the sample: P = F(A|·,·), where F( ) represents the prediction function of the model. The answer distribution of the original sample is regarded as p ori , the answer distribution of positive samples is regarded as ppos v , p pos q , the answer distribution of negative samples is regarded as p neg v , p neg q Here we only describe (p ori , p pos v , p neg v ) loss formula, (p ori , p pos q , p neg q ) has a similar loss formula:

[0224] L c，a v (p ori ;θ)=max(0,Δ m +s(p ori , p pos v )-s(p ori , p neg v ));

[0225] Step 402: In order to better compare the answer predictions between the original sample and the enhanced sample, p should be increased as much as possible. ori and p neg The distance in the dimension of the true answer a, reducing p ori and p pos The distance in the corresponding dimension. Therefore, the following comparative learning objectives are adopted:

[0226] L c，a v (p ori ;θ)=log (σ(p ori )[m]-σ(p pos v )[m])-log(σ(p ori )[m]-σ(p neg v )[m]);

[0227] Where a is the index of the answer in the answer set A. Positive samples are generated by masking irrelevant elements, and the answers between the original sample and the positive sample are usually consistent. Therefore, it is unnecessary to optimize the distance between them, and the formula can be rewritten as:

[0228] L c，a v (p ori;θ)=-log(σ(p ori )[m]-σ(p neg v )[m]);

[0229] Where σ is the sigmoid function and θ represents the model parameters.

[0230] Step 403: For further simplification, directly minimize the possibility of predicting the true answer of the negative sample. - , Q) and (I, Q), (Q - , I))) has the same form of answer level contrast loss, and only one formula is described here. The loss can be expressed as:

[0231] L c，a v =σ(p neg v )[m];

[0232] Step 404, the overall answer level contrast loss is defined as follows:

[0233] L c，a =L c，a v +αL c，a q .

[0234] Step 5: Overall training goals.

[0235] The overall training objective consists of three losses, namely the model prediction loss L vqa , feature level contrast loss L c，f And the answer level comparison loss L c，a , the formula is as follows:

[0236] L=L vqa +λ1L c，f +λ2L c，a .

[0237] Where λ1 and λ2 represent the weight coefficients of feature-level contrastive loss and answer-level contrastive loss, respectively.

[0238] According to one aspect of the present application, a hierarchical contrastive counterfactual learning system for a visual question answering model includes:

[0239] at least one processor; and,

[0240] a memory communicatively connected to at least one of the processors; wherein,

[0241] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the hierarchical contrastive counterfactual learning method for a visual question answering model described in any of the above embodiments.

[0242] The present invention improves the performance and reliability of the visual question answering model through a hierarchical comparative counterfactual learning framework. The overall framework includes a complete process from data preprocessing, feature extraction, knowledge enhancement to model optimization, and each module works together to form a closed-loop learning system. At the data level, the quality of the training data is guaranteed by dynamic sample balancing and multimodal feature fusion; at the model level, the reasoning ability of the model is enhanced by counterfactual learning and multi-level knowledge representation; at the optimization level, the stability and reliability of the model are ensured by adaptive metric learning and comprehensive performance evaluation. The present invention improves the accuracy of the standard VQA dataset by about 15%, improves the performance in complex scenarios by more than 20%, and improves the training efficiency of the model by about 35%. Especially when dealing with open domain problems, the model shows stronger generalization ability and the error rate is reduced by about 25%.

[0243] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A hierarchical contrastive counterfactual learning method for visual question answering models, characterized by: The steps include: S1. Obtain the original image data and question text data and perform preprocessing to construct tensor features; generate a projection matrix based on the tensor features; calculate the probability distribution according to the projection matrix, and screen and obtain samples based on the probability distribution; Based on the density of samples, the sample weights are dynamically adjusted to obtain a balanced set of positive and negative sample pairs; S2. Based on the balanced set of positive and negative sample pairs, construct the adjacency matrix of images, questions and answers; Based on the adjacency matrix, calculate the feature enhancement matrix; Based on the feature enhancement matrix, inter-modal alignment and dynamic feature fusion operations are performed in sequence to obtain enhanced feature representation; based on the enhanced feature representation, feature quality assessment is performed to obtain quality score data; S3. Based on the enhanced feature representation and quality score data, a multi-level probabilistic knowledge structure is constructed. Based on the multi-level probabilistic knowledge structure, counterfactual samples are generated and a probabilistic reasoning framework is constructed. Based on the probabilistic reasoning framework, knowledge fusion operations and uncertainty quantification are performed in sequence to obtain counterfactual knowledge graphs and probabilistic reasoning matrix data; S4. Construct a multi-level distance measurement system based on counterfactual knowledge graph and probabilistic reasoning matrix data; Based on the multi-level distance measurement system, adaptive weight optimization is performed to obtain optimized weight coefficient data; based on the optimized weight coefficient data, the contrast loss is calculated, and spectral clustering optimization and boundary adjustment are performed in sequence to obtain optimized contrast learning representation and distance measurement data; S5. Construct a multi-dimensional evaluation index system based on the optimized contrastive learning representation and distance metric data; Based on the multi-dimensional evaluation index system, stability analysis is performed and robustness indicators are calculated; Based on the robustness index, dynamic performance tracking is performed to obtain tracking results; Comprehensively evaluate the tracking results to obtain the final performance evaluation index data; Step S3 is further as follows: S31. Based on the enhanced feature representation and quality score data, extract entity, relationship and concept hierarchical information; integrate entity, relationship and concept hierarchical information into a three-dimensional probability tensor, and recursively decompose the three-dimensional probability tensor to obtain the decomposition matrix of each level; based on the decomposition matrix, calculate the transition probability between levels; Normalize the transition probability to obtain hierarchical knowledge structure data; S32. Construct a causal relationship matrix based on hierarchical knowledge structure data; Based on the causal relationship matrix, the state change of each sample under the intervention operation is calculated; based on the difference between the state change and the enhanced feature representation, the counterfactual state is generated; the enhanced feature representation and the counterfactual state are combined to obtain the counterfactual sample data; S33. Based on the counterfactual sample data, calculate the conditional probability of the sample; based on the enhanced feature representation, estimate the prior probability of the sample feature; based on the conditional probability and the prior probability, calculate the posterior probability distribution; integrate the conditional probability, the prior probability and the posterior probability distribution to obtain the probability reasoning framework data; S34, based on the probabilistic reasoning framework data, calculate the KL divergence matrix between different knowledge sources and construct a conditional mutual information network; Based on the KL divergence matrix and conditional mutual information network, the knowledge fusion weight is calculated and weighted fusion operation is performed to obtain the fused knowledge representation data; S35. Calculate the entropy matrix of knowledge distribution based on the fused knowledge representation data; Based on the entropy matrix, the information gain rate is calculated and the normalized confidence score is constructed; Based on the confidence score, the fused knowledge representation data and entropy matrix are integrated to obtain the counterfactual knowledge graph and probabilistic reasoning matrix data; Step S31 is further as follows: S311. Based on the enhanced feature representation and quality score data, construct an entity feature matrix and calculate the semantic relevance between entities; based on the semantic relevance, use a hierarchical clustering algorithm to obtain clustering results; perform probability allocation on the clustering results to obtain entity hierarchical probability data; S312, based on the entity level probability data, construct a relationship description matrix and calculate the relationship type distribution; based on the relationship type distribution, perform relationship pattern mining to obtain mining results; perform credibility assessment on the mining results to obtain relationship probability distribution data; S313. Based on the entity hierarchy probability data and the relationship probability distribution data, construct a concept abstraction function and calculate the concept generalization path; based on the concept generalization path, perform concept hierarchy derivation to obtain derivation results; perform consistency check on the derivation results to obtain concept hierarchy structure data; S314, constructing a third-order probability tensor based on the entity hierarchy probability data, the relationship probability distribution data, and the concept hierarchy data, and calculating the initial values ​​of the tensor elements; performing sparsity constraints based on the initial values ​​of the tensor elements to obtain constraint results; Numerically stabilize the constraint results to obtain the initial probability tensor data; S315. Based on the initial probability tensor data, construct a nuclear norm minimization objective function and calculate tensor singular values; Based on the tensor singular value, an accelerated approximation algorithm is used to obtain an approximate result; the approximate result is truncated to obtain low-rank decomposition data; S316. Based on the low-rank decomposition data, construct a recursive decomposition function and calculate sub-tensor features; Based on the sub-tensor characteristics, an alternating optimization algorithm is used to obtain the optimization result; the optimization result is regularized to obtain the hierarchical decomposition matrix data; S317, constructing a state transition function based on the hierarchical decomposition matrix data, and calculating the conditional transition probability; Based on the conditional transition probability, a Markov chain simulation is performed to obtain the simulation results, and the stability test of the simulation results is performed to obtain the initial transition probability data; S318. Based on the initial transfer probability data, a probability flow distribution function is constructed to calculate the flow balance constraint; Based on the flow balance constraint, an iterative balance algorithm is used to obtain the balance result; The balanced results are normalized to obtain hierarchical knowledge structure data.

2. The hierarchical contrastive counterfactual learning method for visual question answering models according to claim 1, characterized in that: Step S1 is further as follows: S11, obtaining original image data and question text data, converting the original image data into three-dimensional tensor data, and converting the question text data into two-dimensional tensor data through word embedding; performing outer product operation based on the three-dimensional tensor data and the two-dimensional tensor data to obtain multimodal joint tensor data; S12. Based on the multimodal joint tensor data, a predetermined mutually independent orthogonal basis vector group is constructed; based on each orthogonal basis vector group, a corresponding projection matrix is ​​calculated, and an orthogonality verification value is calculated for each pair of projection matrices to obtain a set of projection matrices that satisfy complementarity; S13. Based on the projection matrix set, the multimodal joint tensor data is projected into each projection space respectively, and the feature distribution of each projection space is calculated; based on the feature distribution of all projection spaces, a joint probability density function is constructed to obtain the probability distribution data of the sample; S14. Based on the probability distribution data of the samples, calculate the information entropy change value of each sample before and after it is added; based on the information entropy change value, sort all samples, select the predetermined sample with the highest information gain as the positive sample, and select the predetermined sample with the lowest information gain as the negative sample, to obtain a preliminary set of positive and negative sample pairs; S15. Based on the preliminary set of positive and negative sample pairs, calculate the cosine similarity matrix between samples; based on the cosine similarity matrix, use the spectral clustering method to group the samples to obtain grouped samples; based on the grouped samples, calculate the sample density value of each group; Based on the sample density value, the sampling weight coefficient is calculated, and the preliminary set of positive and negative sample pairs is resampled to obtain a balanced set of positive and negative sample pairs.

3. The hierarchical contrastive counterfactual learning method for visual question answering models according to claim 2, characterized in that: Step S2 is further as follows: S21. Based on the balanced positive and negative sample pairs, respectively calculate the similarity between samples of the image, question and answer; based on the similarity between samples, construct an adjacency matrix, and calculate the degree matrix corresponding to each adjacency matrix; Based on the adjacency matrix and degree matrix, the Laplacian matrix is ​​calculated to obtain multimodal graph structure data; S22, based on the multimodal graph structure data, calculating the local structure difference between each sample and its neighboring samples; Based on the local structural differences, the covariance estimation matrix is ​​constructed; Regularization and decomposition operations are performed on the covariance estimation matrix to obtain feature enhancement matrix data; S23, based on the feature enhancement matrix data, calculating the transition probability matrix between different modal features; symmetrizing the transition probability matrix, constructing and solving the generalized eigenvalue equation, and obtaining the eigenvector; Based on the eigenvector, the mapping relationship between modes is constructed to obtain the mode alignment matrix data; S24. Based on the modal alignment matrix data, construct an initial modal fusion weight matrix and calculate the gradient value of the fusion loss; Based on the gradient value, the fusion weight of the initial modality fusion weight matrix is ​​updated, and a weighted fusion operation is performed to obtain the fused feature representation data; S25. Based on the fused feature representation data, calculate the structural consistency index of the feature and the mutual information value between the modes; construct a comprehensive score based on the structural consistency index and the mutual information value; integrate the fused feature representation data and the comprehensive score to obtain enhanced feature representation and quality score data.

4. The hierarchical contrastive counterfactual learning method for visual question answering models according to claim 3, characterized in that: Step S4 is further as follows: S41. Based on the counterfactual knowledge graph and the probability reasoning matrix data, the basic metric matrix, the spectral distance value and the probability distance distribution between the sample pairs are calculated; the basic metric matrix, the spectral distance value and the probability distance distribution are integrated to obtain multi-level distance metric data; S42, constructing an initial weight vector based on multi-level distance measurement data; Based on the initial weight vector, calculate the gradient value of the preset objective function to the weight; Based on the gradient value and the predefined regularization term, the initial weight vector is updated to obtain the optimized weight coefficient data; S43, based on the optimized weight coefficient data, respectively calculating the distance matrix between the positive and negative sample pairs; based on the distance matrix between the positive and negative sample pairs, calculating the contrast loss value, performing temperature scaling processing on the contrast loss value, and obtaining the contrast loss data; S44, based on the contrast loss data, construct an affinity matrix between samples, calculate a normalized Laplace matrix; solve the eigenvalues ​​and eigenvectors of the Laplace matrix, select a predetermined number of eigenvectors, and obtain optimized feature representation data; S45. Based on the optimized feature representation data, the statistical parameters of the feature distribution are calculated to construct an initial boundary function; the dynamic threshold of the initial boundary function is updated according to a predefined time decay factor to obtain an updated boundary function; based on the updated boundary function, an optimized contrastive learning representation and distance measurement data are generated.

5. The hierarchical contrastive counterfactual learning method for visual question answering models according to claim 4, characterized in that: Step S5 is further as follows: S51. Based on the optimized contrastive learning representation and distance metric data, construct a performance indicator tensor including a predetermined time step, and calculate the indicator correlation matrix at different time points; Based on the performance indicator tensor and indicator correlation matrix, dynamic weights are used for weighted combination to obtain time series performance evaluation data; S52. Based on the timing performance evaluation data, a quadratic evaluation function is constructed to calculate the energy value of the system in different states; Based on the energy value, the Jacobian matrix of the system is calculated, the eigenvalue of the Jacobian matrix is ​​extracted, and the system stability index data is obtained; S53, based on the system stability index data, construct an input disturbance matrix, calculate the sensitivity matrix of the system output to the input; perform singular value decomposition on the sensitivity matrix, and calculate the norm after decomposition to obtain robustness evaluation index data; S54. Based on the robustness evaluation index data, record the change trajectory of system performance over time to form time series data; Based on time series data, calculate the performance deviation within adjacent time windows; Based on the performance deviation, the system's performance benchmark is updated to obtain the latest performance benchmark; Based on the latest performance benchmark, the change trajectory is smoothed to obtain dynamic tracking data; S55. Based on the dynamic tracking data, a multi-criteria decision matrix is ​​constructed to calculate the entropy value of each evaluation criterion; Based on the entropy value, the criterion weights are determined, and weighted normalization operations are performed to obtain the final performance evaluation index data.

6. The hierarchical contrastive counterfactual learning method for visual question answering models according to claim 5, characterized in that: Step S41 is further as follows: S411. Based on the counterfactual knowledge graph and the probability reasoning matrix data, a feature space mapping function is constructed to calculate the Euclidean distance matrix between samples; based on the Euclidean distance matrix, a metric learning algorithm is used to obtain the learning results; Verify the learning results for isometry and obtain basic measurement data; S412. Based on the basic metric data, construct a Mahalanobis distance transformation function and calculate the inverse covariance matrix; Based on the inverse covariance matrix, the iterative closest point algorithm is used to obtain the transformation result; Normalize the transformation result to obtain transformation measurement data; S413, based on the transformed metric data, construct a kernel function mapping matrix and calculate the high-dimensional space projection; Based on high-dimensional space projection, kernel principal component analysis is performed to obtain analysis results; Select the dimension of the analysis results to obtain kernel space metric data; S414, constructing a spectral embedding function based on the kernel space metric data, and calculating the local preservation projection; based on the local preservation projection, using a spectral clustering optimization algorithm to obtain a clustering optimization result; performing feature selection on the clustering optimization result to obtain spectral distance data; S415, constructing a probability flow distribution function based on the spectral distance data, and calculating the transition probability between samples; Based on the transfer probability between samples, a random walk algorithm is used to obtain the walk results; the convergence analysis of the walk results is performed to obtain the probability distance data; S416. Based on the probabilistic distance data, a geodesic distance function is constructed to calculate the shortest path on the manifold; Based on the shortest path, the fast marching algorithm is used to obtain the path result; Smooth the path results to obtain geodetic distance data; S417, constructing a multi-scale fusion function based on basic metric data, transformed metric data, kernel space metric data, spectral distance data, probabilistic distance data and geodesic distance data, and calculating a distance correlation matrix; Based on the distance correlation matrix, an adaptive weighting algorithm is used to obtain weighted results; the weighted results are normalized to obtain fused distance data; S418, constructing a distance calibration function based on the fused distance data, and calculating the confidence interval boundary; Based on the confidence interval boundaries, a nonparametric estimation algorithm is used to obtain the estimation results; The estimation results are verified for stability and multi-level distance measurement data are obtained.

7. The hierarchical contrastive counterfactual learning method for visual question answering models according to claim 5, characterized in that: Step S45 is further as follows: S451, constructing a feature distribution mapping function based on the optimized feature representation data, and calculating a multidimensional kernel density estimation; based on the multidimensional kernel density estimation, using an adaptive bandwidth selection algorithm to obtain an estimation result; performing smoothing on the estimation result to obtain feature distribution data; S452. Based on the characteristic distribution data, construct a statistical moment estimation function and calculate high-order statistics; Based on high-order statistics, a robust estimation algorithm is used to extract outliers; Correct the outliers and obtain the distribution parameter data; S453, based on the distribution parameter data, construct a mixed model function and calculate the component distribution weight; based on the component distribution weight, use the expectation maximization algorithm to obtain the model parameters; automatically tune the model parameters to obtain the mixed distribution data; S454, based on the mixed distribution data, construct a boundary initialization function and calculate the decision hyperplane parameters; based on the decision hyperplane parameters, use the support vector positioning algorithm to obtain the boundary position; optimize and adjust the boundary position to obtain initial boundary data; S455. Based on the initial boundary data, a dynamic update function is constructed to calculate time series characteristics; Based on the time series characteristics, an incremental learning algorithm is used to generate an update strategy; the update strategy is adaptively adjusted to obtain dynamic threshold data; S456, constructing a boundary refinement function based on the dynamic threshold data and calculating local curvature information; Based on the local curvature information, a boundary smoothing algorithm is used to obtain a smooth result; the smooth result is topologically preserved to obtain optimized boundary data; S457, based on the optimized boundary data, construct a verification evaluation function and calculate a boundary stability index; based on the boundary stability index, use a cross-validation algorithm to obtain a verification result; Conduct confidence analysis on the verification results to obtain boundary assessment data; S458. Based on the optimized feature representation data and boundary evaluation data, construct a feature integration function and calculate a multi-scale representation; Based on multi-scale representation, the feature fusion algorithm is used to obtain the fusion result; The fusion results are checked for consistency to obtain optimized contrastive learning representation and distance metric data.

8. A hierarchical contrastive counterfactual learning system for visual question answering models, characterized by: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the hierarchical contrastive counterfactual learning method for a visual question answering model as described in any one of claims 1 to 7.

Images