A method for rearranging by counterfactual constraint retrieval of multi-modal data facing computing power

Abstract

The present application relates to a kind of facing computing power multimodal data's counterfactual constraint retrieval rearrangement method, by obtaining the multimodal query data of the computing power network to be queried, the predicted answer corresponding to it is generated using pre-training generation model, coarse-grained retrieval is carried out to external knowledge base to obtain candidate text paragraph set, the image block feature of computing power network image is extracted, semantic slot vector is generated using initialization strategy, the vector after the problem text feature vector and each semantic slot vector are spliced is input into gate network to obtain the final visual selection weight based on dynamic gate strategy mixing and the semantic slot after visual selection weight weighting, based on the problem text in multimodal query data, candidate text paragraph set and the semantic slot obtain the multi-dimensional score feature after updating and the reordering result according to score and the result is used as the final retrieval result of corresponding multimodal query data.Such, the robustness and accuracy of multimodal data retrieval rearrangement model in computing power environment are improved.

Description

Technical Field

[0001] This invention relates to the field of computing power scheduling, and in particular to a counterfactual constraint retrieval and rearrangement method for multimodal computing power data. Background Technology

[0002] As a core component of integrated green computing network construction, computing network scheduling requires allocating computing tasks based on real-time computing network resource status. To achieve intelligent scheduling of the computing network, it is often necessary to refer to massive amounts of historical data, including historical scheduling logs, expert operation and maintenance strategy libraries, and various emergency plans. This historical data exhibits significant multimodal heterogeneity, including visual and textual modalities. The visual modality includes unstructured data reflecting the computing network status, such as server rack heatmaps and energy consumption fluctuation curves; the textual modality includes scheduling task descriptions, historical fault investigation records, and optimization strategy documents.

[0003] However, existing multimodal data retrieval methods have significant limitations when processing such computationally intensive multimodal data: due to insufficient visual feature interaction in the visual modality, abnormal states may occur, and existing methods often cannot accurately decouple and focus on these key abnormal states, resulting in a mismatch between the retrieved historical strategies and the current state; traditional multimodal data retrieval methods tend to score based solely on the question text or high-frequency co-occurring words, which can easily lead to learning false relevance, reducing the reliability of retrieval results, and thus lacking causal robustness.

[0004] Therefore, how to provide a multimodal data retrieval method that can decouple key state features from complex computing network images and conduct deep interaction and causal verification with text queries has become a technical problem that urgently needs to be solved in the field of intelligent scheduling of green computing networks. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a counterfactual constraint retrieval and rearrangement method for computing power multimodal data, which is in contrast to the above-mentioned prior art.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a counterfactual constraint retrieval and reordering method for multimodal computing power data, characterized by comprising the following steps: Step 1: Obtain multimodal query data for the computing power network to be queried; Step 2: Use the pre-trained generative model to generate a predicted answer for the acquired multimodal query data, and introduce the predicted answer into an external knowledge base for text association analysis. Use positive sample paragraphs and hard negative sample paragraphs to perform comparative learning training on the dual encoder model based on sample selection to obtain the trained dual encoder model. Step 3: Use the trained dual encoder model to perform regular dense vector retrieval on the external knowledge base containing massive operation and maintenance strategies and historical scheduling logs for question text in multimodal query data to perform coarse-grained retrieval and obtain a set of candidate text paragraphs. Step 4: Use a visual encoder with frozen parameters to extract image patch features from the computing power network image in the multimodal query data; Step 5: An initial semantic slot vector for the extracted image patch features is generated using an input-related initialization strategy. An orthogonally initialized learnable bias is superimposed on the initial semantic slot vector. The extracted image patch features are aggregated into a set number of complementary and overlapping semantic slot vectors through a multi-round iterative slot attention mechanism. Step 6: Obtain the feature vector of the question text in the multimodal query data through the pre-trained text encoder, and concatenate the feature vector of the question text with the obtained semantic slot vectors to obtain the concatenated vector. Step 7: Input the concatenated vector into a gated network containing a multilayer perceptron and a sigmoid activation function for computation to obtain the importance weight of each semantic slot. Then, use a dynamic gating strategy to mix soft gating weights and hard truncation masks to generate the final visual selection weights. Step 8: Use visual selection weights to weight all semantic slots to obtain the semantic slots weighted by visual selection weights; where each semantic slot corresponds one-to-one with the semantic slots weighted by visual selection weights. Step 9: Based on the question text, candidate text paragraph set and semantic slots weighted by visual selection weights in the acquired multimodal query data, calculate the multidimensional scoring features for the multimodal query data, and sort the multidimensional scoring features in descending order based on the scores to obtain the updated multidimensional scoring features. Step 10: Sort all candidate text paragraphs in the candidate text paragraph set in descending order according to their scores based on the updated multidimensional scoring features, and take the top preset number of candidate text paragraphs with the highest scores in the sorting results as the final retrieval results of the corresponding multimodal query data.

[0007] Improved, in the counterfactual constraint retrieval and reordering method for computing power multimodal data, in step 5, the input-related initialization strategy is set as follows: Step b1: Perform average pooling on all image patch features of the extracted computing power network image to obtain the global visual context vector; Step b2: Input the obtained global visual context vector into the initialized multilayer perceptron for mapping processing to obtain the initial slot reference vector; Step b3: Construct a set of learnable bias parameters that are consistent with the number of semantic slots, and assign values ​​to the learnable bias parameters using an orthogonal initialization method to obtain the assigned learnable bias parameters; Step b4: Add the initial slot reference vector to the assigned learnable bias parameters to obtain the initial semantic slot vector.

[0008] Furthermore, in the counterfactual constraint retrieval and reordering method for computing power multimodal data, in step 5, the extracted image patch features are aggregated into a preset number of semantic slot vectors through a multi-round iterative slot attention mechanism as follows: Step c1: Construct the slot attention module; Step c2: Use all image patch features as the keys and values ​​of the slot attention module, and use the initial semantic slot vector as the query of the slot attention module; Step c3 involves updating the semantic slot vector through a multi-round iterative competitive attention mechanism to aggregate all image patch features into the semantic slot vector.

[0009] Further improvements are made to the counterfactual constraint retrieval and rearrangement method for multimodal computing data. In step 7, the dynamic gating strategy is set as follows: Step d1: Calculate all soft attention weights of the gating network output; Step d2: Select the first K soft attention weights from all the calculated soft attention weights in descending order of their weight values; Step d3: Determine the K semantic slots corresponding to the K soft attention weights, and generate a binary hard truncation mask from the K semantic slots. Step d4: Set a hybrid weight for the attention weight and the hard truncation mask generated corresponding to the first K soft attention weights; wherein, the hybrid weight increases with the number of iterations during the training process; and the value range of the hybrid weight is [0,1].

[0010] In a further improvement, in the counterfactual constraint retrieval and reordering method for multimodal computing power data, the calculation method for the multidimensional scoring features in step 9 is as follows: Step e1, calculate text-paragraph similarity: calculate the cosine similarity between the text question vector and the candidate paragraph vector, and use the cosine similarity as the text-paragraph similarity; Step e2, calculate slot-paragraph interaction similarity: calculate the cosine similarity between each semantic slot and the candidate paragraph vector, use visual selection weights to perform a weighted sum of all the obtained cosine similarities, and use the sum of the weighted sum as the slot-paragraph interaction similarity; Step e3, calculate visual-paragraph convergence similarity: use visual selection weights to perform weighted summation on all semantic slots to obtain converged visual vectors, and calculate the cosine similarity between the converged visual vector and the candidate paragraph vectors, and use the calculated cosine similarity as the visual-paragraph convergence similarity. Step e4: The obtained text-paragraph similarity, slot-paragraph interaction similarity and visual-paragraph convergence similarity are concatenated to obtain the concatenated similarity. The concatenated similarity is then input into a fully connected scoring network for scoring processing to obtain multidimensional scoring features.

[0011] Further improvements are made to the counterfactual constraint retrieval and reordering method for multimodal computing data. In step 2, during the contrastive learning training process based on sample selection: Step a1: Construct a counterfactual complementary path based on complementary weights and a counterfactual erasure path based on all-zero weights; Step a2: Construct a loss function containing necessity constraints and sufficiency constraints by combining the real path, so as to jointly optimize the model parameters of the dual encoder model; Set a necessity constraint loss: Under the counterfactual complementary path, the score difference between positive and negative samples in the positive sample paragraph and the hard negative sample paragraph is smaller than the score difference under the true path. Set a sufficiency constraint loss: Under the true path, the score difference between positive and negative samples is greater than the score difference under the counterfactual erase path using all-zero weights.

[0012] Furthermore, in the counterfactual constraint retrieval and rearrangement method for computing power multimodal data, the counterfactual complementary path uses visual information that is determined to be non-critical by the dual encoder model, and the true path uses only the critical visual information selected by the dual encoder model.

[0013] Compared with the prior art, the advantages of the present invention are as follows: The counterfactual constraint retrieval and reordering method for multimodal computing power data of the present invention obtains multimodal query data of the computing power network to be queried and performs comparative learning training based on sample selection to obtain a trained dual encoder model. The dual encoder model is used to obtain a set of candidate text paragraphs. The visual encoder with frozen parameters is used to extract image patch features of the computing power network image in the multimodal query data. Then, the semantic slot vector is obtained. The question text feature vector and each semantic slot vector are concatenated to obtain the concatenated vector. The final visual selection weight is generated based on the importance weight of each semantic slot. The semantic slots are processed and weighted by the visual selection weight. The updated multidimensional scoring features for the multimodal query data are calculated. The final retrieval result of the multimodal query data is obtained by reordering based on the updated multidimensional scoring features. In this way, based on coarse-grained retrieval using a dual-encoder model, visual entities are decoupled through an improved slot attention mechanism. By combining a gating strategy to screen key evidence and introducing a counterfactual causal constraint training framework, the robustness, accuracy, and interpretability of the multimodal data retrieval rearrangement model in a green computing environment are significantly improved. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the counterfactual constraint retrieval and rearrangement method for multimodal computing data in this embodiment of the invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0016] This embodiment provides a counterfactual constraint retrieval and reordering method for multimodal computing data. See also Figure 1 As shown, the counterfactual constraint retrieval and reordering method for computing power multimodal data in this embodiment includes the following steps: Step 1: Obtain multimodal query data of the computing power network to be queried; wherein, in this embodiment, the multimodal query data includes a computing power network image and question text, and the computing power network image is labeled as follows: I The question text is marked as Q ; Step 2: A pre-trained generative model is used to generate predicted answers for the acquired multimodal query data. These predicted answers are then introduced into an external knowledge base for text association analysis. The dual encoder model is trained using positive and difficult-to-bear sample paragraphs through comparative learning based on sample selection, resulting in a well-trained dual encoder model. This dual encoder model features a dual encoder model similarity score, which measures the degree of matching between the multimodal query and candidate text paragraphs in the same semantic vector space. This score serves as the basis for selecting positive and difficult-to-bear sample paragraphs, as well as for subsequent coarse-grained retrieval ranking. The similarity score formula is defined as follows: , sim ( Q , A,P () indicates the question text Q and predicted answers A The inner product similarity score between the combined query and the candidate text segment P (i.e., the aforementioned similarity score). E Q (·) represents a union query encoder. E P (·) represents the paragraph encoder. E Q ( Q , A ) represents the joint query vector. T This represents the transpose of a vector. E P ( P ) represents the candidate paragraph vector; Represents the join query vector E Q ( Q , A ) and candidate paragraph vector E P ( P The result of the inner product operation between ); In this embodiment, the aforementioned pre-trained generative model is modeled based on the autoregressive sequence generation probability. The method by which this pre-trained generative model generates the aforementioned predicted candidate answers is as follows: based on the computing power network image I and the question text Q, and combined with the image description text C, a target word sequence and the conditional probability of the target word sequence are generated; wherein, the target word sequence is labeled A, and A = { a 1, a 2,···, a T The conditional probability of the target word sequence A is: , Let represent the total number of lexical units contained in the target word sequence A, and t represent the current generation position. a t Let a represent the t-th word element in the target word sequence A.<t This represents the sequence of historical lexical units generated before the t-th generation position. This indicates that, given the question text Q, the image description text C, and the historical word sequence a, <t When generating the t-th word a t The conditional probability; The aforementioned “positive sample paragraph” is defined as a historical strategy document paragraph that scores highly in the external knowledge base and contains the true standard answer; the aforementioned “difficult sample paragraph” is defined as a interference strategy document paragraph that scores highly in the external knowledge base, but whose wording is highly co-occurring with the question text and predicted candidate answers and does not contain the true standard answer. Step 3: Using the trained dual-encoder model, perform regular dense vector retrieval on the external knowledge base containing massive amounts of operation and maintenance strategies and historical scheduling logs to conduct coarse-grained retrieval of the question text in the multimodal query data, obtaining a set of candidate text segments; whereby the set of candidate text segments is labeled as... P , P ={ P 1, P 2,···, P N}; Step 4: Extract image patch features from the computing power network image in the multimodal query data using a visual encoder with frozen parameters; wherein, in this embodiment, the aforementioned visual encoder is labeled Ev(·), and after inputting the computing power network image I into the visual encoder Ev(·), the visual feature vector sequence Ev(I)={v1, v2,···, v L The image patch features here are labeled as follows: V , V ∈ R L×D ; L The total length of the sequence containing global context token features and local image patch features is the total number of visual feature vectors in the visual feature vector sequence Ev(I), and D refers to the hidden dimension size of each visual feature vector embedded in the visual feature vector sequence Ev(I). Step 5: An initial semantic slot vector for the extracted image patch features is generated using an input-related initialization strategy. An orthogonally initialized learnable bias is then superimposed on this initial semantic slot vector. Furthermore, the extracted image patch features are aggregated into a preset number of complementary and overlapping semantic slot vectors through a multi-round iterative slot attention mechanism. Here, the orthogonally initialized learnable bias is superimposed on the initial semantic slot vector to avoid the problem that all slots collapse to the same mode in the early stage of training caused by traditional random initialization. Step 6: Obtain the feature vector of the question text from the aforementioned multimodal query data using a pre-trained text encoder, and concatenate the feature vector of the question text with the obtained semantic slot vectors to obtain the concatenated vector; for example, the obtained feature vector of the question text is labeled as q, and the m-th semantic slot vector is labeled as... S m ; Step 7: Input the resulting concatenated vector into a gated network containing a multilayer perceptron and a sigmoid activation function for computation to obtain the importance weight of each semantic slot. Then, a dynamic gating strategy is used to mix the attention weights and a hard truncation mask to generate the final visual selection weights. The method for inputting the resulting concatenated vector into the gated network containing the multilayer perceptron and sigmoid activation function for computation is as follows: g soft,m =σ(W g [q; S m ]+b g );g soft,m σ represents the soft gating weight corresponding to the m-th semantic slot, used to characterize the degree of correlation between the m-th semantic slot and the current question text; σ(·) represents the Sigmoid activation function, used to map the linear transformation result to between 0 and 1; q represents the feature vector of the question text; S m Represents the m-th semantic slot vector; [q; S m ] represents the joint feature vector obtained by concatenating the question text feature vector q with the m-th semantic slot vector Sm; W g This represents the learnable weight matrix in the gated network, used to adjust the concatenated joint feature vector [q; S m Perform a linear transformation; b g W represents the learnable bias term in a gated network. g [q; S m ]+b g This represents the gated scoring result obtained by the gating network after linearly calculating the correlation between the m-th semantic slot and the question text; Step 8: Use visual selection weights to weight all semantic slots to obtain the semantic slots weighted by visual selection weights; wherein, the aforementioned semantic slots correspond one-to-one with the semantic slots weighted by visual selection weights. Step 9: Based on the question text, candidate text paragraph set, and semantic slots weighted by visual selection weights in the acquired multimodal query data, calculate multidimensional scoring features for the multimodal query data, and sort the multidimensional scoring features in descending order based on the scores to obtain updated multidimensional scoring features; wherein, the multidimensional scoring features include text-paragraph similarity, slot-paragraph interaction similarity, and visual-paragraph convergence similarity, and the reordering is obtained after processing by the scoring network; Step 10: Sort all candidate text paragraphs in the candidate text paragraph set in descending order according to their scores based on the updated multidimensional scoring features, and take the top preset number of candidate text paragraphs with the highest scores in the sorting results as the final retrieval results of the corresponding multimodal query data.

[0017] It should be noted that in step 2 of this embodiment, the contrastive learning training employs a counterfactual constraint-based contrastive learning training method. Specifically, the following measures are taken during this counterfactual constraint-based contrastive learning training process: Step a1: Construct a counterfactual complementary path based on complementary weights and a counterfactual erasure path based on all-zero weights; where "based on complementary weights" means using 1-g sel The semantic slots are weighted; "based on all-zero weights" means that the visual selection weights are forcibly set to 0 to block visual information input. Step a2: Construct a loss function containing necessity and sufficiency constraints based on the real path to jointly optimize the model parameters of the dual encoder model; where "real path" refers to the final visual selection weights calculated and output directly using a gating network without any human intervention; the necessity and sufficiency constraints are set as follows: Set a necessity constraint loss: under the counterfactual complementary path, the score difference between positive and negative samples is smaller than the score difference under the true path; Set a sufficiency constraint loss: Under the true path, the score difference between positive and negative samples is greater than the score difference under the counterfactual erase path using all-zero weights.

[0018] The counterfactual complementary path uses visual information deemed non-critical by the dual-encoder model, while the true path uses only the critical visual information selected by the dual-encoder model. By constructing counterfactual complementary paths, counterfactual erasure paths, and true paths, spurious correlations caused by background noise can be eliminated. That is, the accidental co-occurrence of visual background noise and high-frequency words in the text causes non-critical image regions to interfere with the model's decision-making, thus strengthening the causal argument.

[0019] For the actual path, use g sel Calculate the corresponding score Score main For counterfactual complementary paths, use 1-gsel Calculate the corresponding score Score cf_comp For the counterfactual erasure path, set all visual weights to 0 and calculate the score. Score cf_erase ; The aforementioned loss function is constructed based on the three path scenarios described above; whereby this loss function is denoted as... L total , L total =L rank + λ nec ·L nec + λ suf ·L suf + λ div ·L div ;in: L rank The ranking loss is based on Hinge Loss and is used to optimize. Score main The accuracy of the sorting; L nec For the necessity constraint loss, for the positive sample P + and negative sample P - ,Require[ Score main (P + )- Score main (P - )]-[ Score cf_comp (P + )- Score cf_comp (P - )]>δ nec δ nec The positive margin threshold, representing the necessity constraint, is used to limit the minimum improvement in the positive and negative sample discrimination that the true path should provide relative to the counterfactual complementary path; that is, if key visual information is removed, the discrimination of positive and negative samples must decrease significantly. λ nec Indicates the loss due to necessity constraints L nec In the total loss function L total The weighting coefficients in the equation are used to adjust the degree of influence of necessity constraints on the overall optimization objective.

[0020] Lsuf To satisfy the sufficiency constraint loss, it is required that [ Score main (P + )- Score main (P - )]-[ Score cf_erase (P + )- Score cf_erase (P - )]>δ suf δ suf The positive margin threshold, representing the sufficiency constraint, is used to limit the minimum improvement in positive and negative sample discrimination that the true path should provide relative to the counterfactual erasure path. In other words, the discrimination should be significantly improved after adding key visual information. λ suf Indicates sufficiency constraint loss L suf In the total loss function L total The weighting coefficients in the equation are used to adjust the degree of influence of sufficiency constraints on the overall optimization objective.

[0021] L div The slot diversity loss is used to force different slots to focus on different visual modes by minimizing the off-diagonal elements of the cosine similarity matrix between slot vectors, thus preventing homogenization. λ div Indicates slot diversity loss L div In the total loss function L total The weighting coefficients in the equation are used to adjust the degree of influence of slot diversity constraints on the overall optimization objective.

[0022] Furthermore, in step 5 mentioned above, the input-related initialization strategy is set to include steps b1 to b4: Step b1 involves performing average pooling on all image patch features extracted from the computing power network image to obtain a global visual context vector; wherein, for all image patch features... V After average pooling, the resulting global visual context vector is labeled as follows: v mean Of course, the global visual context vector here can also be called a global feature. Step b2: Input the obtained global visual context vector into the initialized multilayer perceptron for mapping processing to obtain the initial slot reference vector; where the mapping here is MLP mapping; Step b3: Construct a set of learnable bias parameters consistent with the number of semantic slots, and use orthogonal initialization to break the symmetry between multiple semantic slots in the early stage of training. Assign values ​​to these learnable bias parameters to obtain the assigned learnable bias parameters; where the learnable bias parameters constructed here are labeled as μ, μ∈ R M×D ; M This represents the initial number of semantic slots; Step b4: Add the initial slot reference vector to the assigned learnable bias parameters to obtain the initial semantic slot vector. The initial semantic slot vector is denoted as... S init The formula for adding the initial slot reference vector to the assigned learnable bias parameters is as follows: ; v mean MLP(·) represents the global visual context vector obtained by average pooling of features from all image patches; MLP(·) represents the multilayer perceptron mapping function; MLP(·) v mean ) indicates that the global visual context vector v mean The initial slot reference vector obtained after inputting into the multilayer perceptron is used to provide shared initialization information related to the current input image for all semantic slots; μ represents a learnable bias parameter consistent with the number of semantic slots, μ∈R M×D M represents the number of semantic slots, and D represents the hidden dimension of each semantic slot vector; ⊕ indicates that the initial slot reference vector MLP( v mean After expanding according to the number of semantic slots M, the fusion operation is performed by adding the learnable bias parameter μ element by element along the corresponding dimension.

[0023] In step 5 of this embodiment, the extracted image patch features are aggregated into a preset number of semantic slot vectors through a multi-round iterative slot attention mechanism as follows: steps c1~c3: Step c1: Construct the slot attention module; Step c2: Use all image patch features as the keys and values ​​of the slot attention module, and use the initial semantic slot vector as the query of the slot attention module; Step c3 involves updating the semantic slot vector through a multi-round iterative competitive attention mechanism to aggregate all image patch features into the semantic slot vector. Specifically, this is achieved through an iterative update using a slot attention module, where image patch features are calculated in each iteration. V With the initial semantic slot S The attention matrix, through a softmax competition mechanism, assigns different regions (such as foreground devices and background clutter) in the image patch features to different initial semantic slots, ultimately resulting in a semantic slot vector with M decoupled features.S , S∈R M×D Among them, calculating image patch features. V With the initial semantic slot S The formula for the attention matrix is: ; Att This represents the attention matrix between the semantic slot vector and the image patch features; Q S This represents the semantic slot vector of the current round. S The query matrix obtained after linear mapping; K V Represents the features of image patches V The key matrix obtained after linear mapping; Key matrix K V Transpose of; This represents the scaling dimension in attention calculation; represents the scaling factor for scaling the dot product result; Softmax(·) represents the normalization function used to convert the correlation score between semantic slots and image patch features into attention weights.

[0024] To filter out visual slots that are truly relevant to the multimodal query data, this embodiment employs a dynamic gating strategy in step 7. Specifically, the dynamic gating strategy is set as follows: steps d1 to d4: Step d1: Calculate all soft attention weights of the gating network output; where the soft attention weights are calculated using g. soft,m =σ(W g [q; S m ]+b g ); Step d2: Select the first K soft attention weights from all the calculated soft attention weights in descending order of their weight values; Step d3 involves determining the K semantic slots corresponding to the K soft attention weights, and generating a binarized hard truncation mask from these K semantic slots. The hard truncation mask generated here can explicitly remove noise during inference. For example, in this embodiment, the hard truncation mask is denoted as g. hard ; Step d4: Set a hybrid weight for the attention weights and the hard truncation mask generated corresponding to the first K soft attention weights; wherein, the hybrid weight increases with the number of iterations during the training process; and the value range of the hybrid weight is [0,1]. For example, in this embodiment, the set hybrid weight is labeled g. sel g sel =(1-λ)·g soft +λ·ghard λ is the hardening rate, which is zero in the early stages of the training process; that is, λ∈[0,1].

[0025] It should be noted that this embodiment, by setting the dynamic gating strategy in this way, not only ensures gradient propagation in the early stage of training, but also achieves final discretization and denoising.

[0026] Specifically, in step 9 of this embodiment, the calculation method of the multidimensional scoring features is as follows: steps e1 to e4: Step e1, calculate text-paragraph similarity: calculate the cosine similarity between the text question vector and the candidate paragraph vector, and use this cosine similarity as the text similarity; for example, the candidate paragraph vector is labeled as... P , P ={ p 1 , p 2 ,···, p k Text-paragraph similarity is tagged as f. txt f txt =Cosine( q , p k Cosine q , p k ) represents the cosine similarity function. q Represents the text question vector. p k This represents the vector of the k-th candidate paragraph. k Indicates the candidate paragraph number; Step e2, calculate slot-paragraph interaction similarity: calculate the cosine similarity between each semantic slot and the candidate paragraph vector, and use visual selection weights to perform a weighted sum of all the obtained cosine similarities, and use the sum of the weighted sums as the slot-paragraph interaction similarity; wherein, the slot-paragraph interaction similarity is denoted as f. slot , Cosine S m , p k ) represents the m-th semantic slot S m With candidate paragraphs p k cosine similarity, Let M be the visual selection weight for the m-th visual selection, and M be the total number of visual selection weights. Step e3, calculate the visual-paragraph convergence similarity: Use visual selection weights to perform a weighted summation of all semantic slots to obtain a converged visual vector; then calculate the cosine similarity between this converged visual vector and the candidate paragraph vectors, and use this calculated cosine similarity as the visual-paragraph convergence similarity; where the visual-paragraph convergence similarity is denoted as f. pool , ; Step e4 involves concatenating the obtained text-paragraph similarity, slot-paragraph interaction similarity, and visual-paragraph convergence similarity features to obtain a concatenated similarity score. This concatenated similarity score is then input into a fully connected scoring network for scoring, resulting in multi-dimensional scoring features. Specifically, the text-paragraph similarity, slot-paragraph interaction similarity, and visual-paragraph convergence similarity are scaled using a learnable non-linear scaling mechanism before being input into the fully connected layer to obtain the final score. main The final score is calculated using the formula Score. main =W s· φ([f txt ;f slot ;f pool ])+bs;Score main This represents the final score corresponding to the candidate text paragraph under the actual path; f txt Indicates text-paragraph similarity; f slot Indicates slot-paragraph interaction similarity; f pool Indicates visual-paragraph convergence similarity; [f txt ;f slot ;f pool ] represents the joint score feature vector obtained by concatenating text-paragraph similarity, slot-paragraph interaction similarity, and visual-paragraph convergence similarity according to feature dimensions; φ(·) represents a learnable non-linear scaling function used to perform non-linear transformation on the concatenated joint score feature vector; W s represents the learnable weight parameters in the fully connected scoring layer, used to weight and combine different similarity features in the joint scoring feature vector; bs represents the learnable bias term in the fully connected scoring layer, used to adjust the overall bias of the final score; W s· φ([f txt ;f slot ;f pool ])+bs represents the candidate text paragraph scoring result obtained by linearly transforming the concatenated joint scoring feature vector by the fully connected scoring layer.

[0027] Although preferred embodiments of the present invention have been described in detail above, it should be clearly understood that various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A counterfactual constraint retrieval and reordering method for multimodal computing data, characterized in that: Includes the following steps: Step 1: Obtain multimodal query data for the computing power network to be queried; Step 2: Use the pre-trained generative model to generate a predicted answer for the acquired multimodal query data, and introduce the predicted answer into an external knowledge base for text association analysis. Use positive sample paragraphs and hard negative sample paragraphs to perform comparative learning training on the dual encoder model based on sample selection to obtain the trained dual encoder model. Step 3: Use the trained dual encoder model to perform regular dense vector retrieval on the external knowledge base containing massive operation and maintenance strategies and historical scheduling logs for question text in multimodal query data to perform coarse-grained retrieval and obtain a set of candidate text paragraphs. Step 4: Use a visual encoder with frozen parameters to extract image patch features from the computing power network image in the multimodal query data; Step 5: An initial semantic slot vector for the extracted image patch features is generated using an input-related initialization strategy. An orthogonally initialized learnable bias is superimposed on the initial semantic slot vector. The extracted image patch features are aggregated into a set number of complementary and overlapping semantic slot vectors through a multi-round iterative slot attention mechanism. Step 6: Obtain the feature vector of the question text in the multimodal query data through the pre-trained text encoder, and concatenate the feature vector of the question text with the obtained semantic slot vectors to obtain the concatenated vector. Step 7: Input the concatenated vector into a gated network containing a multilayer perceptron and a sigmoid activation function for computation to obtain the importance weight of each semantic slot. Then, use a dynamic gating strategy to mix soft gating weights and hard truncation masks to generate the final visual selection weights. Step 8: Use visual selection weights to weight all semantic slots to obtain the semantic slots weighted by visual selection weights; where each semantic slot corresponds one-to-one with the semantic slots weighted by visual selection weights. Step 9: Based on the question text, candidate text paragraph set and semantic slots weighted by visual selection weights in the acquired multimodal query data, calculate the multidimensional scoring features for the multimodal query data, and sort the multidimensional scoring features in descending order based on the scores to obtain the updated multidimensional scoring features. Step 10: Sort all candidate text paragraphs in the candidate text paragraph set in descending order according to their scores based on the updated multidimensional scoring features, and take the top preset number of candidate text paragraphs with the highest scores in the sorting results as the final retrieval results of the corresponding multimodal query data.

2. The counterfactual constraint retrieval and reordering method for multimodal computing data according to claim 1, characterized in that, In step 5, the input-related initialization strategy is set as follows: Step b1: Perform average pooling on all image patch features of the extracted computing power network image to obtain the global visual context vector; Step b2: Input the obtained global visual context vector into the initialized multilayer perceptron for mapping processing to obtain the initial slot reference vector; Step b3: Construct a set of learnable bias parameters that are consistent with the number of semantic slots, and use an orthogonal initialization method to break the symmetry between multiple semantic slots in the early stage of training. Assign values ​​to the learnable bias parameters to obtain the assigned learnable bias parameters. Step b4: Add the initial slot reference vector to the assigned learnable bias parameters to obtain the initial semantic slot vector.

3. The counterfactual constraint retrieval and reordering method for multimodal computing data according to claim 2, characterized in that, In step 5, the extracted image patch features are aggregated into a preset number of semantic slot vectors through a multi-round iterative slot attention mechanism as follows: Step c1: Construct the slot attention module; Step c2: Use all image patch features as the keys and values ​​of the slot attention module, and use the initial semantic slot vector as the query of the slot attention module; Step c3 involves updating the semantic slot vector through a multi-round iterative competitive attention mechanism to aggregate all image patch features into the semantic slot vector.

4. The counterfactual constraint retrieval and reordering method for multimodal computing data according to claim 2, characterized in that, In step 7, the dynamic gating strategy is set as follows: Step d1: Calculate all soft attention weights of the gating network output; Step d2: Select the first K soft attention weights from all the calculated soft attention weights in descending order of their weight values; Step d3: Determine the K semantic slots corresponding to the K soft attention weights, and generate a binary hard truncation mask from the K semantic slots. Step d4: Set a hybrid weight for the attention weight and the hard truncation mask generated corresponding to the first K soft attention weights; wherein, the hybrid weight increases with the number of iterations during the training process; and the value range of the hybrid weight is [0,1].

5. The counterfactual constraint retrieval and reordering method for multimodal computing data according to claim 1, characterized in that, In step 9, the multidimensional scoring features are calculated as follows: Step e1, calculate text-paragraph similarity: calculate the cosine similarity between the text question vector and the candidate paragraph vector, and use the cosine similarity as the text similarity; Step e2, calculate slot-paragraph interaction similarity: calculate the cosine similarity between each semantic slot and the candidate paragraph vector, and use visual selection weights to perform a weighted sum of all the obtained cosine similarities, and use the sum of the weighted sum as the slot-paragraph interaction similarity; Step e3, calculate visual-paragraph convergence similarity: use visual selection weights to perform weighted summation on all semantic slots to obtain converged visual vectors, and calculate the cosine similarity between the converged visual vector and the candidate paragraph vectors, and use the calculated cosine similarity as the visual-paragraph convergence similarity. Step e4: The obtained text-paragraph similarity, slot-paragraph interaction similarity and visual-paragraph convergence similarity are concatenated to obtain the concatenated similarity. The concatenated similarity is then input into a fully connected scoring network for scoring processing to obtain multidimensional scoring features.

6. The counterfactual constraint retrieval and reordering method for multimodal computing data according to claim 1, characterized in that, In step 2, during the contrastive learning training process based on sample selection: Step a1: Construct a counterfactual complementary path based on complementary weights and a counterfactual erasure path based on all-zero weights; Step a2: Construct a loss function containing necessity constraints and sufficiency constraints based on the real path to jointly optimize the parameters of the dual encoder model; Set a necessity constraint loss: under the counterfactual complementary path, the score difference between positive and negative samples is smaller than the score difference under the true path; Set a sufficiency constraint loss: Under the true path, the score difference between positive and negative samples is greater than the score difference under the counterfactual erase path using all-zero weights.

7. The counterfactual constraint retrieval and reordering method for multimodal computing data according to claim 6, characterized in that, The counterfactual complementary path uses visual information that is determined to be non-critical by the dual encoder model, while the truth path uses only the critical visual information selected by the dual encoder model.

Images