Cross-modal sequence verification method and system based on hypergraph

Abstract

The invention provides a hypergraph-based cross-modal sequence verification method and a hypergraph-based cross-modal sequence verification system, which are used for evaluating whether a sequential task is accurately executed according to a language-based rule or not. The sequence matching method based on the hypergraph comprises the steps that nodes of a cross-modal heterogeneous graph are sent into a Transform model, a connection relation serves as an attention mask, high-level representation of each node is generated, and the high-level representation of each node comprises a video data summary node, a text description summary node and a cross-modal summary node; and inputting the high-level representation of the node into a binary classifier to obtain binary classification prediction, and completing task verification. Through an innovative hypergraph alignment mechanism, the action description can be efficiently aligned with a plurality of video clips, so that the accuracy of task verification is improved. Through the technical means, the cross-modal task verification accuracy can be effectively improved, and the cross-modal task verification method has wide application prospects especially in task verification scenes for processing complex multi-modal and multi-video clips.

Description

Technical Field

[0001] This invention relates to the field of cross-modal task verification, addressing the problem of cross-modal information alignment in video and text pairing. It proposes a novel method and system for establishing a more accurate alignment between video and text descriptions, addressing the shortcomings of existing methods in handling the complex dependencies and temporal ordering between task steps and video clips. Background Art

[0002] Cross-modal task verification is an important research direction in video understanding. It aims to evaluate the alignment between a video and text of a multi-step procedural task. This task requires verifying that the sequence of actions in the video accurately corresponds to the steps described in the text, ensuring that the content in both modalities conveys the same information and is consistent. This task requires not only identifying corresponding steps but also understanding the temporal order and semantic dependencies between these steps.

[0003] Based on videos, some existing methods generate action sequences from target images, some learn sequential information from similar paired videos, and some focus on aligning key steps between different videos. However, these methods often rely on strictly aligned captions or timestamps to understand the video, simplifying the alignment process between the two modalities. However, cross-modal annotation is time-consuming and impractical because in real-world scenarios, video and text often lack synchronous alignment. Therefore, text-based cross-modal task verification for unaligned video and text pairs is a crucial research area.

[0004] To address the misalignment between video and text, researchers have gradually turned to graphical models to model the complex relationships between them. Graph models provide powerful tools for capturing the temporal dependencies and complex relationships between video and text, enabling a more nuanced representation of the interaction between the two modalities. However, traditional directed graph models often represent relationships by simply connecting two nodes, which results in a loss of complexity and high-order structural information in the system, making it difficult to effectively model deeper dependencies and the multi-level complexity of real-world tasks.

[0005] Specifically, existing methods face the following challenges: First, relying solely on the temporal order structure is insufficient to capture the complex underlying relationships between the two modalities, and often ignores the implicit relationships between nodes. For example, some actions in a task step are semantically related, but may be considered independent steps in traditional methods. Second, existing methods usually assume that the text description corresponds one-to-one to a single video clip, but in practical applications, this assumption does not hold. In most cases, a task step may correspond to multiple video clips, forming a one-to-many relationship, and existing methods often cannot effectively handle this situation.

[0006] To overcome these problems, hypergraphs have emerged as an effective mathematical framework for capturing and expressing complex, high-order relationships in networks. Unlike traditional graphs, hyperedges in hypergraphs can connect any number of nodes simultaneously, enabling better modeling of complex relationships between multiple entities, such as high-order interactions involving multiple entities in the same interaction. Hypergraph neural networks can capture complex, high-order, nonlinear relationships between hypergraph nodes, alleviating the need for manual feature annotation and supporting end-to-end learning. Research has shown that hypergraph neural networks outperform traditional graph neural networks in handling complex dependencies, and are therefore widely used in cross-modal tasks such as video and text alignment.

[0007] To address these issues, we propose a hypergraph-based sequence matching method. By incorporating a hypergraph structure, this method can more accurately represent cross-modal alignment between video and text. Experimental results demonstrate that our method outperforms existing state-of-the-art methods on multiple datasets. Summary of the Invention

[0008] The purpose of the present invention is to address the defects of existing multimodal video text matching verification, construct a hypergraph-based model framework, and realize the matching judgment of temporal video and text rules through a temporal supplementary hypergraph module and a step-by-step hypergraph modeling module.

[0009] The technical solutions of the present invention are as follows: Get video data and corresponding text description; Extracting video features of the video data, graph-modeling the video features, and arranging them in chronological order; the representation of each frame of the video is regarded as a node, and the relationship between the nodes reflects the time sequence in the video, thereby structuring the temporal information of the video into a graph structure; Segmenting the text description into individual steps, converting each step in the text description into a feature vector, and constructing a text graph based on the logical relationship between the steps; According to the dynamic programming algorithm based on hypergraph theory, based on the graph structure and text graph, the connection between video nodes and text nodes is established to form a cross-modal heterogeneous graph; Feeding the nodes of the cross-modal heterogeneous graph into a Transformer model and using the connection relationship as an attention mask to generate a high-level representation of each node, wherein the high-level representation of the node includes a summary node of the video data, a summary node of the text description, and a cross-modal summary node; The high-level representation of the node is input into the binary classifier to obtain a binary classification prediction and complete the task verification.

[0010] The task verification method comprises the following steps: Step 1: Select a training dataset, including videos and corresponding text descriptions; Step 2: Extract video features of the video using the pre-trained Clip model; Step 3: Use the pre-trained T5 model to segment the text description into individual steps; Step 4: Use the pre-trained Clip model to extract features of the text steps. Convert each step in the text description into a feature vector that captures the semantic information of each text step. Based on the logical context of the text description, create a corresponding text graph that reflects the order and dependencies between different steps. Step 5: Graph modeling is performed on the video features and arranged in chronological order. The representation of each frame of the video is regarded as a node, and the relationship between the nodes reflects the time sequence in the video, thereby structuring the temporal information of the video into a graph structure. Step 6: Using a dynamic programming algorithm based on hypergraph theory, establish connections between video nodes and text nodes to form a heterogeneous graph of video and text. Connect the video nodes and text nodes to form a heterogeneous graph structure. Step 7: Feed the nodes of the heterogeneous graph into the Transformer model, and use the connectivity of the graph as an attention mask; the Transformer model focuses on the important relationships between different nodes during the learning process; Step 8: The Transformer model generates a high-level representation of each node through its self-attention mechanism, which includes summary nodes for video and text as well as cross-modal summary nodes; Step 9: The high-level representation is fed into a binary classifier, which determines whether the video and text description are correctly matched, completing the multimodal verification task.

[0011] The present invention also includes a cross-modal task verification system based on a hypergraph, wherein the task verification system includes the following units: An acquisition unit, used to acquire video data and corresponding text description; a graph structure construction unit, configured to extract video features of the video data, perform graph modeling on the video features, and arrange the features in chronological order; wherein the representation of each frame of the video is regarded as a node, and the relationship between the nodes reflects the temporal order in the video, thereby structuring the temporal information of the video into a graph structure; a text graph construction unit, configured to segment the text description into individual steps, convert each step in the text description into a feature vector, and construct a text graph based on the logical relationship between the steps; The cross-modal heterogeneous graph construction unit establishes connections between video nodes and text nodes based on the graph structure and text graph using a dynamic programming algorithm based on hypergraph theory to form a cross-modal heterogeneous graph; The task verification unit is used to input the nodes of the cross-modal heterogeneous graph into the Transformer model and use the connection relationship as an attention mask to generate a high-level representation of each node. The high-level representation of the node includes a summary node of the video data, a summary node of the text description, and a cross-modal summary node; the high-level representation of the node is input into a binary classifier to obtain a binary classification prediction to complete the task verification.

[0012] The beneficial effects of the present invention are as follows: This invention, through hypergraph-based modeling, effectively captures high-level interactions within and between video and text modalities, transcending the binary relationship limitations of traditional graph models. It also provides an effective modeling mechanism for the one-to-many relationship between video clips and text descriptions, improving the accuracy of task verification.

[0013] This paper supplements the hypergraph module with a time-series approach and employs a hierarchical sequential hyperedge construction method. This approach effectively mines implicit relationships between non-adjacent nodes while preserving the sequential relationships between task nodes. This approach not only enhances the sequential dependencies between task nodes but also reveals deeper structured relationships.

[0014] This paper innovatively proposes a hypergraph alignment mechanism through a step-by-step hypergraph modeling module, enabling the model to efficiently align action descriptions with multiple video clips. This mechanism addresses the one-to-many relationship that traditional methods struggle to address, significantly improving the accuracy of task verification, especially for cross-modal alignment in complex scenarios.

[0015] Through these technical approaches, the present invention not only improves the accuracy of task verification but also effectively handles complex task verification scenarios, such as multimodal and multi-video segment verification. In particular, the present invention demonstrates a significant advantage in text-based video-text alignment, outperforming existing methods across multiple task verification subtasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments, wherein: Figure 1 Flowchart for the implementation of the present invention; Figure 2 Schematic diagram of the hypergraph-based cross-modal sequence verification framework proposed in this invention; Figure 3 This figure shows the effect of video text matching on the EgoTV and CTV datasets of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0018] The CLIP model, a multimodal model proposed by OpenAI, jointly trains image and text encoders, mapping the two modalities into a shared vector space. It optimizes the similarity of image-text pairs through contrastive learning and supports zero-shot image classification (e.g., direct inference using text labels).

[0019] The T5 model (Text-to-Text Transfer Transformer) is a unified framework for text generation proposed by Google. It converts various NLP tasks (translation, summarization, and classification) into a "text-to-text" format. Based on the Transformer encoder-decoder, it distinguishes task types by prefixes (e.g., "translate EN to DE:").

[0020] like Figure 1 As shown, the present invention can be implemented according to the steps shown in the figure during implementation.

[0021] Step 1: Select a training dataset, including videos and corresponding text descriptions; Step 2: Use the pre-trained Clip model to extract features from the input video data. The goal of this step is to map the video data into a feature space for subsequent processing. Step 3: Use the pre-trained T5 model to segment the text description into individual steps. The T5 model excels at text generation tasks and extracts the independent steps of each task by segmenting the text. Step 4: Use the pre-trained Clip model to extract features of the text steps. Convert each step in the text description into a feature vector. These feature vectors capture the semantic information of each text step. Next, based on the logical context of the text description, create a corresponding text graph that reflects the order and dependencies between different steps. Step 5: Model the video features into a graph and arrange them in chronological order. Each frame of the video is represented as a node, and the relationship between nodes reflects the temporal order of the video. In this way, the temporal information of the video is structured into a graph structure, which facilitates subsequent processing and analysis. Step 6: Using an improved dynamic programming algorithm based on hypergraph theory, connections are established between video nodes and text nodes to form a heterogeneous graph of video and text. In this step, video nodes and text nodes are connected to form a complex heterogeneous graph structure. Hypergraph theory allows us to capture high-order dependencies between video and text, and the dynamic programming algorithm helps optimize the connections between these nodes for more accurate cross-modal matching. Step 7: Feed the nodes of the heterogeneous graph into the Transformer model, and use the graph's connectivity as an attention mask. The Transformer model is capable of handling this complex graph structure and focusing on the important relationships between different nodes during the learning process. By using the graph's connectivity as an attention mask, the Transformer model can effectively focus on the key dependencies between video and text, thereby improving model performance. Step 8: The Transformer model generates a video summary node, a text summary node, and a video-text connection summary node. During this process, the Transformer model uses its self-attention mechanism to generate high-level representations for each node, including summary nodes for video and text, as well as a cross-modal summary node. These nodes represent the core information of their respective modalities while also capturing the connections between video and text. Step 9: The final summary node is fed into a binary classifier to produce a binary prediction. By combining the final representations of the video and text, the binary classifier can make a final prediction, namely whether the video and text description are correctly matched, thus achieving a multimodal verification task.

[0022] like Figure 2 As shown, the core idea of this invention is to achieve a one-to-many text-video connection based on a hypergraph. The Transformer model is used to learn the high-order interactive relationship between text and vision, so that the model can obtain a prediction result for video-text matching. In the step-by-step hypergraph modeling module, we simulate the real-world relationship where one text step corresponds to multiple video sequence frames by introducing a hypergraph. In the temporal supplementary hypergraph module, we pay attention to the possible implicit relationships between non-adjacent nodes by supplementing the connection relationship between them.

[0023] like Figure 3 As shown, the cross-modal sequence verification method based on hypergraph proposed in the present invention can better identify the alignment relationship of cross-modal information and provide good results for timing information matching.

[0024] A cross-modal sequence verification method based on hypergraph, combined with Figure 2As a preferred technical solution, step 5 of the present invention includes in more detail: This step applies the temporal supplementary hypergraph module we proposed. Traditional sequential modeling often only focuses on the relationship between adjacent nodes. By adding corresponding edges between non-adjacent nodes, we can make the model pay attention to the implicit relationship between non-adjacent nodes. Through the attention mechanism of the Transformer model, it will learn the nodes between non-adjacent edges to focus on, thereby supplementing the implicit relationship information. In this step, the adjacency matrix of the corresponding node is initialized as follows: in, It is an upper triangular matrix used to capture the linear time-ordered connections between nodes of the same type. The elements above the diagonal in the first row are set to 1, reflecting the immediate influence of the node on the subsequent nodes. Expressed as ,in is an upper triangular matrix of all ones, is the identity matrix. and After adding, The first row of elements above the mid-diagonal is 2, which is a high weight and emphasizes the main sequential connections between nodes, while the other non-zero elements are 1, which supplement the implicit relationship information.

[0025] As a preferred technical solution, step 6 includes in more detail: To connect the video clips and text steps in the graph, we first follow the video text alignment algorithm based on dynamic programming. This algorithm aims to optimize the matching process by calculating the alignment cost based on the similarity measure. Specifically, we define the cost matrix and the path matrix , which represent the optimal cost of each position and the corresponding backtracking path respectively. The dynamic programming-based algorithm follows the following state transition equation: in, Represents a video text pair Finally, the algorithm uses The path recorded at is taken as the optimal alignment.

[0026] However, this algorithm only achieves one-to-one alignment, which conflicts with the requirement of connecting multiple similar video clips to a single text step. To this end, we further relax the constraint and consider suboptimal paths. Specifically, we not only consider The value at (indicating the optimal alignment from the last video-text pair to the beginning) also evaluates The value at The number of is set according to the video partitioning and task complexity, indicating the number of video clips in a group connected by hyperedges. As an effective alternative depends on and We introduce a tolerance for the difference between ,If the difference is smaller than the tolerance, the paths are considered acceptable.

[0027] As a preferred technical solution, step 8 includes in more detail: the video summary node is sequentially connected to the end of the last video frame, the text summary node is connected to the end of the last step (or connected to multiple last parallel steps at the same time), and the video-text connection summary node is simultaneously connected to the end of the video summary node and the text summary node. Through the attention mechanism and information transmission mechanism of the Transformer model, the video-text connection summary node will integrate the timing information of the video, the timing information of the text, and the cross-modal matching information of the video and text.

[0028] In order to test the model's cross-modal sequence verification effect, the present invention provides different possible matching types for different test situations. On the EgoTV dataset, the test types are new tasks, new steps, new environments, and text simplification. The new task test type represents tasks that have never been encountered during model training. This type can well reflect the generalization of the model. The new step test type refers to the task of swapping the order of certain specific steps in the task during training. This type can test whether the model can determine the relationship between the steps of the serialized task and whether it can recognize incorrect task sequences. The new environment test type refers to changing the operating scenario of the task. This type can test whether the model has a deep understanding of the semantics of the video text and whether the model can recognize the same task in different environments. Text simplification tests whether the model still has excellent results after simplifying the text rules. All tests are evaluated using the F1 score. On the CTV dataset, the test types are new tasks, new steps, and changing steps and deleting steps. The definitions of new tasks and new steps are the same as EgoTV, while changing steps refers to modifying a text step and deleting steps refers to deleting a text step.

[0029] The present invention also includes a cross-modal task verification system based on a hypergraph, wherein the task verification system includes the following units: An acquisition unit, used to acquire video data and corresponding text description; a graph structure construction unit, configured to extract video features of the video data, perform graph modeling on the video features, and arrange the features in chronological order; wherein the representation of each frame of the video is regarded as a node, and the relationship between the nodes reflects the temporal order in the video, thereby structuring the temporal information of the video into a graph structure; a text graph construction unit, configured to segment the text description into individual steps, convert each step in the text description into a feature vector, and construct a text graph based on the logical relationship between the steps; The cross-modal heterogeneous graph construction unit establishes connections between video nodes and text nodes based on the graph structure and text graph using a dynamic programming algorithm based on hypergraph theory to form a cross-modal heterogeneous graph; The task verification unit is used to input the nodes of the cross-modal heterogeneous graph into the Transformer model and use the connection relationship as an attention mask to generate a high-level representation of each node. The high-level representation of the node includes a summary node of the video data, a summary node of the text description, and a cross-modal summary node; the high-level representation of the node is input into a binary classifier to obtain a binary classification prediction to complete the task verification.

[0030] Three scenario cases are given below to more intuitively demonstrate the effects and advantages of the present invention: The first scenario uses the aforementioned approach to train and test the model on the EgoTV dataset, comparing its performance with previous methods. The EgoTV data is divided into five subsets: a training set (10,726 samples) for general task training; a novel task set (1,080 samples); a novel step set (700 samples); a novel environment set (2,164 samples); and a text reduction set (676 samples). This data partitioning ensures a comprehensive evaluation of the model across a variety of unseen and challenging scenarios, supporting extensive generalization testing. The test results are shown in Table 1.

[0031] Table 1 Verification results on the EgoTV dataset The second scenario uses the aforementioned approach to train and test the model on the CTV dataset, comparing its performance with previous methods. The CTV dataset is designed to evaluate the performance of task verification models in real-world videos. It uses videos from the CrossTask dataset, which contains 18 task categories, with approximately 150 videos per category. The CTV dataset is divided into five subsets: 5,308 training samples; 4,514 new task samples; 4,606 new step samples; 4,410 modified step samples; and 4,513 deleted step samples. The CTV dataset complements the EgoTV dataset by providing a more complex and real-world task verification framework, making it suitable for evaluating task verification models.

[0032] Table 2 Verification results on the CTV dataset The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-modal task verification method based on hypergraph, characterized in that: The task verification method comprises the following steps: Get video data and corresponding text description; Extracting video features of the video data, graph-modeling the video features, and arranging them in chronological order; the representation of each frame of the video is regarded as a node, and the relationship between the nodes reflects the time sequence in the video, thereby structuring the temporal information of the video into a graph structure; Segmenting the text description into individual steps, converting each step in the text description into a feature vector, and constructing a text graph based on the logical relationship between the steps; According to the dynamic programming algorithm based on hypergraph theory, based on the graph structure and text graph, the connection between video nodes and text nodes is established to form a cross-modal heterogeneous graph; The nodes of the cross-modal heterogeneous graph are fed into the Transformer model, and the connection relationship is used as an attention mask to generate a high-level representation of each node. The high-level representation of the node includes a summary node of the video data, a summary node of the text description, and a cross-modal summary node; the high-level representation of the node is input into a binary classifier to obtain a binary classification prediction, thereby completing the task verification.

2. A method for cross-modal task verification based on a hypergraph according to claim 1, characterized in that: The task verification method comprises the following steps: Step 1: Select a training dataset, including videos and corresponding text descriptions; Step 2: Extract video features of the video using the pre-trained Clip model; Step 3: Use the pre-trained T5 model to segment the text description into individual steps; Step 4: Use the pre-trained Clip model to extract features of the text steps. Convert each step in the text description into a feature vector that captures the semantic information of each text step. Based on the logical context of the text description, create a corresponding text graph that reflects the order and dependencies between different steps. Step 5: Model the video features into a graph and arrange them in chronological order. The representation of each frame of the video is regarded as a node. The relationship between the nodes reflects the time sequence in the video, thereby structuring the temporal information of the video into a graph structure. Step 6: Using a dynamic programming algorithm based on hypergraph theory, establish connections between video nodes and text nodes to form a heterogeneous graph of video text. Connect the video nodes and text nodes to form a heterogeneous graph structure. Step 7: Feed the nodes of the heterogeneous graph into the Transformer model, and use the connectivity of the graph as an attention mask; the Transformer model focuses on the important relationships between different nodes during the learning process; Step 8: The Transformer model generates a high-level representation of each node through its self-attention mechanism, which includes summary nodes for video and text as well as cross-modal summary nodes; Step 9: The high-level representation is fed into a binary classifier, which determines whether the video and text description are correctly matched, completing the multimodal verification task.

3. A hypergraph-based cross-modal task verification method according to claim 1, characterized in that: In step 5, the adjacency matrix of the node is initialized as follows: in, It is an upper triangular matrix used to capture the linear time-ordered connections between nodes of the same type. The elements above the diagonal in the first row are set to 1, reflecting the immediate influence of the node on the subsequent nodes. Expressed as ,in is an upper triangular matrix of all ones, is the identity matrix. and After adding, The first row of elements above the mid-diagonal is 2, which is a high weight and emphasizes the main sequential connections between nodes, while the other non-zero elements are 1, which supplement the implicit relationship information.

4. A hypergraph-based cross-modal task verification method according to claim 2, characterized in that: In step 6, the dynamic programming algorithm based on hypergraph theory optimizes the matching process by calculating the alignment cost based on the similarity metric, defining the cost matrix and the path matrix , which represent the optimal cost of each position and the corresponding backtracking path respectively; the dynamic programming-based algorithm follows the following state transition equation: in, Represents a video text pair Finally, the algorithm uses The path recorded at is taken as the optimal alignment; Assessment Completed After the value of The value at The number of is set according to the video partition and task complexity, K represents the number of video clips in a group connected by hyperedges; whether to As an effective alternative depends on and Set a tolerance for the difference between If the difference is less than the tolerance , then these paths are considered acceptable.

5. A hypergraph-based cross-modal task verification method according to claim 2, characterized in that: In step 8, the video summary node is sequentially connected to the end of the last video frame, the text summary node is connected to the end of the last step, or multiple last parallel steps are connected at the same time, and the video-text connection summary node is connected to the end of the video summary node and the text summary node at the same time. Through the attention mechanism and information transmission mechanism of the Transformer model, the video-text connection summary node will integrate the timing information of the video, the timing information of the text, and the cross-modal matching information of the video and text.

6. A hypergraph-based cross-modal task verification system, characterized in that: The task verification system includes the following units: An acquisition unit, used to acquire video data and corresponding text description; a graph structure construction unit, configured to extract video features of the video data, perform graph modeling on the video features, and arrange the features in chronological order; wherein the representation of each frame of the video is regarded as a node, and the relationship between the nodes reflects the temporal order in the video, thereby structuring the temporal information of the video into a graph structure; a text graph construction unit, configured to segment the text description into individual steps, convert each step in the text description into a feature vector, and construct a text graph based on the logical relationship between the steps; The cross-modal heterogeneous graph construction unit establishes connections between video nodes and text nodes based on the graph structure and text graph using a dynamic programming algorithm based on hypergraph theory to form a cross-modal heterogeneous graph; The task verification unit is used to input the nodes of the cross-modal heterogeneous graph into the Transformer model and use the connection relationship as an attention mask to generate a high-level representation of each node. The high-level representation of the node includes a summary node of the video data, a summary node of the text description, and a cross-modal summary node; the high-level representation of the node is input into a binary classifier to obtain a binary classification prediction to complete the task verification.

Images