A Deep Learning-Based Multimodal Data Pairing Method and System

Abstract

This invention provides a deep learning-based multimodal data pairing method and system, relating to the field of data processing technology. The method includes acquiring a multi-frame sequence of video and target text, extracting overlapping frame sets and normalized text sequences respectively; performing spatiotemporal feature extraction and text dependency encoding to obtain video temporal vector sequences and text vector sequences; performing cross-modal alignment search to construct a set of monotonic matching paths; calculating the semantic and action entity relationship consistency scores of paired elements on the paths to obtain a comprehensive score; and determining the video-text alignment relationship based on the optimal path. This method achieves accurate matching of video and text, improving cross-modal retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data pairing method and system based on deep learning. Background Technology

[0002] With the widespread adoption of smartphones, smart TVs, and other devices, users' demands for retrieving and understanding video content are increasing daily. Against this backdrop, multimodal data pairing and alignment technologies have become a research hotspot, particularly the pairing and alignment between video and text. This technology can help achieve functions such as accurate video content retrieval, video content understanding, automatic subtitle generation, and video summary extraction, and has broad application prospects in media content production, education and training, and intelligent recommendation systems.

[0003] Currently, multimodal data pairing techniques mainly rely on keyframe-based feature extraction and bag-of-words text models for simple matching. Traditional methods typically employ visual feature extraction combined with text embedding, calculating a similarity matrix to achieve a rough correspondence between video frames and text. With the development of deep learning technology, neural network-based feature representation and attention mechanisms have been introduced into the field of multimodal pairing, but many technical difficulties and challenges still exist.

[0004] Existing video-text matching methods are generally unable to effectively process temporal information. They can only identify the correspondence between static scenes and text, making it difficult to capture the accurate mapping between dynamic behaviors in videos and text descriptions, resulting in low matching accuracy for action-related descriptions.

[0005] Most matching techniques do not consider the syntactic structure and dependency relations of the text, but only use simple word vector representations, ignoring the impact of semantic relations on matching accuracy, and perform poorly, especially in complex narratives and long text scenarios.

[0006] Existing methods typically employ a global matching strategy, lacking the ability to fine-grained alignment between video and text content. This makes it difficult to accurately identify which text segments correspond to specific segments in the video, thus affecting the accuracy and efficiency of downstream tasks such as video Q&A and content moderation. Summary of the Invention

[0007] The embodiments of the present invention provide a multimodal data pairing method and system based on deep learning, which can solve the problems in the prior art.

[0008] A first aspect of the present invention provides a deep learning-based multimodal data pairing method, comprising:

[0009] The original multi-frame sequence of the video to be processed and the target text are obtained. For each frame in the original multi-frame sequence, temporal segments are extracted at a fixed sampling interval to form a set of frame groups containing overlapping adjacent frames. Sentence segmentation and lexical normalization are performed on the target text, word order is preserved and the occurrence position index of each word is recorded to form an aligned text sequence.

[0010] Spatiotemporal features are extracted from the frame set to obtain a video temporal vector sequence corresponding to the time index; simultaneously, embedding mapping and dependency encoding are performed on the text sequence, and syntactic dependency constraints are injected into the word vectors using graph structure message passing operations to obtain a text vector sequence corresponding to the word order index.

[0011] While maintaining the temporal order of video and text, cross-modal alignment search is performed. A hierarchical candidate expansion strategy is adopted to construct a set of monotonic matching paths between the temporal sequence of video vectors and the sequence of text vectors. The cumulative cost of the matching paths is used as the sorting criterion to output candidate paths.

[0012] Calculate the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtain the comprehensive score of the candidate path under multi-objective weighted constraints;

[0013] The alignment relationship between video temporal segments and text segments is determined by the candidate path with the highest comprehensive score. Based on this alignment relationship, the text segments are adaptively weighted to obtain the text matching result for the video content.

[0014] The text sequence is embedded and mapped using dependency encoding. syntactic dependency constraints are injected into word vectors using graph-structured message passing operations, resulting in a text vector sequence corresponding to the word order index, including:

[0015] An initial term feature set and index mapping table are generated for the text sequence using a concatenation strategy of character-level segmentation and word-level segmentation;

[0016] Based on the initial term feature set and the index mapping table, an embedding mapping is performed through a mapping function to generate an initial word vector sequence of the same length as the word order index; the initial word vector sequence and its corresponding word order index and character range are stored together to form an initial word vector table with index tags;

[0017] Based on the initial word vector table, syntactic dependency analysis is performed on the text sequence to obtain a syntactic graph structure. The syntactic graph structure and the initial word vector table are passed forward and backward in a fixed number of rounds. After all rounds of passing are completed, the intermediate word vector sequence corresponding to the word order index and the corresponding index consistency record are output.

[0018] The intermediate word vector sequence is rearranged and checked according to the word order index. After the check is passed, a text vector sequence that corresponds to the word order index and is consistent in semantics and syntax is output.

[0019] Based on the initial word vector table, syntactic dependency analysis is performed on the text sequence to obtain a syntactic graph structure. The syntactic graph structure and the initial word vector table are then passed forward and backward in fixed rounds. After all rounds of passing are completed, the intermediate word vector sequence corresponding to the word order index and the associated index consistency record are output, including:

[0020] The initial word vector table is subjected to index integrity verification, index conflicts are eliminated and missing character range records are filled in to obtain an initial input baseline with consistent indexes; based on the initial input baseline, syntactic dependency analysis is performed on the text sequence to obtain the syntactic graph structure;

[0021] Based on the syntactic graph structure and the initial word vector table, forward propagation based on incoming edges and backward propagation based on outgoing edges are performed in a preset fixed number of rounds. In each round of propagation, a linearly differentiable transformation is performed on the vector propagating along each dependency edge, and the multi-path information reaching the same node is aggregated to obtain the round aggregated vector of that node. The round aggregated vector is aligned and cached according to the index mapping table and the word order index of that node in the initial word vector table to generate a round verification record.

[0022] After completing the fixed rounds of transmission, the word order index is combined with the round verification record to check the consistency between the index mapping relationship and the aggregated value of each node; for nodes that fail the verification, backtracking and interpolation are repeated until the index consistency and numerical stability conditions are met, and an intermediate word vector sequence is obtained.

[0023] Based on the intermediate word vector sequence, the alignment metadata of each node in each round is summarized to form an index consistency record.

[0024] A cross-modal alignment search is performed, employing a hierarchical candidate expansion strategy to construct a set of monotonic matching paths between video vector temporal sequences and text vector sequences. The cumulative cost of the matching paths is used as the ranking criterion, and the output candidate paths include:

[0025] Based on the value ranges of video time index and text word order index, a two-dimensional search grid is constructed; atomic costs are generated for the two-dimensional search grid and cached as a local cost table; the root state is taken as the starting boundary on the two-dimensional search grid and placed into the first-layer candidate set to form a control variable set.

[0026] For each candidate state in the set of control variables, the grid cost at the corresponding grid position is read from the local cost table, the grid cost is combined with the candidate state to calculate a new cumulative cost, the new cumulative cost is appended to the matched pairing sequence to form a partially matched path that satisfies monotonicity, and a new candidate state is formed with the aligned state and added to the extended result set of the current layer.

[0027] The expanded result set is sorted in ascending order according to the new cumulative cost, and a bundle pruning is performed. The pruned candidate states are passed to the next layer, and a termination check is performed based on the level count and the maximum expansion step during the passing.

[0028] When the maximum number of expansion steps is reached or all candidate states are converted to completed states, the current completed state set is aggregated to check each pairing sequence. The checked pairing sequences are written back to the candidate path buffer in the form of grid coordinates to form candidate paths that meet the constraints.

[0029] The semantic consistency score and action entity relationship consistency score of paired elements on the candidate path are calculated, and the comprehensive score of the candidate path is obtained under multi-objective weighted constraints, including:

[0030] Obtain the pairing index sequence corresponding to the candidate path, align the video vector and text vector according to the pairing index sequence to generate a pairing element sequence; perform consistency verification on the pairing index sequence to form a cleaned pairing index; establish a bidirectional mapping table from the pairing index sequence to the pairing element index based on the cleaned pairing index;

[0031] Based on the bidirectional mapping table, the vector similarity and vector residual norm are calculated for each paired element. The vector similarity and vector residual norm are combined into a local semantic score using monotonically increasing boundary constraints. Weighted sliding aggregation with boundary protection is performed on the local semantic score along the path order of the paired elements to obtain the semantic consistency score.

[0032] Using the paired index position in the bidirectional mapping table as the anchor point, the dependency relationship between actions and entities is parsed at the anchor point on the text side. The local relation consistency score is calculated based on the dependency relationship between actions and entities. The local relation consistency score is then accumulated in a chain according to the chronological order and the semantic master-slave order to obtain the action-entity relation consistency score.

[0033] Using the semantic consistency score and the action entity relationship consistency score as target components, a comprehensive score for the candidate path is obtained based on the target components.

[0034] The alignment relationship between video temporal segments and text segments is determined using the candidate path with the highest comprehensive score. Based on this alignment, the text segments are adaptively weighted to obtain text matching results for the video content, including:

[0035] The candidate path with the highest comprehensive score is selected as the target path, and the cross-modal alignment index of the target path is solidified as the alignment index baseline. Based on the alignment index baseline, for each pair of mapping segments, boundary consistency is checked according to the continuous segment boundary and gap mark. Index overlap and omission between each pair of mapping segments are detected and corrected, and an alignment relationship table is generated.

[0036] Using the alignment relationship table as a constraint, term vectors, position features, and dependent role tags of aligned text segments are extracted from the text vector sequence. Combined with the action vectors and scene vectors of the aligned temporal segments on the video side, the alignment quality of each alignment unit is calculated. Based on the alignment quality, a weight generation function is constructed, and the weights of alignment units with gap penalty coefficients are reduced to obtain adaptive weights.

[0037] The corresponding text vectors are weighted according to the adaptive weights to obtain a weighted text representation for the video content; the similarity between the weighted text representation and the video vector is calculated within a window of video time sequence segments, and the results are summarized according to the segment granularity of the alignment table to form a text matching result.

[0038] Based on the alignment index baseline, for each pair of mapped segments, boundary consistency is checked according to the continuous segment boundaries and gap markers. Index overlap and omissions between each pair of mapped segments are detected and corrected, and an alignment relationship table is generated, including:

[0039] Extract continuous segment boundaries and gap markers from the alignment index baseline, and locate the initial start and end indices of each mapped segment accordingly; pair the mapped segments according to the natural order of the pairing indices to form a candidate sequence of segment pairs;

[0040] Taking the candidate sequence of the fragment pair as the object, joint verification is performed on adjacent mapped fragment pairs based on the boundary of the continuous segment and the gap mark. Pairwise detection is performed on the start and end index intersection, time or word order overlap and the missing interval indicated by the gap mark within the same continuous segment. The detection results are attached to the corresponding fragment pair entries in the form of conflict records.

[0041] Based on the conflict record, the mapping segments that have passed the joint verification are corrected at specific points. For cross-boundary overlaps, the continuous segment boundary is used for pruning first. If there is still overlap after pruning, the mapping segments are shrunk back to the start and end boundaries according to the order of the paired indexes, forming a set of corrected segment entries carrying correction parameters.

[0042] A correction fragment sequence is formed based on the set of correction fragment entries, and a global consistency check is performed based on the correction parameter write-back; when global consistency is satisfied, an alignment table is generated using the correction fragment sequence as the sole source.

[0043] A second aspect of the present invention provides a deep learning-based multimodal data pairing system, comprising:

[0044] The first unit is used to acquire the original multi-frame sequence of the video to be processed and the target text. For each frame in the original multi-frame sequence, a temporal segment is extracted at a fixed sampling interval to form a set of frame groups containing overlapping adjacent frames. Sentence segmentation and lexical normalization are performed on the target text, word order is preserved and the occurrence position index of each word is recorded to form an aligned text sequence.

[0045] The second unit is used to extract spatiotemporal features from the frame group set to obtain a video temporal vector sequence corresponding to the time index; at the same time, it performs embedding mapping and dependency encoding on the text sequence, and uses the message passing operation of the graph structure to inject syntactic dependency constraints into the word vectors to obtain a text vector sequence corresponding to the word order index.

[0046] The third unit is used to perform cross-modal alignment search while keeping the temporal order of video and text intact. It adopts a hierarchical candidate expansion strategy to build a set of monotonic matching paths between the temporal sequence of video vectors and the sequence of text vectors, and outputs candidate paths based on the cumulative cost of the matching paths.

[0047] The fourth unit is used to calculate the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtain the comprehensive score of the candidate path under multi-objective weighted constraints.

[0048] The fifth unit is used to determine the alignment relationship between video temporal segments and text segments based on the candidate path with the highest comprehensive score, and to adaptively weight the text segments based on the alignment relationship to obtain the text matching result for the video content.

[0049] A third aspect of the present invention provides an electronic device, comprising:

[0050] processor;

[0051] Memory used to store processor-executable instructions;

[0052] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0054] The beneficial effects of this application are as follows:

[0055] This invention enables accurate matching of multimodal data. Through a hierarchical candidate expansion strategy and a comprehensive scoring mechanism, it ensures the temporal consistency between video and text, effectively solving the problem of insufficient alignment accuracy in complex scenarios using traditional methods.

[0056] This invention introduces syntactic dependency constraints and action entity relationship consistency assessment, which enhances the depth of text semantic understanding, makes the association between text and video content more consistent with actual semantic expression, and improves the accuracy and robustness of cross-modal matching.

[0057] This invention employs an adaptive weighted method to process text fragments, which can dynamically adjust the matching strategy according to different scenarios and content characteristics, thereby improving the system's adaptability in diverse application scenarios and providing high-quality basic data support for downstream tasks such as video content retrieval and automatic annotation. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the multimodal data pairing method based on deep learning, as described in an embodiment of the present invention.

[0059] Figure 2 This is a flowchart of cross-modal path search and alignment generation according to an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0062] Figure 1 This is a flowchart illustrating the deep learning-based multimodal data pairing method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0063] The original multi-frame sequence of the video to be processed and the target text are obtained. For each frame in the original multi-frame sequence, temporal segments are extracted at a fixed sampling interval to form a set of frame groups containing overlapping adjacent frames. Sentence segmentation and lexical normalization are performed on the target text, word order is preserved and the occurrence position index of each word is recorded to form an aligned text sequence.

[0064] Spatiotemporal features are extracted from the frame set to obtain a video temporal vector sequence corresponding to the time index; simultaneously, embedding mapping and dependency encoding are performed on the text sequence, and syntactic dependency constraints are injected into the word vectors using graph structure message passing operations to obtain a text vector sequence corresponding to the word order index.

[0065] While maintaining the temporal order of video and text, cross-modal alignment search is performed. A hierarchical candidate expansion strategy is adopted to construct a set of monotonic matching paths between the temporal sequence of video vectors and the sequence of text vectors. The cumulative cost of the matching paths is used as the sorting criterion to output candidate paths.

[0066] Calculate the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtain the comprehensive score of the candidate path under multi-objective weighted constraints;

[0067] The alignment relationship between video temporal segments and text segments is determined by the candidate path with the highest comprehensive score. Based on this alignment relationship, the text segments are adaptively weighted to obtain the text matching result for the video content.

[0068] In one optional implementation, the text sequence is embedded and dependency encoded, and syntactic dependency constraints are injected into the word vectors using graph structure message passing operations to obtain a text vector sequence corresponding to the word order index, including:

[0069] An initial term feature set and index mapping table are generated for the text sequence using a concatenation strategy of character-level segmentation and word-level segmentation;

[0070] Based on the initial term feature set and the index mapping table, an embedding mapping is performed through a mapping function to generate an initial word vector sequence of the same length as the word order index; the initial word vector sequence and its corresponding word order index and character range are stored together to form an initial word vector table with index tags;

[0071] Based on the initial word vector table, syntactic dependency analysis is performed on the text sequence to obtain a syntactic graph structure. The syntactic graph structure and the initial word vector table are passed forward and backward in a fixed number of rounds. After all rounds of passing are completed, the intermediate word vector sequence corresponding to the word order index and the corresponding index consistency record are output.

[0072] The intermediate word vector sequence is rearranged and checked according to the word order index. After the check is passed, a text vector sequence that corresponds to the word order index and is consistent in semantics and syntax is output.

[0073] The text sequence embedding mapping and dependency encoding process begins with dual segmentation at the character and word levels. The segmentation algorithm employs a sliding window mechanism to perform multi-granularity analysis of the input text. Character-level segmentation is achieved through character-by-character scanning. The program creates a character buffer with a size of 1024 Unicode characters and uses UTF-8 encoding to process multi-byte characters. During scanning, the program identifies the Unicode code point of each character and groups consecutive alphanumeric characters, numeric characters, or Chinese characters into basic character units. Word-level segmentation uses a maximum matching algorithm combined with statistical word frequency information. The program maintains a dictionary tree data structure to store a pre-trained vocabulary. Each dictionary tree node contains a character value, a suffix marker, word frequency statistics, and an array of child node pointers.

[0074] The segmentation algorithm starts from the beginning of the text, setting a maximum matching length of 15 characters. The program searches the dictionary tree for the longest matching word starting from the current position. When multiple matches are found, the algorithm prioritizes the word with the highest frequency; if the frequencies are the same, it selects the longest word. Out-of-vocabulary words are segmented by individual characters, with each character treated as an independent term. A concatenation strategy merges the character-level and word-level segmentation results to generate a composite term set containing both character-level and word-level information.

[0075] The initial term feature set is constructed using a multi-dimensional feature representation method. Each term includes feature dimensions such as lexical representation, character composition, positional information, length attribute, and semantic category. Lexical representation is encoded using lexical identifiers, which are unique integer indices in the dictionary. Character composition features record the Unicode code point sequence of each character within the term. Positional information includes the starting and ending character positions of the term in the original text. The length attribute records the number of characters contained in the term. The semantic category identifies the semantic type of the term using a pre-trained classifier, including basic categories such as nouns, verbs, adjectives, and numerals.

[0076] The index mapping table uses a hash table data structure to store the bidirectional mapping relationship between terms and numerical indices. The hash function uses the FNV-1a algorithm to calculate the hash value of the word strings, and collision resolution adopts a linear probing strategy using open addressing. The mapping table contains a forward mapping from term identifiers to word order indices and a reverse mapping from word order indices to term identifiers. The word order indices are assigned according to the order in which terms appear in the text, starting from zero and increasing sequentially. The mapping table also records the character range information for each term, with the character range stored in the form of start position and length.

[0077] Embedding mapping is implemented using a pre-trained word vector model. The mapping function searches for the vector representation of a word term from the word vector matrix. The word vector matrix is ​​stored using a two-dimensional floating-point array, with the number of rows equal to the size of the vocabulary and the number of columns equal to the word vector dimension. For words present in the vocabulary, the mapping function directly returns the vector value of the corresponding row. Out-of-vocabulary (OV) terms are handled using a sub-word embedding strategy. The algorithm decomposes OV terms into character sequences, searches for the vector representation of each character, and calculates the vector representation of the term by weighted averaging. The weights are determined based on the position of the character within the term, with the starting and ending characters having a weight of 1.5 and the middle characters having a weight of 1.0.

[0078] The initial word vector sequence is generated by arranging the vector representations of all terms in the order of their word indices. The sequence length is equal to the total number of terms in the text. Each vector element corresponds to a term, and the vector dimension remains consistent, typically set to 512 or 768 dimensions. The indexed initial word vector table uses a structured storage format. Each entry contains the word index, term identifier, character start position, character length, word vector data, and additional metadata. The word vector data is stored as a floating-point array, and the additional metadata includes attributes such as term type, frequency statistics, and semantic tags.

[0079] Syntactic dependency parsing is performed based on an initial word vector table, and the parsing algorithm employs a transition-based dependency parser. The parser maintains three basic data structures: a stack to store terms to be processed, a buffer to store unprocessed terms, and a set of dependency arcs to store established dependencies. The parsing process begins by loading all terms into the buffer, initializing the stack and dependency arc set to empty. The parser performs three basic operations: shift, left arc, and right arc. The shift operation moves the top term from the buffer to the top of the stack, the left arc operation establishes a right-to-left dependency between the top two terms on the stack, and the right arc operation establishes a left-to-right dependency.

[0080] The syntactic graph structure uses an adjacency list representation to store dependency relations. Each node corresponds to a term, and each node contains a word index, a term identifier, a dependency label, and a list of adjacent nodes. The dependency label identifies the type of syntactic relation, including specific types such as subject-verb, verb-object, attributive-head, and adverbial-head. The list of adjacent nodes stores other nodes directly connected to the current node, and each element in the list contains the target node index and edge direction information. The graph structure also maintains a root node list, recording the main predicate or sentence root node in the sentence.

[0081] The message passing operation in the graph structure employs a multi-round iterative forward and backward pass mechanism. Forward pass starts from the root node and propagates information along the dependency arcs to the leaf nodes. Each node receives a message from its parent node and calculates its updated node representation by combining it with its own word vector information. Message computation is implemented through linear transformations and activation functions. The linear transformation maps the parent node vector to the message space, and the activation function uses the ReLU function to maintain non-linearity. Node updates are achieved by weighted fusion of the received message and the current node vector; the fusion weights are dynamically calculated using an attention mechanism.

[0082] Backpropagation propagates information from leaf nodes to root nodes in the opposite direction of the dependency arc, similar to forward propagation but in the opposite direction. Each node collects messages from all its child nodes and calculates a comprehensive message representation using an aggregation function employing either max pooling or average pooling strategies. The attention mechanism's weights are calculated based on the dependency relationship type and vector similarity between nodes, and the weights are normalized using a softmax function to ensure that the sum of the weights is 1.

[0083] The fixed-round propagation process is set to 3 to 5 rounds, with each round containing a complete forward and backward propagation process. Information accumulation between rounds is achieved through residual connections, where the node representation after each round is added to the representation of the previous round to obtain the accumulated result. During the propagation process, the node state history is maintained, recording the vector change magnitude before and after each round of propagation. The propagation process is terminated early when the change magnitude is less than a preset threshold.

[0084] The output of the intermediate word vector sequence contains the final node vector representation after all propagation rounds. The sequence is arranged according to the original word index order to ensure consistency. The index consistency record uses a checksum mechanism to verify the integrity of the vector sequence. The checksum is calculated by summing and moduloing all vector elements. The record contains the original index sequence, the processed index sequence, vector dimension information, the number of propagation rounds, and the checksum value.

[0085] The rearrangement and verification process ensures the correct sequence order by comparing the word indices before and after processing. The verification algorithm traverses the intermediate word vector sequence, verifying whether the word index at each position matches the expected value. If there is a mismatch, the algorithm reconstructs the sequence based on the index consistency record, rearranging the vectors according to the correct word index. The verification process also verifies the consistency of vector dimensions, ensuring that all vectors have the same dimensions and meet the expected specifications.

[0086] Semantic and syntactic consistency verification is achieved through vector similarity calculation and dependency preservation checks. Vector similarity calculation compares the cosine similarity of vectors before and after processing, with a similarity threshold set to 0.8. Vectors below the threshold are marked as abnormal and require reprocessing. Dependency preservation is calculated by comparing the original dependency graph with the syntactic information implied by the vectors, with a preservation threshold set to 0.9. The final output of the text vector sequence adopts a standardized format, with each vector normalized to a unit length. The entire sequence is output to a specified storage location after passing integrity checks.

[0087] In one optional implementation, syntactic dependency analysis is performed on the text sequence based on the initial word vector table to obtain a syntactic graph structure. The syntactic graph structure and the initial word vector table are then passed forward and backward in fixed rounds. After all rounds of passing are completed, the intermediate word vector sequence corresponding to the word order index and the associated index consistency record are output, including:

[0088] The initial word vector table is subjected to index integrity verification, index conflicts are eliminated and missing character range records are filled in to obtain an initial input baseline with consistent indexes; based on the initial input baseline, syntactic dependency analysis is performed on the text sequence to obtain the syntactic graph structure;

[0089] Based on the syntactic graph structure and the initial word vector table, forward propagation based on incoming edges and backward propagation based on outgoing edges are performed in a preset fixed number of rounds. In each round of propagation, a linearly differentiable transformation is performed on the vector propagating along each dependency edge, and the multi-path information reaching the same node is aggregated to obtain the round aggregated vector of that node. The round aggregated vector is aligned and cached according to the index mapping table and the word order index of that node in the initial word vector table to generate a round verification record.

[0090] After completing the fixed rounds of transmission, the word order index is combined with the round verification record to check the consistency between the index mapping relationship and the aggregated value of each node; for nodes that fail the verification, backtracking and interpolation are repeated until the index consistency and numerical stability conditions are met, and an intermediate word vector sequence is obtained.

[0091] Based on the intermediate word vector sequence, the alignment metadata of each node in each round is summarized to form an index consistency record.

[0092] Syntactic dependency parsing and graph-structured message passing based on the initial word vector table begin with index integrity verification. The verification algorithm performs multi-dimensional verification on each term in the initial word vector table. Index conflict detection is achieved by constructing an index occupancy bitmap, the size of which is equal to the total number of terms, with each bit corresponding to a word order index position. The program traverses the word vector table, marking the word order index position of each term in the bitmap. When a position is found to be marked, it is determined to be an index conflict. Conflict resolution adopts a remapping strategy. The algorithm allocates a new available index to the conflicting term, and the remapping process maintains the relative order of terms in the text.

[0093] The integrity verification of character interval records is achieved through interval coverage checks. The program constructs a character position array, the length of which is equal to the total number of characters in the original text. It iterates through the character interval records of all terms, marking the covered character positions in the array as covered. After scanning, the unmarked positions in the array represent missing character intervals. Missing interval completion is achieved through text re-segmentation. The algorithm performs word recognition on the characters within the missing intervals, generating corresponding term records and inserting them into the word vector table. During completion, the word vectors of new terms are generated using a character-level embedding strategy, weighted by averaging the vector representations of each character within the interval to obtain the term vector.

[0094] The process of generating an index-consistent initial input baseline includes two stages: index reordering and data structure standardization. The index reordering algorithm sorts the word vector table according to the corrected word order index, using a stable sorting algorithm to ensure that terms with the same index maintain their original order. Data structure standardization unifies the storage format of word vectors. Each term entry includes a word order index, lexical identifier, character start position, character length, word vector data, and syntactic attribute fields required for dependency parsing. During standardization, the algorithm verifies the dimensionality consistency of each word vector, padding or truncating vectors with mismatched dimensions.

[0095] Syntactic dependency parsing employs an incremental parser based on transition actions. The parser maintains a configuration state including a stack structure, an input buffer, and a dependency arc set. The stack structure is implemented using a dynamic array, supporting push and pop operations; elements in the stack are the index values ​​of terms. The input buffer stores terms to be processed in word index order, with the buffer pointer pointing to the current position to be processed. The dependency arc set is stored using an adjacency list structure; each node maintains a list of directed edges pointing to other nodes, with each edge containing a dependency relation type label.

[0096] The analyzer executes three basic transition actions: a shift action pushes the top term from the buffer onto the stack; a left-arc action establishes a right-to-left dependency between the two top terms and pops the left term; and a right-arc action establishes a left-to-right dependency and pops the right term. Action selection is determined by a trained classifier, which takes the feature vector of the current configuration state as input and outputs the probability distribution of each action. The feature vector contains syntactic information such as the part-of-speech, lexical form, and dependency tags of the top term, as well as structural state information of the stack and buffer.

[0097] The syntactic graph structure is constructed using a directed graph data structure, where nodes represent terms and edges represent dependencies. The graph structure includes a node array and an adjacency list. The node array is arranged by word index, and each node stores a term identifier, part-of-speech tag, dependency head node index, and a list of child nodes. The adjacency list is implemented using a hash table, where the key is the node index and the value is a list of all adjacent nodes for that node. Dependency relation tags use a unified annotation system, including basic syntactic relation types such as subject, object, modifier, adverbial, and complement.

[0098] The forward pass algorithm, based on incoming edges, starts from the root node of the graph and propagates information to child nodes along the dependency relationships. It employs a breadth-first traversal strategy, maintaining a node access queue to ensure each node is processed in the correct order. Each node receives a message vector from its dependent head node. Message computation is achieved through a linear transformation, with the transformation matrix using different parameter sets depending on the dependency type. The transformation process maps the vector representation of the head node to the message space, maintaining the same mapping dimension as the original vector.

[0099] Backward propagation based on outgoing edges propagates information from leaf nodes to root nodes in the reverse direction of dependency relationships. The propagation strategy employs depth-first traversal to ensure that child node information is fully aggregated before being passed to the parent node. Each node collects message vectors from all child nodes and calculates a comprehensive message representation using an aggregation function. The aggregation function supports multiple strategies, including element-wise summation, element-wise maximum calculation, and weighted average, with weights determined based on the strength of the dependency relationship between the child node and the current node.

[0100] The linearly differentiable transformation adopts an affine transformation form. The transformation parameters include a weight matrix and a bias vector. The dimension of the weight matrix is ​​a square matrix equal to the vector dimension multiplied by the vector dimension, and the length of the bias vector is equal to the vector dimension. The transformation calculation is implemented through matrix-vector multiplication and vector addition. The transformation result is processed by an activation function to introduce nonlinear characteristics. The activation function uses a gated linear unit, controlling the flow of information through a gating mechanism. The gating value is calculated using the sigmoid function.

[0101] The multi-path information aggregation process handles multiple message vectors arriving at the same node. The aggregation algorithm supports an attention mechanism to dynamically assign importance weights to each message. Attention weights are calculated based on the dot product similarity between the query vector and the key vector, where the query vector is the vector representation of the current node, and the key vectors are the message vectors. The similarity scores are normalized using a softmax function to obtain the weight distribution, and then weighted and summed to obtain the aggregated vector. The aggregated vector for each round is obtained by performing a residual concatenation between the aggregation result and the current vector representation of the node. This residual concatenation helps with gradient propagation and training stability.

[0102] The index mapping table maintains a bidirectional mapping between word order indexes and graph node indexes, and uses a hash table for fast lookup. The alignment cache allocates independent cache space for each node, and the cached content includes the aggregation vector for each round, timestamps, and verification information. The cache uses a circular buffer structure; when the number of rounds exceeds the buffer capacity, the oldest record is overwritten. The round verification record contains the round number, node index, aggregate vector checksum, and processing timestamp, etc. The checksum is calculated by summing and moduloing the vector elements.

[0103] The consistency check process verifies the reasonableness of the index mapping relationship and numerical changes of each node in each round. Index mapping relationship verification ensures consistency by comparing the node's position in the graph structure with its position in the word vector table. Numerical stability checking assesses convergence by calculating the norm of vector changes between adjacent rounds; the value is considered stable when the norm is less than a preset threshold. Nodes that fail the check enter a repeated backtracking process. The backtracking algorithm resets the node's state to a specified round and re-executes message passing computation.

[0104] Interpolation is used to correct node vectors with numerical anomalies. The interpolation algorithm calculates the correction value based on the vector values ​​of neighboring nodes and dependency weights. The correction value is obtained through weighted interpolation, with weights determined based on the syntactic distance and semantic relevance between nodes. Syntactic distance is defined as the length of the dependency path between two nodes, and semantic relevance is calculated using the cosine similarity of pre-trained word vectors. The interpolated and corrected vectors need to be re-validated for consistency conditions; once the conditions are met, they are added to the intermediate word vector sequence.

[0105] The intermediate word vector sequence is generated by arranging all nodes according to the original word order index, resulting in the final vector representation. The sequence format includes fields such as word order index, vocabulary identifier, vector data, and processing tags. The processing tags record the state changes of nodes during transmission, including normal processing, anomaly correction, and interpolation processing. Sequence integrity is verified through checksums, which cover all vector data and index information.

[0106] The index consistency record summarizes the alignment metadata of each node across all rounds. The record structure includes basic node information, round processing history, anomaly handling records, and final state information. The round processing history includes statistics such as vector state before and after each round's transmission, the number of aggregated messages, and processing time. The anomaly handling records contain detailed information such as anomaly type, round in which it occurred, handling method, and correction results. The final state information includes evaluation results such as convergence rounds, stability metrics, and quality scores, providing a reference for subsequent processing.

[0107] In one optional implementation, a cross-modal alignment search is performed, employing a hierarchical candidate expansion strategy to construct a set of monotonic matching paths between the video vector temporal sequence and the text vector sequence, and using the cumulative cost of the matching paths as the ranking criterion, the output candidate paths include:

[0108] Based on the value ranges of video time index and text word order index, a two-dimensional search grid is constructed; atomic costs are generated for the two-dimensional search grid and cached as a local cost table; the root state is taken as the starting boundary on the two-dimensional search grid and placed into the first-layer candidate set to form a control variable set.

[0109] For each candidate state in the set of control variables, the grid cost at the corresponding grid position is read from the local cost table, the grid cost is combined with the candidate state to calculate a new cumulative cost, the new cumulative cost is appended to the matched pairing sequence to form a partially matched path that satisfies monotonicity, and a new candidate state is formed with the aligned state and added to the extended result set of the current layer.

[0110] The expanded result set is sorted in ascending order according to the new cumulative cost, and a bundle pruning is performed. The pruned candidate states are passed to the next layer, and a termination check is performed based on the level count and the maximum expansion step during the passing.

[0111] When the maximum number of expansion steps is reached or all candidate states are converted to completed states, the current completed state set is aggregated to check each pairing sequence. The checked pairing sequences are written back to the candidate path buffer in the form of grid coordinates to form candidate paths that meet the constraints.

[0112] like Figure 2 As shown, the method includes:

[0113] The two-dimensional search grid for cross-modal alignment search is constructed based on the range of values ​​for the video time index and the text word order index, determining the grid's row and column dimensions. The video time index range is determined by the start and end frame numbers of the video frame sequence, with frame rate information used to calculate the mapping relationship between timestamps and frame indices. The text word order index range is determined based on the length of the text vector sequence, starting from zero and ending at the sequence length minus one. The two-dimensional grid is represented using a matrix data structure, with the number of rows equal to the maximum value of the video time index plus one, and the number of columns equal to the maximum value of the text word order index plus one. Each position in the grid corresponds to a potential alignment point, with the position coordinates being an ordered pair of row and column indices.

[0114] Atomic costs are generated by calculating the similarity between video vectors and text vectors at corresponding positions, using a cosine similarity algorithm. The algorithm extracts the video frame vector and text word vector corresponding to the grid position, and divides the dot product of the two vectors by the product of their respective norms to obtain the similarity value. The atomic cost is defined as one minus the similarity value, with higher similarity positions having lower costs. The cost ranges from zero to two, where zero represents perfect similarity and two represents complete dissimilarity. The local cost table uses a two-dimensional floating-point array to store atomic costs, with the array's row and column indices directly corresponding to grid coordinates, supporting cost lookup operations at constant time.

[0115] The root state is set at the starting boundary of the 2D grid, specifically at coordinates zero row and zero column, representing the alignment point between the start time of the video sequence and the start position of the text sequence. The root state data structure includes fields such as current grid coordinates, cumulative cost value, matched sequences, and state identifier. The cumulative cost value is initialized to zero, the matched sequences are initialized to an empty list, and the state identifier uses a unique integer number. The first-level candidate set is stored using a priority queue data structure, sorted by cumulative cost value, with lower-cost states having higher priority.

[0116] The management of the control set employs a hierarchical processing mechanism, with each layer corresponding to an expansion step in the search process. The expansion of candidate states follows the monotonicity constraint of dynamic programming, allowing movement from the current grid position in only three directions: right, down, or down-right. Moving right indicates an increase in the text index while the video index remains unchanged, corresponding to an alignment pattern where text progresses and video pauses. Moving down indicates an increase in the video index while the text index remains unchanged, corresponding to an alignment pattern where video progresses and text pauses. Moving down-right indicates a simultaneous increase in both the video and text indices, corresponding to an alignment pattern where video and text progress synchronously.

[0117] Mesh cost is retrieved via index access to the local cost table. The retrieval operation locates the corresponding cost based on the mesh coordinates of the candidate state. The new cumulative cost is calculated by adding the cumulative cost of the current candidate state to the retrieved mesh cost. The cumulative cost reflects the total alignment cost from the root state to the current position; a smaller value indicates better alignment quality. The construction of partially matched paths is achieved by adding new mesh coordinates to the matched pairing sequence, which records the complete path from the root state to the current position.

[0118] Monotonicity verification ensures that the generated matching paths meet temporal constraints. The verification algorithm checks the monotonicity of adjacent coordinate pairs in the path. For any adjacent coordinate pair in the path, the row index of the latter coordinate must be greater than or equal to the row index of the former coordinate, and the column index must also be greater than or equal to the column index of the former coordinate. Paths that violate monotonicity are discarded and not added to the expanded result set. Alignment state generation creates a new candidate state for each valid expanded position. The new state inherits the pairing sequence of the parent state and adds the coordinate information of the current position.

[0119] The expanded result set uses a dynamic array to store all candidate states generated in the current layer. This array supports dynamic expansion to accommodate different expansion scales. States in the set are arranged in the order of generation, and each state contains complete path information and a cumulative cost value. The sorting operation uses a quicksort algorithm to arrange candidate states in ascending order of cumulative cost value, with states of the same cost arranged in order of generation time. A stable sorting algorithm is used during the sorting process to ensure that the relative order of states with the same cost remains unchanged.

[0120] The bundle pruning mechanism limits the number of candidate states at each layer to control search complexity. The pruning parameter is set to the bundle width value, typically between 50 and 200. The pruning algorithm retains the first bundle width of candidate states with the minimum cumulative cost, discarding the remaining states to reduce memory usage and computational overhead. During the pruning process, the algorithm records statistical information about the discarded states, including the number of discarded states, average cost, and optimal cost.

[0121] The hierarchical transfer process moves the pruned candidate states to the processing queue of the next level. The transfer operation uses a shallow copy mechanism to reduce memory allocation overhead. A level counter records the current level number being processed, incrementing from zero, and is used for termination condition checks. The maximum expansion step parameter limits the maximum search depth to prevent the search process from getting stuck in an infinite loop or consuming excessive computational resources.

[0122] The termination check mechanism monitors the end conditions of the search process, including reaching the maximum number of expansion steps and all candidate states transitioning to the completion state. The completion state is defined as reaching the lower right boundary of the 2D grid, indicating that both the video and text sequences have been fully processed. The state transition check is implemented by comparing the grid coordinates of the candidate states with the grid boundary coordinates; a completion state is determined when the row index equals the maximum row index and the column index equals the maximum column index.

[0123] The aggregation of the completed state set uses a list data structure to store all candidate paths leading to the completed state. Each element in the set contains a complete matching path and its corresponding cumulative cost. The pairing sequence verification process validates the validity and completeness of each path, including path length, coordinate range, monotonicity constraints, and start and end point positions. Path length verification ensures that the path contains sufficient alignment points, with a minimum length requirement of half the sum of the number of grid rows and columns.

[0124] The coordinate range check verifies that all coordinates in the path are within the valid grid range; paths outside the range are marked as invalid and removed from the results. The start and end point check confirms that the path starts at the top left corner of the grid and ends at the bottom right corner; paths that do not meet this requirement are discarded. The checked pairing sequence is converted into grid coordinate form, and the coordinate sequence records all grid positions traversed by the path.

[0125] The candidate path cache uses a hash table data structure to store valid candidate paths. The hash key is the unique identifier of the path, and the hash value is the complete path information. The path identifier is generated by hashing all coordinates in the path, ensuring that different paths have different identifiers. The cache supports fast path lookup, insertion, and deletion operations, and provides path deduplication functionality to avoid storing duplicate paths.

[0126] The constraint satisfaction verification ensures that the candidate path meets the business constraints for cross-modal alignment. These constraints include requirements for alignment density, jump amplitude, and coverage integrity. Alignment density constraints require that alignment points in the path be relatively evenly distributed, avoiding overly concentrated or overly sparse alignment patterns. Jump amplitude constraints limit the coordinate span between adjacent alignment points to prevent excessive temporal jumps. Coverage integrity constraints ensure that the path covers the main content areas of the video and text sequences, with a coverage rate of at least 85%.

[0127] Path quality assessment is achieved by comprehensively considering multiple indicators such as cumulative cost, path smoothness, and alignment consistency. Path smoothness is measured by calculating the variance of coordinate changes between adjacent aligned points; the smaller the variance, the smoother the path. Alignment consistency is assessed by analyzing the semantic relevance of aligned points in the path, with the relevance calculated based on the semantic matching degree between video and text content. Candidate paths are ranked according to their comprehensive quality scores, and the path with the highest score is output as the optimal alignment result.

[0128] In one optional implementation, calculating the semantic consistency score and action entity relationship consistency score of paired elements on the candidate path, and obtaining the comprehensive score of the candidate path under multi-objective weighted constraints includes:

[0129] Obtain the pairing index sequence corresponding to the candidate path, align the video vector and text vector according to the pairing index sequence to generate a pairing element sequence; perform consistency verification on the pairing index sequence to form a cleaned pairing index; establish a bidirectional mapping table from the pairing index sequence to the pairing element index based on the cleaned pairing index;

[0130] Based on the bidirectional mapping table, the vector similarity and vector residual norm are calculated for each paired element. The vector similarity and vector residual norm are combined into a local semantic score using monotonically increasing boundary constraints. Weighted sliding aggregation with boundary protection is performed on the local semantic score along the path order of the paired elements to obtain the semantic consistency score.

[0131] Using the paired index position in the bidirectional mapping table as the anchor point, the dependency relationship between actions and entities is parsed at the anchor point on the text side. The local relation consistency score is calculated based on the dependency relationship between actions and entities. The local relation consistency score is then accumulated in a chain according to the chronological order and the semantic master-slave order to obtain the action-entity relation consistency score.

[0132] Using the semantic consistency score and the action entity relationship consistency score as target components, a comprehensive score for the candidate path is obtained based on the target components.

[0133] The candidate path pairing index sequence extraction begins with the grid coordinate sequence of the path, where each grid coordinate contains a tuple of a video time index and a text word order index. The pairing index sequence generation algorithm traverses all coordinate points in the path, extracting the row index of each coordinate point as the video index and the column index as the text index, forming an ordered sequence of index pairs. Each index pair in the sequence represents a potential alignment relationship between a video frame and a text word, and the length of the sequence is equal to the number of coordinate points traversed by the path.

[0134] The alignment operation between video vectors and text vectors extracts corresponding elements from the respective vector sequences based on the index values ​​in the paired index sequence. Video vector extraction locates the feature vector of the corresponding frame in the video vector temporal sequence using the video index, while text vector extraction locates the vector representation of the corresponding word in the text vector sequence using the text index. The generation of the paired element sequence combines the video vector and text vector corresponding to each index pair into a paired element. The paired element is stored in a structure form, containing the video vector, text vector, paired index, and metadata fields.

[0135] The consistency check of paired index sequences ensures the validity and rationality of the indexes through a multi-factor validation mechanism. Index range verification verifies whether each video index is within the valid range of the video sequence, and whether the text index is within the valid range of the text sequence. Temporal continuity verification checks the temporal relationship between adjacent paired indexes, ensuring that both video and text indexes maintain a monotonically non-decreasing trend. Jump magnitude verification limits the jump distance between adjacent indexes; when the jump distance exceeds a preset threshold, it is marked as an abnormal index pair.

[0136] The generation of clean paired indexes is achieved through the identification and correction of anomalous indexes, including those that are out of range, violate temporal continuity, or have excessively large jumps. The correction strategy uses interpolation to handle missing intermediate indexes, generating reasonable intermediate index values ​​between the positions before and after the anomalous index through linear interpolation. Duplicate index removal is achieved through hash sets, merging duplicate index pairs into single index pairs while retaining the earliest appearing index pair in time sequence.

[0137] The bidirectional mapping table uses a hash table data structure to store the mapping relationship between paired index sequences and paired element indices. Forward mapping uses the position in the paired index sequence as the key and the index of the paired element in the paired element sequence as the value. Reverse mapping uses the paired element index as the key and the position in the paired index sequence as the value. The mapping table supports constant-time lookup operations and provides an efficient implementation of index transformation. The mapping table also records metadata about the mapping relationship, including the mapping creation time, index validity flags, and mapping quality scores.

[0138] Vector similarity calculation uses the cosine similarity algorithm to measure the semantic relevance between video and text vectors in a pair. The algorithm calculates the dot product of the two vectors and divides it by the product of their respective vector norms to obtain the similarity value. The similarity value ranges from -1 to +1, with positive values ​​indicating similar vector directions, negative values ​​indicating opposite directions, and zero indicating orthogonal vectors. The vector residual norm calculates the Euclidean norm of the difference between two vectors, reflecting the absolute distance difference between them. The residual norm value is always non-negative; the smaller the value, the closer the vectors are.

[0139] The application of monotonically increasing boundary constraints ensures that the local semantic score increases with vector similarity and decreases with increasing vector residual norm. The constraint function maps vector similarity to the range of zero to one, achieving a smooth mapping through the sigmoid function or a piecewise linear function. The vector residual norm is mapped to the range of zero to one through a reciprocal transformation, the transformation formula being one divided by one plus the residual norm value. The local semantic score is calculated as a weighted average of the mapped values ​​of vector similarity and vector residual norm, with the weight parameters adjusted according to the application scenario.

[0140] A weighted sliding aggregation algorithm with boundary protection processes local semantic scores in paired element sequences. The aggregation process uses a sliding window mechanism to calculate the average score of local regions. The sliding window size is set to 5 to 10 elements, and the weighted average of all local semantic scores within the window is calculated as the window slides across the sequence. The boundary protection mechanism handles special cases at sequence boundaries; when the window exceeds the sequence boundary, it automatically adjusts its size to accommodate the number of available elements. A Gaussian weighting function is used for weight allocation, with the highest weight at the center of the window and decreasing weight towards the edges.

[0141] The semantic consistency score is calculated by weighted summation of the aggregated results of all sliding windows, with the weighting coefficients determined by the window's position in the sequence. Windows in the middle of the sequence have higher weights, while windows at the sequence boundaries have lower weights. The final semantic consistency score is normalized to the range of zero to one using a minimum-maximum normalization method. Intermediate results and statistical information, including the number of windows, average local score, and variance, are recorded during the score calculation process.

[0142] Action-entity dependency parsing is performed on the text side based on anchor positions in a bidirectional mapping table. Anchor positions correspond to specific word positions in the text sequence. The parsing algorithm analyzes the role and relationship of each anchor word in the syntactic tree. Action recognition is achieved through part-of-speech tagging and semantic role tagging, identifying predicate words as action candidates. Entity recognition is achieved through named entity recognition and noun phrase extraction, identifying nominal components as entity candidates.

[0143] Dependency relation calculation analyzes the syntactic dependency paths and semantic relation strengths between actions and entities. Syntactic dependency paths are calculated using a syntactic tree traversal algorithm, and path length is defined as the number of edges traversed in the shortest dependency path between two words. Semantic relation strength is calculated based on dependency relation type and path length, with direct dependencies having the highest strength, and indirect dependencies decreasing in strength with increasing path length. Relation type weights are assigned according to the importance of syntactic relations, with subject-verb and verb-object relations having higher weights.

[0144] The local relation consistency score comprehensively considers factors such as the number of action entity pairs, relation strength, and relation type distribution. The score is calculated by weighted summation of the relation strengths of all action entity pairs within a window surrounding the anchor point. Weight allocation considers the positional importance and semantic centrality of action entity pairs in the text; action entity pairs that are closer to the anchor point and semantically important receive higher weights.

[0145] Chained consistency accumulation processes the local relation consistency scores across multiple anchor points, traversing all anchor points in chronological order. This chronological order is determined by the anchor point's position in the paired index sequence, with earlier anchor points having earlier timestamps. The semantic hierarchy is determined by the dependency level of the action entity relationship, with primary actions and core entities having higher semantic priority. The accumulation algorithm employs a recursive accumulation mechanism, weighted and fused with the accumulated score of the previous anchor point.

[0146] The consistency score for action entity relationships is obtained by normalizing the chained cumulative results across all anchor points. This normalization method ensures the score falls within the range of zero to one. The score calculation process considers the global consistency of action entity relationships, verifying the existence of conflicts or contradictions in relationships at different anchor points through consistency checks. Conflict detection is based on logical constraints of action entity relationships, such as the requirement that the same entity cannot simultaneously execute mutually exclusive actions.

[0147] The weighted fusion of target components employs a multi-objective optimization weight allocation strategy, with semantic consistency score and action entity relationship consistency score participating in the fusion as two independent target components. The weight parameters are adjusted according to the needs of the application scenario; applications that emphasize semantic matching are assigned higher semantic consistency weights, while applications that emphasize action understanding are assigned higher relationship consistency weights.

[0148] The comprehensive score of the candidate path is calculated through a linear weighted combination of the target components, reflecting the overall quality of the candidate path across multiple evaluation dimensions. The score calculation formula is: semantic consistency score multiplied by semantic weight, plus action entity relationship consistency score multiplied by relationship weight. The weight sum equals one to ensure the boundedness of the score. The final score is normalized to a range of zero to one hundred to facilitate comparison and ranking among different candidate paths. The scoring process records detailed scoring information, including the original scores of each target component, weight allocation, and the final fusion result, providing a basis for result analysis and parameter tuning.

[0149] In one optional implementation, the alignment relationship between the video temporal segment and the text segment is determined using the candidate path with the highest comprehensive score. Based on this alignment relationship, the text segment is adaptively weighted to obtain the text matching result for the video content, including:

[0150] The candidate path with the highest comprehensive score is selected as the target path, and the cross-modal alignment index of the target path is solidified as the alignment index baseline. Based on the alignment index baseline, for each pair of mapping segments, boundary consistency is checked according to the continuous segment boundary and gap mark. Index overlap and omission between each pair of mapping segments are detected and corrected, and an alignment relationship table is generated.

[0151] Using the alignment relationship table as a constraint, term vectors, position features, and dependent role tags of aligned text segments are extracted from the text vector sequence. Combined with the action vectors and scene vectors of the aligned temporal segments on the video side, the alignment quality of each alignment unit is calculated. Based on the alignment quality, a weight generation function is constructed, and the weights of alignment units with gap penalty coefficients are reduced to obtain adaptive weights.

[0152] The corresponding text vectors are weighted according to the adaptive weights to obtain a weighted text representation for the video content; the similarity between the weighted text representation and the video vector is calculated within a window of video time sequence segments, and the results are summarized according to the segment granularity of the alignment table to form a text matching result.

[0153] The candidate path with the highest overall score is selected by ranking and comparing the overall scores of all candidate paths. The ranking algorithm uses quicksort to arrange all candidate paths from highest to lowest score. Paths with the same score are ranked according to secondary indicators such as path length and semantic consistency score to ensure the uniqueness and stability of the selection result. After the target path is determined, the algorithm extracts the complete grid coordinate sequence of the path, which contains all alignment point information from the start position to the end position.

[0154] The solidification process for cross-modal alignment indexing converts the coordinate sequence of the target path into a standardized alignment index format. This solidification operation creates a persistent index data structure to store the alignment relationships. The alignment index baseline uses a compressed storage format, reducing storage space usage through differential encoding. Differential encoding records the index increment between adjacent alignment points rather than the absolute index value; the increment value is typically small and can be represented using fewer bits. The baseline data structure includes fields such as version information, compression algorithm identifier, original path length, and the compressed index sequence.

[0155] The generation of mapped segments is based on the combination of consecutive alignment points in the aligned index baseline. The segment partitioning algorithm identifies continuous intervals and discontinuities in the index sequence. A continuous interval is defined as the set of adjacent alignment points within a preset threshold, and a discontinuity indicates a position where the index jumps beyond the threshold. Each mapped segment contains attributes such as a starting index, an ending index, length information, and the number of alignment points it contains. The segment identifier is generated using a hash algorithm based on the segment's index range and content characteristics to create a unique identifier.

[0156] The identification of continuous segment boundaries is achieved by analyzing the index distribution pattern in the aligned index baseline. The boundary detection algorithm calculates the interval statistics between adjacent indices. The mean and variance of the intervals are used to determine the range of normal intervals, and intervals exceeding the normal range are identified as segment boundaries. Gap markers are generated by detecting blank regions in the index sequence, defined as regions where the range of consecutively missing indices exceeds the minimum gap length. Boundary markers and gap markers are stored in the baseline data using special encodings to facilitate identification and parsing by subsequent processing algorithms.

[0157] Boundary consistency verification checks whether the index range of each pair of mapped segments conforms to the continuous segment boundary constraints. The verification algorithm checks whether the segment boundaries are aligned with the segment boundary markers. Inconsistent segment boundaries are corrected by a boundary adjustment algorithm, prioritizing the preservation of semantic integrity. Index overlap detection identifies overlapping areas by comparing the index ranges of adjacent mapped segments. The overlap length is calculated as the intersection length of the index ranges of the two segments. Overlap correction employs a priority strategy: segments with higher overall scores retain their original ranges, while segments with lower scores undergo boundary shrinkage.

[0158] Index omission detection identifies missing regions by comparing the total coverage of mapped segments with the expected coverage. Omission correction generates missing segments using an interpolation algorithm. The interpolation algorithm estimates the attributes of the missing segments based on feature information of segments before and after the missing region, including features such as alignment quality and content type. The alignment relationship table is generated using a tabular data structure to store the corrected mapped segment information. The table includes fields such as segment identifier, index range, alignment quality, and correction record.

[0159] Word vector extraction from aligned text fragments is based on extracting corresponding vectors from the text vector sequence from the text index range in the alignment table. The extraction algorithm traverses all text index positions contained in the fragment and collects the corresponding word vectors to form a fragment vector set. Positional features are calculated using the absolute and relative positions of the terms in the text sequence. The absolute position is the index value of the term in the entire text, and the relative position is the proportion of the term's position in the current fragment.

[0160] Dependency role tag extraction is based on syntactic dependency analysis results. The tags contain information about the syntactic and semantic roles of the terms. Syntactic roles identify the function of a term in the syntactic tree, such as basic roles like subject, object, and modifier. Semantic roles identify the function of a term in the semantic framework, such as agent, patient, and instrument. The tag information is represented using one-hot encoding or embedded vectors for easier subsequent numerical computation.

[0161] Feature extraction for video-side aligned temporal segments comprises two dimensions: action vectors and scene vectors. Feature extraction is based on the video index range in the alignment table. Action vectors are extracted from the video frame sequence using an action recognition model, with the model output including action category probability distribution and action intensity information. Scene vectors are extracted using a scene understanding model, containing scene description information such as scene category, object distribution, and spatial layout. The vector dimensions are uniformly set to 512 or 1024 dimensions to facilitate cross-modal feature fusion computation.

[0162] The alignment quality calculation of the alignment unit comprehensively considers the matching degree between text features and video features, and the quality assessment adopts multi-dimensional similarity calculation. The semantic similarity between term vectors and action vectors is calculated using the cosine similarity algorithm to measure the semantic consistency between text content and video actions. The spatial similarity between positional features and scene vectors is calculated using Euclidean distance to reflect the degree of matching between the spatial information of the text description and the video scene. The structural similarity between dependent roles and action categories is calculated using a role mapping table, which defines the correspondence between grammatical roles and action categories.

[0163] The weight generation function constructs a non-linear mapping relationship based on alignment quality. The function design ensures that high-quality aligned units receive higher weights. The mapping function uses a sigmoid function or a piecewise linear function to achieve a smooth quality-to-weight conversion, and the function parameters are optimized using training data. A quality threshold is set to distinguish between high-quality and low-quality aligned units; units exceeding the threshold receive weights close to one, while units below the threshold have their weights rapidly decayed.

[0164] The gap penalty coefficient is applied to perform weight reduction on alignment units containing gap markings. The penalty strength is determined based on the gap length and type. Short gaps are lightly penalized, while long gaps are severely penalized, with penalty coefficients ranging from 0.1 to 0.9. The reduction calculation is performed by multiplying the original weights by the penalty coefficient. The reduced weights ensure that the negative impact of the gap region on the final result is controlled. Adaptive weight normalization ensures that all weight values ​​are within a reasonable range; normalization uses the softmax function or the minimum-maximum normalization method.

[0165] The weighted text representation is calculated through element-wise multiplication of adaptive weights with the corresponding text vectors. The weighting operation assigns a weight value to each dimension of each text vector. A weight broadcasting mechanism handles cases where the weight dimension and vector dimension do not match; the broadcasting operation extends the low-dimensional weights to the high-dimensional vector space. The weighted text vectors retain the original semantic information while highlighting the contribution of high-quality alignment. The weighted results are normalized to avoid numerical overflow.

[0166] The video temporal segment window is set based on the temporal range information in the alignment table, and the window size is dynamically adjusted according to the segment length and content complexity. Short segments use smaller windows to maintain temporal accuracy, while long segments use larger windows to capture global features. The window sliding strategy adopts an overlapping sliding mechanism, maintaining a 50% overlap between adjacent windows to ensure feature continuity. Similarity calculation is performed within each window, calculating the average similarity between the weighted text representation and the video vector within the window.

[0167] Similarity aggregation is performed at the segment level of the alignment table. The aggregation algorithm collects the similarity scores of all windows within the same segment. The aggregation strategy uses a weighted average method, with weights allocated according to the importance of the window in the segment. Center windows of the segment receive higher weights, while boundary windows receive lower weights. The aggregation result forms a segment-level matching score, ranging from zero to one. A higher score indicates a better match between the text and video content.

[0168] The text matching results are output in a structured format, including fragment identifiers, matching scores, confidence assessments, and detailed matching information. The matching information includes sub-results such as keyword matching, action matching, and scene matching, facilitating result analysis and application integration. The confidence assessment is calculated based on statistical information during the matching process, including indicators such as alignment quality variance and weight distribution uniformity. The final results are output in sorted order of matching scores, providing high-quality cross-modal matching results for downstream applications.

[0169] In one optional implementation, for each pair of mapped segments based on the alignment index baseline, boundary consistency is checked according to the continuous segment boundaries and gap markers, and index overlap and omissions between each pair of mapped segments are detected and corrected to generate an alignment relationship table, including:

[0170] Extract continuous segment boundaries and gap markers from the alignment index baseline, and locate the initial start and end indices of each mapped segment accordingly; pair the mapped segments according to the natural order of the pairing indices to form a candidate sequence of segment pairs;

[0171] Taking the candidate sequence of the fragment pair as the object, joint verification is performed on adjacent mapped fragment pairs based on the boundary of the continuous segment and the gap mark. Pairwise detection is performed on the start and end index intersection, time or word order overlap and the missing interval indicated by the gap mark within the same continuous segment. The detection results are attached to the corresponding fragment pair entries in the form of conflict records.

[0172] Based on the conflict record, the mapping segments that have passed the joint verification are corrected at specific points. For cross-boundary overlaps, the continuous segment boundary is used for pruning first. If there is still overlap after pruning, the mapping segments are shrunk back to the start and end boundaries according to the order of the paired indexes, forming a set of corrected segment entries carrying correction parameters.

[0173] A correction fragment sequence is formed based on the set of correction fragment entries, and a global consistency check is performed based on the correction parameter write-back; when global consistency is satisfied, an alignment table is generated using the correction fragment sequence as the sole source.

[0174] The data structure parsing of the alignment index baseline begins with the baseline file in binary or text format. The parser creates a file read buffer, setting its size to 4096 bytes to optimize read performance. The program scans the baseline data byte by byte, identifying specific byte sequences that mark the boundaries of consecutive segments. The start marker for a consecutive segment is typically 0x53544152544D4152 (corresponding to ASCII "STARTMAR"), and the end marker is 0x454E444D41524B45 (corresponding to ASCII "ENDMARKE"). When a start marker is detected, the program reads the last 4 bytes as the segment identifier and then reads 8 bytes as the segment start index, parsing it into a 64-bit unsigned integer using big-endian byte order. The end marker is handled similarly; after extracting the segment identifier and end index, the program stores the segment information in a hash table data structure, using the segment identifier as the key and the start and end index pairs as the values.

[0175] The notch markers are identified using the sequence 0x4741504D41524B45 (corresponding to the ASCII code "GAPMARKE"). Each notch marker is followed by 16 bytes of data: the first 8 bytes indicate the notch start position, and the last 8 bytes indicate the notch length. The program calculates the notch end position by adding the length value to the start position and adds the notch information to a dedicated notch list structure. The initial index extraction for each mapped segment is achieved through the metadata area of ​​the segment header. The segment data begins with a fixed 32-byte header containing a unique segment identifier (16-byte UUID), a start index (8 bytes), and a length field (8 bytes). The program iterates through all segments, extracts the start index and length information, and calculates the end index by adding one to the start index and subtracting one from the length.

[0176] The mapping fragment pairing algorithm is based on the sorting of pairing indices. The program creates a fragment information array, where each array element contains attributes such as fragment identifier, pairing index value, and start and end index range. The quicksort algorithm sorts the array in ascending order by pairing index values, using a three-way quicksort algorithm to optimize the handling of cases where pairing index values ​​are the same. After sorting, the program traverses the array to generate a candidate sequence of fragment pairs. Adjacent array elements are automatically paired, with the pairing rule being that the fragment at index i is paired with the fragment at index i+1. For cases where there is a gap in pairing index values, the program checks the difference between two consecutive pairing index values. If the difference is greater than 1, it indicates a gap, and fragments within the gap are paired sequentially until the gap is filled.

[0177] The fragment pair data structure uses a linked list node format. Each node contains a source fragment pointer, a target fragment pointer, a pairing type identifier, and a pointer to the head of the conflict record linked list. The pairing type identifier is represented by an enumeration value, including types such as sequential pairing, cross pairing, and nested pairing. The program assigns a unique 64-bit identifier to each fragment pair, which is composed of the pairing type (8 bits), the source fragment index (28 bits), and the target fragment index (28 bits).

[0178] The joint verification process executes multiple detection algorithms for each segment pair. Start-end index cross-validation is achieved through interval intersection judgment. The program compares the end index of segment A with the start index of segment B. Cross-validation is determined when the end index of A is greater than or equal to the start index of B and the start index of A is less than or equal to the end index of B. The cross-validation length is calculated as the length of the overlapping portion of the two intervals, specifically the smaller end index minus the larger start index plus one. Time overlap detection applies to segments containing timestamps. The program extracts timestamp information from the extended header of the segment, using a 64-bit integer with microsecond precision. Time overlap judgment uses the same interval intersection algorithm as index cross-validation, and the overlap duration is the number of microseconds of the overlapping time interval.

[0179] Word overlap detection is performed on text content segments. The program divides the segment content into a vocabulary array based on spaces and punctuation marks, and calculates a hash value for each word to construct a vocabulary hash set. Word overlap between two segments is detected by intersecting the hash sets; the overlap coefficient is obtained by dividing the intersection size by the union size. Significant word overlap is determined when the overlap coefficient exceeds a preset threshold of 0.3. Gap and omission detection is achieved by comparing the coverage range of a segment with the complete range of a continuous segment. The program constructs a Boolean array to represent the coverage status of the index range, and the array length is equal to the length of the index range of the continuous segment. All segments are traversed, and the index positions covered by the segment are marked as true in the Boolean array. After detection, the continuous intervals in the array with false values ​​are scanned as omission intervals.

[0180] Conflict records are stored in a structured format. Each record contains fields such as conflict type enumeration value, conflict start position, conflict length, conflict severity score, and timestamp. The conflict type enumeration defines specific types such as index intersection, time overlap, word order overlap, and gap omission. The conflict severity score uses an integer range from 0 to 100, and the scoring algorithm is calculated based on the proportion of the conflict length to the total fragment length; the higher the proportion, the higher the score. Conflict records are attached to the corresponding fragment pair entries through a linked list structure, and the head pointer of the linked list is stored in the conflict record field of the fragment pair node.

[0181] The fixed-point correction algorithm implements differentiated processing strategies based on the type and severity of conflict records. For cross-boundary overlap pruning, a boundary constraint priority principle is adopted. The program checks whether the start and end indices of a fragment cross the boundary of a continuous segment. Cross-boundary detection is achieved by comparing the fragment index range with the position of the continuous segment boundary. A cross-boundary is determined when the fragment start index is less than the start boundary of the continuous segment or the end index is greater than the end boundary of the continuous segment. The pruning algorithm removes the cross-boundary portion by adjusting the fragment start index to the larger value between the start boundary of the continuous segment and the original start index, and adjusting the end index to the smaller value between the end boundary of the continuous segment and the original end index.

[0182] The boundary shrinking mechanism handles overlaps that persist after trimming. Shrinking priority is determined by the numerical value of the pairing index; segments with smaller pairing index values ​​have higher priority. The program calculates the overlap interval length and determines the shrinking strategy based on the ratio of the overlap length to the remaining segment length. When the overlap length is less than 20% of the remaining segment length, the lower-priority segment moves its starting boundary backward by a distance equal to the overlap length plus a safety interval. When the overlap length exceeds 20% but is less than 50%, a bidirectional shrinking strategy is used, with each segment sharing half the shrinking distance. When the overlap length exceeds 50%, the program marks the segment pair as a high-conflict pair, requiring manual intervention.

[0183] The correction parameter records adopt a correction log structure. Each log entry contains fields such as the original start index, original end index, corrected start index, corrected end index, correction type identifier, correction timestamp, and correction reason description. The correction type identifier includes enumerated values ​​such as boundary clipping, start shrinking, end shrinking, and bidirectional shrinking. Correction fragment entries add a correction parameter field to the original fragment data structure; this field is a pointer to the correction log entry. The correction fragment entry set is stored using a dynamic array structure, supporting fast insertion and lookup operations.

[0184] The reorganization of the corrected fragment sequence is based on reordering the corrected index values. The program uses a merge sort algorithm to ensure sorting stability, with the corrected starting index value as the sort key. During sequence generation, the program verifies the index validity of each corrected fragment, checking if the starting index is less than the ending index, if the index value is within the boundary of consecutive segments, and if it overlaps with gap regions. Invalid fragments are marked and removed from the sequence. The correction parameter write-back operation updates the corrected index values ​​to the fragment's main index field while retaining the original index values ​​in the history field. The write-back process uses atomic operations to ensure data consistency.

[0185] Global consistency verification ensures the correctness of the correction results through multi-stage validation. The first stage performs an index continuity check, traversing the corrected fragment sequence and verifying whether the index intervals between adjacent fragments meet expectations; the allowed interval range is 0 to the maximum allowed interval value. The second stage performs range coverage verification, calculating the union of the index ranges of all fragments and comparing it with the expected coverage range. A coverage rate of 95% or higher is required to pass the verification. The third stage performs constraint verification, checking whether the corrected fragments violate continuous segment boundary constraints and gap mark constraints; any violation will result in verification failure.

[0186] The alignment table is generated using a tabular data structure. The table header defines fields such as source fragment identifier, target fragment identifier, alignment type, source start index, source end index, target start index, target end index, and confidence score. The program iterates through the corrected fragment sequence verified by global consistency checks, generating an alignment record for each valid fragment pair. The alignment type is determined based on the index relationship between fragments, including one-to-one alignment, one-to-many alignment, and many-to-one alignment. The confidence score comprehensively considers factors such as correction magnitude, number of conflicts, and coverage integrity, and is calculated using a weighted average algorithm with weights of 0.4, 0.3, and 0.3, respectively.

[0187] The index mapping relationship is calculated using a linear interpolation algorithm. When the source fragment index range is [s1, s2] and the target fragment index range is [t1, t2], the formula for mapping the source index s to the target index is the target starting index plus the result of multiplying the relative position of the source index by the target range length and dividing by the source range length. Mapping precision is maintained to two decimal places, and the final result is rounded to the nearest integer. After the alignment table is generated, the program performs integrity verification, checking whether the index mapping relationship of each record in the table is monotonically increasing, whether the confidence score is within a valid range, and whether there are duplicate records. After successful verification, the relationship table is serialized and stored in binary format for easy subsequent loading and use.

[0188] A second aspect of the present invention provides a deep learning-based multimodal data pairing system, comprising:

[0189] The first unit is used to acquire the original multi-frame sequence of the video to be processed and the target text. For each frame in the original multi-frame sequence, a temporal segment is extracted at a fixed sampling interval to form a set of frame groups containing overlapping adjacent frames. Sentence segmentation and lexical normalization are performed on the target text, word order is preserved and the occurrence position index of each word is recorded to form an aligned text sequence.

[0190] The second unit is used to extract spatiotemporal features from the frame group set to obtain a video temporal vector sequence corresponding to the time index; at the same time, it performs embedding mapping and dependency encoding on the text sequence, and uses the message passing operation of the graph structure to inject syntactic dependency constraints into the word vectors to obtain a text vector sequence corresponding to the word order index.

[0191] The third unit is used to perform cross-modal alignment search while keeping the temporal order of video and text intact. It adopts a hierarchical candidate expansion strategy to build a set of monotonic matching paths between the temporal sequence of video vectors and the sequence of text vectors, and outputs candidate paths based on the cumulative cost of the matching paths.

[0192] The fourth unit is used to calculate the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtain the comprehensive score of the candidate path under multi-objective weighted constraints.

[0193] The fifth unit is used to determine the alignment relationship between video temporal segments and text segments based on the candidate path with the highest comprehensive score, and to adaptively weight the text segments based on the alignment relationship to obtain the text matching result for the video content.

[0194] A third aspect of the present invention provides an electronic device, comprising:

[0195] processor;

[0196] Memory used to store processor-executable instructions;

[0197] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0198] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0199] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based multimodal data pairing method, characterized in that, include: Obtain the original multi-frame sequence of the video to be processed and the target text. Extract time segments from each frame in the original multi-frame sequence at a fixed sampling interval to form a set of frame groups containing overlapping adjacent frames. The target text is processed by sentence segmentation and lexical normalization, word order is preserved and the occurrence index of each word is recorded to form an aligned text sequence; Spatiotemporal features are extracted from the frame set to obtain a video temporal vector sequence corresponding to the time index; simultaneously, embedding mapping and dependency encoding are performed on the text sequence, and syntactic dependency constraints are injected into the word vectors using graph structure message passing operations to obtain a text vector sequence corresponding to the word order index. While maintaining the temporal order of video and text, a cross-modal alignment search is performed. A hierarchical candidate expansion strategy is used to construct a set of monotonic matching paths between the video vector temporal sequence and the text vector sequence. The cumulative cost of the matching paths is used as the sorting criterion to output candidate paths, including: Based on the value ranges of video time index and text word order index, a two-dimensional search grid is constructed; atomic costs are generated for the two-dimensional search grid and cached as a local cost table; the root state is taken as the starting boundary on the two-dimensional search grid and placed into the first-layer candidate set to form a control variable set. For each candidate state in the set of control variables, the grid cost at the corresponding grid position is read from the local cost table, the grid cost is combined with the candidate state to calculate a new cumulative cost, the new cumulative cost is appended to the matched pairing sequence to form a partially matched path that satisfies monotonicity, and a new candidate state is formed with the aligned state and added to the extended result set of the current layer. The expanded result set is sorted in ascending order according to the new cumulative cost, and a bundle pruning is performed. The pruned candidate states are passed to the next layer, and a termination check is performed based on the level count and the maximum expansion step during the passing. When the maximum number of expansion steps is reached or all candidate states are converted to the completed state, the current completed state set is aggregated to check each pairing sequence. The checked pairing sequences are written back to the candidate path buffer in the form of grid coordinates to form candidate paths that meet the constraints. Calculate the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtain the comprehensive score of the candidate path under multi-objective weighted constraints; The alignment relationship between video temporal segments and text segments is determined by the candidate path with the highest comprehensive score. Based on this alignment relationship, the text segments are adaptively weighted to obtain the text matching result for the video content.

2. The method according to claim 1, characterized in that, The text sequence is embedded and mapped using dependency encoding. syntactic dependency constraints are injected into word vectors using graph-structured message passing operations, resulting in a text vector sequence corresponding to the word order index, including: An initial term feature set and index mapping table are generated for the text sequence using a concatenation strategy of character-level segmentation and word-level segmentation; Based on the initial term feature set and the index mapping table, an embedding mapping is performed through a mapping function to generate an initial word vector sequence of the same length as the word order index; the initial word vector sequence and its corresponding word order index and character range are stored together to form an initial word vector table with index tags; Based on the initial word vector table, syntactic dependency analysis is performed on the text sequence to obtain a syntactic graph structure. The syntactic graph structure and the initial word vector table are passed forward and backward in a fixed number of rounds. After all rounds of passing are completed, the intermediate word vector sequence corresponding to the word order index and the corresponding index consistency record are output. The intermediate word vector sequence is rearranged and checked according to the word order index. After the check is passed, a text vector sequence that corresponds to the word order index and is consistent in semantics and syntax is output.

3. The method according to claim 2, characterized in that, Based on the initial word vector table, syntactic dependency analysis is performed on the text sequence to obtain a syntactic graph structure. The syntactic graph structure and the initial word vector table are then passed forward and backward in fixed rounds. After all rounds of passing are completed, the intermediate word vector sequence corresponding to the word order index and the associated index consistency record are output, including: The initial word vector table is subjected to index integrity verification, index conflicts are eliminated and missing character range records are filled in to obtain an initial input baseline with consistent indexes; based on the initial input baseline, syntactic dependency analysis is performed on the text sequence to obtain the syntactic graph structure; Based on the syntactic graph structure and the initial word vector table, forward propagation based on incoming edges and backward propagation based on outgoing edges are performed in a preset fixed number of rounds. In each round of propagation, a linearly differentiable transformation is performed on the vector propagating along each dependency edge, and the multi-path information reaching the same node is aggregated to obtain the round aggregated vector of that node. The round aggregated vector is aligned and cached according to the index mapping table and the word order index of that node in the initial word vector table to generate a round verification record. After completing the fixed rounds of transmission, the word order index is combined with the round verification record to check the consistency between the index mapping relationship and the aggregated value of each node; for nodes that fail the verification, backtracking and interpolation are repeated until the index consistency and numerical stability conditions are met, and an intermediate word vector sequence is obtained. Based on the intermediate word vector sequence, the alignment metadata of each node in each round is summarized to form an index consistency record.

4. The method according to claim 1, characterized in that, The semantic consistency score and action entity relationship consistency score of paired elements on the candidate path are calculated, and the comprehensive score of the candidate path is obtained under multi-objective weighted constraints, including: Obtain the pairing index sequence corresponding to the candidate path, align the video vector and text vector according to the pairing index sequence to generate a pairing element sequence; perform consistency verification on the pairing index sequence to form a cleaned pairing index; establish a bidirectional mapping table from the pairing index sequence to the pairing element sequence based on the cleaned pairing index; Based on the bidirectional mapping table, the vector similarity and vector residual norm are calculated for each paired element. The vector similarity and vector residual norm are combined into a local semantic score using monotonically increasing boundary constraints. Weighted sliding aggregation with boundary protection is performed on the local semantic score along the path order of the paired elements to obtain the semantic consistency score. Using the paired index position in the bidirectional mapping table as the anchor point, the dependency relationship between actions and entities is parsed at the anchor point on the text side. The local relation consistency score is calculated based on the dependency relationship between actions and entities. The local relation consistency score is then accumulated in a chain according to the chronological order and the semantic master-slave order to obtain the action-entity relation consistency score. Using the semantic consistency score and the action entity relationship consistency score as target components, a comprehensive score for the candidate path is obtained based on the target components.

5. The method according to claim 1, characterized in that, The alignment relationship between video temporal segments and text segments is determined using the candidate path with the highest comprehensive score. Based on this alignment, the text segments are adaptively weighted to obtain text matching results for the video content, including: The candidate path with the highest comprehensive score is selected as the target path, and the cross-modal alignment index of the target path is solidified as the alignment index baseline. Based on the alignment index baseline, for each pair of mapping segments, boundary consistency is checked according to the continuous segment boundary and gap mark. Index overlap and omission between each pair of mapping segments are detected and corrected, and an alignment relationship table is generated. Using the alignment relationship table as a constraint, term vectors, position features, and dependent role tags of aligned text segments are extracted from the text vector sequence. Combined with the action vectors and scene vectors of the aligned temporal segments on the video side, the alignment quality of each alignment unit is calculated. Based on the alignment quality, a weight generation function is constructed, and the weights of alignment units with gap penalty coefficients are reduced to obtain adaptive weights. The corresponding text vectors are weighted according to the adaptive weights to obtain a weighted text representation for the video content; the similarity between the weighted text representation and the video vector is calculated within a window of video time sequence segments, and the results are summarized according to the segment granularity of the alignment table to form a text matching result.

6. The method according to claim 5, characterized in that, Based on the alignment index baseline, for each pair of mapped segments, boundary consistency is checked according to the continuous segment boundaries and gap markers. Index overlap and omissions between each pair of mapped segments are detected and corrected, and an alignment relationship table is generated, including: Extract continuous segment boundaries and gap markers from the alignment index baseline, and locate the initial start and end indices of each mapped segment accordingly; pair the mapped segments according to the natural order of the pairing indices to form a candidate sequence of segment pairs; Taking the candidate sequence of the fragment pair as the object, joint verification is performed on adjacent mapped fragment pairs based on the boundary of the continuous segment and the gap mark. Pairwise detection is performed on the start and end index intersection, time or word order overlap and the missing interval indicated by the gap mark within the same continuous segment. The detection results are attached to the corresponding fragment pair entries in the form of conflict records. Based on the conflict record, the mapping segments that have passed the joint verification are corrected at specific points. For cross-boundary overlaps, the continuous segment boundary is used for pruning first. If there is still overlap after pruning, the mapping segments are shrunk back to the start and end boundaries according to the order of the paired indexes, forming a set of corrected segment entries carrying correction parameters. A correction fragment sequence is formed based on the set of correction fragment entries, and a global consistency check is performed based on the correction parameter write-back; when global consistency is satisfied, an alignment table is generated using the correction fragment sequence as the sole source.

7. A deep learning-based multimodal data pairing system for implementing the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to acquire the original multi-frame sequence of the video to be processed and the target text, and to extract time segments for each frame in the original multi-frame sequence at a fixed sampling interval to form a set of frame groups containing overlapping adjacent frames. The target text is processed by sentence segmentation and lexical normalization, word order is preserved and the occurrence index of each word is recorded to form an aligned text sequence; The second unit is used to extract spatiotemporal features from the frame group set to obtain a video temporal vector sequence corresponding to the time index; at the same time, it performs embedding mapping and dependency encoding on the text sequence, and uses the message passing operation of the graph structure to inject syntactic dependency constraints into the word vectors to obtain a text vector sequence corresponding to the word order index. The third unit is used to perform cross-modal alignment search while keeping the temporal order of video and text intact. It adopts a hierarchical candidate expansion strategy to build a set of monotonic matching paths between the temporal sequence of video vectors and the sequence of text vectors, and outputs candidate paths based on the cumulative cost of the matching paths. The fourth unit is used to calculate the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and to obtain the comprehensive score of the candidate path under multi-objective weighted constraints. The fifth unit is used to determine the alignment relationship between video temporal segments and text segments based on the candidate path with the highest comprehensive score, and to adaptively weight the text segments based on the alignment relationship to obtain the text matching result for the video content.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images