Multi-modal data pairing method and system based on deep learning

Abstract

The invention provides a multi-modal data pairing method and system based on deep learning, and relates to the technical field of data processing, and the method comprises the steps: obtaining a video multi-frame sequence and a target text, and respectively extracting an overlapped frame group set and a standardized text sequence; performing spatio-temporal feature extraction and text dependency relationship coding to obtain a video time sequence vector sequence and a text vector sequence; executing cross-modal alignment search, and constructing a monotonic matching path set; calculating a semantic and action entity relationship consistency score of the paired elements on the path to obtain a comprehensive score; and determining an alignment relationship between the video and the text based on the optimal path. According to the method, accurate matching of the video and the text is realized, and the cross-modal retrieval efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a multi-modal data pairing method and system based on deep learning. BACKGROUND

[0002] With the popularity of smart phones, smart TVs and other devices, users' demand for video content retrieval and understanding is growing. In this context, multi-modal data pairing and alignment technology has become a research hotspot, especially the pairing and alignment between video and text. This technology can help achieve precise retrieval of video content, video content understanding, automatic generation of subtitles, and video summary extraction, and has wide application prospects in media content production, education and training, intelligent recommendation, etc.

[0003] Currently, multi-modal data pairing technology mainly relies on key frame-based feature extraction and text bag-of-words model for simple matching. Traditional methods usually use visual feature extraction plus text embedding, and calculate a similarity matrix to achieve rough correspondence between video frames and text. With the development of deep learning technology, neural network-based feature representation and attention mechanism have been introduced into the field of multi-modal pairing, but there are still many technical difficulties and challenges.

[0004] Existing video-text pairing methods usually cannot effectively handle temporal information, and can only identify the correspondence between static scenes and text, making it difficult to capture the precise mapping between dynamic behaviors in the video and text descriptions, resulting in low accuracy in pairing action descriptions.

[0005] Most pairing technologies do not consider the syntactic structure and dependency relationships of text, and only use simple word vector representation, ignoring the impact of semantic relationships on pairing accuracy, especially in complex narrative and long text scenarios.

[0006] Existing methods usually use a global matching strategy, lack fine-grained alignment capability for video and text content, and cannot accurately identify which text segments correspond to specific segments in the video, thereby affecting the accuracy and efficiency of downstream tasks such as video question answering, content review, etc. SUMMARY

[0007] The embodiments of the present application provide a multi-modal data pairing method and system based on deep learning, which can solve the problems in the prior art.

[0008] In a first aspect, the embodiments of the present application provide a multi-modal data pairing method based on deep learning, comprising:

[0009] Obtaining an original multi-frame sequence of a video to be processed and a target text, extracting a time sequence segment at a fixed sampling interval for each frame in the original multi-frame sequence to form a frame group set containing overlapping adjacent frames; performing sentence processing and lexical normalization processing on the target text, retaining word order and recording the appearance position index of each word to form an alignable text sequence;

[0010] Performing spatio-temporal feature extraction on the frame group set to obtain a video time sequence vector sequence corresponding to a time index; simultaneously performing embedding mapping and dependency relationship coding on the text sequence, and injecting syntax dependency relationship constraints into word vectors using message passing operations of graph structure to obtain a text vector sequence corresponding to a word order index;

[0011] Under the premise of keeping the time order of video and text unchanged, performing cross-modal alignment search, using a hierarchical candidate expansion strategy to construct a monotonic matching path set between the video vector time sequence and the text vector sequence, and taking the cumulative cost of the matching path as the basis for sorting, outputting a candidate path;

[0012] Calculating the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtaining the comprehensive score of the candidate path under multi-objective weighted constraint;

[0013] Determining the alignment relationship between the video time sequence segment and the text segment with the candidate path with the highest comprehensive score, and performing adaptive weight weighting on the text segment based on the alignment relationship to obtain a text matching result for the video content.

[0014] Performing embedding mapping and dependency relationship coding on the text sequence, and injecting syntax dependency relationship constraints into word vectors using message passing operations of graph structure to obtain a text vector sequence corresponding to a word order index includes:

[0015] Generating an initial word feature set and an index mapping table according to a concatenation strategy of character-level segmentation and word-level segmentation on the text sequence;

[0016] According to the initial word feature set and the index mapping table, embedding mapping is performed through a mapping function to generate an initial word vector sequence with the same length as the word order index; the initial word vector sequence, its corresponding word order index and character interval are stored together to form an initial word vector table with index markers;

[0017] Performing syntax dependency analysis on the text sequence based on the initial word vector table to obtain a syntax graph structure, and performing forward and backward transmission of the syntax graph structure and the initial word vector table at a fixed round, and outputting an intermediate word vector sequence corresponding to the word order index and an index consistency record matched therewith after all transmission rounds are completed;

[0018] The intermediate word vector sequence is rearranged according to the word sequence index. After the rearrangement passes, a text vector sequence that is consistent in semantics and syntax and corresponds to the word sequence index is output.

[0019] Performing syntax dependency analysis on the text sequence based on the initial word vector table obtains a syntax graph structure. The syntax graph structure and the initial word vector table are subjected to forward and backward transmission under a fixed round. After all transmission rounds are completed, an intermediate word vector sequence corresponding to the word sequence index and an index consistency record matched therewith are output, including:

[0020] Performing index integrity check on the initial word vector table eliminates index conflicts and completes missing character interval records to obtain an index-consistent initial input baseline. Based on the initial input baseline, syntax dependency analysis is performed on the text sequence to obtain a syntax graph structure.

[0021] According to the syntax graph structure and the initial word vector table, forward transmission based on the in-edge and backward transmission based on the out-edge are respectively performed under a preset fixed round. In each transmission round, for each dependency edge, a linear differentiable transformation is performed on the vector propagated along the edge, and the multiple information reaching the same node is aggregated to obtain the round aggregation vector of the node. The round aggregation vector is aligned and cached according to the index mapping table and the word sequence index of the node in the initial word vector table to generate a round check record.

[0022] After completing the transmission of the fixed round, the word sequence index is combined with the round check record to check the index mapping relationship and aggregated value of each node for consistency. The nodes that do not pass the check are repeatedly backtracked and interpolated until the index consistency and value stability conditions are met to obtain an intermediate word vector sequence.

[0023] Based on the intermediate word vector sequence, the alignment metadata of each node in each round is summarized to form an index consistency record.

[0024] Performing cross-modal alignment search, a hierarchical candidate expansion strategy is adopted to construct a monotonic matching path set between the video vector time sequence and the text vector sequence, and the cumulative cost of the matching path is taken as the basis for sorting to output a candidate path, including:

[0025] Based on the value range of the video time index and the text word sequence index, a two-dimensional search grid is constructed. The atomic cost of the two-dimensional search grid is generated, and the atomic cost is cached as a local cost table. The starting boundary is taken as the root state on the two-dimensional search grid, and the root state is placed into the first layer candidate set to form a control quantity set.

[0026] For each candidate state in the control quantity set, read the grid cost of the corresponding grid position from the local cost table, calculate a new cumulative cost by combining the grid cost with the candidate state, append the new cumulative cost to the matched pairing sequence to form a partial matching path satisfying monotonicity, and form a new candidate state with the alignment state to join the expansion result set of the current layer;

[0027] Sort the expansion result set according to the new cumulative cost from small to large, perform beam pruning, pass the pruned candidate state to the next layer, and perform termination checking according to the level count and the maximum expansion step number during transmission;

[0028] When the maximum expansion step number is reached or all candidate states become completed states, check each pairing sequence in the current completed state set, write the pairing sequence that passes the check to the candidate path buffer area in the form of grid coordinates to form a candidate path that satisfies the constraints.

[0029] Calculate the semantic consistency score of the pairing elements on the candidate path, the action entity relationship consistency score, and obtain the comprehensive score of the candidate path under the multi-objective weighted constraint, including:

[0030] Obtain the pairing index sequence corresponding to the candidate path, align the video vector and the text vector according to the pairing index sequence to generate a pairing element sequence, perform consistency checking on the pairing index sequence to form a purified pairing index, and establish a bidirectional mapping table from the pairing index sequence to the pairing element index based on the purified pairing index;

[0031] According to the bidirectional mapping table, calculate the vector similarity and vector residual norm for each pairing element, combine the vector similarity and the vector residual norm into a local semantic score using a monotonically increasing boundary constraint; perform a weighted sliding aggregation with boundary protection on the local semantic score in the order of the pairing element path to obtain a semantic consistency score;

[0032] Using the pairing index position in the bidirectional mapping table as an anchor point, analyze the dependency relationship between actions and entities at the anchor point on the text side, calculate a local relationship consistency score according to the dependency relationship between actions and entities, and apply chain consistency accumulation to the local relationship consistency score according to the time sequence and semantic master-slave sequence to obtain an action entity relationship consistency score;

[0033] Using the semantic consistency score and the action entity relationship consistency score as target components, obtain the comprehensive score of the candidate path according to the target components.

[0034] determine the alignment relationship between the video time sequence segments and the text segments based on the alignment relationship, and obtain a text matching result for the video content by adaptively weighting the text segments based on the alignment relationship, including:

[0035] select the candidate path with the highest comprehensive score as a target path, solidify the cross-modal alignment index of the target path as an alignment index baseline, and perform boundary consistency checking on each pair of mapping segments according to the continuous segment boundary and the gap marker, detect and correct index overlap and omission between each pair of mapping segments according to the continuous segment boundary and the gap marker, and generate an alignment relationship table;

[0036] extract the term vector, position feature and dependency role label of the aligned text segment from the text vector sequence as a constraint based on the alignment relationship table, combine the action vector and scene vector of the video side aligned time sequence segment, calculate the alignment quality of each alignment unit, construct a weight generation function based on the alignment quality, and reduce the weight of the alignment unit with a gap penalty coefficient to obtain an adaptive weight;

[0037] weight the corresponding text vector according to the adaptive weight to obtain a weighted text representation for the video content; calculate the similarity between the weighted text representation and the video vector in the window of the video time sequence segment, and aggregate the similarity according to the segment granularity of the alignment relationship table to form a text matching result.

[0038] According to the alignment index baseline, the boundary consistency of each pair of mapping segments is checked according to the continuous segment boundary and the gap marker, and the index overlap and omission between each pair of mapping segments is detected and corrected to generate an alignment relationship table, including:

[0039] extract the continuous segment boundary and the gap marker from the alignment index baseline, and locate the initial start and end index of each mapping segment according to the continuous segment boundary and the gap marker; pair the mapping segments in natural order according to the pairing index to form a segment pair candidate sequence;

[0040] According to the continuous segment boundary and the gap marker, the joint checking of adjacent mapping segment pairs is performed, the start and end index intersection, time or word sequence overlap in the same continuous segment, and the missing interval indicated by the gap marker are detected pair by pair, and the detection results are attached to the corresponding segment pair entries in the form of conflict records;

[0041] Based on the conflict record, the mapping segments that pass the joint checking are modified at the specified point, the cross-border overlap is preferentially cut according to the continuous segment boundary, and if there is still overlap after cutting, the start and end boundary of the mapping segment pair is retracted according to the order of the pairing index, and a modified segment entry set carrying the modification parameters is formed;

[0042] According to the modified segment entry set, a modified segment sequence is formed, and global consistency checking is performed on the basis of the modified parameter write-back; when the global consistency is satisfied, an alignment relationship table is generated from the modified segment sequence as a unique source.

[0043] In a second aspect, the embodiment of the present application provides a multi-modal data pairing system based on deep learning, which comprises:

[0044] A first unit is configured to acquire an original multi-frame sequence of a to-be-processed video and a target text, extract a time sequence segment from each frame in the original multi-frame sequence at a fixed sampling interval, form a frame group set containing overlapping adjacent frames, perform sentence processing and lexical normalization processing on the target text, retain word order and record the appearance position index of each word, and form an alignable text sequence;

[0045] A second unit is configured to perform spatio-temporal feature extraction on the frame group set to obtain a video time sequence vector sequence corresponding to a time index, and perform embedding mapping and dependency relationship coding on the text sequence, inject syntax dependency relationship constraints into word vectors by using message passing operation of a graph structure, and obtain a text vector sequence corresponding to a word order index;

[0046] A third unit is configured to perform cross-modal alignment search under the premise that the time sequences of the video and the text are not disturbed, construct a monotonic matching path set between the video vector time sequence and the text vector sequence by using a hierarchical candidate expansion strategy, and output a candidate path by taking the cumulative cost of the matching path as a sorting basis;

[0047] A fourth unit is configured to calculate the semantic consistency score and the action entity relationship consistency score of the paired elements on the candidate path, and obtain a comprehensive score of the candidate path under multi-objective weighted constraints;

[0048] A fifth unit is configured to determine the alignment relationship between the video time sequence segment and the text segment by using the candidate path with the highest comprehensive score, and perform adaptive weight weighting on the text segment based on the alignment relationship to obtain a text matching result for the video content.

[0049] In a third aspect, the embodiment of the present application provides an electronic device, which comprises:

[0050] A processor;

[0051] A memory for storing processor-executable instructions;

[0052] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0053] In a fourth aspect, the present application provides a computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method described above.

[0054] The present application has the following advantages:

[0055] The present application can achieve accurate matching of multi-modal data, ensure the consistency of video and text in time sequence through hierarchical candidate expansion strategy and comprehensive scoring mechanism, and effectively solve the problem of insufficient alignment accuracy of traditional methods in processing complex scenes.

[0056] The present application introduces syntactic dependency relationship constraint and action entity relationship consistency evaluation, enhances the depth of text semantic understanding, makes the association of text and video content more in line with actual semantic expression, and improves the accuracy and robustness of cross-modal matching.

[0057] The present application adopts an adaptive weight weighting method to process text segments, can dynamically adjust the matching strategy according to different scenes and content characteristics, improves the adaptability of the system in diversified application scenarios, and provides high-quality basic data support for video content retrieval, automatic labeling and other downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The figure is a flowchart of the multi-modal data pairing method based on deep learning of the embodiment of the present application.

[0059] Figure 2 The figure is a flowchart of the cross-modal path search and alignment generation of the embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0061] The technical scheme of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0062] Figure 1 The figure is a flowchart of the multi-modal data pairing method based on deep learning of the embodiment of the present application, as shown in Figure 1 The method comprises:

[0063] The original multi-frame sequence of the video to be processed and the target text are acquired, a time sequence fragment is extracted at a fixed sampling interval for each frame in the original multi-frame sequence to form a frame group set containing overlapping adjacent frames, sentence processing and lexical normalization processing are performed on the target text, the word order is preserved and the appearance position index of each word is recorded to form an alignable text sequence;

[0064] Temporal-spatial feature extraction is performed on the frame group set to obtain a video time sequence vector sequence corresponding to a time index, embedding mapping and dependency relationship coding are performed on the text sequence, and syntactic dependency relationship constraints are injected into word vectors by using message passing operations of a graph structure to obtain a text vector sequence corresponding to a word order index;

[0065] Under the premise that the time sequences of the video and the text are not disturbed, cross-modal alignment search is performed, a monotonous matching path set is constructed between the video vector time sequence and the text vector sequence by using a hierarchical candidate expansion strategy, and the cumulative cost of the matching path is taken as a sorting basis to output a candidate path;

[0066] The semantic consistency score and the action entity relationship consistency score of the paired elements on the candidate path are calculated, and the comprehensive score of the candidate path is obtained under multi-target weighted constraints;

[0067] The alignment relationship between the video time sequence fragment and the text fragment is determined according to the candidate path with the highest comprehensive score, and the text matching result for the video content is obtained based on the adaptive weight weighting of the text fragment according to the alignment relationship.

[0068] In an optional implementation, the embedding mapping and the dependency relationship coding of the text sequence, the injection of syntactic dependency relationship constraints into word vectors by using message passing operations of a graph structure to obtain a text vector sequence corresponding to a word order index include the following steps:

[0069] An initial word feature set and an index mapping table are generated according to a concatenation strategy of character-level segmentation and word-level segmentation of the text sequence;

[0070] An initial word vector sequence with the same length as the word order index is generated by embedding mapping through a mapping function according to the initial word feature set and the index mapping table, and the initial word vector sequence, the corresponding word order index and character interval are stored together to form an initial word vector table with index markers;

[0071] Syntactic dependency analysis is performed on the text sequence based on the initial word vector table to obtain a syntactic graph structure, and the syntactic graph structure and the initial word vector table are subjected to forward transmission and backward transmission in a fixed round, and after all transmission rounds are completed, an intermediate word vector sequence corresponding to the word order index and an index consistency record matched therewith are outputted;

[0072] The intermediate word vector sequence is rearranged according to the word order index. After the rearrangement is passed, the text vector sequence corresponding to the word order index and maintaining consistency in semantics and syntax is output.

[0073] The embedding mapping and dependency relation encoding of the text sequence starts from double segmentation at the character level and the word level. The sliding window mechanism is used in the segmentation algorithm to analyze the input text at multiple granularities. Character-level segmentation is achieved by scanning characters one by one. The program creates a character buffer with a size of 1024 Unicode characters and uses UTF-8 encoding format to handle multi-byte characters. During the scanning process, the program identifies the Unicode code points of each character and merges consecutive alphabetic characters, numeric characters, or Chinese characters into basic character units. The word-level segmentation uses the maximum matching algorithm combined with statistical word frequency information. The program maintains a trie data structure to store the pre-trained vocabulary. The trie nodes contain character values, end-of-word markers, word frequency statistics, and sub-node pointer arrays.

[0074] The segmentation algorithm starts from the beginning of the text and sets the maximum matching length to 15 characters. The program searches for the longest matching word starting from the current position in the trie. When multiple matches are found, the algorithm prioritizes the word with the highest frequency, and when the frequencies are the same, it selects the longest word. Unregistered words are processed by single-character segmentation, with each single character as an independent term. The concatenation strategy combines the character-level segmentation results with the word-level segmentation results to generate a composite term set containing character granularity and word granularity information.

[0075] The initial term feature set is constructed using a multi-dimensional feature representation method. Each term contains vocabulary representation, character composition, position information, length attribute, and semantic category features. The vocabulary representation is encoded by a vocabulary identifier, which is a unique integer index in the dictionary. The character composition feature records the Unicode code point sequence of the characters in the term. The position information includes the starting and ending character positions of the term in the original text. The length attribute records the number of characters in the term. The semantic category is identified by a pre-trained classifier to identify the semantic type of the term, including basic categories such as nouns, verbs, adjectives, and numerals.

[0076] The index mapping table uses a hash table data structure to store the bidirectional mapping relationship between terms and numerical indexes. The hash function uses the FNV-1a algorithm to calculate the hash value of the word string, and the collision resolution uses the linear probing strategy of open addressing. The mapping table contains forward mapping from term identifiers to word order indexes and reverse mapping from word order indexes to term identifiers. The word order index is assigned according to the order of term appearance in the text, starting from zero and increasing continuously. The mapping table also records the character interval information of each term in the form of starting position and length.

[0077] The embedding mapping is implemented by a pre-trained word vector model, and the mapping function looks up the vector representation corresponding to the word item from the word vector matrix. The word vector matrix is stored in a two-dimensional floating-point number array, with the number of rows equal to the size of the vocabulary and the number of columns equal to the word vector dimension. For a word item existing in the vocabulary, the mapping function directly returns the vector value of the corresponding row. The out-of-vocabulary word item is processed by the sub-word embedding strategy, and the algorithm decomposes the out-of-vocabulary word item into a character sequence, looks up the vector representation of each character, and calculates the vector representation of the word item by weighted average. The weight is determined according to the position of the character in the word item, and the weight of the starting character and the ending character is 1.5, and the weight of the middle character is 1.0.

[0078] The generation of the initial word vector sequence arranges the vector representations of all word items in the order of word sequence index, and the sequence length is equal to the total number of word items in the text. Each vector element corresponds to a word item, and the vector dimension remains consistent, usually set to 512 or 768 dimensions. The initial word vector table with index mark adopts a structured storage format, and each table entry contains word sequence index, word item identifier, character starting position, character length, word vector data, and additional meta information. The word vector data is stored in the form of a floating-point number array, and the additional meta information includes word item type, frequency statistics, semantic label, etc.

[0079] The syntax dependency analysis is performed based on the initial word vector table, and the analysis algorithm adopts a transition-based dependency syntax analyzer. The analyzer maintains three basic data structures: a stack structure stores word items to be processed, a buffer stores word items to be processed, and a dependency arc set stores determined dependency relationships. The analysis process starts with loading all word items into the buffer, the stack structure is initialized to empty, and the dependency arc set is initialized to empty. The analyzer performs three basic operations: shift, left arc, and right arc. The shift operation moves the word item at the top of the buffer to the top of the stack, the left arc operation establishes a right-to-left dependency relationship between the two word items at the top of the stack, and the right arc operation establishes a left-to-right dependency relationship.

[0080] The syntax graph structure stores the dependency relationship using the adjacency list representation method, each node corresponds to a word item, and the node contains the word sequence index, the word item identifier, the dependency label, and the adjacency node list. The dependency label identifies the syntax relationship type, including subject-predicate relationship, verb-object relationship, determiner-noun relationship, and adverbial relationship. The adjacency node list stores other nodes directly connected to the current node, and each element in the list contains the target node index and the direction information of the edge. The graph structure also maintains a root node list to record the main predicate or sentence root node in the sentence.

[0081] The message passing operation of the graph structure adopts a multi-round iteration mechanism of forward and backward passing. The forward passing starts from the root node and propagates information to leaf nodes along the direction of dependency arcs. Each node receives messages from parent nodes, combines its own word vector information to calculate the updated node representation. Message calculation is realized through linear transformation and activation function. Linear transformation maps parent node vectors to message space, and activation function adopts ReLU function to maintain non-linear characteristics. Node update is realized by weighting and fusing received messages with the current node vector. The fusion weight is dynamically calculated through attention mechanism.

[0082] The backward passing propagates information from leaf nodes to root nodes along the reverse direction of dependency arcs. The passing mechanism is similar to forward passing but in the opposite direction. Each node collects messages from all child nodes, calculates the comprehensive message representation through the aggregation function, and adopts the maximum pooling or average pooling strategy. The weight calculation of the attention mechanism is based on the dependency relationship type and vector similarity between nodes. The weight value is normalized through the softmax function to ensure the weight sum is 1.

[0083] The fixed round passing process sets the passing round number to 3 to 5 rounds, each round containing complete forward and backward passing processes. Information accumulation between rounds is realized through residual connection. The node representation after each passing round is added to the representation of the last round to obtain the accumulated result. The node state history is maintained during the passing process, recording the vector change amplitude before and after each passing round. When the change amplitude is less than the preset threshold, the passing process is terminated in advance.

[0084] The output of the intermediate word vector sequence contains the final node vector representation after all passing rounds are completed. The sequence is arranged according to the original word order index to ensure consistency. The index consistency record verifies the integrity of the vector sequence through the checksum mechanism. The checksum is calculated by summing all vector elements and taking the modulus. The record content includes the original index sequence, the processed index sequence, the vector dimension information, the passing round number, and the checksum value.

[0085] The rearrangement checking process ensures the correctness of the sequence order by comparing the word order index before and after processing. The checking algorithm traverses the intermediate word vector sequence to verify whether the word order index at each position matches the expected value. In the case of mismatch, the algorithm reconstructs the sequence according to the index consistency record, rearranging the vectors according to the correct word order index. The checking process also verifies the consistency of the vector dimension, ensuring that all vectors have the same dimension and meet the expected specifications.

[0086] The semantic and syntactic consistency verification is realized by vector similarity calculation and dependency relation maintenance degree check. The vector similarity calculation compares the cosine similarity of vectors before and after processing. The similarity threshold is set to 0.8. Vectors below the threshold are marked as abnormal and need to be reprocessed. The dependency relation maintenance degree is calculated by comparing the original dependency graph and the syntactic information contained in the vector. The maintenance degree threshold is set to 0.9. The final output of the text vector sequence uses a standardized format. Each vector is normalized to unit length. The sequence as a whole is checked for integrity and then output to the specified storage location.

[0087] In an alternative embodiment, syntactic dependency analysis is performed on the text sequence based on the initial word vector table to obtain a syntactic graph structure. Forward and backward passes are performed on the syntactic graph structure and the initial word vector table in a fixed number of rounds. After all transmission rounds are completed, an intermediate word vector sequence corresponding to the word order index and an index consistency record are output.

[0088] Index integrity check is performed on the initial word vector table to eliminate index conflicts and complete missing character interval records to obtain an index-consistent initial input baseline. Syntactic dependency analysis is performed on the text sequence based on the initial input baseline to obtain a syntactic graph structure.

[0089] According to the syntactic graph structure and the initial word vector table, forward pass based on incoming edges and backward pass based on outgoing edges are respectively performed in a predetermined fixed number of rounds. During each round of transmission, a linearly differentiable transformation is performed on the vector propagated along the edge for each dependency edge. The multiple information arriving at the same node is aggregated to obtain the round aggregation vector of the node. The round aggregation vector is aligned and cached according to the index mapping table and the word order index of the node in the initial word vector table to generate round check records.

[0090] After completing the transmission of the fixed number of rounds, the index mapping relationship and aggregated values of each node are checked for consistency by combining the word order index and the round check records. Nodes that do not pass the check are repeatedly backtracked and interpolated until the index consistency and value stability conditions are met to obtain an intermediate word vector sequence.

[0091] Based on the intermediate word vector sequence, the alignment metadata of each node in each round is summarized to form an index consistency record.

[0092] The syntax dependency analysis based on the initial word vector table and the graph structure message passing process starts with index integrity checking. The checking algorithm performs multi-dimensional verification on each word item in the initial word vector table. Index conflict detection is achieved by constructing an index occupancy bitmap. The bitmap size is equal to the total number of word items, and each bit corresponds to a word sequence index position. The program iterates through the word vector table and marks the word sequence index position of each word item in the bitmap. When a certain position is found to be marked, it is determined to be an index conflict. The conflict resolution uses a remapping strategy, which assigns a new available index to the conflict word item. The remapping process keeps the relative order of the word items in the text unchanged.

[0093] The integrity verification of the character interval record is achieved through interval coverage checking. The program constructs a character position array with a length equal to the total number of characters in the original text. It iterates through all the character interval records of the word items and marks the covered character positions in the array as covered. After scanning, the unmarked positions in the array represent missing character intervals. The completion of the missing intervals is achieved by re-splitting the text. The algorithm performs lexical recognition on the characters in the missing interval to generate corresponding word item records and insert them into the word vector table. During the completion process, the word vector of the new word item is generated by a character-level embedding strategy, and the vector representation of each character in the interval is weighted and averaged to obtain the word vector.

[0094] The initial input baseline generation process with consistent indexes includes two steps: index rearrangement and data structure standardization. The index rearrangement algorithm sorts the word vector table according to the corrected word sequence index. The sorting algorithm uses a stable sorting algorithm to ensure that word items with the same index maintain their original order. The data structure standardization unifies the storage format of the word vector. Each word item entry contains the word sequence index, lexical identifier, character start position, character length, word vector data, and syntax attribute fields required for dependency analysis. During the standardization process, the algorithm verifies the dimension consistency of each word vector and performs zero padding or truncation on the vectors with inconsistent dimensions.

[0095] The syntax dependency analysis uses an incremental analyzer based on shift-reduce actions. The analyzer maintains configuration states including stack structure, input buffer, and dependency arc set. The stack structure is implemented as a dynamic array, supporting push and pop operations. The elements in the stack are index values of word items. The input buffer stores word items to be processed in the order of word sequence index. The buffer pointer points to the current processing position. The dependency arc set is stored in an adjacency list structure. Each node maintains a list of directed edges pointing to other nodes. The edges contain dependency relationship type labels.

[0096] The parser performs three basic shift actions: a shift action pushes the top word in the buffer onto the stack, a left arc action establishes a right-to-left dependency between the top two words in the stack and pops the left word, and a right arc action establishes a left-to-right dependency and pops the right word. Action selection is determined by a trained classifier, which takes as input a feature vector of the current configuration state and outputs a probability distribution over actions. The feature vector contains syntactic information of the top word in the stack, such as its part-of-speech, lemma, and dependency label, as well as structural state information of the stack and buffer.

[0097] The syntactic graph structure is constructed using a directed graph data structure, with nodes representing words and edges representing dependencies. The graph structure contains a node array and an adjacency list. The node array is indexed by word order, and each node stores a word identifier, a part-of-speech label, a dependency head node index, and a list of child nodes. The adjacency list is implemented using a hash table, with node indices as keys and lists of all adjacent nodes for each node as values. Dependency labels use a unified annotation scheme, including basic grammatical relation types such as subject, object, attributive, adverbial, and complement.

[0098] Forward propagation based on incoming edges starts from the root node of the graph and propagates information to child nodes in the direction of dependencies. The propagation algorithm uses a breadth-first traversal strategy and maintains a node visit queue to ensure that each node is processed in the correct order. Each node receives a message vector from its dependency head node, and message computation is achieved through a linear transformation. The transformation matrix selects different parameter sets based on the dependency relation type. The transformation process maps the head node's vector representation to the message space, maintaining the same dimensionality as the original vector.

[0099] Backward propagation based on outgoing edges propagates information from leaf nodes to the root node in the opposite direction of dependencies. The propagation strategy uses a depth-first traversal to ensure that child node information is fully aggregated before being passed to the parent node. Each node collects message vectors from all child nodes and computes a comprehensive message representation using an aggregation function. The aggregation function supports multiple strategies, including element-wise summation, element-wise maximum, and weighted averaging, with weights determined based on the dependency strength between child nodes and the current node.

[0100] The linear differentiable transformation uses an affine transformation form, with transformation parameters including a weight matrix and a bias vector. The weight matrix has a dimension of the square of the vector dimension, and the bias vector has a length equal to the vector dimension. Transformation computation is achieved through matrix-vector multiplication and vector addition, and the transformation result is processed through an activation function to introduce non-linear characteristics. The activation function uses a gated linear unit, which controls the flow of information through a gating mechanism. The gating value is calculated using a sigmoid function.

[0101] The multi-path information aggregation process reaches multiple message vectors of the same node, and the aggregation algorithm supports attention mechanism to dynamically assign importance weights to each message. The attention weight calculation is based on the dot product similarity between the query vector and the key vector, the query vector being the vector representation of the current node, and the key vector being each message vector. The similarity score is normalized by the softmax function to obtain the weight distribution, and the weighted sum is calculated to obtain the aggregation vector. The round aggregation vector is obtained by connecting the aggregation result and the current vector representation of the node through a residual connection, which helps gradient propagation and training stability.

[0102] The index mapping table maintains a bidirectional mapping relationship between the word order index and the graph node index, and the mapping table is implemented by a hash table for fast lookup. The alignment cache allocates independent cache space for each node, and the cache content includes the aggregation vector of each round, the timestamp and the verification information. The cache uses a ring buffer structure, and when the number of rounds exceeds the buffer capacity, the earliest record is overwritten. The round verification record contains the round number, node index, aggregation vector checksum and processing timestamp, etc. The checksum is calculated by summing the vector elements and taking the modulus.

[0103] The consistency check process verifies the rationality of the index mapping relationship and the numerical change of each node in each round. The index mapping relationship check ensures consistency by comparing the position of the node in the graph structure with the position in the word vector table. The numerical stability check evaluates the convergence by calculating the norm of the vector change between adjacent rounds, and when the change norm is less than the preset threshold, the numerical stability is considered. The nodes that fail the check enter the repeated backtracking processing flow, and the backtracking algorithm resets the node state to a specified round and re-executes the message passing calculation.

[0104] The interpolation process is used to correct the node vector of the numerical anomaly, and the interpolation algorithm calculates the correction value based on the vector value and the dependency relationship weight of the adjacent node. The correction value is obtained by weighted interpolation, and the weight is determined according to the syntactic distance and semantic correlation between nodes. The syntactic distance is defined as the length of the dependency path between two nodes, and the semantic correlation is calculated by the cosine similarity of the pre-trained word vector. The interpolated and corrected vector needs to be re-verified for consistency, and after meeting the conditions, it is added to the intermediate word vector sequence.

[0105] The generation of the intermediate word vector sequence arranges the final vector representation of all nodes according to the original word order index, and the sequence format includes word order index, word identifier, vector data and processing mark fields. The processing mark records the state change of the node in the transmission process, including normal processing, abnormal correction, interpolation processing, etc. The sequence integrity is verified by the checksum, which covers all vector data and index information.

[0106] The index consistency record summarizes the alignment metadata of each node in all rounds, and the record structure includes node basic information, round processing history, abnormal processing record, and final state information. The round processing history records the vector state before and after each round transmission, the number of aggregated messages, and the processing time consumption, etc. The abnormal processing record includes the abnormal type, the occurrence round, the processing method, and the correction result, etc. The final state information includes the convergence round, the stability index, and the quality score, etc. evaluation results, which provide reference basis for subsequent processing.

[0107] In an optional implementation, a cross-modal alignment search is performed, a hierarchical candidate expansion strategy is adopted to construct a monotonic matching path set between the video vector time sequence and the text vector sequence, and the cumulative cost of the matching path is taken as the basis for sorting, and the candidate path includes:

[0108] Based on the value range of the video time index and the text word sequence index, a two-dimensional search grid is constructed; atomic costs are generated for the two-dimensional search grid, and the atomic costs are cached as a local cost table; the starting boundary is taken as the root state on the two-dimensional search grid, the root state is placed in the first layer candidate set, and a control variable set is formed;

[0109] For each candidate state in the control variable set, the grid cost of the corresponding grid position is read from the local cost table, the grid cost is combined with the candidate state to calculate a new cumulative cost, the new cumulative cost is appended to the matched pairing sequence, a partial matching path satisfying monotonicity is formed, and an alignment state is used to form a new candidate state, which is added to the expansion result set of the current layer;

[0110] The expansion result set is sorted in ascending order according to the new cumulative cost, beam pruning is performed, the pruned candidate states are passed to the next layer, and termination checking is performed according to the level count and the maximum expansion step number during transmission;

[0111] When the maximum expansion step number is reached or all candidate states become completed states, the current completed state set is aggregated to check each pairing sequence, the pairing sequence that passes the check is written back to the candidate path cache area in the form of grid coordinates, and a candidate path satisfying the constraint is formed.

[0112] As shown in Figure 2 the method comprises:

[0113] The two-dimensional search grid of the cross-modal alignment search is constructed based on the value range determination of the video time index and the text word sequence index. The video time index range is determined by the start frame number and the end frame number of the video frame sequence, and the frame rate information is used to calculate the mapping relationship between the timestamp and the frame index. The text word sequence index range is determined according to the length of the text vector sequence, starting from zero to the sequence length minus one. The two-dimensional grid adopts a matrix data structure, the number of rows is equal to the maximum value of the video time index plus one, and the number of columns is equal to the maximum value of the text word sequence index plus one. Each position in the grid corresponds to a potential alignment point, and the position coordinates are an ordered pair of row index and column index.

[0114] The generation of the atomic cost is achieved by calculating the similarity between the video vector and the text vector at the corresponding position, and the cosine similarity algorithm is used for similarity calculation. The algorithm extracts the video frame vector and the text word vector corresponding to the grid position, and obtains the similarity value by dividing the dot product of the two vectors by the product of the respective norms. The atomic cost is defined as one minus the similarity value, so that the higher the similarity, the lower the position cost. The value range of the cost is between zero and two, zero represents complete similarity, and two represents complete dissimilarity. The local cost table stores the atomic cost using a two-dimensional floating point number array, the row and column indexes of the array directly correspond to the grid coordinates, and the constant time cost query operation is supported.

[0115] The root state is set at the starting boundary position of the two-dimensional grid, and the specific coordinates are zero row and zero column, representing the alignment point of the start time of the video sequence and the start position of the text sequence. The root state data structure includes the current grid coordinates, the cumulative cost value, the matched paired sequence, and the state identifier fields. The cumulative cost value is initialized to zero, the matched paired sequence is initialized to an empty list, and the state identifier uses a unique integer number. The first layer candidate set uses a priority queue data structure to store, and the queue is sorted according to the cumulative cost value, and the state with lower cost has higher priority.

[0116] The management of the control quantity set adopts a hierarchical processing mechanism, and each layer corresponds to an expansion step in the search process. The expansion of the candidate state follows the monotonicity constraint of dynamic programming, and can only move to the right, down or right down from the current grid position. Moving to the right means that the text index increases while the video index remains unchanged, corresponding to the alignment mode of advancing the text while pausing the video. Moving down means that the video index increases while the text index remains unchanged, corresponding to the alignment mode of advancing the video while pausing the text. Moving to the right and down means that the video index and the text index increase simultaneously, corresponding to the alignment mode of advancing the video and the text synchronously.

[0117] The reading of the grid cost is achieved by the index access of the local cost table, and the reading operation locates the corresponding cost value according to the grid coordinates of the candidate state. The calculation of the new cumulative cost adds the cumulative cost of the current candidate state and the read grid cost. The cumulative cost reflects the total alignment cost from the root state to the current position, and the smaller the value, the better the alignment quality. The construction of the partial matching path is achieved by adding the new grid coordinates to the matched pairing sequence, which records the complete path from the root state to the current position.

[0118] The monotonicity verification ensures that the generated matching path meets the timing constraint condition, and the verification algorithm checks the monotonicity of adjacent coordinate pairs in the path. For any adjacent coordinate pair in the path, the row index of the latter coordinate must be greater than or equal to the row index of the former coordinate, and the column index must also be greater than or equal to the column index of the former coordinate. Paths that violate monotonicity are directly discarded and not added to the expansion result set. The generation of alignment states creates new candidate states for each valid expansion position, and the new state inherits the pairing sequence of the parent state and adds the coordinate information of the current position.

[0119] The expansion result set uses a dynamic array to store all candidate states generated in the current layer, and the array supports dynamic expansion to adapt to different expansion scales. The states in the set are arranged in the order of generation, and each state contains complete path information and cumulative cost value. The sorting operation uses the quicksort algorithm to arrange the candidate states in ascending order of cumulative cost value, and the states with the same cost are arranged in chronological order of generation time. The sorting process uses a stable sorting algorithm to ensure that the relative order of states with the same cost remains unchanged.

[0120] The beam pruning mechanism limits the number of candidate states in each layer to control the search complexity, and the pruning parameter is set to the beam width value, which is usually between 50 and 200. The pruning algorithm retains the first beam width of candidate states with the smallest cumulative cost, and discards the remaining states to reduce memory occupancy and computational overhead. During the pruning process, the algorithm records statistical information of the discarded states, including the number of discarded states, average cost value, and optimal cost value, etc.

[0121] The hierarchical transfer process moves the pruned candidate states to the processing queue of the next layer, and the transfer operation uses the shallow copy mechanism to reduce memory allocation overhead. The layer counter records the current processing layer number, which starts from zero and increments, and is used for termination condition checking. The maximum expansion step parameter limits the maximum depth of the search to prevent the search process from falling into an infinite loop or consuming excessive computational resources.

[0122] The termination checking mechanism monitors the end conditions of the search process, including reaching the maximum expansion step number and all candidate states turning into the completed state. The completed state is defined as the state reaching the right-bottom corner boundary of the two-dimensional grid, indicating that both the video sequence and the text sequence have been completely processed. The state transition check is implemented by comparing the grid coordinates of the candidate state with the grid boundary coordinates, and the completed state is determined when the row index is equal to the maximum row index of the grid and the column index is equal to the maximum column index of the grid.

[0123] The aggregation of the completed state set uses a list data structure to store all candidate paths that reach the completed state, and each element in the set contains a complete matching path and the corresponding cumulative cost value. The paired sequence verification process verifies the validity and integrity of each path, including path length, coordinate range, monotonicity constraint, and start and end point position. The path length verification ensures that the path contains enough alignment points, with a minimum length requirement of half the sum of the grid row and column numbers.

[0124] The coordinate range verification verifies that all coordinates in the path are within the valid grid range, and the path that exceeds the range is marked as invalid and removed from the result. The start and end point position verification confirms that the path starts at the top-left corner of the grid and ends at the right-bottom corner of the grid, and the path that does not meet the requirements is discarded. The paired sequence that passes the verification is converted to grid coordinates, and the coordinate sequence records all grid positions passed by the path.

[0125] The candidate path cache area uses a hash table data structure to store valid candidate paths, with the hash key being the unique identifier of the path and the hash value being the complete path information. The path identifier is generated by hashing all coordinates in the path, ensuring that different paths have different identifiers. The cache area supports fast path lookup, insertion, and deletion operations, and provides path deduplication function to avoid storing duplicate paths.

[0126] The constraint satisfaction verification ensures that the candidate path meets the business constraint conditions of cross-modal alignment, including alignment density, jump amplitude, and coverage integrity requirements. The alignment density constraint requires that the alignment points in the path be distributed relatively uniformly, avoiding excessive concentration or excessive sparsity of alignment patterns. The jump amplitude constraint limits the coordinate span between adjacent alignment points to prevent large temporal jumps. The coverage integrity constraint ensures that the path covers the main content area of the video and text sequences, with a coverage rate requirement of more than 85%.

[0127] The path quality evaluation is realized by comprehensively considering multiple indexes such as accumulated cost, path smoothness and alignment consistency. The path smoothness is measured by calculating the variance of coordinate changes between adjacent alignment points, and the smaller the variance is, the smoother the path is. The alignment consistency is evaluated by analyzing the semantic correlation of the alignment points in the path, and the correlation is calculated based on the semantic matching degree of the video content and the text content. The candidate paths are ranked according to the comprehensive quality scores, and the path with the highest score is output as the optimal alignment result.

[0128] In an optional implementation, the computing of the semantic consistency score of the paired elements on the candidate path, the action-entity relationship consistency score, and the obtaining of the comprehensive score of the candidate path under the multi-target weighted constraint include:

[0129] The paired index sequence corresponding to the candidate path is obtained, the video vector and the text vector are aligned according to the paired index sequence, and a paired element sequence is generated. The paired index sequence is checked for consistency to form a purified paired index. A bidirectional mapping table from the paired index sequence to the paired element index is established based on the purified paired index;

[0130] According to the bidirectional mapping table, the vector similarity and the vector residual norm of each paired element are calculated, the vector similarity and the vector residual norm are combined into a local semantic score using a monotonically increasing boundary constraint. The local semantic score is executed with a weighted sliding aggregation with boundary protection in the order of the paired element path to obtain a semantic consistency score;

[0131] Taking the paired index position in the bidirectional mapping table as an anchor point, the dependency relationship between the action and the entity at the anchor point is analyzed on the text side, the local relationship consistency score is calculated according to the dependency relationship between the action and the entity, and the local relationship consistency score is accumulated according to the time sequence and the semantic master-slave sequence to obtain an action-entity relationship consistency score;

[0132] The semantic consistency score and the action-entity relationship consistency score are used as target components, and the comprehensive score of the candidate path is obtained according to the target components.

[0133] The paired index sequence of the candidate path is extracted from the grid coordinate sequence of the path, and each grid coordinate contains a two-tuple of video time index and text word sequence index. The paired index sequence generation algorithm traverses all coordinate points in the path, extracts the row index of each coordinate point as the video index and the column index as the text index, and forms an ordered index pair sequence. Each index pair in the sequence represents a potential alignment relationship between a video frame and a text word, and the length of the sequence is equal to the number of coordinate points passed by the path.

[0134] The alignment operation of video vectors and text vectors extracts corresponding elements from respective vector sequences based on index values in the pair index sequence. Video vector extraction locates the feature vector of the corresponding frame in the video vector timeline through the video index, and text vector extraction locates the vector representation of the corresponding vocabulary in the text vector sequence through the text index. The generation of the pair element sequence combines the corresponding video vector and text vector of each index pair into a pair element, which is stored in the form of a structure and contains a video vector, a text vector, a pair index, and a metadata field.

[0135] The consistency check of the pair index sequence ensures the validity and reasonableness of the indexes through multiple verification mechanisms. The index range check verifies whether each video index is within the valid range of the video sequence and whether the text index is within the valid range of the text sequence. The temporal continuity check examines the temporal relationship between adjacent pair indexes to ensure that both the video index and the text index maintain a monotonically non-decreasing change trend. The jump amplitude check limits the jump distance between adjacent indexes, and marks an abnormal index pair when the jump distance exceeds a preset threshold.

[0136] The generation of the purified pair index is achieved through the identification and correction of abnormal indexes, including out-of-range indexes, indexes violating temporal continuity, and indexes with excessively large jump amplitudes. The correction strategy uses an interpolation method to handle missing intermediate indexes by generating reasonable intermediate index values between the positions before and after the abnormal indexes through linear interpolation. The removal of duplicate indexes is achieved through a hash set, which merges duplicate index pairs into a single index pair while retaining the earliest appearing index pair in the time sequence.

[0137] The establishment of the bidirectional mapping table uses a hash table data structure to store the mapping relationship between the pair index sequence and the pair element index. The forward mapping takes the position in the pair index sequence as the key and the index of the pair element in the pair element sequence as the value. The reverse mapping takes the pair element index as the key and the position in the pair index sequence as the value. The mapping table supports constant-time lookup operations, providing an efficient implementation of index conversion. The mapping table also records the metadata of the mapping relationship, including the mapping establishment time, index validity markers, and mapping quality scores.

[0138] The vector similarity calculation uses the cosine similarity algorithm to measure the semantic correlation between the video vector and the text vector in the pair element. The algorithm calculates the dot product of the two vectors, divides by the product of the respective vector norms to obtain the similarity value. The similarity value ranges from negative one to positive one, with positive values indicating similar vector directions, negative values indicating opposite vector directions, and zero indicating orthogonal vectors. The vector residual norm calculates the Euclidean norm of the difference between the two vectors, reflecting the absolute distance difference between the vectors. The residual norm value is always non-negative, with smaller values indicating closer vectors.

[0139] The application of monotonically increasing boundary constraints ensures that the local semantic score increases with the vector similarity and decreases with the vector residual norm. The constraint function maps the vector similarity to the range of zero to one, and achieves smooth mapping through the sigmoid function or piecewise linear function. The vector residual norm is mapped to the range of zero to one through inverse transformation, and the transformation formula is one divided by one plus the residual norm value. The local semantic score is calculated by the weighted average of the mapping value of the vector similarity and the mapping value of the vector residual norm, and the weight parameter is adjusted according to the application scene.

[0140] The weighted sliding aggregation algorithm with boundary protection processes the local semantic scores in the paired element sequence, and the aggregation process calculates the average score of the local area using a sliding window mechanism. The sliding window size is set to 5 to 10 elements, and the weighted average value of all local semantic scores in the window is calculated when the window slides on the sequence. The boundary protection mechanism handles special cases at the sequence boundary, and automatically adjusts the window size to adapt to the number of available elements when the window exceeds the sequence boundary. The weight distribution uses a Gaussian weight function, with the weight of the center position of the window being the largest and decreasing towards the edge position.

[0141] The calculation of the semantic consistency score is realized by weighted summation of the aggregation results of all sliding windows, and the weighting coefficients are determined according to the position of the window in the sequence. The windows in the middle position of the sequence have higher weights, and the windows in the boundary position of the sequence have lower weights. The final semantic consistency score is normalized to the range of zero to one, and the normalization process uses the minimum-maximum value normalization method. Intermediate results and statistical information are recorded during the score calculation process, including the number of windows, the average local score, the variance, etc.

[0142] The dependency relationship analysis between actions and entities is based on the anchor position in the bidirectional mapping table to perform syntactic analysis on the text side, and the anchor position corresponds to a specific word position in the text sequence. The analysis algorithm analyzes the role and relationship of the anchor word in the syntax tree from the anchor word. Action recognition is achieved through part-of-speech tagging and semantic role labeling, and predicate words are identified as action candidates. Entity recognition is achieved through named entity recognition and noun phrase extraction, and noun components are identified as entity candidates.

[0143] Dependency relationship calculation analyzes the syntactic dependency path and semantic relationship strength between actions and entities. The syntactic dependency path is calculated by a syntax tree traversal algorithm, and the path length is defined as the number of edges passed by the shortest dependency path between two words. The semantic relationship strength is calculated based on the dependency relationship type and path length, and the direct dependency relationship has the highest strength, and the indirect dependency relationship strength decreases with the path length. The relationship type weight is assigned according to the importance of the grammatical relationship, and the subject-predicate relationship and the verb-object relationship have higher weights.

[0144] The local relation consistency score comprehensively considers factors such as the number of action entity pairs, relation strength, and relation type distribution. The score is calculated by weighted summation of the relation strengths of all action entity pairs within a window surrounding the anchor point. Weight allocation considers the positional importance and semantic centrality of action entity pairs in the text; action entity pairs that are closer to the anchor point and semantically important receive higher weights.

[0145] Chained consistency accumulation processes the local relation consistency scores across multiple anchor points, traversing all anchor points in chronological order. This chronological order is determined by the anchor point's position in the paired index sequence, with earlier anchor points having earlier timestamps. The semantic hierarchy is determined by the dependency level of the action entity relationship, with primary actions and core entities having higher semantic priority. The accumulation algorithm employs a recursive accumulation mechanism, weighted and fused with the accumulated score of the previous anchor point.

[0146] The consistency score for action entity relationships is obtained by normalizing the chained cumulative results across all anchor points. This normalization method ensures the score falls within the range of zero to one. The score calculation process considers the global consistency of action entity relationships, verifying the existence of conflicts or contradictions in relationships at different anchor points through consistency checks. Conflict detection is based on logical constraints of action entity relationships, such as the requirement that the same entity cannot simultaneously execute mutually exclusive actions.

[0147] The weighted fusion of target components employs a multi-objective optimization weight allocation strategy, with semantic consistency score and action entity relationship consistency score participating in the fusion as two independent target components. The weight parameters are adjusted according to the needs of the application scenario; applications that emphasize semantic matching are assigned higher semantic consistency weights, while applications that emphasize action understanding are assigned higher relationship consistency weights.

[0148] The comprehensive score of the candidate path is calculated through a linear weighted combination of the target components, reflecting the overall quality of the candidate path across multiple evaluation dimensions. The score calculation formula is: semantic consistency score multiplied by semantic weight, plus action entity relationship consistency score multiplied by relationship weight. The weight sum equals one to ensure the boundedness of the score. The final score is normalized to a range of zero to one hundred to facilitate comparison and ranking among different candidate paths. The scoring process records detailed scoring information, including the original scores of each target component, weight allocation, and the final fusion result, providing a basis for result analysis and parameter tuning.

[0149] In one optional implementation, the alignment relationship between the video temporal segment and the text segment is determined using the candidate path with the highest comprehensive score. Based on this alignment relationship, the text segment is adaptively weighted to obtain the text matching result for the video content, including:

[0150] selecting a candidate path with the highest comprehensive score as a target path, solidifying a cross-modal alignment index of the target path as an alignment index baseline, checking boundary consistency of each pair of mapping segments according to a continuous segment boundary and a gap mark based on the alignment index baseline, detecting and correcting index overlap and omission between each pair of the mapping segments, and generating an alignment relationship table;

[0151] extracting a word vector, a position feature, and a dependency role label of an aligned text segment from a text vector sequence as a constraint based on the alignment relationship table, combining an action vector and a scene vector of a video-side aligned time sequence segment, calculating an alignment quality of each alignment unit, constructing a weight generation function based on the alignment quality, and reducing a weight of an alignment unit with a gap penalty coefficient to obtain an adaptive weight;

[0152] weighting a text vector corresponding to the adaptive weight to obtain a weighted text representation for the video content, calculating a similarity between the weighted text representation and a video vector within a window of the video time sequence segment, and aggregating the similarity according to a segment granularity of the alignment relationship table to form a text matching result.

[0153] The candidate path with the highest comprehensive score is selected by sorting and comparing the comprehensive scores of all candidate paths. The sorting algorithm arranges all candidate paths in descending order of score using quicksort. Paths with the same score are sorted according to secondary indicators such as path length and semantic consistency score to ensure the uniqueness and stability of the selected result. After the target path is determined, the algorithm extracts the complete grid coordinate sequence of the path, which contains all alignment point information from the starting position to the ending position.

[0154] The solidification process of the cross-modal alignment index converts the coordinate sequence of the target path into a standardized alignment index format. The solidification operation creates a persistent index data structure to store the alignment relationship. The alignment index baseline uses a compressed storage format to reduce storage space. Differential encoding records the index increment between adjacent alignment points rather than the absolute index value. The increment value is usually small and can be represented using fewer bits. The baseline data structure includes fields such as version information, compression algorithm identification, original path length, and compressed index sequence.

[0155] The generation of mapping segments is based on the combination of consecutive alignment points in the alignment index baseline. The segment division algorithm identifies continuous intervals and discontinuity points in the index sequence. A continuous interval is defined as a set of adjacent alignment points with an index increment within a preset threshold, and a discontinuity point identifies a position where the index jump exceeds the threshold. Each mapping segment contains attributes such as start index, end index, length information, and the number of included alignment points. The segment identifier uses a hash algorithm to generate a unique identifier based on the index range and content features of the segment.

[0156] The identification of the continuous segment boundary is achieved by analyzing the index distribution pattern in the alignment index baseline. The boundary detection algorithm calculates the interval statistics between adjacent indexes. The mean and variance of the interval are used to determine the range of normal intervals, and intervals that exceed the normal range are identified as segment boundaries. The gap marker is generated by detecting the blank area in the index sequence, which is defined as the range of consecutive missing indexes exceeding the minimum gap length. The boundary marker and the gap marker are stored in the baseline data with special encoding, which facilitates the subsequent processing algorithm to identify and parse.

[0157] The boundary consistency verification verifies whether the index range of each pair of mapping segments meets the continuous segment boundary constraint. The verification algorithm checks whether the segment boundary is aligned with the segment boundary marker. The inconsistent segment boundary is corrected by the boundary adjustment algorithm, and the adjustment strategy prioritizes maintaining the semantic integrity of the segment. The index overlap detection identifies the overlapping area by comparing the index ranges of adjacent mapping segments, and the overlap length is calculated as the intersection length of the two segment index ranges. The overlap correction uses a priority strategy, and the segment with a higher score retains the original range, and the segment with a lower score performs boundary retraction.

[0158] The index omission detection identifies the missing area by comparing the total coverage range of the mapping segment with the expected coverage range, and the omission correction generates the missing segment by the interpolation algorithm. The interpolation algorithm estimates the properties of the missing segment based on the characteristics of the segments before and after the missing area, including alignment quality, content type, and other features. The generation of the alignment relationship table uses a table data structure to store the corrected mapping segment information, and the table contains fields such as segment identification, index range, alignment quality, and correction record.

[0159] The word vector extraction of the aligned text segment is based on the text index range in the alignment relationship table to extract the corresponding vector from the text vector sequence. The extraction algorithm traverses all the text index positions contained in the segment, collects the corresponding word vectors to form a segment vector set. The position feature is calculated by the absolute position and relative position of the word item in the text sequence. The absolute position is the index value of the word item in the entire text, and the relative position is the position proportion of the word item in the current segment.

[0160] The extraction of dependency role labels is based on the results of syntactic dependency analysis, and the labels contain syntactic roles and semantic roles of word items. The syntactic role identifies the function of the word item in the syntactic tree, such as subject, object, and attributive. The semantic role identifies the function of the word item in the semantic framework, such as agent, patient, and instrument. The label information is represented by one-hot encoding or embedding vector, which facilitates subsequent numerical calculation processing.

[0161] The feature extraction of the video side-aligned timing segment includes two dimensions of action vectors and scene vectors, and the feature extraction is based on the video index range in the alignment relationship table. The action vector is extracted from the video frame sequence by an action recognition model, and the model output includes action category probability distribution and action intensity information. The scene vector is extracted by a scene understanding model, including scene category, object distribution, spatial layout and other scene description information. The vector dimensions are unified to 512 dimensions or 1024 dimensions, which is convenient for cross-modal feature fusion calculation.

[0162] The alignment quality calculation of the alignment unit comprehensively considers the matching degree of text features and video features, and the quality evaluation adopts multi-dimensional similarity calculation. The semantic similarity of the word vector and the action vector is calculated by the cosine similarity algorithm, which measures the semantic consistency of the text content and the video action. The spatial similarity of the position feature and the scene vector is calculated by the Euclidean distance, which reflects the matching degree of the spatial information of the text description and the video scene. The structural similarity of the dependency role and the action category is calculated by the role mapping table, which defines the corresponding relationship between the syntax role and the action category.

[0163] The weight generation function constructs a nonlinear mapping relationship based on the alignment quality, and the function design ensures that high-quality alignment units obtain higher weights. The mapping function adopts sigmoid function or piecewise linear function to realize smooth quality to weight conversion, and the function parameters are obtained by training data optimization. The quality threshold is set to distinguish high-quality and low-quality alignment units, and the units exceeding the threshold obtain a weight close to one, and the units below the threshold have a rapid weight decay.

[0164] The application of gap penalty coefficient reduces the weight of alignment units containing gap marks, and the penalty strength is determined according to the gap length and type. Short gaps use slight punishment, and long gaps use severe punishment, with a penalty coefficient range of 0.1 to 0.9. The reduction calculation is realized by multiplying the original weight by the penalty coefficient, and the reduced weight ensures that the negative impact of the gap area on the final result is controlled. The normalization processing of adaptive weight ensures that all weight values are within a reasonable range, and the normalization adopts softmax function or minimum maximum value normalization method.

[0165] The calculation of weighted text representation is realized by the element-by-element multiplication of adaptive weight and corresponding text vector, and the weighting operation assigns corresponding weight values to each dimension of each text vector. The weight broadcast mechanism handles the case where the weight dimension and the vector dimension do not match, and the broadcast operation expands the low-dimensional weight to the high-dimensional vector space. The weighted text vector retains the original semantic information while highlighting the contribution of high-quality alignment parts, and the weighted result avoids numerical overflow problem through normalization processing.

[0166] The setting of the video timing segment window is based on the timing range information in the alignment relationship table, and the window size is dynamically adjusted according to the segment length and content complexity. Short segments use smaller windows to maintain timing accuracy, and long segments use larger windows to capture global features. The window sliding strategy uses an overlapping sliding mechanism, maintaining a 50% overlap rate between adjacent windows to ensure feature continuity. Similarity calculation is performed within each window, calculating the weighted text representation and the average similarity of the video vectors within the window.

[0167] The similarity summary is performed at the segment granularity of the alignment relationship table, and the summary algorithm collects the similarity scores of all windows within the same segment. The summary strategy uses a weighted average method, and the weights are assigned according to the importance of the window in the segment. The center window of the segment has a higher weight, and the boundary window has a lower weight. The summary result forms a matching score at the segment level, and the score ranges from zero to one. The higher the score, the better the matching degree of the text and the video content.

[0168] The output of the text matching result uses a structured format, including segment identification, matching score, confidence evaluation, and detailed matching information. The matching information includes keyword matching, action matching, scene matching, and other sub-results, which facilitates result analysis and application integration. Confidence evaluation is calculated based on statistical information during the matching process, including alignment quality variance, weight distribution uniformity, and other indicators. The final result is sorted and output according to the matching score, providing high-quality cross-modal matching results for downstream applications.

[0169] In an optional implementation, according to the alignment index baseline, the boundary consistency of each pair of mapping segments is checked according to the continuous segment boundary and the gap mark, the index overlap and the missing between each pair of the mapping segments are detected and corrected, and an alignment relationship table is generated, including:

[0170] The continuous segment boundary and the gap mark are extracted from the alignment index baseline, and the initial start and end indexes of each mapping segment are located according to this; the mapping segments are paired according to the natural order of the pairing indexes, forming a segment pair candidate sequence;

[0171] Taking the segment pair candidate sequence as the object, the joint checking of adjacent mapping segment pairs is performed according to the continuous segment boundary and the gap mark, the cross of the start and end indexes within the same continuous segment, the time or word sequence overlap, and the missing interval indicated by the gap mark are detected pair by pair, and the detection results are attached to the corresponding segment pair entries in the form of conflict records;

[0172] Based on the conflict records, the mapping segments that pass the joint checking are pointed and corrected, the cross-border overlap is preferentially cut according to the continuous segment boundary, and if there is still overlap after cutting, the start and end boundary of the mapping segment pair is retracted according to the order of the pairing indexes, forming a corrected segment entry set carrying correction parameters;

[0173] According to the modified fragment entry set, a modified fragment sequence is composed, and global consistency checking is performed on the basis of the modified parameter write-back; when the global consistency is satisfied, an alignment relationship table is generated from the modified fragment sequence as the only source.

[0174] The data structure analysis of the alignment index baseline starts from a binary or text format baseline file, and the parsing program creates a file reading buffer, sets the buffer size to 4096 bytes to optimize reading performance. The program scans the baseline data byte by byte, identifies the specific byte sequence of the continuous segment boundary marker, and the continuous segment start marker is usually 0x53544152544D4152 (corresponding to ASCII code "STARTMAR"), and the continuous segment end marker is 0x454E444D41524B45 (corresponding to ASCII code "ENDMARKE"). When the start marker is detected, the program reads the 4 bytes after the marker as the segment identifier, and then reads 8 bytes as the segment start index value, which is parsed into a 64-bit unsigned integer using big-endian byte order. The processing of the end marker is similar, after extracting the segment identifier and the end index value, the program stores the segment information into a hash table data structure, taking the segment identifier as the key and the start and end index pair as the value.

[0175] The gap marker identification uses 0x4741504D41524B45 (corresponding to ASCII code "GAPMARKE") as the identification sequence, and each gap marker is followed by 16 bytes of data, the first 8 bytes representing the gap start position and the last 8 bytes representing the gap length. The program calculates the gap termination position as the start position plus the length value, and adds the gap information to the dedicated gap list structure. The initial index extraction of each mapping fragment is realized through the metadata area of the segment header, and the segment data starts with a 32-byte fixed header, which contains the segment unique identifier (16-byte UUID), the start index (8 bytes) and the length field (8 bytes). The program traverses all the segments, extracts the start index and length information, and calculates the termination index as the start index plus the length minus one.

[0176] The mapping fragment pairing algorithm is based on the sorting of the pairing index, and the program creates a fragment information array, each array element containing the segment identifier, the pairing index value, the start and end index range, etc. The quicksort algorithm sorts the array in ascending order according to the pairing index value, and uses three-way quicksort optimization to handle the case where the pairing index value is the same. After sorting, the program traverses the array to generate a candidate sequence of fragment pairs, and the adjacent array elements are automatically paired, with the pairing rule being that the fragment at index position i and the fragment at index position i+1 form a pair. For the case where there is a gap in the pairing index value, the program detects the difference between the two consecutive pairing index values, and when the difference is greater than 1, it indicates that there is a gap, and the fragments in the gap are paired in order until the gap is filled.

[0177] The segment pair data structure adopts a linked list node form, each node containing a source segment pointer, a target segment pointer, a pairing type identifier, and a conflict record linked list head pointer. The pairing type identifier adopts an enumeration value, including sequential pairing, cross pairing, nested pairing, and the like. The program assigns each segment pair a unique 64-bit identifier, which is composed of the pairing type (8 bits), the source segment index (28 bits), and the target segment index (28 bits).

[0178] The joint checking process performs multiple detection algorithms for each segment pair. The start and end index intersection detection is achieved by interval intersection judgment. The program compares the end index of segment A with the start index of segment B. When the end index of A is greater than or equal to the start index of B and the start index of A is less than or equal to the end index of B, it is determined that there is an intersection. The intersection length is calculated as the length of the overlapping part of the two intervals, specifically, the smaller end index minus the larger start index plus one. The time overlap detection is applicable to segments containing timestamps. The program extracts timestamp information from the segment extension header. The timestamp is represented by a 64-bit integer with microsecond precision. The time overlap judgment uses the same interval intersection algorithm as the index intersection. The overlap duration is the number of microseconds of the overlapping time interval.

[0179] The word sequence overlap detection is performed for text content segments. The program divides the segment content into an array of words based on spaces and punctuation marks. For each word, a hash value is calculated to construct a word hash set. The word sequence overlap between two segments is detected by the intersection operation of the hash sets. The overlap coefficient is obtained by dividing the intersection size by the union size. When the overlap coefficient exceeds the preset threshold of 0.3, it is determined that there is significant word sequence overlap. The gap omission detection is achieved by comparing the coverage range of the segments with the complete range of the continuous segments. The program constructs a Boolean array to represent the coverage state of the index range, and the array length is equal to the index range length of the continuous segments. All segments are traversed, and the index positions covered by the segments are marked as true in the Boolean array. After detection, the array is scanned to identify the continuous intervals with false values as the missing intervals.

[0180] The conflict record is stored in a structured format, each record containing a conflict type enumeration value, a conflict start position, a conflict length, a conflict severity score, and a timestamp. The conflict type enumeration defines specific types such as index intersection, time overlap, word sequence overlap, and gap omission. The conflict severity score adopts an integer range of 0 to 100. The scoring algorithm calculates the proportion of the conflict length to the total length of the segment. The higher the proportion, the higher the score. The conflict record is attached to the corresponding segment pair entry through a linked list structure, and the linked list head pointer is stored in the conflict record field of the segment pair node.

[0181] The point correction algorithm performs differentiated processing strategies according to the type and severity of the conflict record. The boundary constraint priority principle is adopted for the cutting processing of cross-border overlap. The program checks whether the start and end indexes of the segment cross the continuous segment boundary. Cross-border detection is achieved by comparing the segment index range with the continuous segment boundary position. When the segment start index is less than the continuous segment start boundary or the end index is greater than the continuous segment end boundary, it is determined as cross-border. The cross-border part is cut off by the cutting algorithm. The specific operation is to adjust the segment start index to the larger value of the continuous segment start boundary and the original start index, and adjust the end index to the smaller value of the continuous segment end boundary and the original end index.

[0182] The boundary retraction mechanism handles the overlap that still exists after cutting. The retraction priority is determined according to the value of the paired index. The segment with a smaller paired index value has a higher priority. The program calculates the length of the overlap interval and determines the retraction strategy according to the proportion of the overlap length and the remaining length of the segment. When the overlap length is less than 20% of the remaining length of the segment, the segment with lower priority is given priority to move the start boundary backward. The distance of backward movement is equal to the overlap length plus a safety interval value. When the overlap length exceeds 20% but is less than 50%, a bidirectional retraction strategy is adopted. The two segments bear half of the retraction distance respectively. When the overlap length exceeds 50%, the program marks the pair as a high conflict pair and requires manual intervention for processing.

[0183] The record of the correction parameter adopts a correction log structure. Each log contains fields such as original start index, original end index, corrected start index, corrected end index, correction type identifier, correction timestamp, and correction reason description. The correction type identifier includes enumeration values such as boundary cutting, start retraction, end retraction, and bidirectional retraction. The corrected segment entry adds a correction parameter field to the original segment data structure. This field is a pointer to the correction log entry. The corrected segment entry set is stored in a dynamic array structure, supporting fast insertion and lookup operations.

[0184] The reorganization of the corrected segment sequence is based on the reordered index values after correction. The program uses the merge sorting algorithm to ensure the stability of the sorting. The sorting key is the corrected start index value. During the sequence generation process, the program verifies the index validity of each corrected segment, checks whether the start index is less than the end index, whether the index value is within the continuous segment boundary range, and whether it overlaps with the gap area. Invalid segments will be marked and removed from the sequence. The correction parameter write-back operation updates the corrected index value to the main index field of the segment, while retaining the original index value in the history field. The write-back process uses atomic operations to ensure data consistency.

[0185] The global consistency check ensures the correctness of the correction result through multi-stage verification. In the first stage, the index continuity check is performed. The program traverses the sequence of corrected segments to verify whether the index interval between adjacent segments meets the expectation. The allowed interval range is 0 to the maximum allowed interval value. In the second stage, the range coverage verification is performed. The union of all segment index ranges is calculated and compared with the expected coverage range. The coverage rate must reach more than 95% to pass the verification. In the third stage, the constraint condition verification is performed. The corrected segments are checked for violation of the continuous segment boundary constraint and the gap marker constraint. Any violation will result in a failed check.

[0186] The generation of the alignment relationship table adopts a table data structure. The table header defines fields including source segment identifier, target segment identifier, alignment type, source start index, source end index, target start index, target end index, and confidence score. The program traverses the sequence of corrected segments that pass the global consistency check to generate an alignment relationship record for each valid segment pair. The alignment type is determined according to the index relationship between the segments, including one-to-one alignment, one-to-many alignment, and many-to-one alignment. The confidence score considers factors such as correction amplitude, conflict number, and coverage integrity. The weighted average algorithm is used for calculation, with weights of 0.4, 0.3, and 0.3, respectively.

[0187] The calculation of the index mapping relationship is implemented through the linear interpolation algorithm. When the source segment index range is [s1, s2] and the target segment index range is [t1, t2], the calculation formula for mapping the source index s to the target index is the target start index plus the relative position of the source index multiplied by the target range length divided by the source range length. The mapping precision is maintained to two decimal places, and the final result is rounded to the nearest integer. After the generation of the alignment relationship table, the program performs integrity verification to check whether the index mapping relationship of each record in the table is monotonically increasing, whether the confidence score is within the valid range, and whether there are duplicate records. After passing the verification, the relationship table is serialized and stored in binary format for subsequent quick loading and use.

[0188] In a second aspect of the embodiment of the present application, a multi-modal data pairing system based on deep learning is provided, comprising:

[0189] The first unit is configured to obtain an original multi-frame sequence of a video to be processed and a target text, extract a time sequence segment from each frame in the original multi-frame sequence at a fixed sampling interval, form a frame group set containing overlapping adjacent frames, perform sentence processing and lexical normalization processing on the target text, retain the word order and record the appearance position index of each word, and form an alignable text sequence.

[0190] The second unit is configured to perform spatio-temporal feature extraction on the frame set to obtain a video time sequence vector sequence corresponding to a time index; and perform embedding mapping and dependency relationship coding on the text sequence, inject syntax dependency relationship constraints into word vectors by using message passing operation of a graph structure to obtain a text vector sequence corresponding to a word sequence index;

[0191] The third unit is configured to perform cross-modal alignment search while keeping the time sequences of the video and the text unchanged, construct a set of monotonous matching paths between the video vector time sequence and the text vector sequence by using a hierarchical candidate expansion strategy, and output a candidate path by taking the cumulative cost of the matching path as a sorting basis.

[0192] The fourth unit is configured to calculate a semantic consistency score and an action entity relationship consistency score of a pair of elements on the candidate path, and obtain a comprehensive score of the candidate path under a multi-target weighted constraint.

[0193] The fifth unit is configured to determine the alignment relationship between the video time sequence segment and the text segment by using the candidate path with the highest comprehensive score, and perform adaptive weight weighting on the text segment based on the alignment relationship to obtain a text matching result for the video content.

[0194] In a third aspect, an electronic device is provided, including:

[0195] a processor;

[0196] a memory for storing processor-executable instructions;

[0197] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0198] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0199] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for implementing various aspects of the present application.

[0200] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep learning-based multi-modal data pairing method, characterized in that, The method comprises the following steps: obtaining an original multi-frame sequence of a video to be processed and a target text, extracting a time sequence segment at a fixed sampling interval for each frame in the original multi-frame sequence to form a frame group set containing adjacent frame overlaps; performing sentence processing and lexical normalization processing on the target text, retaining word order and recording the appearance position index of each word, to form an alignable text sequence; extracting spatiotemporal features from the frame group set to obtain a video time sequence vector sequence corresponding to the time index; meanwhile, embedding mapping and dependency relationship coding are performed on the text sequence, and the syntactic dependency relationship constraints are injected into the word vector by using the message passing operation of the graph structure to obtain a text vector sequence corresponding to the word sequence index; under the premise that the time sequence of the video and the text is not disturbed, cross-modal alignment search is performed, a monotonous matching path set is constructed between the video vector time sequence and the text vector sequence by using a hierarchical candidate expansion strategy, and the cumulative cost of the matching path is taken as the basis for sorting, and a candidate path is output; the semantic consistency score and the action entity relationship consistency score of the paired elements on the candidate path are calculated, and the comprehensive score of the candidate path is obtained under the multi-target weighted constraint; the alignment relationship between the video time sequence segment and the text segment is determined by the candidate path with the highest comprehensive score, and the text matching result for the video content is obtained based on the adaptive weight weighting of the text segment based on the alignment relationship.

2. The method of claim 1, wherein, The embedding mapping and dependency relationship coding of the text sequence, the injection of the syntactic dependency relationship constraints into the word vector by using the message passing operation of the graph structure, and the obtaining of the text vector sequence corresponding to the word sequence index comprise: generating an initial word feature set and an index mapping table according to the concatenation strategy of character-level segmentation and word-level segmentation on the text sequence; embedding mapping is performed on the initial word feature set and the index mapping table by using a mapping function to generate an initial word vector sequence with the same length as the word sequence index; the initial word vector sequence, the corresponding word sequence index and the character interval are stored together to form an initial word vector table with index markers; performing syntactic dependency analysis on the text sequence based on the initial word vector table to obtain a syntactic graph structure, performing forward and backward transmission of the syntactic graph structure and the initial word vector table at a fixed number of rounds, and outputting an intermediate word vector sequence corresponding to the word sequence index and an index consistency record matched therewith after all transmission rounds are completed; the intermediate word vector sequence is rearranged and checked according to the word sequence index, and after the check is passed, a text vector sequence corresponding to the word sequence index and maintaining consistency in semantics and syntax is output.

3. The method of claim 2, wherein, performing syntactic dependency analysis on the text sequence based on the initial word vector table to obtain a syntactic graph structure, performing forward and backward transmission of the syntactic graph structure and the initial word vector table at a fixed number of rounds, and outputting an intermediate word vector sequence corresponding to the word sequence index and an index consistency record matched therewith after all transmission rounds are completed, Perform index integrity check on the initial word vector table, eliminate index conflicts and complete missing character interval records to obtain an initial input baseline with consistent indexes; Perform syntax dependency analysis on the text sequence based on the initial input baseline to obtain a syntax graph structure; According to the syntax graph structure and the initial word vector table, perform forward pass based on the in-edge and backward pass based on the out-edge under a preset fixed round; In each round of transmission process, for each dependency edge, perform linear differentiable transformation on the vector propagated along the edge, and aggregate the multiple information reaching the same node to obtain the round aggregation vector of the node; Align and cache the round aggregation vector according to the index mapping table and the word order index of the node in the initial word vector table to generate a round check record; After completing the transmission of the fixed round, check the index mapping relationship and aggregated value of each node for consistency based on the word order index and the round check record; Repeat backtracking and interpolation for the nodes that do not pass the check until the index consistency and value stability conditions are met to obtain an intermediate word vector sequence; Based on the intermediate word vector sequence, aggregate the alignment metadata of each node in each round to form an index consistency record.

4. The method of claim 1, wherein, Perform cross-modal alignment search, use a hierarchical candidate expansion strategy to construct a monotonic matching path set between the video vector time sequence and the text vector sequence, and output the candidate path based on the cumulative cost of the matching path as the sorting basis, including: Based on the value range of the video time index and the text word order index, construct a two-dimensional search grid; Generate atomic cost for the two-dimensional search grid, and cache the atomic cost as a local cost table; Take the starting boundary as the root state on the two-dimensional search grid, and put the root state into the first layer candidate set to form a control quantity set; For each candidate state in the control quantity set, read the grid cost of the corresponding grid position from the local cost table, calculate the new cumulative cost by combining the grid cost with the candidate state, and append the new cumulative cost to the matched pairing sequence to form a partial matching path that satisfies monotonicity, and form a new candidate state with the alignment state, and add it to the expansion result set of the current layer; Sort the expansion result set according to the new cumulative cost from small to large, perform beam pruning, and pass the pruned candidate state to the next layer, and perform termination check according to the layer count and the maximum expansion step number during transmission; When the maximum expansion step number is reached or all candidate states are converted to completed states, check each pairing sequence for the current completed state set, and write the pairing sequence that passes the check to the candidate path cache area in the form of grid coordinates to form a candidate path that satisfies the constraints.

5. The method of claim 1, wherein, Calculate the semantic consistency score and action entity relationship consistency score of the paired elements in the candidate path, and obtain the comprehensive score of the candidate path under the multi-objective weighted constraint, including: obtaining a pair index sequence corresponding to the candidate path, aligning the video vector and the text vector according to the pair index sequence to generate a pair element sequence; performing consistency checking on the pair index sequence to form a purified pair index; and establishing a bidirectional mapping table from the pair index sequence to the pair element index based on the purified pair index; calculating vector similarity and vector residual norm for each pair element according to the bidirectional mapping table, combining the vector similarity and the vector residual norm into a local semantic score using a monotonically increasing boundary constraint; and performing weighted sliding aggregation with boundary protection on the local semantic score in the order of the path of the pair element to obtain a semantic consistency score; taking the pair index position in the bidirectional mapping table as an anchor point, analyzing the dependency relationship between the action and the entity at the anchor point on the text side, calculating a local relationship consistency score according to the dependency relationship between the action and the entity, and imposing chain consistency accumulation on the local relationship consistency score according to the time sequence and the semantic master-slave sequence to obtain an action-entity relationship consistency score; taking the semantic consistency score and the action-entity relationship consistency score as target components, and obtaining a comprehensive score of the candidate path according to the target components.

6. The method of claim 1, wherein, determining the alignment relationship between the video time sequence segment and the text segment by selecting the candidate path with the highest comprehensive score as the target path, and solidifying the cross-modal alignment index of the target path into an alignment index baseline; performing boundary consistency checking on each pair of mapping segments according to the continuous segment boundary and the gap marker based on the alignment index baseline, detecting and correcting the index overlap and omission between each pair of the mapping segments to generate an alignment relationship table; extracting the term vector, position feature and dependency role label of the aligned text segment from the text vector sequence, combining the action vector and the scene vector of the aligned time sequence segment on the video side, and calculating the alignment quality of each alignment unit under the constraint of the alignment relationship table; constructing a weight generation function based on the alignment quality, and reducing the weight of the alignment unit with a gap penalty coefficient to obtain an adaptive weight; weighting the corresponding text vector according to the adaptive weight to obtain a weighted text representation for the video content; and calculating the similarity between the weighted text representation and the video vector in the window of the video time sequence segment, and aggregating the similarity according to the segment granularity of the alignment relationship table to form a text matching result. performing boundary consistency checking on each pair of mapping segments according to the continuous segment boundary and the gap marker based on the alignment index baseline, detecting and correcting the index overlap and omission between each pair of the mapping segments to generate an alignment relationship table includes:

7. The method of claim 6, wherein, extracting the continuous segment boundary and the gap marker from the alignment index baseline, and positioning the initial start and end index of each mapping segment according to the extracted continuous segment boundary and the gap marker; and pairing the mapping segments according to the natural order of the pair index to form a segment pair candidate sequence; ​ The candidate sequence is subjected to the fragment, and joint checking is performed on adjacent mapping fragment pairs according to the continuous segment boundary and the gap mark pair, and the detection results are attached to the corresponding fragment pair entry in the form of conflict records; Based on the conflict records, the mapping fragments subjected to joint checking are corrected, and the cross-border overlap is preferentially subjected to cutting according to the continuous segment boundary. If there is still overlap after cutting, the start and end boundary of the mapping fragment pair is subjected to retraction according to the order of the pair index, and a corrected fragment entry set carrying a correction parameter is formed; According to the corrected fragment entry set, a corrected fragment sequence is formed, and global consistency checking is performed on the basis of correction parameter rewriting; when the global consistency is satisfied, an alignment relationship table is generated from the corrected fragment sequence as the only source.

8. A deep learning based multi-modal data pairing system for implementing the method of any one of claims 1-7, characterized in that, Comprise: A first unit for obtaining an original multi-frame sequence of a video to be processed and a target text, extracting a time sequence fragment at a fixed sampling interval for each frame in the original multi-frame sequence to form a frame group set containing adjacent frame overlaps; Perform sentence processing and lexical normalization processing on the target text, retain the word order and record the appearance position index of each word, and form an alignable text sequence; A second unit for performing spatiotemporal feature extraction on the frame group set to obtain a video time sequence vector sequence corresponding to the time index; while performing embedding mapping and dependency relationship coding on the text sequence, injecting the syntactic dependency relationship constraint into the word vector using the message passing operation of the graph structure to obtain a text vector sequence corresponding to the word sequence index; A third unit for performing cross-modal alignment search while keeping the time order of video and text unchanged, using a hierarchical candidate expansion strategy to construct a monotonic matching path set between the video vector time sequence and the text vector sequence, and taking the cumulative cost of the matching path as the basis for sorting to output the candidate path; A fourth unit for calculating the semantic consistency score and action entity relationship consistency score of the paired elements on the candidate path, and obtaining the comprehensive score of the candidate path under the multi-objective weighted constraint; A fifth unit for determining the alignment relationship between the video time sequence fragment and the text fragment with the highest comprehensive score of the candidate path, and obtaining the text matching result for the video content based on the adaptive weight weighting of the text fragment.

9. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1-7.

Images