Methods, devices and electronic equipment for matching video data

Abstract

This application discloses a video data matching method, apparatus, and electronic device, belonging to the field of video processing technology. The video data matching method includes: obtaining a first initial vector and a second initial vector, wherein the first initial vector includes modal features of at least two preset modalities for each video frame in a first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in a second video sequence; processing the first initial vector and the second initial vector using an attention mechanism to obtain a first target vector and a second target vector; constructing a similarity matrix between the first video sequence and the second video sequence based on the first target vector and the second target vector; and determining matching video segments in the first video sequence and the second video sequence based on the similarity matrix.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and more specifically to a method, apparatus, and electronic device for matching video data. Background Technology

[0002] In related technologies, when determining whether a video is original or whether two videos are duplicates, it is usually necessary to compare the videos to determine if their content is consistent. A common method for comparing two videos is keyword matching; however, this method suffers from low accuracy in the matching results. Summary of the Invention

[0003] This application provides a video data matching method, apparatus, and electronic device, which can solve the problem of low accuracy of matching results in existing video matching methods.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for matching video data, including:

[0006] Obtain a first initial vector and a second initial vector, wherein the first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence;

[0007] The first initial vector and the second initial vector are processed using an attention mechanism to obtain the first target vector and the second target vector;

[0008] Based on the first target vector and the second target vector, a similarity matrix between the first video sequence and the second video sequence is constructed;

[0009] Based on the similarity matrix, the video segments that match in the first video sequence and the second video sequence are determined.

[0010] Secondly, embodiments of this application provide a video data matching device, comprising:

[0011] The acquisition module is used to acquire a first initial vector and a second initial vector, wherein the first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence;

[0012] The processing module is used to process the first initial vector and the second initial vector using an attention mechanism to obtain the first target vector and the second target vector;

[0013] A construction module is used to construct a similarity matrix between the first video sequence and the second video sequence based on the first target vector and the second target vector;

[0014] The determination module is used to determine the video segments that match the first video sequence and the second video sequence based on the similarity matrix.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a readable storage medium, characterized in that a program or instructions are stored on the readable storage medium, and when the program or instructions are executed by a processor, they implement the steps described in the first aspect above.

[0017] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run a program or instructions, the program or instructions being executed by the processor to implement the steps described in the first aspect above.

[0018] In this embodiment, by making the first initial vector include modal features of at least two preset modalities of each video frame in the first video sequence, and the second initial vector include the modal features of the at least two preset modalities of each video frame in the second video sequence, and by using an attention mechanism to perceive the modal features of different modalities in the first and second initial vectors, a similarity matrix is ​​constructed. This allows the constructed similarity matrix to reflect the similarity of different modalities in the two video sequences, thus making the matched video segments determined based on the similarity matrix more accurate. Attached Figure Description

[0019] Figure 1 This is one of the flowcharts illustrating the video data matching method provided in the embodiments of this application;

[0020] Figure 2 This is a similarity chart constructed based on a similarity matrix in the embodiments of this application;

[0021] Figure 3 This is the chart after normalization of the similarity chart in the embodiments of this application;

[0022] Figure 4 This embodiment of the application uses a greedy search algorithm to search for the... Figure 3 The search matrix is ​​obtained after processing the chart;

[0023] Figure 5 In the embodiments of this application Figure 4 The best matching segment was determined based on this;

[0024] Figure 6 yes Figure 3 Any four adjacent matrix elements;

[0025] Figure 7 This is a schematic diagram illustrating the merging of sequence pairs in an embodiment of this application;

[0026] Figure 8 This is a second schematic flowchart of the video data matching method provided in the embodiments of this application;

[0027] Figure 9 This is a schematic diagram illustrating the 3D attention mechanism processing of the first video sequence and the second video sequence in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of the video data matching device provided in the embodiments of this application;

[0029] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0030] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The video data matching method, apparatus, and electronic device provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0034] Please see Figure 1 , Figure 1 This is a flowchart illustrating a video data matching method provided in an embodiment of the present disclosure. The video data matching method includes the following steps:

[0035] Step S101: Obtain a first initial vector and a second initial vector. The first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence.

[0036] Step S102: Process the first initial vector and the second initial vector using an attention mechanism to obtain the first target vector and the second target vector;

[0037] Step S103: Based on the first target vector and the second target vector, construct a similarity matrix between the first video sequence and the second video sequence;

[0038] Step S104: Based on the similarity matrix, determine the video segments that match in the first video sequence and the second video sequence.

[0039] In this context, one of the first video sequence and the other of the second video sequence can be the video sequence to be identified, and the other can be a reference video sequence. Specifically, it can be determined whether the video sequence to be identified is an original video sequence or a duplicate video sequence by judging whether the first video sequence and the second video sequence are repeated.

[0040] The aforementioned at least two modal features are modal features of different types. Furthermore, the modal features can be used to characterize the modal features of the video frame in a specific modality. For example, the preset modality may include image modality, audio modality, and subtitle modality, etc. Thus, by determining whether the modal features of each video frame in different video sequences match, the matching video segments in the first video sequence and the second video sequence can be identified.

[0041] The video segment matching the first video sequence and the second video sequence can refer to a video segment that is similar to or identical to the first video sequence and the second video sequence. When the first video sequence and the second video sequence are completely identical, the video segment matching the first video sequence and the second video sequence is either the first video sequence or the second video sequence.

[0042] Specifically, the similarity between any video frame in the first video sequence and any video frame in the second video sequence can be calculated based on the modal features of the preset modality to obtain the similarity matrix. For example, the similarity matrix can be represented as:

[0043]

[0044] Among them, Z i *Z j The similarity is defined as the similarity between the i-th frame in the first video sequence and the j-th frame in the second video sequence, where i ranges from 1 to 3 and j ranges from 4 to 6.

[0045] In one embodiment of this disclosure, after obtaining the similarity matrix, the similarity between each video frame of the first video sequence and each video frame of the second video sequence can be directly determined based on the similarity matrix, thereby identifying video segments with high similarity between the first and second video sequences, and defining the identified video segments with high similarity as matching video segments between the first and second video sequences. For example, video segments formed by video frames in the first and second video sequences with a similarity exceeding a preset similarity can be defined as the matching video segments.

[0046] In addition, in another embodiment of this disclosure, after obtaining the similarity matrix, mathematical operations can also be performed based on the similarity matrix using algorithms in related technologies to obtain video segments that match the first video sequence and the second video sequence.

[0047] In this embodiment, by making the first initial vector include modal features of at least two preset modalities of each video frame in the first video sequence, and the second initial vector include the modal features of the at least two preset modalities of each video frame in the second video sequence, and by using an attention mechanism to perceive the modal features of different modalities in the first and second initial vectors, a similarity matrix is ​​constructed. This allows the constructed similarity matrix to reflect the similarity of different modalities in the two video sequences, thus making the matched video segments determined based on the similarity matrix more accurate.

[0048] Optionally, the step of processing the first initial vector and the second initial vector using an attention mechanism to obtain the first target vector and the second target vector includes:

[0049] The modal features of at least two preset modalities in each video frame in the first initial vector are processed by an intermodal attention mechanism to obtain a first vector; and the modal features of at least two preset modalities in each video frame in the second initial vector are processed by an intermodal attention mechanism to obtain a second vector.

[0050] The first target vector and the second target vector are generated based on the first vector and the second vector.

[0051] In one embodiment of this application, the first video sequence and the second video sequence each include three video frames, wherein the first video sequence can be represented as follows:<f1,f2,f3> The second video sequence includes what can be represented as<f4,f5,f6> .

[0052] The initial vector corresponding to the i-th video frame can be represented as: The initial vector can have a dimension of 1*128, where a i b i c i These are the modal features of three different preset modes for the i-th video frame, for example, a. i b i c i These represent the modal features of the image modality, audio modality, and subtitle modality, respectively. Thus, the first initial vector can be represented as:

[0053]

[0054] The second initial vector can be expressed as:

[0055]

[0056] The specific process of performing inter-modal attention mechanism processing on the modal features of at least two preset modalities in each video frame of the first initial vector to obtain the first vector may include:

[0057] For any video frame, mutual attention is performed among the three preset modalities to obtain the embedding after perceiving other modalities, i.e., the first vector. Taking frame f4 as an example, the multimodal vector of frame f4 is... The following formula can be used to perform intermodal attention on the f4 frame:

[0058]

[0059] Where X'4 is the frame embedding of the f4th frame after intermodal attention. It is the transpose of X4 matrix, where a'4, b'4, and c'4 are the embeddings of image modality a4, audio modality b4, and subtitle modality c4 after inter-modal attention processing, respectively. Softmax: This function normalizes each element of a set of inputs to a real number between 0 and 1, and the sum is 1.

[0060] The same principle applies to other frames. After obtaining the intermodal attention, seq1_emb' and seq2_emb' are the sequence embeddings of video sequence 1 and video sequence 2, respectively. That is, seq1_emb' represents the first vector, and seq2_emb' represents the second vector, as shown below:

[0061]

[0062]

[0063] In this embodiment, by performing mutual attention among the three preset modalities of any video frame, any modality can be made capable of perceiving other modalities, thereby improving the accuracy of the inter-frame similarity calculation results of each video frame in the similarity matrix, and further improving the accuracy of the matched video segments determined based on the similarity matrix.

[0064] Optionally, generating the first target vector and the second target vector based on the first vector and the second vector includes:

[0065] The modal features of each video frame in the first vector are processed by an inter-frame attention mechanism to obtain a third vector, and the modal features of each video frame in the second vector are processed by an inter-frame attention mechanism to obtain a fourth vector.

[0066] The first target vector and the second target vector are generated based on the third vector and the fourth vector.

[0067] In this embodiment, by performing inter-frame attention mechanism processing on each video frame in the first vector and the second vector, any video frame in the video sequence can perceive the modal information of other video frames in the same video sequence, thereby further improving the accuracy of the inter-frame similarity calculation results of each video frame in the similarity matrix, and further improving the accuracy of the matched video segments determined based on the similarity matrix.

[0068] Optionally, the step of performing inter-frame attention processing on the modal features of each video frame in the first vector to obtain a third vector, and performing inter-frame attention processing on the modal features of each video frame in the second vector to obtain a fourth vector, includes:

[0069] The modal features in the first vector are concatenated to obtain a first intermediate vector, and the modal features in the second vector are concatenated to obtain a second intermediate vector. Each row in the first intermediate vector corresponds to a video frame in the first video sequence, and any row in the first intermediate vector is formed by concatenating the modal features of the corresponding video frame in the first vector. Each row in the second intermediate vector corresponds to a video frame in the second video sequence, and any row in the second intermediate vector is formed by concatenating the modal features of the corresponding video frame in the second vector.

[0070] The modal features of each video frame in the first intermediate vector are processed by an inter-frame attention mechanism to obtain a third intermediate vector, and the modal features of each video frame in the second intermediate vector are processed by an inter-frame attention mechanism to obtain a fourth intermediate vector.

[0071] The third intermediate vector is segmented according to video frames to obtain the third vector, and the fourth intermediate vector is segmented according to video frames to obtain the fourth vector.

[0072] Before performing inter-frame attention on the video frames in the first video sequence, at least two modal features from each video frame in the first vector are concatenated to obtain a first intermediate vector. This ensures that each video frame in the first vector has only one concatenated modal feature, facilitating subsequent inter-frame attention. Similarly, before performing inter-frame attention on the video frames in the second video sequence, at least two modal features from each video frame in the second vector are concatenated to obtain a second intermediate vector. This ensures that each video frame in the second vector has only one concatenated modal feature, facilitating subsequent inter-frame attention.

[0073] For ease of understanding, the following text will still refer to the first video sequence as...<f1,f2,f3> The second video sequence is<f4,f5,f6> For example, let's further explain the process of inter-frame attention mechanism:

[0074] After the above intermodal attention is applied, the video sequence can be subjected to intra-sequence attention at the granularity of video frame embedding, so that each frame embedding can perceive the influence of other frame embeddings.

[0075] Taking the second video sequence as an example, X'4, X'5, and X'6 are the video frame embeddings of X4, X5, and X6 after intermodal attention.

[0076] First, inter-modal embeddings within the frame are concatenated to obtain the second intermediate vector. Concatenate(X'4) concatenates the three vectors within the vector X'4 of frame f4 into a single vector. The vector dimension of each preset modality feature within X'4 is 1*128, meaning the vector dimension of X'4 is 3*128. After concatenation, the resulting concatenated vector Concatenate(X'4) has a dimension of 1*384. Concatenate(X'5) and Concatenate(X'6) are the results of processing the embedding vectors of frames f5 and f6, respectively.

[0077] Then, intra-frame inter-attention is applied to the second intermediate vector X' to obtain the fourth intermediate vector X''. The specific process of intra-frame inter-attention is as follows:

[0078]

[0079] Where X' TX' is the matrix transpose of X', and X"4, ​​X"5, X"6 are the embeddings of frames f4, f5, and f6 in the second video sequence after inter-frame attention within the sequence.

[0080] Finally, the vectors X”4, X”5, and X”6 are split into equal-length segments. Taking X”4 as an example, the X”4 vector is split into three vectors a”4, b4”, and c”4, representing the embeddings of the image modality, audio modality, and subtitle modality in frame f4 after two rounds of attention:

[0081]

[0082] Embed the f4, f5, and f6 frames. Assemble into the fourth vector "seq2_emb", where:

[0083]

[0084] Similarly, by performing the above processing on the first video sequence, we can obtain the third vector "seq1_emb", where:

[0085]

[0086] In this embodiment, by performing inter-frame attention mechanism processing on each video frame in the first vector and the second vector, any video frame in the video sequence can perceive the modal information of other video frames in the same video sequence, thereby further improving the accuracy of the inter-frame similarity calculation results of each video frame in the similarity matrix, and further improving the accuracy of the matched video segments determined based on the similarity matrix.

[0087] Optionally, generating the first target vector and the second target vector based on the third vector and the fourth vector includes:

[0088] Based on the third vector and the fourth vector, the first video sequence and the second video sequence are processed by an inter-sequence attention mechanism to obtain the first target vector and the second target vector.

[0089] In this embodiment, based on the third vector and the fourth vector, the first video sequence and the second video sequence are processed by an inter-sequence attention mechanism, so that video frames in any video sequence can perceive the modal features of other video sequences, thereby further improving the accuracy of the inter-frame similarity calculation results of each video frame in the similarity matrix, and further improving the accuracy of the matched video segments determined based on the similarity matrix.

[0090] Optionally, each column of the third vector corresponds to a video frame in the first video sequence, and each row of the third vector corresponds to a preset mode; each column of the fourth vector corresponds to a video frame in the second video sequence, and each row of the fourth vector corresponds to a preset mode;

[0091] The step of performing inter-sequence attention processing on the first video sequence and the second video sequence based on the third vector and the fourth vector to obtain the first target vector and the second target vector includes:

[0092] In the third vector, at least two modal features of the same video frame are concatenated to obtain a fifth intermediate vector, wherein each row of the fifth intermediate vector corresponds to a video frame in the first video sequence; and in the fourth vector, at least two modal features of the same video frame are concatenated to obtain a sixth intermediate vector, wherein each row of the sixth intermediate vector corresponds to a video frame in the second video sequence.

[0093] Each column of the third vector is processed with the sixth intermediate vector using an inter-sequence attention mechanism to obtain the first target vector, wherein the nth column of the first target vector is a column vector obtained by processing the nth column of the third vector with the sixth intermediate vector using an inter-sequence attention mechanism; and each column of the fourth vector is processed with the fifth intermediate vector using an inter-sequence attention mechanism to obtain the second target vector, wherein the mth column of the second target vector is a column vector obtained by processing the mth column of the fourth vector with the fifth intermediate vector using an inter-sequence attention mechanism.

[0094] For ease of understanding, the following text will still refer to the first video sequence as...<f1,f2,f3> The second video sequence is<f4,f5,f6> For example, let's further explain the process of inter-sequence attention mechanism:

[0095] Taking the f4th frame of the second video sequence as an example, the steps for performing inter-sequence attention are as follows:

[0096] First, the seq1_emb of the first video sequence is assembled to obtain the fifth intermediate vector:

[0097]

[0098] The vector of frame f4 is X”4, which becomes X'”4 after inter-sequence attention:

[0099]

[0100] Where mean is the average value, and a'”4, b'”4, and c'”4 are the embeddings of the image modality, audio modality, and subtitle modality after three rounds of attention, respectively.

[0101] The other frames f5 and f6 of the second video sequence also use this method to obtain the frame embeddings after inter-sequence attention. The frame embeddings of f4, f5, and f6 are assembled into the second target vector seq2_emb”' after inter-sequence attention. The first target vector seq1_emb”' of frames f1, f2, and f3 in video sequence 1 is also obtained using this method, where:

[0102]

[0103]

[0104] In this embodiment, based on the third vector and the fourth vector, the first video sequence and the second video sequence are processed by an inter-sequence attention mechanism, so that video frames in any video sequence can perceive the modal features of other video sequences, thereby further improving the accuracy of the inter-frame similarity calculation results of each video frame in the similarity matrix, and further improving the accuracy of the matched video segments determined based on the similarity matrix.

[0105] Optionally, the value of the i-th row and j-th column in the similarity matrix is: the similarity value between the i-th video frame in the first video sequence and the j-th video frame in the second video sequence; determining the matching video segments in the first video sequence and the second video sequence based on the similarity matrix includes:

[0106] Based on a preset threshold, each similarity value in the similarity matrix is ​​normalized to obtain the first similarity matrix;

[0107] Adding a first row and a first column to the first similarity matrix yields a second similarity matrix, where the values ​​in the first row and the first column are both 0.

[0108] The second similarity matrix is ​​processed using a greedy search algorithm to obtain the search matrix;

[0109] The video segment formed by the video sequence with the highest similarity in the search matrix is ​​determined as the matched video segment.

[0110] The similarity matrix can be a matrix calculated based on an initial similarity matrix. Specifically, the values ​​in the i-th row and j-th column of the initial similarity matrix can be an array formed by at least two sub-similarity values, which correspond to the at least two preset modalities.

[0111] Specifically, the similarity matrix can be calculated based on the initial similarity matrix. The similarity matrix includes similarity values ​​corresponding to any array in the initial similarity matrix, and the similarity values ​​are the average of the sub-similarity values ​​in the corresponding array.

[0112] In one embodiment of this application, let Z1 represent the vector of video frame f1 after the above three rounds of attention processing, that is, Z1 represents: the vector after sequentially performing inter-modal attention, inter-frame attention, and inter-sequence attention on video frame f1, where:

[0113]

[0114] Similarly, Z2, Z3, Z4, Z5, and Z6 represent the vectors of video frames f2, f3, f4, f5, and f6 after the above three rounds of attention processing, respectively.

[0115] Thus, seq1_emb”' and seq2_emb”' can be simplified to:

[0116]

[0117]

[0118] The initial similarity matrix sim is constructed using seq1_emb”' and seq2_emb”'. 3D It can be represented as:

[0119]

[0120] Among them, seq1_emb”' T It is the transpose of 'seq1_emb', sim 3D The elements are:

[0121]

[0122] sim 3D It is a 3D matrix, Z i *Z j Cos(a') represents the similarity between the i-th video frame in the first video sequence and the j-th video frame in the second video sequence. i,a'” j ) represents the similarity between the i-th video frame and the j-th video frame in terms of image modality, cos(b') i ,b'” j ) represents the similarity between the i-th video frame and the j-th video frame in audio modality, cos(c') i ,c'” j ) represents the similarity between the i-th video frame and the j-th video frame in the subtitle modality.

[0123] Based on the initial similarity matrix sim 3D The similarity matrix sim is calculated. 2D The specific process is as follows:

[0124]

[0125] Wherein, the mean(Z) i *Z j ) is cos(a'” i ,a'” j ), cos(b') i ,b'” j ) and cos(c'” i ,c'” j The mean (Z) is the average of the similarity of the three modalities between two video frames. i *Z j This can be represented as:

[0126] mean(Z i *Z j )=(cos(a'” i ,a'” j )+cos(b'” i ,b'” j )+cos(c'” i ,c'” j )) / 3

[0127] In this implementation, the similarity of the three modalities in the same frame is summed and averaged to become one dimension, thus compressing the 3D similarity into a 2D matrix, which helps to simplify the subsequent calculation process.

[0128] Optionally, determining the matching video segments in the first video sequence and the second video sequence based on the similarity matrix includes:

[0129] Based on the similarity matrix, the Smith-Waterman algorithm is used to determine the video segments that match the first video sequence and the second video sequence.

[0130] The Smith-Waterman algorithm is an algorithm for local sequence alignment (as opposed to global alignment) used to find similar regions between two sequences.

[0131] The following section uses the example of a first video sequence comprising five video frames "f1, f2, f3, f4, f5" and a second video sequence comprising four video frames "F1, F2, F3, F4" to further explain the specific process of determining the matching video segments in the first and second video sequences based on the similarity matrix using the Smith-Waterman algorithm:

[0132] Please see Figure 2 A similarity chart is constructed by comparing the similarity matrix of the first video sequence "f1, f2, f3, f4, f5" with the second video sequence "F1, F2, F3, F4". This similarity matrix can be constructed based on the method described in the above embodiments. In the chart, the value in row i and column j represents the similarity between the i-th video frame in the first video sequence and the j-th video frame in the second video sequence. Figure 2 The chart shown is the similarity matrix.

[0133] a. Similarity Normalization: In this embodiment, the preset threshold is 0.5. For each similarity value in the matrix, a value greater than or equal to 0.5 is assigned a value of 1, and a value less than 0.5 is assigned a value of 0. This results in the following: Figure 3 The normalized chart shown is the first similarity matrix.

[0134] b. Greedy search algorithm: such as Figure 4 As shown, based on the normalized matrix from step a, a new row and column with a value of 0 are added to obtain the second similarity matrix. Then, according to the calculation formula of the greedy search algorithm, the element values ​​in the second similarity matrix are calculated from left to right and from top to bottom, finally obtaining the following... Figure 4 The search matrix is ​​shown.

[0135] The calculation process for the formula based on the greedy search algorithm is as follows:

[0136] Please see Figure 6 ,for Figure 3 The calculation process for any four adjacent matrix elements, where the matrix element values ​​Hi and j are Hi and j, is as follows:

[0137] Permutation matrix: Open shot penalty: W k =kW1,W1=2;

[0138]

[0139] c. Backtracking to find the best matching segment: Find the maximum value based on the search matrix, and backtrack to the cell with a value of 0 according to the calculation path. At this point, output the backtracking path. The row and column segments corresponding to the backtracking path are the matched video segments, such as... Figure 5 The figure shows the best matching segment determined in this embodiment, that is, the matching segment between the first video sequence and the second video sequence is the second video frame to the fourth video frame.

[0140] In this embodiment, by using the Smith-Waterman algorithm to perform similarity normalization, greedy search, and backtracking on the similarity matrix in sequence, the matching video segments in the first video sequence and the second video sequence can be determined.

[0141] Optionally, obtaining the first initial vector and the second initial vector includes:

[0142] Obtain the first initial video sequence and the second initial video sequence;

[0143] A first video sequence is determined in the first initial video sequence, and a second video sequence is determined in the second initial video sequence, wherein the first video sequence and the second video sequence are matched;

[0144] The first initial vector is generated based on the modal features of at least two preset modalities of each image frame in the first video sequence, and the second initial vector is generated based on the modal features of at least two preset modalities of each image frame in the second video sequence.

[0145] Specifically, the first initial video sequence and the second initial video sequence are video sequences from two video files to be compared. Both the first initial video sequence and the second initial video sequence have relatively long sequence lengths. In this case, a first video sequence and a second initial video sequence that may match can be determined based on matching methods in related technologies. The first video sequence is a portion of the first initial video sequence, and the second video sequence is a segment of the second initial video sequence that may match the first video sequence.

[0146] Then, based on the method in the above embodiments, precise matching segments are further determined in the first video sequence and the second video sequence. Since the number of video frames in the first video sequence is less than the number of video frames in the first initial video sequence, and the number of video frames in the second video sequence is less than the number of video frames in the second initial video sequence, compared with directly using the method in the above embodiments to determine precise matching segments in the first initial video sequence and the second initial video sequence, the computational amount of the process of determining precise matching segments can be effectively reduced, thereby improving the efficiency of the process of determining matching segments.

[0147] In this embodiment, a first and second video sequence that may match are initially determined from the first and second initial video sequences using methods based on related technologies. Then, the precise matching segments are further determined from the first and second video sequences using the method described above. This helps to reduce the computational load in the process of determining precise matching segments, thereby improving the efficiency of the process of determining matching segments.

[0148] Optionally, determining the first video sequence in the first initial video sequence and determining the second video sequence in the second initial video sequence includes:

[0149] Extract modal features of at least two preset modalities from each frame of the first initial video series, and extract modal features of the at least two preset modalities from each frame of the second initial video series;

[0150] In the first initial video series and the second initial video series, at least two matching sequence pairs are determined, wherein the at least two matching sequence pairs correspond one-to-one with the at least two preset modalities, and the matching sequence pair corresponding to the target modal is: a sequence pair determined in the first initial video series and the second initial video series based on the target modal, wherein the target modal is any preset modal among the at least two preset modalities, and the matching sequence pair includes a first subsequence in the first video series and a second subsequence in the second video series, wherein the first subsequence matches the second subsequence;

[0151] The first subsequence of each of the at least two matching sequences is merged to obtain the first video sequence, and the second subsequence of each of the at least two matching sequences is merged to obtain the second video sequence.

[0152] The process of obtaining the first initial video sequence can be as follows: acquiring a first video file, extracting one video frame per second, extracting one audio segment per second, and extracting frame images and subtitles for each video frame to obtain the first initial video sequence, as well as the modal features of three preset modalities for each video frame in the first initial video sequence. Correspondingly, the process of obtaining the second initial video sequence can be as follows: acquiring a second video file, extracting one video frame per second, extracting one audio segment per second, and extracting frame images and subtitles for each video frame to obtain the second initial video sequence, as well as the modal features of three preset modalities for each video frame in the second initial video sequence. The three preset modalities are image modality, audio modality, and subtitle modality.

[0153] The aforementioned modal features can be extracted from the video frames using a deep model. For example, a convolutional network can be used to extract the frame vector of the image modality, a BERT network can be used to extract the vector of the subtitle modality, and a VGGish network can be used to extract the vector of the audio modality to form the aforementioned first initial vector and second initial vector.

[0154] After obtaining the modal features of the three preset modes for each video frame of the first and second initial video sequences, the matching video segments between the first and second initial video sequences can be analyzed from the perspective of a single modality, thus obtaining three sequence pairs. Then, the union of these three sequence pairs can be taken to obtain the first and second video sequences.

[0155] For example, see Figure 7 The first initial video sequence consists of 6 video frames, and the second initial video sequence consists of 8 video frames. From an image modality perspective, the matching video segments between the first and second initial video sequences are: frames 1-3 of the first initial video sequence match frames 1-3 of the second initial video sequence. From an audio modality perspective, the matching video segments between the first and second initial video sequences are: frames 2-5 of the first initial video sequence match frames 4-7 of the second initial video sequence. From a subtitle modality perspective, the matching video segments between the first and second initial video sequences are: frames 2-5 of the first initial video sequence match frames 4-7 of the second initial video sequence.

[0156] like Figure 7 As shown, after merging, the first video sequence can be output as frames 1-5 of the first initial video sequence, and the second video sequence can be output as frames 1-7 of the second initial video sequence.

[0157] The specific process of analyzing the matching video segments between the first initial video sequence and the second initial video sequence from the perspective of a single modality can be similar to the above embodiments. For example, taking the image modality as an example, the process of determining the sequence pair corresponding to the image modality will be further explained:

[0158] Modal features of the image modality for each video frame in the first initial video sequence are obtained, and modal features of the image modality for each video frame in the second initial video sequence are also obtained. The cosine similarity between the modal features of the image modality for each video frame in the first initial video sequence and the modal features of the video modality for each video frame in the second initial video sequence is calculated. Then, similar to... Figure 2 A similarity chart was generated. The Smith-Waterman algorithm was then used to find matching segments for each individual modality, i.e., to find the sequence pairs corresponding to the image modality. The process for determining the sequence pairs corresponding to the other two modalities is similar to that for determining the sequence pairs corresponding to the image modality, and will not be described further here to avoid repetition.

[0159] In this implementation, by analyzing the matching video segments between the first initial video sequence and the second initial video sequence from a single modal perspective, three sequence pairs are obtained. Then, the union of these three sequence pairs is taken to obtain the first video sequence and the second video sequence. This ensures that the first video sequence and the second video sequence include as many potentially matching video segments as possible from the first initial video sequence and the second initial video sequence, thereby allowing for the determination of more precise matching segments from the first video sequence and the second video sequence.

[0160] Please see Figure 8 The following is a flowchart illustrating a video data matching method according to another embodiment of this application. The method includes the following steps:

[0161] The first and second initial video sequences are each extracted as video frames at 1 second, and an audio segment is extracted at 1 second.

[0162] Retrieve the corresponding frame image, subtitle, and audio for each frame and each audio segment;

[0163] Modal features of image modality, audio modality, and subtitle modality are extracted for each video frame;

[0164] Each video is matched by a single modality to find the matching segment, that is, the sequence pair corresponding to the preset modality;

[0165] Merge the three sequence pairs to obtain the first video sequence and the second video sequence;

[0166] Inter-frame similarity is calculated using a 3D attention mechanism on the first and second video sequences to obtain a similarity matrix, where, for example... Figure 9 As shown, the 3D attention mechanism performs inter-frame similarity calculation, which includes three processes: inter-modal attention mechanism processing, inter-frame attention mechanism processing, and inter-sequence and inter-sequence attention mechanism processing.

[0167] The 3D similarity is compressed to 2D similarity to obtain the target pixel matrix;

[0168] The best matching segment is found based on the Smith-Waltman algorithm.

[0169] The specific implementation process of this embodiment is similar to that of the above embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0170] Please see Figure 10 This application provides a video data matching device 1000, which includes:

[0171] The acquisition module 1001 is used to acquire a first initial vector and a second initial vector. The first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence.

[0172] Processing module 1002 is used to process the first initial vector and the second initial vector using an attention mechanism to obtain a first target vector and a second target vector;

[0173] The construction module 1003 is used to construct a similarity matrix between the first video sequence and the second video sequence based on the first target vector and the second target vector;

[0174] The determination module 1004 is used to determine the video segments that match the first video sequence and the second video sequence based on the similarity matrix.

[0175] Optionally, the processing module 1002 includes:

[0176] The first processing submodule is configured to perform inter-modal attention mechanism processing on the modal features of at least two preset modalities of each video frame in the first initial vector to obtain a first vector, and to perform inter-modal attention mechanism processing on the modal features of at least two preset modalities of each video frame in the second initial vector to obtain a second vector.

[0177] The first generation submodule is used to generate the first target vector and the second target vector based on the first vector and the second vector.

[0178] Optionally, the first generation submodule includes:

[0179] The splicing unit is used to splice the modal features in the first vector to obtain a first intermediate vector, and to splice the modal features in the second vector to obtain a second intermediate vector. Each row in the first intermediate vector corresponds to a video frame in the first video sequence, and any row in the first intermediate vector is formed by splicing the modal features of the corresponding video frame in the first vector. Each row in the second intermediate vector corresponds to a video frame in the second video sequence, and any row in the second intermediate vector is formed by splicing the modal features of the corresponding video frame in the second vector.

[0180] The first processing unit is configured to perform inter-frame attention mechanism processing on the modal features of each video frame in the first intermediate vector to obtain a third intermediate vector, and to perform inter-frame attention mechanism processing on the modal features of each video frame in the second intermediate vector to obtain a fourth intermediate vector.

[0181] A segmentation unit is configured to segment the third intermediate vector according to video frames to obtain the third vector, and to segment the fourth intermediate vector according to video frames to obtain the fourth vector;

[0182] A generation unit is configured to generate the first target vector and the second target vector based on the third vector and the fourth vector.

[0183] Optionally, each column of the third vector corresponds to a video frame in the first video sequence, and each row of the third vector corresponds to a preset mode; each column of the fourth vector corresponds to a video frame in the second video sequence, and each row of the fourth vector corresponds to a preset mode;

[0184] The generation unit includes:

[0185] The splicing subunit is used to splice at least two modal features of the same video frame in the third vector to obtain a fifth intermediate vector, wherein each row of the fifth intermediate vector corresponds to a video frame in the first video sequence; and to splice at least two modal features of the same video frame in the fourth vector to obtain a sixth intermediate vector, wherein each row of the sixth intermediate vector corresponds to a video frame in the second video sequence.

[0186] The processing subunit is configured to perform inter-sequence attention mechanism processing on each column of the third vector and the sixth intermediate vector respectively to obtain the first target vector, wherein the nth column of the first target vector is a column vector obtained by performing inter-sequence attention mechanism processing on the nth column of the third vector and the sixth intermediate vector; and to perform inter-sequence attention mechanism processing on each column of the fourth vector and the fifth intermediate vector respectively to obtain the second target vector, wherein the mth column of the second target vector is a column vector obtained by performing inter-sequence attention mechanism processing on the mth column of the fourth vector and the fifth intermediate vector.

[0187] Optionally, the value of the i-th row and j-th column in the similarity matrix is: the similarity value between the i-th video frame in the first video sequence and the j-th video frame in the second video sequence;

[0188] The determining module includes:

[0189] The normalization submodule is used to normalize each similarity value in the similarity matrix based on a preset threshold to obtain a first similarity matrix.

[0190] The second processing submodule is used to add a first row and a first column to the first similarity matrix to obtain a second similarity matrix, wherein the values ​​in the first row and the first column are 0.

[0191] The second processing submodule is further configured to process the second similarity matrix based on a greedy search algorithm to obtain a search matrix;

[0192] The first determining submodule is used to determine the video segment formed by the video sequence with the highest similarity in the search matrix as the matching video segment.

[0193] Optionally, the acquisition module 1001 includes:

[0194] The acquisition submodule is used to acquire the first initial video sequence and the second initial video sequence;

[0195] The second determining submodule is configured to determine a first video sequence in the first initial video sequence and to determine a second video sequence in the second initial video sequence, wherein the first video sequence and the second video sequence are matched.

[0196] The second generation submodule is configured to generate the first initial vector based on the modal features of at least two preset modalities of each image frame in the first video sequence, and to generate the second initial vector based on the modal features of at least two preset modalities of each image frame in the second video sequence.

[0197] Optionally, the second determining submodule includes:

[0198] An extraction unit is configured to extract modal features of at least two preset modalities from each frame of the first initial video series, and to extract the modal features of the at least two preset modalities from each frame of the second initial video series.

[0199] A determining unit is configured to determine at least two matching sequence pairs in the first initial video series and the second initial video series, wherein the at least two matching sequence pairs correspond one-to-one with the at least two preset modalities, and the matching sequence pair corresponding to the target modal is: a sequence pair determined in the first initial video series and the second initial video series based on the target modal, wherein the target modal is any preset modal among the at least two preset modalities, and the matching sequence pair includes a first subsequence in the first video series and a second subsequence in the second video series, wherein the first subsequence matches the second subsequence;

[0200] The merging unit is configured to merge the first subsequences of each of the at least two matching sequences to obtain the first video sequence, and to merge the second subsequences of each of the at least two matching sequences to obtain the second video sequence.

[0201] In this embodiment, by making the first initial vector include modal features of at least two preset modalities of each video frame in the first video sequence, and the second initial vector include the modal features of the at least two preset modalities of each video frame in the second video sequence, and by using an attention mechanism to perceive the modal features of different modalities in the first and second initial vectors, a similarity matrix is ​​constructed. This allows the constructed similarity matrix to reflect the similarity of different modalities in the two video sequences, thus making the matched video segments determined based on the similarity matrix more accurate.

[0202] The video data matching device 1000 in this embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment does not impose specific limitations.

[0203] The video data matching device 1000 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0204] The video data matching device 1000 provided in this application embodiment can achieve... Figure 1 and Figure 8 The various processes implemented in the method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0205] Optionally, such as Figure 11 As shown, this application embodiment also provides an electronic device 1100, including a processor 1101, a memory 1102, and a program or instructions stored in the memory 1102 and executable on the processor 1101. When the program or instructions are executed by the processor 1101, they implement the various processes of the above-described video data matching method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0206] It should be noted that the application management device in this application embodiment includes the mobile electronic device and non-mobile electronic device described above.

[0207] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0208] The electronic device 1200 includes, but is not limited to, components such as: radio frequency unit 1201, network module 1202, audio output unit 1203, input unit 1204, sensor 1205, display unit 1206, user input unit 1207, interface unit 1208, memory 1209, and processor 1210.

[0209] The processor 1210 is configured to acquire a first initial vector and a second initial vector, wherein the first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence.

[0210] The first initial vector and the second initial vector are processed using an attention mechanism to obtain the first target vector and the second target vector;

[0211] Based on the first target vector and the second target vector, a similarity matrix between the first video sequence and the second video sequence is constructed;

[0212] Based on the similarity matrix, the video segments that match in the first video sequence and the second video sequence are determined.

[0213] Optionally, the processor 1210 is configured to perform inter-modal attention mechanism processing on the modal features of the at least two preset modalities of each video frame in the first initial vector to obtain a first vector, and to perform inter-modal attention mechanism processing on the modal features of the at least two preset modalities of each video frame in the second initial vector to obtain a second vector;

[0214] The processor 1210 is configured to generate the first target vector and the second target vector based on the first vector and the second vector.

[0215] Optionally, the processor 1210 is configured to concatenate the modal features in the first vector to obtain a first intermediate vector, and to concatenate the modal features in the second vector to obtain a second intermediate vector, wherein each row in the first intermediate vector corresponds to a video frame in the first video sequence, and any row in the first intermediate vector is formed by concatenating the modal features of the corresponding video frame in the first vector; each row in the second intermediate vector corresponds to a video frame in the second video sequence, and any row in the second intermediate vector is formed by concatenating the modal features of the corresponding video frame in the second vector;

[0216] The processor 1210 is configured to perform inter-frame attention mechanism processing on the modal features of each video frame in the first intermediate vector to obtain a third intermediate vector, and to perform inter-frame attention mechanism processing on the modal features of each video frame in the second intermediate vector to obtain a fourth intermediate vector.

[0217] The processor 1210 is configured to segment the third intermediate vector according to video frames to obtain the third vector, and to segment the fourth intermediate vector according to video frames to obtain the fourth vector;

[0218] The processor 1210 is configured to generate the first target vector and the second target vector based on the third vector and the fourth vector.

[0219] Optionally, the processor 1210 is configured to concatenate at least two modal features of the same video frame in the third vector to obtain a fifth intermediate vector, wherein each row of the fifth intermediate vector corresponds to a video frame in the first video sequence; and to concatenate at least two modal features of the same video frame in the fourth vector to obtain a sixth intermediate vector, wherein each row of the sixth intermediate vector corresponds to a video frame in the second video sequence.

[0220] The processor 1210 is configured to perform inter-sequence attention processing on each column of the third vector and the sixth intermediate vector to obtain the first target vector, wherein the nth column of the first target vector is a column vector obtained by performing inter-sequence attention processing on the nth column of the third vector and the sixth intermediate vector; and to perform inter-sequence attention processing on each column of the fourth vector and the fifth intermediate vector to obtain the second target vector, wherein the mth column of the second target vector is a column vector obtained by performing inter-sequence attention processing on the mth column of the fourth vector and the fifth intermediate vector.

[0221] Optionally, the processor 1210 is configured to normalize each similarity value in the similarity matrix based on a preset threshold to obtain a first similarity matrix;

[0222] The processor 1210 is configured to add a first row and a first column to the first similarity matrix to obtain a second similarity matrix, wherein the values ​​in the first row and the first column are 0;

[0223] The processor 1210 is used to process the second similarity matrix based on a greedy search algorithm to obtain a search matrix;

[0224] The processor 1210 is used to determine the video segment formed by the video sequence with the highest similarity in the search matrix as the matching video segment.

[0225] Optionally, the processor 1210 is configured to acquire a first initial video sequence and a second initial video sequence;

[0226] The processor 1210 is configured to determine a first video sequence in the first initial video sequence, and to determine a second video sequence in the second initial video sequence, wherein the first video sequence and the second video sequence are matched.

[0227] The processor 1210 is configured to generate a first initial vector based on the modal features of at least two preset modalities of each image frame in the first video sequence, and to generate a second initial vector based on the modal features of at least two preset modalities of each image frame in the second video sequence.

[0228] Optionally, the processor 1210 is configured to extract modal features of at least two preset modalities in each frame of the first initial video series, and to extract the modal features of the at least two preset modalities in each frame of the second initial video series.

[0229] The processor 1210 is configured to determine at least two matching sequence pairs in the first initial video series and the second initial video series, wherein the at least two matching sequence pairs correspond one-to-one with the at least two preset modalities, and the matching sequence pair corresponding to the target modal is: a sequence pair determined in the first initial video series and the second initial video series based on the target modal, wherein the target modal is any preset modal among the at least two preset modalities, and the matching sequence pair includes a first subsequence in the first video series and a second subsequence in the second video series, wherein the first subsequence matches the second subsequence;

[0230] The processor 1210 is configured to merge the first subsequences of each of the at least two matching sequences to obtain the first video sequence, and to merge the second subsequences of each of the at least two matching sequences to obtain the second video sequence.

[0231] Those skilled in the art will understand that the electronic device 1200 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1210 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0232] It should be understood that, in this embodiment, the input unit 1204 may include a graphics processing unit (GPU) 12041 and a microphone 12042. The GPU 12041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1206 may include a display panel 12061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here. The memory 1209 can be used to store software programs and various data, including but not limited to applications and operating systems. Processor 1210 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 1210.

[0233] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video data matching method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0234] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0235] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described video data matching method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0236] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0237] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0238] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0239] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for matching video data, characterized in that, include: Obtain a first initial vector and a second initial vector, wherein the first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence; The modal features of at least two preset modalities in each video frame in the first initial vector are processed by an intermodal attention mechanism to obtain a first vector; and the modal features of at least two preset modalities in each video frame in the second initial vector are processed by an intermodal attention mechanism to obtain a second vector. The modal features in the first vector are concatenated to obtain a first intermediate vector, and the modal features in the second vector are concatenated to obtain a second intermediate vector. Each row in the first intermediate vector corresponds to a video frame in the first video sequence, and any row in the first intermediate vector is formed by concatenating the modal features of the corresponding video frame in the first vector. Each row in the second intermediate vector corresponds to a video frame in the second video sequence, and any row in the second intermediate vector is formed by concatenating the modal features of the corresponding video frame in the second vector. The modal features of each video frame in the first intermediate vector are processed by an inter-frame attention mechanism to obtain a third intermediate vector, and the modal features of each video frame in the second intermediate vector are processed by an inter-frame attention mechanism to obtain a fourth intermediate vector. The third intermediate vector is segmented according to video frames to obtain a third vector, and the fourth intermediate vector is segmented according to video frames to obtain a fourth vector; Generate a first target vector and a second target vector based on the third vector and the fourth vector; Based on the first target vector and the second target vector, a similarity matrix between the first video sequence and the second video sequence is constructed; Based on the similarity matrix, the video segments that match in the first video sequence and the second video sequence are determined.

2. The method according to claim 1, characterized in that, Each column of the third vector corresponds to a video frame in the first video sequence, and each row of the third vector corresponds to a preset mode; each column of the fourth vector corresponds to a video frame in the second video sequence, and each row of the fourth vector corresponds to a preset mode; The step of generating the first target vector and the second target vector based on the third vector and the fourth vector includes: In the third vector, at least two modal features of the same video frame are concatenated to obtain a fifth intermediate vector, wherein each row of the fifth intermediate vector corresponds to a video frame in the first video sequence; and in the fourth vector, at least two modal features of the same video frame are concatenated to obtain a sixth intermediate vector, wherein each row of the sixth intermediate vector corresponds to a video frame in the second video sequence. Each column of the third vector is processed with the sixth intermediate vector using an inter-sequence attention mechanism to obtain the first target vector, wherein the nth column of the first target vector is a column vector obtained by processing the nth column of the third vector with the sixth intermediate vector using an inter-sequence attention mechanism; and each column of the fourth vector is processed with the fifth intermediate vector using an inter-sequence attention mechanism to obtain the second target vector, wherein the mth column of the second target vector is a column vector obtained by processing the mth column of the fourth vector with the fifth intermediate vector using an inter-sequence attention mechanism.

3. The method according to any one of claims 1 to 2, characterized in that, The value in the i-th row and j-th column of the similarity matrix is ​​the similarity value between the i-th video frame in the first video sequence and the j-th video frame in the second video sequence. The step of determining the matching video segments in the first video sequence and the second video sequence based on the similarity matrix includes: Based on a preset threshold, each similarity value in the similarity matrix is ​​normalized to obtain the first similarity matrix; Adding a first row and a first column to the first similarity matrix yields a second similarity matrix, where the values ​​in the first row and the first column are both 0. The second similarity matrix is ​​processed using a greedy search algorithm to obtain the search matrix; The video segment formed by the video sequence with the highest similarity in the search matrix is ​​determined as the matched video segment.

4. The method according to any one of claims 1 to 2, characterized in that, The process of obtaining the first initial vector and the second initial vector includes: Obtain the first initial video sequence and the second initial video sequence; A first video sequence is determined in the first initial video sequence, and a second video sequence is determined in the second initial video sequence, wherein the first video sequence and the second video sequence are matched; The first initial vector is generated based on the modal features of at least two preset modalities of each image frame in the first video sequence, and the second initial vector is generated based on the modal features of at least two preset modalities of each image frame in the second video sequence.

5. The method according to claim 4, characterized in that, Determining the first video sequence in the first initial video sequence and determining the second video sequence in the second initial video sequence includes: Extract modal features of at least two preset modalities from each frame of the first initial video series, and extract modal features of the at least two preset modalities from each frame of the second initial video series; In the first initial video series and the second initial video series, at least two matching sequence pairs are determined, wherein the at least two matching sequence pairs correspond one-to-one with the at least two preset modalities, and the matching sequence pair corresponding to the target modal is: a sequence pair determined in the first initial video series and the second initial video series based on the target modal, wherein the target modal is any preset modal among the at least two preset modalities, and the matching sequence pair includes a first subsequence in the first video series and a second subsequence in the second video series, wherein the first subsequence matches the second subsequence; The first subsequence of each of the at least two matching sequences is merged to obtain the first video sequence, and the second subsequence of each of the at least two matching sequences is merged to obtain the second video sequence.

6. A video data matching device, characterized in that, include: The acquisition module is used to acquire a first initial vector and a second initial vector, wherein the first initial vector includes modal features of at least two preset modalities for each video frame in the first video sequence, and the second initial vector includes the modal features of the at least two preset modalities for each video frame in the second video sequence; The processing module is used to process the first initial vector and the second initial vector using an attention mechanism to obtain the first target vector and the second target vector; A construction module is used to construct a similarity matrix between the first video sequence and the second video sequence based on the first target vector and the second target vector; The determining module is used to determine the video segments that match the first video sequence and the second video sequence based on the similarity matrix; The processing module includes: The first processing submodule is configured to perform inter-modal attention mechanism processing on the modal features of at least two preset modalities of each video frame in the first initial vector to obtain a first vector, and to perform inter-modal attention mechanism processing on the modal features of at least two preset modalities of each video frame in the second initial vector to obtain a second vector. A first generation submodule is configured to generate a first target vector and a second target vector based on the first vector and the second vector; the first generation submodule includes: The splicing unit is used to splice the modal features in the first vector to obtain a first intermediate vector, and to splice the modal features in the second vector to obtain a second intermediate vector. Each row in the first intermediate vector corresponds to a video frame in the first video sequence, and any row in the first intermediate vector is formed by splicing the modal features of the corresponding video frame in the first vector. Each row in the second intermediate vector corresponds to a video frame in the second video sequence, and any row in the second intermediate vector is formed by splicing the modal features of the corresponding video frame in the second vector. The first processing unit is configured to perform inter-frame attention mechanism processing on the modal features of each video frame in the first intermediate vector to obtain a third intermediate vector, and to perform inter-frame attention mechanism processing on the modal features of each video frame in the second intermediate vector to obtain a fourth intermediate vector. The segmentation unit is used to segment the third intermediate vector according to video frames to obtain a third vector, and to segment the fourth intermediate vector according to video frames to obtain a fourth vector; A generation unit is configured to generate the first target vector and the second target vector based on the third vector and the fourth vector.

7. The apparatus according to claim 6, characterized in that, Each column of the third vector corresponds to a video frame in the first video sequence, and each row of the third vector corresponds to a preset mode; each column of the fourth vector corresponds to a video frame in the second video sequence, and each row of the fourth vector corresponds to a preset mode; The generation unit includes: The splicing subunit is used to splice at least two modal features of the same video frame in the third vector to obtain a fifth intermediate vector, wherein each row of the fifth intermediate vector corresponds to a video frame in the first video sequence; and to splice at least two modal features of the same video frame in the fourth vector to obtain a sixth intermediate vector, wherein each row of the sixth intermediate vector corresponds to a video frame in the second video sequence. The processing subunit is configured to perform inter-sequence attention mechanism processing on each column of the third vector and the sixth intermediate vector respectively to obtain the first target vector, wherein the nth column of the first target vector is a column vector obtained by performing inter-sequence attention mechanism processing on the nth column of the third vector and the sixth intermediate vector; and to perform inter-sequence attention mechanism processing on each column of the fourth vector and the fifth intermediate vector respectively to obtain the second target vector, wherein the mth column of the second target vector is a column vector obtained by performing inter-sequence attention mechanism processing on the mth column of the fourth vector and the fifth intermediate vector.

8. The apparatus according to any one of claims 6 to 7, characterized in that, The value in the i-th row and j-th column of the similarity matrix is ​​the similarity value between the i-th video frame in the first video sequence and the j-th video frame in the second video sequence. The determining module includes: The normalization submodule is used to normalize each similarity value in the similarity matrix based on a preset threshold to obtain a first similarity matrix. The second processing submodule is used to add a first row and a first column to the first similarity matrix to obtain a second similarity matrix, wherein the values ​​in the first row and the first column are 0. The second processing submodule is further configured to process the second similarity matrix based on a greedy search algorithm to obtain a search matrix; The first determining submodule is used to determine the video segment formed by the video sequence with the highest similarity in the search matrix as the matching video segment.

9. The apparatus according to any one of claims 6 to 7, characterized in that, The acquisition module includes: The acquisition submodule is used to acquire the first initial video sequence and the second initial video sequence; The second determining submodule is configured to determine a first video sequence in the first initial video sequence and to determine a second video sequence in the second initial video sequence, wherein the first video sequence and the second video sequence are matched. The second generation submodule is configured to generate the first initial vector based on the modal features of at least two preset modalities of each image frame in the first video sequence, and to generate the second initial vector based on the modal features of at least two preset modalities of each image frame in the second video sequence.

10. The apparatus according to claim 9, characterized in that, The second determining submodule includes: An extraction unit is configured to extract modal features of at least two preset modalities from each frame of the first initial video series, and to extract the modal features of the at least two preset modalities from each frame of the second initial video series. A determining unit is configured to determine at least two matching sequence pairs in the first initial video series and the second initial video series, wherein the at least two matching sequence pairs correspond one-to-one with the at least two preset modalities, and the matching sequence pair corresponding to the target modal is: a sequence pair determined in the first initial video series and the second initial video series based on the target modal, wherein the target modal is any preset modal among the at least two preset modalities, and the matching sequence pair includes a first subsequence in the first video series and a second subsequence in the second video series, wherein the first subsequence matches the second subsequence; The merging unit is configured to merge the first subsequences of each of the at least two matching sequences to obtain the first video sequence, and to merge the second subsequences of each of the at least two matching sequences to obtain the second video sequence.

11. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the video data matching method as described in any one of claims 1 to 5.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video data matching method as described in any one of claims 1 to 5.

13. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run a program or instructions, the program or instructions being executed by the processor to implement the steps of the video data matching method as described in any one of claims 1 to 5.

Images