Video copy segment positioning method and device based on spatiotemporal instance enhancement

Abstract

The application provides a video copy segment positioning method and device based on a space-time domain instance enhancement, wherein the method comprises the following steps: extracting regional features and global features of target frames in a query video and a reference video, and constructing an instance relationship graph based on the regional features and the global features; based on the instance relationship graph, obtaining local alignment information and global alignment information through a space branch and a time branch respectively; through a complementarity perception alignment module, generating a final inter-frame alignment result based on the local alignment information and the global alignment information; and the inter-frame alignment result is used for copy segment positioning of the query video and the reference video. The application comprehensively considers global semantic information and fine-grained and more robust local representation, improves the matching accuracy of video frames in the query video and the reference video, and can also improve the accuracy of the matching result of the copy segment in the query video and the reference video.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for locating video copy segments based on spatiotemporal domain instance enhancement. Background Technology

[0002] Video copy segment localization (VSL) requires identifying time segments within a pair of videos containing copied content. Existing VSL techniques mainly consist of three steps: extracting keyframes from the videos, extracting features from the keyframes and modeling temporal dependencies, and using time alignment methods to extract aligned segments.

[0003] In the keyframe extraction part, due to the specific nature of video copy localization, video-level and segment-level features cannot meet the requirements for accurate localization. Current methods generally use frames per second as keyframes. Some works have attempted to model keyframe extraction as a multi-frame classification problem to select more representative keyframes. Feature extraction typically uses pre-trained models to extract frame-level features and then models temporal dependencies through an attention module. Similarity calculations are performed on the features of the query video and the reference video to obtain a frame similarity matrix. Traditional methods use Hough voting, temporal networks, dynamic programming, and other schemes to locate the common segments of two videos. Recent methods employ more advanced data-driven approaches, such as: parameterized time-sensitive similarity measures based on the Fourier domain; constructing a spatiotemporal similarity step prediction graph; and modeling the time alignment task as an object detection task, using an object detector to predict the bounding boxes of the copied segments.

[0004] Current VSL methods largely rely on spatiotemporal global features. While these global features excel at capturing high-level information and long-term temporal relationships, they primarily reflect a coarse semantic granularity. Therefore, these methods often fail to provide a deep understanding of complex, fine-grained details crucial for achieving accurate results in VSL tasks. Some related methods attempt to introduce local spatial information using pixel-level partitioning, but this disrupts the spatial semantic structure, leading to decreased matching accuracy. Summary of the Invention

[0005] This invention provides a video copy segment localization method and apparatus based on spatiotemporal domain instance enhancement, which addresses the shortcomings of existing technologies that rely on global features and cannot deeply understand complex, fine-grained details. It achieves a more robust local representation that comprehensively considers global semantic information and fine-grained details, thereby improving matching accuracy and the accuracy of matching results.

[0006] This invention provides a method for locating copied video segments based on spatiotemporal domain instance enhancement, comprising the following steps:

[0007] Extract the regional and global features of the target frames in the query video and the reference video, and construct an instance relationship graph based on the regional and global features;

[0008] Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branching and temporal branching, respectively.

[0009] The complementarity-aware alignment module generates the final inter-frame alignment result based on the local alignment information and the global alignment information; the inter-frame alignment result is used to locate the copied segments of the query video and the reference video.

[0010] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, which obtains local alignment information through spatial branching based on the instance relationship graph, specifically including:

[0011] The nodes and edges in the instance relationship graph are encoded by an instance-based graph neural network, and the information is propagated through the nodes and edges in the instance relationship graph using a message passing mechanism to obtain an updated local feature matrix.

[0012] Based on the updated local feature matrix, depth map matching calculation is performed to obtain the local alignment information of the query video and the reference video.

[0013] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, which obtains global alignment information through time branching based on the instance relationship graph, specifically including:

[0014] The global features are enhanced based on the instance relationship graph to obtain enhanced global features;

[0015] Based on the enhanced global features, temporal information is captured through temporal position encoding and a temporal attention module;

[0016] Based on the time sequence information, the global alignment information of the query video and the reference video is calculated.

[0017] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, which performs depth map matching calculation based on the updated local feature matrix to obtain local alignment information between the query video and the reference video, specifically including:

[0018] Based on the updated local feature matrix, calculate the affinity matrix between the query video and the reference video;

[0019] Based on the affinity matrix, the inter-frame matching matrix is ​​calculated using the pseudo-Hungarian algorithm;

[0020] Based on the inter-frame matching matrix and the local scoring function, the local alignment information of the query video and the reference video is obtained.

[0021] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, wherein the complementarity-aware alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0022] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, wherein the regional features and global features are extracted based on region of interest pooling.

[0023] The present invention also provides a video copy segment localization device based on spatiotemporal domain instance enhancement, comprising the following modules:

[0024] A construction module is used to extract regional and global features of target frames in query and reference videos, and construct an instance relationship graph based on the regional and global features;

[0025] The acquisition module is used to acquire local alignment information and global alignment information based on the instance relationship graph through spatial branches and temporal branches, respectively.

[0026] The generation module is used to generate the final inter-frame alignment result based on the local alignment information and the global alignment information through the complementarity-aware alignment module; the inter-frame alignment result is used for locating the copied segments of the query video and the reference video.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video copy segment localization method based on spatiotemporal domain instance enhancement as described above.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video copy segment localization method based on spatiotemporal domain instance enhancement as described above.

[0029] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video copy segment localization method based on spatiotemporal domain instance enhancement as described above.

[0030] The present invention provides a video copy segment localization method and apparatus based on spatiotemporal domain instance enhancement. This method extracts regional and global features of target frames from query and reference videos, constructs an instance relationship graph based on these features, and then obtains local and global alignment information through spatial and temporal branches. Finally, a complementarity-aware alignment module generates the final inter-frame alignment result based on the local and global alignment information, thereby locating copied segments in the query and reference videos. This method comprehensively considers global semantic information and a more robust, fine-grained local representation, improving the matching accuracy of video frames in the query and reference videos, and also enhancing the accuracy of the matching results for copied segments in the query and reference videos. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the video copy segment localization method based on spatiotemporal domain instance enhancement provided by the present invention.

[0033] Figure 2 This is a schematic diagram of the video copy segment positioning method provided by the present invention.

[0034] Figure 3 This is a schematic diagram of the video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention.

[0035] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Figure 1 This is a flowchart illustrating the video copy segment localization method based on spatiotemporal domain instance enhancement provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0038] Step 100: Extract the regional and global features of the target frames in the query video and reference video, and construct an instance relationship graph based on the regional and global features.

[0039] Specifically, in this embodiment of the invention, it is necessary to locate the copied segments of the query video and the reference video. First, it is necessary to extract the regional features and global features from the query video and the reference video respectively.

[0040] The feature extraction part can be performed using an object-aware method, thereby ensuring the connection between local features and objects within the frame.

[0041] In some implementations, regional features and global features are extracted based on Region of Interest Pooling (ROI-Pooling).

[0042] Specifically, this can be achieved by detecting objects in video frames and extracting the regional features of these objects as well as global frame features. For each frame... You can extract the collection of object boxes. ,in This represents the number of objects detected within the frame. Local features for each object can then be extracted using ROI-Pooling techniques. At the same time, the global features of the frame are obtained. By using the ROI-Pooling method to extract local features of each object, it is possible to ensure that the extracted local features are associated with objects within the frame, so as to facilitate subsequent processing.

[0043] After extracting the regional and global features of the target frame from the query video and the reference video, the regional and global features can be jointly processed to construct an instance-relation graph (IRG) to enhance expressive power.

[0044] For each frame Local features extracted They can be stacked into a feature matrix. , This represents the feature dimension. The matrix contains feature information for each object, serving as a feature representation of nodes in the instance relationship graph.

[0045] Next, edges in the instance relationship graph can be constructed by calculating the similarity between object features. Cosine similarity can be used to measure the similarity between two objects. Calculate node (local) features. and The cosine similarity is used as the adjacency matrix. elements in The adjacency matrix describes the adjacency of frames. The relationships between objects in the game.

[0046] In instance relationship graphs, it's not enough to model the similarity between nodes (objects); each edge also needs to be assigned higher-order information. To achieve this, edge features can be computed using cross-attention mechanisms and global features.

[0047] Through cross attention between adjacent nodes and global features As virtual nodes, calculate the edge feature matrix of higher-order relationships between nodes. The edge features contain relationship information between nodes and global context information, as shown in the following formula:

[0048]

[0049]

[0050] in, These are edge features; MLP stands for Multilayer Perceptron. It is attention weight. It is a learnable weight matrix. This is a node feature matrix that incorporates global features.

[0051] Ultimately, this can be achieved through the node feature matrix. Adjacency matrix Sum of edge feature matrices The three together describe the frame. Generate an instance relationship diagram by analyzing the spatial relationships between objects in the diagram. , represented as .

[0052] Step 101: Based on the instance relationship graph, obtain local alignment information and global alignment information through spatial branch and temporal branch respectively.

[0053] Specifically, in this embodiment of the invention, a new VSL framework is designed, which performs targeted spatiotemporal modeling of two types of features (i.e., regional features and global features) through a dual-branch network structure, obtains local alignment information through the spatial branch, and obtains global alignment information through the temporal branch.

[0054] Among them, the spatial branch in the dual-branch network structure is the instance-relation graph-based spatial branch (IGS), which can process the instance-relation graph to obtain local alignment information between frames, that is, the similarity alignment matrix of local objects in the query video and the reference video at the frame level. This matrix describes the spatial matching of similar objects in the query video and the reference video.

[0055] The temporal branch in the dual-branch network structure is the Instance-enhanced Temporal Branch (IET), which can process global features based on the instance relationship graph. The resulting global frame-level features contain the visual semantic information of the frame itself and the temporal dependencies between video frames. Then, inter-frame similarity is calculated, that is, by comparing the frame-level features of the query video and the reference video, their similarity in the temporal dimension is calculated to obtain the global alignment information between frames.

[0056] Step 102: The complementaryness-aware alignment module generates the final inter-frame alignment result based on local and global alignment information; the inter-frame alignment result is used to locate the copied segments of the query video and the reference video.

[0057] Specifically, local features capture important local details in an image, while global features capture high-level visual semantic information of the frame. Therefore, after obtaining local and global alignment information, the complementarity-perception alignment module (CAM) can leverage the complementarity between local and global features to combine the broad perspective provided by global features with the detailed insights of local features. This ensures that the necessary information is obtained in a comprehensive and balanced manner for accurate VSL analysis, as shown in the following formula:

[0058]

[0059] in, This is the final inter-frame alignment result. This is a local feature similarity matrix (local feature information). is the global feature similarity matrix (global feature information), and CAM is the complementary sensing alignment module.

[0060] After obtaining the inter-frame alignment results of the query video and the reference video, the copied segments of the query video and the reference video can be located.

[0061] The video copy segment localization method based on spatiotemporal domain instance enhancement provided by this invention extracts regional and global features of target frames in query and reference videos, constructs an instance relationship graph based on the regional and global features, and then obtains local and global alignment information through spatial and temporal branches based on the instance relationship graph. Finally, a complementarity-aware alignment module generates the final inter-frame alignment result based on the local and global alignment information, thereby locating the copy segments in the query and reference videos. It comprehensively considers global semantic information and more robust local representations with finer granularity, improving the matching accuracy of video frames in the query and reference videos, and also improving the accuracy of the matching results of copy segments in the query and reference videos.

[0062] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, which obtains local alignment information through spatial branching based on an instance relationship graph, specifically including:

[0063] The nodes and edges in the instance relationship graph are encoded by an instance-based graph neural network, and the updated local feature matrix is ​​obtained by using a message passing mechanism to propagate information through the nodes and edges in the instance relationship graph.

[0064] Based on the updated local feature matrix, depth map matching calculation is performed to obtain local alignment information between the query video and the reference video.

[0065] Specifically, the spatial branch in this embodiment of the invention may include an instance-based graph neural network (Instance-GNN) structure and a depth graph matching module.

[0066] Specifically, the spatial structural relationships between nodes and edges in the instance relation graph can be encoded using Instance-GNN, and a message-passing mechanism can be used to communicate between nodes and edges (i.e., the adjacency matrix) in the instance relation graph. This process propagates information, enabling the modeling of relationships across the entire graph, thereby improving the model's understanding of instance relationships in the video and obtaining an updated local feature matrix. The formula is shown below:

[0067]

[0068] in, It is the first The feature matrix output by the layer GNN, It is an adjacency matrix. It is the first Learnable weights of a layer.

[0069] The node features have a self-updating mechanism. In this embodiment of the invention, nonlinear transformation and adaptive weights are combined to update the feature representation of each node, and the formula is as follows:

[0070]

[0071] in, This represents the node characteristics after self-updating. Indicates the first Video frames in layered graph neural networks Middle node The initial characteristics. It is an activation function. It is a learnable temperature parameter. These are the learnable weights and biases, respectively.

[0072] This self-updating mechanism ensures that nodes can retain their important information during multi-layer GNN propagation, while adapting to changes in local and global context.

[0073] After obtaining the updated local feature matrix through multiple layers of GNN, the depth map matching module can then find the corresponding feature point pairs from images from different perspectives, thereby capturing the spatial matching of similar objects between the query video and the reference video, and obtaining the local alignment information between the query video and the reference video.

[0074] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, which obtains global alignment information through time branching based on an instance relationship graph, specifically including:

[0075] Enhanced global features are obtained by augmenting global features based on instance relationship graphs;

[0076] Based on enhanced global features, temporal information is captured through temporal position encoding and a temporal attention module;

[0077] Based on temporal information, global alignment information between the query video and the reference video is calculated.

[0078] Specifically, the time branch in this embodiment of the invention may include a temporal position encoding process and a temporal attention module.

[0079] First, global features can be modeled to represent temporal relationships using a transformer. For each frame, the temporal branch enhances the global representation of the frame by adding fine-grained local information (i.e., the enhancement of global features by local features) to the global features, resulting in enhanced global features. .

[0080] Then, to enhance global features, temporal location encoding and temporal attention modules can be used to capture frame-level temporal dependencies. In this way, the model can better understand long-term dependencies and dynamic changes between video frames.

[0081] This process can be accomplished using multi-head self-attention (MHA) and temporal position encoding (TPM), as shown in the following formula:

[0082]

[0083] in, This refers to the temporal information obtained after performing temporal modeling on the enhanced global features. To enhance global features.

[0084] Once the temporal information is obtained, the global frame-to-frame temporal similarity can be calculated based on this information, resulting in the global feature similarity matrix (i.e., global alignment information), as shown in the following formula:

[0085]

[0086] in, This is the global feature similarity matrix. To query the timing information of the video, The timing information is for reference video.

[0087] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided. Based on an updated local feature matrix, depth map matching calculation is performed to obtain local alignment information between the query video and the reference video. Specifically, the method includes:

[0088] Based on the updated local feature matrix, calculate the affinity matrix between the query video and the reference video;

[0089] Based on the affinity matrix, the inter-frame matching matrix is ​​calculated using the pseudo-Hungarian algorithm;

[0090] Based on the inter-frame matching matrix and the local scoring function, the local alignment information between the query video and the reference video is obtained.

[0091] Specifically, in the process of depth map matching calculation based on the updated local feature matrix, the affinity matrix between the query video and the reference video can first be calculated based on the updated local feature matrix, as shown in the following formula:

[0092]

[0093] in, It is an affinity matrix. It is the transpose of the local feature matrix updated in the query video. It is a learnable weight matrix. It is a local feature matrix updated from the reference video.

[0094] Obtain the affinity matrix Then, the pseudo-Hungarian algorithm can be used to calculate the inter-frame matching matrix between the query video and the reference video. The specific formula is as follows:

[0095]

[0096] In the formula, This indicates that a pseudo-Hungarian matching algorithm is applied to determine the matching of local objects.

[0097] for A Hungarian loss can be designed. The specific formula for supervising it is shown below:

[0098]

[0099] in, It represents the actual matching relationship between the query video and the reference video. It is a predicted matching relationship between the query video and the reference video. They are the first The number of nodes in a frame.

[0100] Finally, based on the inter-frame matching matrix and the local scoring function, the local feature similarity matrix (i.e., local alignment information) can be obtained, as shown in the following formula:

[0101]

[0102] in, Indicates querying video frames and reference video frames Local similarity between them Indicates querying video frames Local features, Indicates reference video frame Local features, It is the first The number of nodes in a frame.

[0103] According to the present invention, a video copy segment localization method based on spatiotemporal domain instance enhancement is provided, wherein the complementarity-aware alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0104] Specifically, in this embodiment of the invention, the complementarity-aware alignment module is trained using differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0105] This invention employs a differentiable dynamic time warping (DTW) method, utilizing potential time alignment as a weak supervision signal. This approach addresses the inherent limitations of VSL task annotations (typical VSL annotations only provide the start and end positions of similar segments between two videos).

[0106] Previous methods approximate element-level constraints by constructing a diagonal matrix, assuming the two sequences are perfectly synchronized. This assumption is oversimplified because similar video clips may exhibit temporal variations due to content editing, playback speed changes, noise, or distortion, resulting in complex alignments that cannot be described by a diagonal matrix.

[0107] Therefore, this embodiment of the invention introduces an alignment loss based on smoothDTW, the specific formula of which is as follows:

[0108]

[0109]

[0110]

[0111] The final DTW loss can be expressed as:

[0112]

[0113] in, This represents the inter-frame similarity matrix between the query video and the reference video. It is the alignment cost matrix. It is a cumulative distance matrix. It is a smooth minimum function. It's a temperature parameter. It is the size of the similar submatrices.

[0114] In this embodiment of the invention, a weighted mean square error (MSE) loss can be further introduced to constrain the non-copying segments of the video, and its formula is as follows:

[0115]

[0116] in, It is the first Frame and the Predicted similarity between frames It is the first Frame and the Ground truth similarity between frames. It is a weighting factor. These are the frame rates of the query video and the reference video, respectively.

[0117] In some embodiments, the alignment loss function of the complementarity-aware alignment module It can be designed as:

[0118]

[0119] in, It is a weighting factor.

[0120] The following examples from specific application scenarios further illustrate the video copy segment localization method based on spatiotemporal domain instance enhancement provided by the present invention.

[0121] Figure 2 This is a schematic diagram of the video copy segment positioning method provided by the present invention, as shown below. Figure 2 As shown, this embodiment designs a new VSL framework to extract video frame-level and region-level features. A dual-branch network structure is used to perform targeted spatiotemporal modeling on both types of features. For the feature extraction part, this embodiment uses an object-aware method to ensure the connection between local features and intra-frame objects. For each frame... This embodiment extracts the object box set. , This indicates the number of objects detected within the frame. Region features are extracted using ROI-Pooling technology. At the same time, global features are obtained. .

[0122] The dual-branch structure design is not completely decoupled. In this embodiment, the two types of information are jointly processed in the instance relationship graph construction part to enhance expressive power. IGS models the IRG and calculates the inter-frame alignment relationship through graph matching. For each frame... Local features Stacked into feature matrices , Representing feature dimension. Adjacency matrix. medium elements Obtained from the cosine similarity of node (local) features. Edge feature matrix of higher-order relationships between nodes. Through cross attention between adjacent nodes and global features Computation as a virtual node:

[0123]

[0124]

[0125] in, These are edge features; MLP stands for Multilayer Perceptron. It is attention weight. It is a learnable weight matrix. This is a node feature matrix that incorporates global features.

[0126] The final instance relationship diagram is represented as follows .

[0127] This embodiment utilizes an IRG to represent the video feature space, which can also be used to refine the IET branch. This embodiment designs an Instance-GNN structure, which encodes the spatial structural relationships between nodes and edges, and uses a message-passing mechanism to propagate information through nodes and edges, achieving modeling relationships across the entire graph. This improves the model's understanding of instance relationships in the video.

[0128]

[0129] in, It is the first The feature matrix output by the layer GNN, It is the first Learnable weights of a layer.

[0130] The node self-updating mechanism dynamically adjusts features based on their relative importance in the graph:

[0131]

[0132] in, It is an activation function. It is a learnable temperature parameter. These are learnable weights and biases.

[0133] The updated nodes in the Q and R sequences of the videos possess rich structural and contextual information. This embodiment captures inter-video correlations through depth map matching. First, the updated nodes are used to calculate the affinity matrix:

[0134]

[0135] in, It is an affinity matrix. It is a learnable weight matrix.

[0136] Calculate the inter-frame matching matrix using the pseudo-Hungarian algorithm. :

[0137]

[0138] Aggregate the prediction matrix using local scoring functions:

[0139]

[0140] It is the first The number of nodes in a frame.

[0141] IET models temporal relationships using global features through a transformer, followed by frame-level similarity calculation. Given a query video Q and a reference video R, IET first calculates similarity using the equation... Add a virtual node for global features to enhance its global representation using fine-grained information from local features. Further utilize MHA and TPM to capture timing information:

[0142]

[0143] And calculate global frame-to-frame temporal similarity:

[0144]

[0145] Local features capture important local details in an image, while global features capture high-level visual semantic information of the frame. CAM leverages the complementarity between local and global features in IRG and IET, combining the broad perspective provided by global features with the detailed insights of local features to ensure a comprehensive and balanced approach to acquiring the necessary information for accurate VSL analysis.

[0146]

[0147] in This is the final inter-frame alignment map, where CAM stands for Complementary Sensing Alignment Module.

[0148] For the formula In In this embodiment, a Hungarian loss is designed for supervision:

[0149]

[0150] in, It is the matched real value. They are the first The number of nodes in a frame.

[0151] This embodiment employs a differentiable dynamic time warping method that utilizes latent time alignment as a weak supervision signal. It addresses the inherent limitations of VSL task annotations (typical VSL annotations only provide the start and end positions of similar segments between two videos). Previous methods approximate element-level constraints by constructing diagonal matrices, assuming the two sequences are perfectly synchronized. This assumption is overly simplistic, as similar video segments may exhibit temporal variations due to content editing, playback speed changes, noise, or distortion, leading to complex alignments that cannot be described by a diagonal matrix. Therefore, this embodiment introduces an alignment loss based on smoothDTW:

[0152]

[0153]

[0154]

[0155] The final DTW loss is expressed as:

[0156]

[0157] Where C is the alignment cost matrix, D is the cumulative distance matrix, and SmoothMin is the smoothing minimum function. It's a temperature parameter. It is the size of the similar submatrices.

[0158] This embodiment further introduces a weighted mean squared error (MSE) loss to constrain the non-replicated segment:

[0159]

[0160] It is a weighting factor.

[0161] The alignment loss in this embodiment is designed as follows:

[0162]

[0163] It is a weighting factor.

[0164] The network in this embodiment captures instance-level features and fine-grained interactions by constructing an IRG, thereby preserving the integrity of local information. It employs an Instance-GNN to learn inter-instance relationships and structured information at an appropriate representational granularity. Inter-frame similarity alignment is achieved through depth map matching. Furthermore, this embodiment captures long-term temporal relationships by modeling instance-enhanced frame-level features and effectively processes and integrates complementary spatiotemporal information using a complementary perceptual alignment module, thereby generating accurate inter-frame alignment maps. The method in this embodiment also incorporates differentiable DTW, utilizing latent temporal alignment as a weak supervision signal to improve the accuracy of the matching process.

[0165] The video copy segment localization device based on spatiotemporal domain instance enhancement provided by the present invention is described below. The video copy segment localization device based on spatiotemporal domain instance enhancement described below and the video copy segment localization method based on spatiotemporal domain instance enhancement described above can be referred to in correspondence with each other.

[0166] Figure 3 This is a schematic diagram of the video copy segment localization device based on spatiotemporal domain instance enhancement provided by the present invention, as shown below. Figure 3 As shown, the device includes the following modules:

[0167] The construction module 300 is used to extract the regional features and global features of the target frame in the query video and the reference video, and construct an instance relationship graph based on the regional features and the global features;

[0168] The acquisition module 310 is used to acquire local alignment information and global alignment information based on the instance relationship graph through spatial branch and temporal branch respectively;

[0169] The generation module 320 is used to generate a final inter-frame alignment result based on the local alignment information and the global alignment information through the complementarity-aware alignment module; the inter-frame alignment result is used for locating the copied segments of the query video and the reference video.

[0170] According to the present invention, a video copy segment localization device based on spatiotemporal domain instance enhancement obtains local alignment information through spatial branching based on an instance relationship graph, specifically including:

[0171] The nodes and edges in the instance relationship graph are encoded by an instance-based graph neural network, and the updated local feature matrix is ​​obtained by using a message passing mechanism to propagate information through the nodes and edges in the instance relationship graph.

[0172] Based on the updated local feature matrix, depth map matching calculation is performed to obtain local alignment information between the query video and the reference video.

[0173] According to the present invention, a video copy segment localization device based on spatiotemporal domain instance enhancement obtains global alignment information through time branching based on an instance relationship graph, specifically including:

[0174] Enhanced global features are obtained by augmenting global features based on instance relationship graphs;

[0175] Based on enhanced global features, temporal information is captured through temporal position encoding and a temporal attention module;

[0176] Based on temporal information, global alignment information between the query video and the reference video is calculated.

[0177] According to the present invention, a video copy segment localization device based on spatiotemporal domain instance enhancement performs depth map matching calculation based on an updated local feature matrix to obtain local alignment information between the query video and the reference video, specifically including:

[0178] Based on the updated local feature matrix, calculate the affinity matrix between the query video and the reference video;

[0179] Based on the affinity matrix, the inter-frame matching matrix is ​​calculated using the pseudo-Hungarian algorithm;

[0180] Based on the inter-frame matching matrix and the local scoring function, the local alignment information between the query video and the reference video is obtained.

[0181] According to the present invention, a video copy segment localization device based on spatiotemporal domain instance enhancement is provided, wherein the complementarity-aware alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0182] According to the present invention, a video copy segment localization device based on spatiotemporal domain instance enhancement is provided, wherein regional features and global features are extracted based on region of interest pooling.

[0183] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a video copy segment localization method based on spatiotemporal domain instance enhancement, the method including:

[0184] Extract the regional and global features of the target frames in the query video and reference video, and construct an instance relationship graph based on the regional and global features;

[0185] Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branching and temporal branching, respectively.

[0186] The complementaryness-aware alignment module generates the final inter-frame alignment result based on local and global alignment information; the inter-frame alignment result is used to locate duplicate segments in the query video and the reference video.

[0187] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the video copy segment localization method based on spatiotemporal domain instance enhancement provided by the above methods, the method comprising:

[0189] Extract the regional and global features of the target frames in the query video and reference video, and construct an instance relationship graph based on the regional and global features;

[0190] Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branching and temporal branching, respectively.

[0191] The complementaryness-aware alignment module generates the final inter-frame alignment result based on local and global alignment information; the inter-frame alignment result is used to locate duplicate segments in the query video and the reference video.

[0192] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video copy segment localization method based on spatiotemporal domain instance enhancement provided by the methods described above, the method comprising:

[0193] Extract the regional and global features of the target frames in the query video and reference video, and construct an instance relationship graph based on the regional and global features;

[0194] Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branching and temporal branching, respectively.

[0195] The complementaryness-aware alignment module generates the final inter-frame alignment result based on local and global alignment information; the inter-frame alignment result is used to locate duplicate segments in the query video and the reference video.

[0196] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for locating copied video segments based on spatiotemporal domain instance enhancement, characterized in that, include: Extract the regional and global features of the target frames in the query video and the reference video, and construct an instance relationship graph based on the regional and global features; Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branching and temporal branching, respectively. The complementaryness-aware alignment module generates the final inter-frame alignment result based on the local alignment information and the global alignment information. The inter-frame alignment result is used to locate the copied segments of the query video and the reference video; Based on the instance relationship graph, local alignment information is obtained through spatial branching, specifically including: The nodes and edges in the instance relationship graph are encoded by an instance-based graph neural network, and the information is propagated through the nodes and edges in the instance relationship graph using a message passing mechanism to obtain an updated local feature matrix. Based on the updated local feature matrix, depth map matching calculation is performed to obtain the local alignment information of the query video and the reference video.

2. The video copy segment localization method based on spatiotemporal domain instance enhancement according to claim 1, characterized in that, Based on the instance relationship graph, global alignment information is obtained through time branches, specifically including: The global features are enhanced based on the instance relationship graph to obtain enhanced global features; Based on the enhanced global features, temporal information is captured through temporal position encoding and a temporal attention module; Based on the time sequence information, the global alignment information of the query video and the reference video is calculated.

3. The video copy segment localization method based on spatiotemporal domain instance enhancement according to claim 1, characterized in that, Based on the updated local feature matrix, depth map matching calculation is performed to obtain local alignment information between the query video and the reference video, specifically including: Based on the updated local feature matrix, calculate the affinity matrix between the query video and the reference video; Based on the affinity matrix, the inter-frame matching matrix is ​​calculated using the pseudo-Hungarian algorithm; Based on the inter-frame matching matrix and the local scoring function, the local alignment information of the query video and the reference video is obtained.

4. The video copy segment localization method based on spatiotemporal domain instance enhancement according to claim 1, characterized in that, The complementary sensing alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

5. The video copy segment localization method based on spatiotemporal domain instance enhancement according to any one of claims 1-4, characterized in that, The regional and global features are extracted based on region-of-interest pooling.

6. A video copy segment localization device based on spatiotemporal domain instance enhancement, characterized in that, include: A construction module is used to extract regional and global features of target frames in query and reference videos, and construct an instance relationship graph based on the regional and global features; The acquisition module is used to acquire local alignment information and global alignment information based on the instance relationship graph through spatial branches and temporal branches, respectively. The generation module is used to generate the final inter-frame alignment result based on the local alignment information and the global alignment information through the complementarity-aware alignment module. The inter-frame alignment result is used to locate the copied segments of the query video and the reference video; Based on the instance relationship graph, local alignment information is obtained through spatial branching, specifically including: The nodes and edges in the instance relationship graph are encoded by an instance-based graph neural network, and the information is propagated through the nodes and edges in the instance relationship graph using a message passing mechanism to obtain an updated local feature matrix. Based on the updated local feature matrix, depth map matching calculation is performed to obtain the local alignment information of the query video and the reference video.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video copy segment localization method based on spatiotemporal domain instance enhancement as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video copy segment localization method based on spatiotemporal domain instance enhancement as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the video copy segment localization method based on spatiotemporal domain instance enhancement as described in any one of claims 1 to 5.

Images