Video copy fragment positioning method and device based on time-space domain instance enhancement

Abstract

The invention provides a video copy fragment positioning method and device based on time-space domain instance enhancement, and the method comprises the steps: extracting the regional features and global features of target frames in a query video and a reference video, and constructing an instance relation graph based on the regional features and the global features; obtaining local alignment information and global alignment information through a spatial branch and a time branch based on the instance relation graph; generating a final inter-frame alignment result based on the local alignment information and the global alignment information through a complementarity perception alignment module; the inter-frame alignment result is used for positioning duplicated fragments of the query video and the reference video. According to the method, global semantic information and fine-grained more robust local representation are comprehensively considered, the matching precision of video frames in a query video and a reference video is improved, and the accuracy of a matching result of copied fragments in the query video and the reference video can also be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for locating a video copy segment based on spatiotemporal domain instance enhancement. Background Art

[0002] The task of video copy segment localization (VSL) requires identifying time periods in a pair of videos that contain copied content. Existing video copy segment localization technology (VSL) is mainly divided into three steps: extracting video key frames, extracting features from key frames and modeling temporal dependencies, and extracting aligned segments using time alignment methods.

[0003] In the key frame extraction part, due to the particularity of the video copy positioning task, the video-level and segment-level features cannot meet the precise positioning requirements. The current methods generally use frames per second as key frames. Some work attempts to model key frame extraction as a multi-frame classification problem to select more representative key frames; feature extraction usually uses a pre-trained model, and after extracting frame-level features, the temporal dependency is modeled through the attention module; the query video and reference video features are similarly calculated to obtain a frame similarity matrix. Traditional methods use Hough voting, temporal networks, dynamic programming and other schemes to achieve the location of the homologous segments of two videos. Recent methods use more advanced data-driven methods, such as: parameterized time-sensitive similarity metrics based on the Fourier domain; constructing a spatiotemporal similarity step prediction graph; modeling the time alignment task as an object detection task, and using an object detector to predict the bounding box of the copy segment.

[0004] Most current VSL methods rely on spatiotemporal global features. Although these global features are good at capturing high-level information and long-term temporal relationships, they mainly reflect a coarse semantic granularity. Therefore, these methods often fail to provide a deep understanding of complex, fine-grained details, which are crucial to achieving accurate results in VSL tasks. Some related methods attempt to introduce local spatial information by using pixel-level partitioning, but this destroys the spatial semantic structure and leads to a decrease in matching accuracy. Summary of the invention

[0005] The present invention provides a video copy segment positioning method and device based on spatiotemporal domain instance enhancement, which is used to solve the defects of the prior art that it relies on global features and cannot deeply understand complex and fine-grained details, and realizes a more robust local representation that comprehensively considers global semantic information and fine-grained, thereby improving the matching precision and the accuracy of the matching results.

[0006] The present invention provides a video copy segment positioning method based on spatiotemporal domain instance enhancement, comprising the following steps: Extracting regional features and global features of target frames in the query video and the reference video, and constructing an instance relationship graph based on the regional features and the global features; Based on the instance relationship graph, obtaining local alignment information and global alignment information through a spatial branch and a temporal branch respectively; A final inter-frame alignment result is generated based on the local alignment information and the global alignment information through a complementarity-aware alignment module; the inter-frame alignment result is used for locating the duplicated segments of the query video and the reference video.

[0007] According to a video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention, based on the instance relationship graph, local alignment information is obtained through spatial branches, specifically including: Encoding the nodes and edges in the instance relationship graph by using an instance-based graph neural network, and propagating information through the nodes and edges in the instance relationship graph by using a message passing mechanism to obtain an updated local feature matrix; Based on the updated local feature matrix, a depth map matching calculation is performed to obtain local alignment information of the query video and the reference video.

[0008] According to a video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention, based on the instance relationship graph, global alignment information is obtained through time branches, specifically including: Enhance the global feature based on the instance relationship graph to obtain an enhanced global feature; Based on the enhanced global features, temporal information is captured through temporal position encoding and temporal attention modules; Based on the timing information, global alignment information of the query video and the reference video is calculated.

[0009] According to a video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention, a depth map matching calculation is performed based on the updated local feature matrix to obtain local alignment information of the query video and the reference video, specifically comprising: Based on the updated local feature matrix, calculating the affinity matrix of the query video and the reference video; Based on the affinity matrix, calculating the inter-frame matching matrix by using a pseudo-Hungarian algorithm; Based on the inter-frame matching matrix and the local scoring function, local alignment information of the query video and the reference video is obtained.

[0010] According to a video copy segment positioning method based on spatiotemporal instance enhancement provided by the present invention, the complementarity-aware alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0011] According to a video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention, the regional features and global features are extracted based on interest region pooling.

[0012] The present invention also provides a video copy segment positioning device based on spatiotemporal domain instance enhancement, comprising the following modules: A construction module, used to extract regional features and global features of target frames in the query video and the reference video, and construct an instance relationship graph based on the regional features and the global features; An acquisition module, used for acquiring local alignment information and global alignment information through a spatial branch and a temporal branch respectively based on the instance relationship graph; A generation module is used to generate a final inter-frame alignment result based on the local alignment information and the global alignment information through a complementarity-aware alignment module; the inter-frame alignment result is used for locating the duplicated segments of the query video and the reference video.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for locating a video copy segment based on spatiotemporal instance enhancement as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for locating video copy segments based on spatiotemporal domain instance enhancement.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for locating video copy segments based on spatiotemporal domain instance enhancement.

[0016] The video duplicate segment positioning method and device based on spatiotemporal instance enhancement provided by the present invention extracts regional features and global features of target frames in query videos and reference videos, constructs an instance relationship graph according to the regional features and global features, and then obtains local alignment information and global alignment information through spatial branches and temporal branches respectively according to the instance relationship graph. Finally, a complementary perception alignment module is used to generate a final inter-frame alignment result according to the local alignment information and the global alignment information, thereby locating the duplicate segments of the query video and the reference video. It comprehensively considers global semantic information and fine-grained more robust local representation, improves the matching accuracy of video frames in the query video and the reference video, and can also improve the accuracy of the matching results of the duplicate segments in the query video and the reference video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a method for locating video copy segments based on spatiotemporal domain instance enhancement provided by the present invention.

[0019] Figure 2 The present invention provides a flow chart of a method for locating a video copy segment.

[0020] Figure 3 It is a structural schematic diagram of a video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Figure 1 FIG. 1 is a flow chart of a method for locating video copy segments based on spatiotemporal domain instance enhancement provided by the present invention. Figure 1 As shown, the method comprises the following steps: Step 100: extract regional features and global features of target frames in the query video and the reference video, and construct an instance relationship graph based on the regional features and the global features.

[0024] Specifically, in the embodiment of the present invention, it is necessary to locate the duplicated segments of the query video and the reference video. First, it is necessary to extract the regional features and global features in the query video and the reference video respectively.

[0025] The feature extraction part can be performed using object-aware methods to ensure the association of local features with objects within the frame.

[0026] In some implementations, regional features and global features are extracted based on a region of interest pooling (ROI-Pooling).

[0027] Specifically, by detecting objects in video frames, we can extract the regional features of these objects and the global frame features. Can extract object frame collection ,in Indicates the number of objects detected in the frame. Then the local features of each object can be extracted by ROI-Pooling technique. , and obtain the global features of the frame By extracting the local features of each object through the ROI-Pooling method, it can be ensured that the extracted local features are associated with the objects in the frame for subsequent processing.

[0028] After extracting the regional features and global features of the target frames in the query video and the reference video, the regional features and the global features can be jointly processed to construct an instance-relation graph (IRG) to enhance the expression ability.

[0029] For each frame , the local features extracted Can be stacked into a feature matrix , Represents the feature dimension. This matrix contains the feature information of each object and serves as the feature representation of the node in the instance relationship graph.

[0030] Next, we can construct the edges in the instance relationship graph by calculating the similarity between object features. We can use cosine similarity to measure the similarity between two objects. and The cosine similarity of Elements in The adjacency matrix describes the The relationship between objects in .

[0031] In the instance relationship graph, it is necessary not only to model the similarity between nodes (objects), but also to give each edge higher-order information. To achieve this, the edge features can be calculated through the cross-attention mechanism and the global features.

[0032] By cross-attention between adjacent nodes and global features As a virtual node, calculate the edge feature matrix of the high-order relationship between nodes , the edge feature contains the relationship information between nodes and the global context information. The specific formula is as follows:

[0033] in, is the edge feature, MLP means multi-layer perceptron, is the attention weight, is a learnable weight matrix, is the node feature matrix superimposed with global features.

[0034] Finally, the node feature matrix , adjacency matrix and edge feature matrix The three describe the frame together The spatial relationship between objects in the example is generated , expressed as .

[0035] Step 101: Based on the instance relationship graph, local alignment information and global alignment information are obtained through a spatial branch and a temporal branch respectively.

[0036] Specifically, in an embodiment of the present invention, a new VSL framework is designed, which uses a dual-branch network structure to perform targeted spatiotemporal modeling on two types of features (i.e., regional features and global features), obtains local alignment information through the spatial branch, and obtains global alignment information through the temporal branch.

[0037] Among them, the spatial branch in the dual-branch network structure is the instance-relation graph-based spatial branch (IGS), which can process the instance relationship graph to obtain local alignment information between frames, that is, the similarity alignment matrix of local objects in the query video and the reference video at the frame level, which describes the spatial matching of similar objects in the query video and the reference video.

[0038] The temporal branch in the dual-branch network structure is an instance-enhanced temporal branch (IET), which can process global features according to the instance relationship graph. The global frame-level features obtained after processing contain the visual semantic information of the frame itself and the temporal dependency between video frames, and then perform inter-frame similarity calculation, that is, by comparing the frame-level features of the query video and the reference video, their similarity in the temporal dimension is calculated to obtain the global alignment information between frames.

[0039] Step 102: Generate a final inter-frame alignment result based on the local alignment information and the global alignment information through a complementarity-aware alignment module; the inter-frame alignment result is used to locate the duplicated segments of the query video and the reference video.

[0040] Specifically, local features capture important local details in the image, while global features capture high-level visual semantic information of the frame. Therefore, after obtaining local and global alignment information, the complementarity between local and global features can be exploited through the Complementarity-perception Alignment Module (CAM), combining the broad perspective provided by global features with the detailed insights of local features, ensuring a comprehensive and balanced approach to obtain the necessary information for accurate VSL analysis. The formula is as follows:

[0041] in, is the final frame alignment result, is the local feature similarity matrix (local feature information), is the global feature similarity matrix (global feature information), and CAM is the complementary perception alignment module.

[0042] After obtaining the inter-frame alignment results of the query video and the reference video, the duplicate segments of the query video and the reference video can be located.

[0043] The video duplicate segment positioning method based on spatiotemporal instance enhancement provided by the present invention extracts regional features and global features of target frames in query videos and reference videos, constructs an instance relationship graph according to the regional features and global features, and then obtains local alignment information and global alignment information through spatial branches and temporal branches respectively according to the instance relationship graph. Finally, a complementary perception alignment module is used to generate a final inter-frame alignment result according to the local alignment information and the global alignment information, thereby locating the duplicate segments of the query video and the reference video. It comprehensively considers global semantic information and fine-grained more robust local representation, improves the matching accuracy of video frames in the query video and the reference video, and can also improve the accuracy of the matching results of the duplicate segments in the query video and the reference video.

[0044] According to a video copy segment positioning method based on spatiotemporal instance enhancement provided by the present invention, local alignment information is obtained through spatial branches based on an instance relationship graph, specifically including: The nodes and edges in the instance relationship graph are encoded through an instance-based graph neural network, and the message passing mechanism is used to propagate information through the nodes and edges in the instance relationship graph to obtain an updated local feature matrix. Based on the updated local feature matrix, depth map matching calculation is performed to obtain the local alignment information of the query video and the reference video.

[0045] Specifically, the spatial branch in the embodiment of the present invention may include an instance-based graph neural network (Instance-GNN) structure and a deep graph matching module.

[0046] Among them, the spatial structural relationship between nodes and edges in the instance relationship graph can be encoded by Instance-GNN, and the message passing mechanism can be used to encode the spatial structural relationship between nodes and edges in the instance relationship graph (i.e., the adjacency matrix ) propagates information to achieve modeling relationships across the entire graph, thereby improving the model’s understanding of instance relationships in the video and obtaining an updated local feature matrix, the formula of which is as follows:

[0047] in, It is The feature matrix of the layer GNN output, is the adjacency matrix, It is The learnable weights of the layer.

[0048] Among them, the node feature has a self-update mechanism. In the embodiment of the present invention, nonlinear transformation and adaptive weight are combined to update the feature representation of each node. The formula is as follows:

[0049] in, represents the node features after self-update, Indicates Video frames in layered graph neural networks Midpoint initial features. is the activation function, is a learnable temperature parameter, are the learnable weights and biases respectively.

[0050] This self-updating mechanism ensures that during the multi-layer GNN propagation process, nodes can maintain their important information while adapting to changes in local and global context.

[0051] After the updated local feature matrix is ​​obtained through multiple layers of GNN, the corresponding feature point pairs can be found from images of different perspectives through the deep graph matching module, thereby capturing the spatial matching of similar objects between the query video and the reference video, and obtaining the local alignment information of the query video and the reference video.

[0052] According to a video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention, based on an instance relationship graph, global alignment information is obtained through time branches, specifically including: The global features are enhanced based on the instance relationship graph to obtain enhanced global features; Based on enhanced global features, temporal information is captured through temporal position encoding and temporal attention modules; Based on the temporal information, the global alignment information of the query video and the reference video is calculated.

[0053] Specifically, the time branch in the embodiment of the present invention may include a temporal position encoding process and a temporal attention module.

[0054] First, the global features can be modeled into temporal relationships through transformers. For each frame, the temporal branch enhances the global representation of the frame by adding fine-grained local information to the global features (i.e., the enhancement of the global features by local features), thus obtaining the enhanced global features. .

[0055] Then, for enhancing global features, temporal position encoding and temporal attention modules can be used to capture frame-level temporal dependencies. In this way, the model can better understand the long-term dependencies and dynamic changes between video frames.

[0056] This process can be accomplished using Multi-Head self-Attention (MHA) and Temporal Position Encoding (TPM), as shown below:

[0057] in, To obtain the temporal information after temporal modeling of enhanced global features, To enhance the global features.

[0058] After obtaining the timing information, the global frame-to-frame temporal similarity can be calculated based on the timing information to obtain the global feature similarity matrix (i.e., global alignment information), the formula of which is as follows:

[0059] in, is the global feature similarity matrix, To query the timing information of the video, It is the timing information of the reference video.

[0060] According to a video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention, a depth map matching calculation is performed based on an updated local feature matrix to obtain local alignment information of a query video and a reference video, specifically including: Based on the updated local feature matrix, the affinity matrix of the query video and the reference video is calculated; Based on the affinity matrix, the inter-frame matching matrix is ​​calculated using the pseudo-Hungarian algorithm; Based on the inter-frame matching matrix and the local scoring function, the local alignment information of the query video and the reference video is obtained.

[0061] Specifically, in the process of performing depth map matching calculation according to the updated local feature matrix, the affinity matrix of the query video and the reference video can be calculated according to the updated local feature matrix. The specific formula is as follows:

[0062] in, is the affinity matrix, is the transpose of the updated local feature matrix of the query video, is a learnable weight matrix, is the updated local feature matrix of the reference video.

[0063] Get the affinity matrix After that, the inter-frame matching matrix between the query video and the reference video can be calculated using the pseudo-Hungarian algorithm , the specific formula is as follows:

[0064] In the formula, Indicates that the pseudo-Hungarian matching algorithm is applied to determine the matching of local objects.

[0065] for , a Hungarian loss can be designed To supervise it, the specific formula is as follows:

[0066] in, is the true matching relationship between the query video and the reference video, is the predicted matching relationship between the query video and the reference video, They are The number of nodes in the frame.

[0067] Finally, the local feature similarity matrix (i.e. local alignment information) can be obtained based on the inter-frame matching matrix and the local scoring function. The formula is as follows:

[0068] in, Represents the query video frame and reference video frame The local similarity between Represents the query video frame The local characteristics of Represents the reference video frame The local characteristics of It is The number of nodes in the frame.

[0069] According to a video copy segment positioning method based on spatiotemporal instance enhancement provided by the present invention, a complementarity-aware alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0070] Specifically, the complementarity-aware alignment module in the embodiment of the present invention is trained based on the differentiable dynamic time warping loss and the weighted mean square error loss as the alignment loss function.

[0071] In the embodiment of the present invention, a differentiable dynamic time warping (DTW) method is used to use potential time alignment as a weak supervision signal. In this way, the inherent limitations in VSL task annotations (typical VSL annotations only provide the start and end positions of similar segments between two videos) can be solved.

[0072] Some previous methods approach element-wise constraints by constructing diagonal matrices, assuming that the two sequences are perfectly synchronized. This assumption is an oversimplification, as similar video clips may exhibit temporal variations due to content editing, changes in playback speed, noise, or distortion, leading to complex alignments that cannot be described by diagonal matrices.

[0073] Therefore, the embodiment of the present invention introduces the alignment loss based on smoothDTW, and the specific formula is as follows:

[0074] The final DTW loss can be expressed as:

[0075] in, represents the inter-frame similarity matrix between the query video and the reference video, is the alignment cost matrix, is the cumulative distance matrix, is a smooth minimum function, is the temperature parameter, is the size of the similarity submatrix.

[0076] In the embodiment of the present invention, a weighted mean square error (MSE) loss may be further introduced to constrain the non-copy segments of the video, and the formula thereof is as follows:

[0077] in, It is Frame and Prediction similarity between frames, It is Frame and The true similarity between frames (ground truth), is the weight factor, are the frame numbers of the query video and the reference video, respectively.

[0078] In some embodiments, the alignment loss function of the complementarity-aware alignment module is Can be designed as:

[0079] in, is the weight factor.

[0080] The video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the present invention is further explained below through embodiments in specific application scenarios.

[0081] Figure 2 The flowchart of the video duplication segment positioning method provided by the present invention is as follows: Figure 2 As shown in the figure, this embodiment designs a new VSL framework to extract video frame-level and region-level features. Through the dual-branch network structure, the two features are respectively modeled in a targeted spatiotemporal manner. For the feature extraction part, this embodiment uses the object-aware method to ensure the connection between local features and objects in the frame. This embodiment extracts the object frame set , Indicates the number of objects detected in the frame. Use ROI-Pooling technology to extract the corresponding regional features , and obtain global features .

[0082] The dual-branch structure design is not completely decoupled. In this embodiment, the two types of information are jointly processed in the instance relationship graph construction part to enhance the expression ability. IGS models IRG and calculates the alignment relationship between frames by graph matching. Local features Stacking into feature matrix , Represents the feature dimension. Adjacency matrix Medium Element Obtained from the cosine similarity of node (local) features. Edge feature matrix of high-order relationships between nodes By cross-attention between adjacent nodes and global features Compute as a virtual node:

[0083] in, is the edge feature, MLP means multi-layer perceptron, is the attention weight, is a learnable weight matrix, is the node feature matrix superimposed with global features.

[0084] The final instance relationship diagram is represented as .

[0085] This embodiment uses IRG to represent the video feature space, which can also be used to refine the IET branch. This embodiment designs an Instance-GNN structure, which encodes the spatial structural relationship between nodes and edges and uses a message passing mechanism to propagate information through nodes and edges to achieve modeling relationships across the entire graph. This improves the model's understanding of instance relationships in the video:

[0086] in, It is The feature matrix of the layer GNN output, It is The learnable weights of the layer.

[0087] The node self-update mechanism dynamically adjusts features based on their relative importance in the graph:

[0088] in, is the activation function, is a learnable temperature parameter, are learnable weights and biases.

[0089] The updated nodes of videos Q and R have rich structure and context information. This embodiment captures the correlation between videos through deep graph matching. First, the updated nodes are used to calculate the affinity matrix:

[0090] in, is the affinity matrix, is a learnable weight matrix.

[0091] Use the pseudo-Hungarian algorithm to calculate the inter-frame matching matrix :

[0092] Aggregate the prediction matrix by the local scoring function:

[0093] It is The number of nodes in the frame.

[0094] IET models the temporal relationship of global features through transformers, and then performs frame-level similarity calculation. Given a pair of query video Q and reference video R, IET first calculates the similarity by A virtual node for global features is added to enhance its global representation by using fine-grained information from local features. . Further use of MHA and TPM to capture timing information:

[0095] And calculate the global frame-to-frame temporal similarity:

[0096] Local features capture important local details in an image, while global features capture high-level visual semantic information of a frame. CAM leverages the complementarity between local and global features in IRG and IET, combining the broad perspective provided by global features with the detailed insights of local features, ensuring a comprehensive and balanced approach to obtain the necessary information for accurate VSL analysis:

[0097] in It is the final inter-frame alignment map, and CAM is the complementary perception alignment module.

[0098] For the formula In , this embodiment designs a Hungarian loss for supervision:

[0099] in, is the true value of the match, They are The number of nodes in the frame.

[0100] This embodiment adopts a differentiable dynamic time warping method that uses potential temporal alignment as a weak supervision signal. It solves the inherent limitations in VSL task annotations (typical VSL annotations only provide the start and end positions of similar segments between two videos). Some previous methods approached element-level constraints by constructing diagonal matrices, assuming that the two sequences are completely synchronized. This assumption is an oversimplification because similar video segments may exhibit temporal changes due to content editing, changes in playback speed, noise, or distortion, resulting in complex alignments that cannot be described by diagonal matrices. Therefore, this embodiment introduces an alignment loss based on smoothDTW:

[0101] The final DTW loss is expressed as:

[0102] Among them, C is the alignment cost matrix, D is the cumulative distance matrix, SmoothMin is the smooth minimum function, is the temperature parameter, is the size of the similarity submatrix.

[0103] This embodiment further introduces a weighted mean square error (MSE) loss to constrain the non-replicated segments:

[0104] is the weight factor.

[0105] The alignment loss of this embodiment is designed as:

[0106] is the weight factor.

[0107] The network of this embodiment captures instance-level features and fine-grained interactions by constructing IRG, thereby preserving local information integrity. It adopts Instance-GNN to learn instance-to-instance relationships and structured information at an appropriate representation granularity. Inter-frame similarity alignment is achieved through deep graph matching. In addition, this embodiment captures long-term temporal relationships by modeling instance-enhanced frame-level features, and uses a complementary perceptual alignment module to effectively process and integrate complementary spatiotemporal information to generate an accurate inter-frame alignment map. The method of this embodiment also combines differentiable DTW to utilize potential time alignment as a weak supervisory signal, thereby improving the accuracy of the matching process.

[0108] The video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention is described below. The video copy segment positioning device based on spatiotemporal domain instance enhancement described below and the video copy segment positioning method based on spatiotemporal domain instance enhancement described above can refer to each other.

[0109] Figure 3 The schematic diagram of the structure of the video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention is as follows: Figure 3 As shown, the device includes the following modules: A construction module 300 is used to extract regional features and global features of target frames in the query video and the reference video, and construct an instance relationship graph based on the regional features and the global features; An acquisition module 310, configured to acquire local alignment information and global alignment information through a spatial branch and a temporal branch respectively based on the instance relationship graph; The generation module 320 is used to generate a final inter-frame alignment result based on the local alignment information and the global alignment information through a complementarity-aware alignment module; the inter-frame alignment result is used for locating the duplicate segments of the query video and the reference video.

[0110] According to the present invention, a video copy segment positioning device based on spatiotemporal instance enhancement is provided, which obtains local alignment information through spatial branches based on an instance relationship graph, and specifically includes: The nodes and edges in the instance relationship graph are encoded through an instance-based graph neural network, and the message passing mechanism is used to propagate information through the nodes and edges in the instance relationship graph to obtain an updated local feature matrix. Based on the updated local feature matrix, depth map matching calculation is performed to obtain the local alignment information of the query video and the reference video.

[0111] According to the present invention, a video copy segment positioning device based on spatiotemporal instance enhancement is provided, which obtains global alignment information through time branches based on an instance relationship graph, specifically comprising: The global features are enhanced based on the instance relationship graph to obtain enhanced global features; Based on enhanced global features, temporal information is captured through temporal position encoding and temporal attention modules; Based on the temporal information, the global alignment information of the query video and the reference video is calculated.

[0112] According to a video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention, a depth map matching calculation is performed based on an updated local feature matrix to obtain local alignment information of a query video and a reference video, specifically including: Based on the updated local feature matrix, the affinity matrix of the query video and the reference video is calculated; Based on the affinity matrix, the inter-frame matching matrix is ​​calculated using the pseudo-Hungarian algorithm; Based on the inter-frame matching matrix and the local scoring function, the local alignment information of the query video and the reference video is obtained.

[0113] According to a video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention, the complementarity-aware alignment module is trained based on differentiable dynamic time warping loss and weighted mean square error loss as alignment loss functions.

[0114] According to a video copy segment positioning device based on spatiotemporal domain instance enhancement provided by the present invention, regional features and global features are extracted based on interest region pooling.

[0115] Figure 4 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the video copy segment positioning method based on spatiotemporal domain instance enhancement, the method comprising: Extract regional features and global features of target frames in query videos and reference videos, and construct instance relationship graphs based on regional features and global features; Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branches and temporal branches respectively; Through the complementarity-aware alignment module, the final inter-frame alignment result is generated based on the local alignment information and the global alignment information; the inter-frame alignment result is used to locate the copied segments of the query video and the reference video.

[0116] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0117] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the above methods, the method comprising: Extract regional features and global features of target frames in query videos and reference videos, and construct instance relationship graphs based on regional features and global features; Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branches and temporal branches respectively; Through the complementarity-aware alignment module, the final inter-frame alignment result is generated based on the local alignment information and the global alignment information; the inter-frame alignment result is used to locate the copied segments of the query video and the reference video.

[0118] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the video copy segment positioning method based on spatiotemporal domain instance enhancement provided by the above methods, the method comprising: Extract regional features and global features of target frames in query videos and reference videos, and construct instance relationship graphs based on regional features and global features; Based on the instance relationship graph, local alignment information and global alignment information are obtained through spatial branches and temporal branches respectively; Through the complementarity-aware alignment module, the final inter-frame alignment result is generated based on the local alignment information and the global alignment information; the inter-frame alignment result is used to locate the copied segments of the query video and the reference video.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video copy segment location method based on spatiotemporal instance enhancement, characterized in that: include: Extracting regional features and global features of target frames in the query video and the reference video, and constructing an instance relationship graph based on the regional features and the global features; Based on the instance relationship graph, obtaining local alignment information and global alignment information through a spatial branch and a temporal branch respectively; Generate a final inter-frame alignment result based on the local alignment information and the global alignment information through a complementarity-aware alignment module; The inter-frame alignment result is used for locating the duplicate segments of the query video and the reference video.

2. The video copy segment positioning method based on spatiotemporal domain instance enhancement according to claim 1 is characterized in that: Based on the instance relationship graph, local alignment information is obtained through spatial branches, specifically including: Encoding the nodes and edges in the instance relationship graph by using an instance-based graph neural network, and propagating information through the nodes and edges in the instance relationship graph by using a message passing mechanism to obtain an updated local feature matrix; Based on the updated local feature matrix, a depth map matching calculation is performed to obtain local alignment information of the query video and the reference video.

3. The video copy segment positioning method based on spatiotemporal domain instance enhancement according to claim 1 is characterized in that: Based on the instance relationship graph, global alignment information is obtained through time branching, specifically including: Enhance the global feature based on the instance relationship graph to obtain an enhanced global feature; Based on the enhanced global features, temporal information is captured through temporal position encoding and temporal attention modules; Based on the timing information, global alignment information of the query video and the reference video is calculated.

4. The video duplication segment positioning method based on spatiotemporal domain instance enhancement according to claim 2 is characterized in that: Based on the updated local feature matrix, a depth map matching calculation is performed to obtain local alignment information of the query video and the reference video, specifically including: Based on the updated local feature matrix, calculating the affinity matrix of the query video and the reference video; Based on the affinity matrix, calculating the inter-frame matching matrix by using a pseudo-Hungarian algorithm; Based on the inter-frame matching matrix and the local scoring function, local alignment information of the query video and the reference video is obtained.

5. The video copy segment positioning method based on spatiotemporal domain instance enhancement according to claim 1 is characterized in that: The complementarity-aware alignment module is trained based on the differentiable dynamic time warping loss and the weighted mean square error loss as the alignment loss function.

6. The video copy segment positioning method based on spatiotemporal instance enhancement according to any one of claims 1 to 5, characterized in that: The regional features and global features are extracted based on the region of interest pooling method.

7. A video copy segment positioning device based on spatiotemporal instance enhancement, characterized in that: include: A construction module, used to extract regional features and global features of target frames in the query video and the reference video, and construct an instance relationship graph based on the regional features and the global features; An acquisition module, used for acquiring local alignment information and global alignment information through a spatial branch and a temporal branch respectively based on the instance relationship graph; A generating module, configured to generate a final inter-frame alignment result based on the local alignment information and the global alignment information through a complementarity-aware alignment module; The inter-frame alignment result is used for locating the duplicate segments of the query video and the reference video.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for locating a video copy segment based on spatiotemporal domain instance enhancement is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for locating a video copy segment based on spatiotemporal instance enhancement as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for locating a video copy segment based on spatiotemporal instance enhancement as claimed in any one of claims 1 to 6 is implemented.

Images