A video copy detection method based on attention enhanced multi-modal features

Abstract

The application provides a video copy detection method based on attention-enhanced multi-modal features, comprising: extracting frame image features and audio features in video data; performing self-attention enhancement and cross-attention enhancement on the extracted frame image features and audio features to obtain long-range dependency information hidden in the features; using a multi-modal factor decomposition bilinear pooling layer to perform feature fusion on the enhanced frame image features and audio features; using a local feature matching network to generate a similarity matrix of a query video and a copy video; using the difference between the detected video copy region and the label true value on the similarity matrix to form a training loss, and learning the neural network in a data-driven manner. The application not only has faster extraction speed and fused multi-modal information, but also can accurately detect copy videos with complex transformations, can better guide downstream tasks such as video retrieval, video action understanding, video copy, and has the characteristics of high robustness.

Description

Technical Field

[0001] This invention relates to the field of multimedia information processing technology, and more particularly to a video copy detection method based on attention-enhanced multimodal features. Background Technology

[0002] Video copying is primarily achieved by transforming source video through optical, geometric, or temporal transformations, including methods such as inserting icons, simulating recording, scaling, and picture-in-picture. Due to the sheer volume of network data, manually identifying copied videos within massive datasets is impractical; therefore, video copy detection technology has been developed.

[0003] Patent CN 111709945 A discloses a video copy detection method based on deep local features, comprising: extracting frame images from video data, and then constructing image pyramids using different scales; constructing a deep convolutional neural network model to extract feature maps from the input image pyramids, and fusing the feature maps to obtain a fused feature map; training the deep convolutional neural network model using metric learning; extracting the fused feature map from the image pyramids using the trained deep convolutional neural network model; extracting key points from the fused feature map using maximum suppression, and extracting corresponding local features based on the key points; and performing video copy detection based on the local features. This invention not only has a faster extraction speed but also stronger local feature representation. Therefore, it can accurately detect copied videos with various complex transformations, exhibiting high robustness. The above-disclosed scheme only considers a single video modality feature, which has a certain bottleneck in the accuracy of video copy detection. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a video copy detection method based on attention-enhanced multimodal features. This invention enhances the attention of extracted initial video and audio features using a Transformer model, better considering long-range dependencies between features. Then, multimodal factorization and bilinear pooling are used to fuse visual and audio information, thus fusing video information across multiple modal dimensions. This more fully captures the similarity representation of different modalities within the feature space. Finally, an object detection algorithm is used to cleverly identify specific similarity patterns between video pairs.

[0005] The technical means employed in this invention are as follows:

[0006] A video copy detection method based on attention-enhanced multimodal features includes:

[0007] Extract frame image features and audio features from video data;

[0008] Self-attention enhancement and cross-attention enhancement are applied to the extracted frame image features and audio features to obtain the long-range dependency information hidden in the features;

[0009] Multimodal factor decomposition is used to fuse the enhanced frame image features and audio features using a bilinear pooling layer;

[0010] A similarity matrix between the query video and the copied video is generated using a local feature matching network;

[0011] The training loss is constructed by using the difference between the detected video copy regions and the ground truth labels on the similarity matrix, thus enabling the neural network to learn in a data-driven manner.

[0012] Furthermore, the extraction of frame image features and audio features from the video data includes:

[0013] A convolutional neural network with contrastive learning is used to extract video frame images into 256-dimensional visual feature vectors that can fully represent the image features;

[0014] An audio fingerprint generation network is used to extract audio features in seconds, with one 128-dimensional audio feature vector extracted per second.

[0015] Furthermore, the step of performing self-attention enhancement and cross-attention enhancement on the extracted frame image features and audio features to obtain the long-range dependency information hidden in the features includes:

[0016] Fixed sine and cosine time codes are added to the extracted initial features to make them visual and auditory feature codes with time information;

[0017] Construct a Transformer model and use the model's self-cross-attention module to enhance the self-cross-attention of visual and auditory feature encodings;

[0018] After the cross-attention enhancement is completed, cross-attention enhancement is performed again between the query video and the reference video pair;

[0019] Furthermore, the feature fusion of the enhanced frame image features and audio features using multimodal factorization bilinear pooling includes:

[0020] By sequentially combining fully connected layers, element-wise multiplication layers, and pooling layers, a multimodal factorization bilinear pooling layer is constructed.

[0021] The enhanced frame image features and audio features are expanded and contracted in the multimodal factorization bilinear pooling layer to obtain robust and expressive fusion features.

[0022] Furthermore, the enhanced frame image features and audio features are expanded and contracted in a multimodal factorization bilinear pooling layer to obtain robust and expressive fusion features, including:

[0023] In the expansion phase, features from different modalities are expanded to a high-dimensional space and then combined through an element-wise multiplication layer. A Dropout layer is added after the element-wise multiplication layer to prevent overfitting.

[0024] During the shrinkage phase, pooling and normalization layers are executed to compress high-dimensional features into compact output features. Due to the introduction of element-wise multiplication, the size of the output neurons changes significantly, and the model cannot converge to a local optimum. Power normalization and L2 normalization layers are introduced at the end of the network.

[0025] Furthermore, the length and width of the similarity matrix are the number of seconds of the query video and the copied video, respectively. The similarity matrix contains specific infringement patterns between copied fragment pairs. The 4D copied fragment pairs with start and end timestamps are formulated as bounding boxes with upper-left and lower-right coordinates in the similarity mapping.

[0026] Furthermore, the method of using the difference in similarity matrix between the detected video copy regions and the ground truth labels to construct the training loss, and learning the neural network in a data-driven manner, includes:

[0027] The temporal boundaries of the truth-marked copy fragments are represented as bounding box coordinates on the similarity matrix;

[0028] The task of locating all duplicated fragment pairs is considered as an object detection task on the similarity matrix;

[0029] Based on the YOLO object detection network, target detection is performed on bounding boxes that represent copy relationships on the similarity matrices generated from querying and copying videos, and specific similarity patterns between video pairs on the similarity map are detected.

[0030] The regression loss between the detected bounding boxes with upper-left and lower-right coordinates and the ground truth bounding boxes is calculated, and the regression loss is used as the training loss to train the network model.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. The video copy detection method based on attention-enhanced multimodal features provided by this invention not only utilizes attention enhancement strategies to better leverage long-range dependencies between features, but also fully integrates information from video and audio features. The visual and auditory feature vectors contain both intermodal and intramodal information, and the fused audiovisual information can better guide downstream tasks such as video retrieval, video action understanding, and video copying. Learning based on multimodal features is gradually becoming a new trend in the field of multimedia information processing technology.

[0033] 2. Compared with traditional single-modal video copy detection algorithms, this invention not only extracts data faster but also integrates multimodal information, enabling accurate detection of copied videos with complex transformations. It is highly robust and provides a feasible technical solution for network regulatory departments to monitor the large amount of tampered and rampant multimedia video data on the Internet. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the method of the present invention.

[0036] Figure 2 This is a schematic diagram illustrating the principle of attention enhancement in this invention.

[0037] Figure 3 This is a schematic diagram illustrating the feature fusion principle of the present invention.

[0038] Figure 4 This is a schematic diagram illustrating the principle of copy region detection in this invention. Detailed Implementation

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0042] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0044] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0045] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0046] like Figure 1 As shown, this invention provides a video copy detection method based on attention-enhanced multimodal features, comprising:

[0047] S1. Extract frame image features and audio features from video data;

[0048] S2. Perform self-attention enhancement and cross-attention enhancement on the extracted frame image features and audio features to obtain the long-range dependency information hidden in the features;

[0049] S3. Multimodal factor decomposition is used to fuse the enhanced frame image features and audio features using a bilinear pooling layer.

[0050] S4. Use a local feature matching network to generate a similarity matrix between the query video and the copied video;

[0051] S5. The difference between the detected video copy region and the ground truth label on the similarity matrix is ​​used to form the training loss, and the neural network is learned in a data-driven manner.

[0052] In a specific implementation, as a preferred embodiment of the present invention, step S1, extracting frame image features and audio features from the video data, includes:

[0053] S11. A convolutional neural network with contrastive learning is used to extract video frame images into 256-dimensional visual feature vectors that can fully represent image features.

[0054] S12. An audio fingerprint generation network is used to extract audio features in seconds, with one 128-dimensional audio feature vector extracted per second.

[0055] In this embodiment, two feature information can be extracted from a video file, with lengths corresponding to video features of 256 seconds and audio features of 128 seconds. The feature extraction networks mentioned above are all mature feature extraction networks currently in use, and are only used in this preprocessing step, so they will not be described in detail here.

[0056] In a specific implementation, as a preferred embodiment of the present invention, in step S2, as follows: Figure 2 As shown, self-attention enhancement and cross-attention enhancement are applied to the extracted frame image features and audio features to obtain the long-range dependency information hidden in the features, including:

[0057] S21. Add fixed sine and cosine time codes to the extracted initial features to make them visual and auditory feature codes with time information;

[0058] S22. Construct a Transformer model and use the model's self-cross-attention module to enhance the self-cross-attention of visual and auditory feature encodings;

[0059] S23. After the cross-attention enhancement is completed, cross-attention enhancement is performed between the query video and the reference video pair.

[0060] In this embodiment, taking video frame features as an example, the frame-level feature pairs are represented as frame-level features of the form F^Q = {f_m^Q}_(m=1)^q from the query video V^Q and frame-level features of the form F^R = {f_n^R}_(n=1)^r from the reference video V^R, where f_m^Q is the m-th frame feature of the query video and f_n^R is the n-th frame feature of the reference video. To further consider the interdependencies and internal dependencies between these features, the Transformer model is particularly suitable for modeling the internal or inter-dependent relationships between F^Q and F^R using an attention mechanism. Since frame-level features do not have the concept of temporal information, fixed sine and cosine time codes need to be added to the initial features. The sum of the initial frame-level features and the time codes also means that the next matching process must consider not only the similarity of features but also their temporal distance.

[0061] In a specific implementation, as a preferred embodiment of the present invention, in step S3, as follows: Figure 3 As shown, multimodal factorization and bilinear pooling are used to fuse the enhanced frame image features and audio features, including:

[0062] S31. Combining the fully connected layer, the element-wise multiplication layer, and the pooling layer in sequence, a multimodal factorization bilinear pooling layer is constructed.

[0063] S32. The enhanced frame image features and audio features are expanded and contracted in a multimodal factorization bilinear pooling layer to obtain robust and expressive fusion features. Among them:

[0064] In the expansion phase, features from different modalities are expanded to a high-dimensional space and then combined through an element-wise multiplication layer. A Dropout layer is added after the element-wise multiplication layer to prevent overfitting.

[0065] During the shrinkage phase, pooling and normalization layers are executed to compress high-dimensional features into compact output features. Due to the introduction of element-wise multiplication, the size of the output neurons changes significantly, and the model cannot converge to a local optimum. Power normalization and L2 normalization layers are introduced at the end of the network.

[0066] In a specific implementation, as a preferred embodiment of the present invention, in step S4, the length and width of the similarity matrix are the number of seconds of the query video and the copied video, respectively. The similarity matrix contains specific infringement patterns between copied fragment pairs. The 4D copied fragment pairs with start and end timestamps are formulated as bounding boxes with upper-left and lower-right coordinates in the similarity mapping.

[0067] In a specific implementation, as a preferred embodiment of the present invention, in step S5, as follows: Figure 4 As shown, the training loss is constructed using the difference in similarity matrix between the detected video copy regions and the ground truth labels, and the neural network is learned in a data-driven manner, including:

[0068] S51. Represent the temporal boundaries of the truth-marked copy fragments as bounding box coordinates on the similarity matrix;

[0069] S52. The task of locating all duplicated fragment pairs is regarded as an object detection task on the similarity matrix;

[0070] S53. Based on the YOLO object detection network, target detection is performed on the bounding boxes that represent the copy relationship on the similarity matrix generated by querying and copying videos, and specific similarity patterns between video pairs on the similarity map are detected.

[0071] S54. Calculate the regression loss between the detected bounding boxes with upper-left and lower-right coordinates and the ground truth bounding boxes, and input the regression loss as the training loss into the network model for training.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video copy detection method based on attention-enhanced multimodal features, characterized in that, include: Extract frame image features and audio features from video data; Self-attention enhancement and cross-attention enhancement are applied to the extracted frame image features and audio features to obtain the long-range dependency information hidden in the features; Multimodal factor decomposition is used to fuse the enhanced frame image features and audio features using a bilinear pooling layer; A similarity matrix between a query video and a copied video is generated using a local feature matching network. The length and width of the similarity matrix are the number of seconds in the query video and the copied video, respectively. The similarity matrix contains specific infringement patterns between copied fragment pairs. A 4D copied fragment pair with start and end timestamps is formulated as a bounding box with upper-left and lower-right coordinates in the similarity map. The training loss is constructed using the difference in similarity matrix between the detected video copy regions and the ground truth labels, and the neural network is learned in a data-driven manner, including: The temporal boundaries of the truth-marked copy fragments are represented as bounding box coordinates on the similarity matrix; The task of locating all duplicated fragment pairs is considered as an object detection task on the similarity matrix; Based on the YOLO object detection network, target detection is performed on bounding boxes that represent copy relationships on the similarity matrices generated from querying and copying videos, and specific similarity patterns between video pairs on the similarity map are detected. The regression loss between the detected bounding boxes with upper-left and lower-right coordinates and the ground truth bounding boxes is calculated, and the regression loss is used as the training loss to train the network model.

2. The video copy detection method based on attention-enhanced multimodal features according to claim 1, characterized in that, The extraction of frame image features and audio features from video data includes: A convolutional neural network with contrastive learning is used to extract video frame images into 256-dimensional visual feature vectors that can fully represent the image features; An audio fingerprint generation network is used to extract audio features in seconds, with one 128-dimensional audio feature vector extracted per second.

3. The video copy detection method based on attention-enhanced multimodal features according to claim 1, characterized in that, The step of performing self-attention enhancement and cross-attention enhancement on the extracted frame image features and audio features to obtain the long-range dependency information hidden in the features includes: Fixed sine and cosine time codes are added to the extracted initial features to make them visual and auditory feature codes with time information; Construct a Transformer model and use the model's self-cross-attention module to enhance the self-cross-attention of visual and auditory feature encodings; After the cross-attention enhancement is completed, cross-attention enhancement is performed again between the query video and the reference video pair.

4. The video copy detection method based on attention-enhanced multimodal features according to claim 1, characterized in that, The feature fusion of enhanced frame image features and audio features using multimodal factorization bilinear pooling includes: By sequentially combining fully connected layers, element-wise multiplication layers, and pooling layers, a multimodal factorization bilinear pooling layer is constructed. The enhanced frame image features and audio features are expanded and contracted in the multimodal factorization bilinear pooling layer to obtain robust and expressive fusion features.

5. The video copy detection method based on attention-enhanced multimodal features according to claim 4, characterized in that, The enhanced frame image features and audio features are expanded and contracted in a multimodal factorization bilinear pooling layer to obtain robust and expressive fusion features, including: In the expansion phase, features from different modalities are expanded to a high-dimensional space and then combined through an element-wise multiplication layer. A Dropout layer is added after the element-wise multiplication layer to prevent overfitting. During the contraction phase, pooling and normalization layers compress high-dimensional features into compact output features. Due to the introduction of element-wise multiplication, the size of the output neurons changes significantly, and the model fails to converge to a local optimum. A power normalization layer is then introduced at the end of the network. Normalized layer.

Images