Video Editing Detection Method Based on Frequency Domain Aware Spatiotemporal Self-Attention Transformer Network

Abstract

The present application proposes a video editing detection method based on a frequency domain-aware spatiotemporal self-attention transformation network, including: obtaining input video frames; performing image serialization on the input video frames to generate complete image serialization codes; constructing a spatiotemporal self-attention transformation network; encoding the complete image serialization codes, and downsampling the encoding results to generate encoded image representations; progressively upsampling the encoded image representations to obtain a first feature map combination; performing frequency domain feature perception on the input video frames to obtain frequency domain-aware feature maps, and then downsampling the frequency domain-aware feature maps a preset number of times, and forming all feature maps generated by the downsampling process into a second feature map combination; cascading and fusing the feature maps of the same size in the first feature map combination and the second feature map combination to obtain the final prediction result. The present application improves the robustness and accuracy of the scheme and reduces the impact of class imbalance in the data set.

Description

Technical Field

[0001] The present application relates to the field of computer vision and machine learning technology, and in particular to a video editing detection method based on a frequency-domain-aware spatiotemporal self-attention transformation network and a non-temporary computer-readable storage medium. Background Art

[0002] Video editing technology has attracted more and more attention in recent years. Video editing technology mainly involves repairing missing image areas in video sequences or restoring damaged pixels. Video editing technology has been widely used in multiple applications, such as video content restoration and virtual reality. With the recent progress of deep learning technology, various studies have made amazing progress in video editing technology, which can edit any area of ​​the video, such as removing the object area that may be key evidence. However, edited videos are becoming more and more difficult to distinguish from real videos, even for human eyes. Moreover, the continuous advancement and rapid development of video editing technology will also induce malicious attackers to forge false videos and publish some fake news to mislead public opinion. In addition, the abuse of video editing technology may pose a potential threat and lead to legal and social problems. Therefore, it is urgent to propose an effective video editing detection method to try to detect whether the presented video is original and real or edited and false.

[0003] There are various studies on video editing techniques, which can be divided into two categories: region association-based methods and data-driven methods. However, both of the above methods have an important problem - video editing techniques directly refer to or even copy corresponding pixels from similar regions or frames, or learn related distributions from similar scenes. Therefore, these methods inevitably leave clues and artifacts, such as inconsistencies between regional pixels, sharp changes in regional edges, and regional blurring caused by the inability to learn complete distributions. Therefore, recent studies have begun to focus on video editing detection, but most of the existing video editing detection methods are based on single-frame detection methods that only input one image. In addition, Peng Zhou first proposed an LSTM-based framework that combines RGB images and ELA information to extract spatial and temporal features for video editing detection, but this method is still significantly away from practical use.

[0004] The self-attention transformer network structure has now been successfully applied to natural language processing and machine translation tasks. Since the core of the self-attention transformer network is the self-attention mechanism, researchers prefer to use the self-attention transformer network to model long-range dependencies. Recently, the self-attention transformer network structure has begun to be applied to a series of computer vision tasks and has brought great breakthroughs. DETR uses the encoder-decoder architecture to build a self-attention transformer network structure for object detection. In addition, IPT has developed a new pre-trained self-attention transformer network structure as a model for low-level computer vision tasks. At the same time, the self-attention transformer network structure has also attracted more and more attention in video processing. VisTR regards the video instance segmentation task as a direct end-to-end sequence decoding problem and uses the self-attention transformer network structure to complete the sequence instance segmentation. However, so far, there is no application of the self-attention transformer network structure in video inpainting detection. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first purpose of this application is to propose a video editing detection method based on a frequency-domain-aware spatiotemporal self-attention transformation network, which solves the problem of inconsistent prediction results in the time dimension in existing video editing detection technologies. By proposing a frequency-domain-aware spatiotemporal self-attention transformation network, features are extracted from three dimensions: time domain, frequency domain, and spatial domain to improve the temporal consistency and accuracy of the prediction results.

[0007] This application introduces the results of the self-attention transformation network into video editing detection technology for the first time. The two-dimensional network structure of time and space enables the simultaneous extraction of temporal information and spatial information. At the same time, the overall network structure is assisted by extracting frequency domain perception information, which improves the robustness and accuracy of the scheme, and optimizes the overall framework under the guidance of the hybrid loss function, which is directly related to the evaluation index and reduces the impact of class imbalance in the data set.

[0008] A second object of the present application is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above-mentioned objectives, the first aspect of the present application proposes a video editing detection method based on a frequency-domain-aware spatiotemporal self-attention transformation network, including: obtaining an input video frame; performing image serialization on the input video frame to generate a complete image serialization code; constructing a spatiotemporal self-attention transformation network, wherein the spatiotemporal self-attention transformation network includes an encoder and a decoder; inputting the complete image serialization code into the encoder for encoding, and downsampling the encoding result to generate an encoded image representation; using the decoder to progressively upsample the encoded image representation to obtain a first feature map combination; performing frequency-domain feature perception on the input video frame to obtain a frequency-domain-aware feature map, and then downsampling the frequency-domain-aware feature map a preset number of times, and forming a second feature map combination from all feature maps generated by the downsampling process, wherein the preset number of times is the same as the progressive upsampling number; cascading and fusing feature maps of the same size in the first feature map combination and the second feature map combination to obtain a final prediction result.

[0010] Optionally, in one embodiment of the present application, performing image serialization on the input video frame includes the following steps:

[0011] Each input image in the input video frame is cut into small blocks, and then the cut small blocks are stretched into vectors, wherein the small blocks are squares of a preset size;

[0012] Concatenate the vectors to get the initialization sequence encoding of the image;

[0013] The initialization sequence code is subjected to time domain linear mapping and space domain linear mapping, and the time domain linear mapping results and the space domain linear mapping results are directly added together to form a complete image serialization code.

[0014] Optionally, in one embodiment of the present application, the encoder is composed of a plurality of stacked self-attention transformation network layers, and each self-attention transformation network layer includes two core modules: a multi-head self-attention module and a multi-layer perceptron.

[0015] Optionally, in one embodiment of the present application, the encoding process of the encoder is expressed as:

[0016]

[0017]

[0018] Among them, LN(·) represents the layer regularization operation, is the output variable of the MSA layer, z l-1 and z l Represents the encoded image representation of the input and output.

[0019] Optionally, in one embodiment of the present application, downsampling is performed by extracting feature maps using a convolutional neural network.

[0020] Optionally, in one embodiment of the present application, progressive upsampling of the encoded image representation using a decoder comprises the following steps:

[0021] Reassemble the encoded image representation output by the encoder into a square shape in sequence;

[0022] The combined image is upsampled a fixed number of times using a convolutional neural network so that the final feature map has the same size as the original image, and all feature maps generated by the upsampling process are combined into a first feature map combination.

[0023] Optionally, in one embodiment of the present application, frequency domain feature perception is performed on the input video frame, comprising the following steps:

[0024] Perform discrete cosine transform on the input video frame to convert the image in the input video frame into a spectrum graph;

[0025] According to the energy averaging principle, the spectrum graph is divided into three parts: low frequency part, medium frequency part and high frequency part;

[0026] Perform inverse discrete cosine transform on the three parts of the spectrum graph to obtain the low-frequency part, medium-frequency part, and high-frequency part corresponding to the input image.

[0027] Optionally, in one embodiment of the present application, the method further includes training the spatiotemporal self-attention transformation network to generate a predictor, and the composite loss function used in the training process is expressed as:

[0028]

[0029] Among them, λ1 and λ2 are hyperparameters used to control the weights of Focal loss function and mIoU loss function.

[0030] The Focal loss function is expressed as:

[0031]

[0032] in, represents the predicted pixel value, y represents the corresponding binary edit region label, α and γ represent the hyperparameters in the Focal loss function,

[0033] The mIoU loss function is expressed as:

[0034]

[0035] in, Represents the predicted pixel value, y represents the corresponding binary editing area label, and ∈ is a very small number to ensure that there is no division by 0 error.

[0036] In order to achieve the above-mentioned objectives, the second aspect embodiment of the present application proposes a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor, a video editing detection method based on a frequency-domain-aware spatiotemporal self-attention transformation network can be executed.

[0037] The video editing detection method based on the frequency-domain-aware spatiotemporal self-attention transformation network and the non-temporary computer-readable storage medium of the embodiments of the present application solve the problem of inconsistent prediction results in the time dimension in the existing video editing detection technology, and improve the time consistency and accuracy of the prediction results by proposing a frequency-domain-aware spatiotemporal self-attention transformation network to extract features from three dimensions: time domain, frequency domain, and spatial domain.

[0038] This application introduces the results of the self-attention transformation network into video editing detection technology for the first time. The two-dimensional network structure of time and space enables the simultaneous extraction of temporal information and spatial information. At the same time, the overall network structure is assisted by extracting frequency domain perception information, which improves the robustness and accuracy of the scheme, and optimizes the overall framework under the guidance of the hybrid loss function, which is directly related to the evaluation index and reduces the impact of class imbalance in the data set.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A flowchart of a video editing detection method based on a frequency domain-aware spatiotemporal self-attention transformation network provided in Example 1 of the present application;

[0042] Figure 2 Another flowchart of the video editing detection method based on the frequency domain-aware spatiotemporal self-attention transformation network according to an embodiment of the present application;

[0043] Figure 3 Schematic diagram of extracting frequency domain perception features in a video editing detection method based on a frequency domain perception spatiotemporal self-attention transformation network in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0045] The following describes a video editing detection method based on a frequency-domain-aware spatiotemporal self-attention transformation network according to an embodiment of the present application with reference to the accompanying drawings.

[0046] Figure 1 A flowchart of a video editing detection method based on a frequency-domain-aware spatiotemporal self-attention transformation network provided in Example 1 of the present application.

[0047] like Figure 1 As shown, the video editing detection method based on frequency domain perception spatiotemporal self-attention transformation network includes the following steps:

[0048] Step 101, obtaining an input video frame;

[0049] Step 102, performing image serialization on the input video frame to generate a complete image serialization code;

[0050] Step 103, constructing a spatiotemporal self-attention transformation network, wherein the spatiotemporal self-attention transformation network includes an encoder and a decoder;

[0051] Step 104, inputting the complete image serialization code into the encoder for encoding, and downsampling the encoding result to generate a coded image representation;

[0052] Step 105, using a decoder to progressively upsample the encoded image representation to obtain a first feature map combination;

[0053] Step 106, performing frequency domain feature perception on the input video frame to obtain a frequency domain perception feature map, then downsampling the frequency domain perception feature map a preset number of times, and forming a second feature map combination from all feature maps generated by the downsampling process, wherein the preset number of times is the same as the number of progressive upsampling times;

[0054] Step 107, cascading and fusing the feature maps of the same size in the first feature map combination and the second feature map combination to obtain a final prediction result.

[0055] The video editing detection method based on the frequency domain perception spatiotemporal self-attention transformation network of the embodiment of the present application is obtained by obtaining an input video frame; performing image serialization on the input video frame to generate a complete image serialization code; constructing a spatiotemporal self-attention transformation network, wherein the spatiotemporal self-attention transformation network includes an encoder and a decoder; inputting the complete image serialization code into the encoder for encoding, and downsampling the encoding result to generate a coded image representation; using the decoder to progressively upsample the coded image representation to obtain a first feature map combination; performing frequency domain feature perception on the input video frame to obtain a frequency domain perception feature map, and then downsampling the frequency domain perception feature map a preset number of times, and forming all feature maps generated by the downsampling process into a second feature map combination, wherein the preset number of times is the same as the progressive upsampling number of times; cascading and fusing the feature maps of the same size in the first feature map combination and the second feature map combination to obtain the final prediction result. Thus, the problem of inconsistent prediction results in the time dimension in the existing video editing detection technology can be solved, and the time consistency and accuracy of the prediction results can be improved by proposing a frequency domain perception spatiotemporal self-attention transformation network to extract features from the three dimensions of time domain, frequency domain and space domain.

[0056] This application introduces the results of the self-attention transformation network into video editing detection technology for the first time. The two-dimensional network structure of time and space enables the simultaneous extraction of temporal information and spatial information. At the same time, the overall network structure is assisted by extracting frequency domain perception information, which improves the robustness and accuracy of the scheme, and optimizes the overall framework under the guidance of the hybrid loss function, which is directly related to the evaluation index and reduces the impact of class imbalance in the data set.

[0057] In fact, one of the most important things in video editing detection is to find the associations between video frames and the differences between regions. Existing methods usually use attention models and memory mechanisms, but the rigid combination of the two will lead to inconsistent prediction results in the time dimension. This application constructs a self-attention transformation network including an encoder and a decoder to capture spatial and temporal artifacts using a multi-head self-attention mechanism. In addition, frequency-aware features are used as auxiliary information to supervise the training in order to regularize the operation of upsampling prediction masks to improve the generalization of the algorithm. Afterwards, the overall framework is optimized under the guidance of a hybrid loss function that can reduce the impact of class imbalance in the data set. After the network training is completed, a generalized predictor is obtained that can predict the authenticity of a video given a video.

[0058] Further, in the embodiment of the present application, in order to meet the input format of the general self-attention transformation network, the video frame needs to be serialized first, and the input video frame is serialized into images, including the following steps:

[0059] Each input image in the input video frame is cut into small blocks, and then the cut small blocks are stretched into vectors, wherein the small blocks are squares of a preset size;

[0060] Concatenate the vectors to get the initialization sequence encoding of the image;

[0061] The initialization sequence code is subjected to time domain linear mapping and space domain linear mapping, and the time domain linear mapping results and the space domain linear mapping results are directly added together to form a complete image serialization code.

[0062] Image serialization encoding:

[0063]

[0064] Among them, z0 represents the complete image serialization encoding, is the encoding of the i-th block area of ​​the t-th frame image, and E and E0 represent the linear mapping in the time domain and the linear mapping in the spatial domain respectively. pos represents a learnable positional encoding.

[0065] Furthermore, in an embodiment of the present application, the encoder includes a plurality of stacked self-attention transformation network layers, and each self-attention transformation network layer includes two core modules: a multi-head self-attention module and a multi-layer perceptron.

[0066] The encoder consists of a series of stacked self-attention transformation network layers (set to 8 layers in the experiment), and each layer of self-attention transformation network layer contains two core modules: a multi-head self-attention module and a multi-layer perceptron.

[0067] Further, in the embodiment of the present application, the encoding process of the encoder is expressed as:

[0068]

[0069]

[0070] Among them, LN(·) represents the layer regularization operation, is the output variable of the MSA layer, z l-1 and z l Represents the encoded image representation of the input and output.

[0071] In the self-attention transformation network layer of the present application, an L-layer multi-head self-attention module and a multi-layer perceptron are used.

[0072] Furthermore, in the embodiment of the present application, downsampling is accomplished by extracting feature maps using a convolutional neural network.

[0073] Using convolutional neural networks, feature maps are gradually extracted. As the convolution operation proceeds, the size of the feature map will gradually decrease exponentially, such as 1 / 2, 1 / 4, 1 / 8, which is called a downsampling operation.

[0074] Further, in an embodiment of the present application, progressive upsampling of the encoded image representation using a decoder comprises the following steps:

[0075] Reassemble the encoded image representation output by the encoder into a square shape in sequence;

[0076] The combined image is upsampled a fixed number of times using a convolutional neural network so that the final feature map has the same size as the original image, and all feature maps generated by the upsampling process are combined into a first feature map combination.

[0077] First, the encoded image representation output by the last encoder is reassembled into a square shape in order, which is 1 / 16 of the size of the input image. Then, the convolutional neural network is used to expand the representation by two times in sequence. After four upsampling operations, a feature map of the same size as the original image can be obtained.

[0078] Furthermore, in an embodiment of the present application, frequency domain feature perception is performed on an input video frame, including the following steps:

[0079] Perform discrete cosine transform on the input video frame to convert the image in the input video frame into a spectrum graph;

[0080] According to the energy averaging principle, the spectrum graph is divided into three parts: low frequency part, medium frequency part and high frequency part;

[0081] Perform inverse discrete cosine transform on the three parts of the spectrum graph to obtain the low-frequency part, medium-frequency part, and high-frequency part corresponding to the input image.

[0082] Performing inverse discrete cosine transform (IDCT) on the three parts in turn can obtain the low-frequency part, medium-frequency part and high-frequency part corresponding to the input image:

[0083]

[0084] Among them, I i is the low-frequency part, medium-frequency part or high-frequency part of the image after frequency domain transformation, I is the input image, is the discrete cosine transform, ⊙ is the element-by-element operation, f i is a low frequency, medium frequency or high frequency filter, and is the inverse discrete cosine transform.

[0085] After obtaining the parts of the image in different frequency bands, the components of different frequency bands are cascaded to form the final frequency domain perception features. At this time, the frequency domain perception features have the same size as the original image. The frequency domain perception features can be encoded using a convolutional neural network, and the feature map is reduced by two times through downsampling operations in turn. Finally, after four downsampling operations, the original image can be obtained. Figure 1 / 16 size feature map, so that different size features can be constructed in the spatiotemporal self-attention transformation network Figure 1 Correspondingly, by cascading and fusing feature maps of the same size, the frequency domain perception features can be used to regularize the decoding process of the decoder of the self-attention transformer network, so that the final prediction result has higher accuracy and robustness.

[0086] Furthermore, in the embodiment of the present application, the spatiotemporal self-attention transformation network is trained to generate a predictor, and the composite loss function used in the training process is expressed as:

[0087]

[0088] Among them, λ1 and λ2 are hyperparameters used to control the weights of Focal loss function and mIoU loss function.

[0089] The Focal loss function is expressed as:

[0090]

[0091] in, represents the predicted pixel value, y represents the corresponding binary edit region label, α and γ represent the hyperparameters in the Focal loss function,

[0092] The mIoU loss function is expressed as:

[0093]

[0094] in, Represents the predicted pixel value, y represents the corresponding binary editing area label, and ∈ is a very small number to ensure that there is no division by 0 error.

[0095] In order to further improve the accuracy and generalization of the model, a hybrid loss function is proposed to guide the optimization process of the overall framework. The Focal loss function is used to solve the problem of the imbalance of the number of positive and negative samples (that is, edited areas and unedited areas) in the data, where the two learnable hyperparameters α and γ can be automatically adjusted according to the number of positive and negative samples. On the other hand, in order to improve the accuracy of the prediction results, the evaluation indicator mIoU (meanIntersection of Union) is used as the second loss function.

[0096] After the network training is completed, a predictor with high accuracy and generalization is obtained, which can predict the authenticity and edited areas of a video based on a given video.

[0097] For a given video, this application models long-range dependencies through a spatiotemporal self-attention transform network to learn temporal information across multiple frames, and uses a self-attention mechanism to explore the spatial features between patches. At the same time, frequency-domain perceptual features are extracted from video frames as auxiliary information to supervise the prediction process, so that the operation of upsampling prediction masks is regularized to improve the generalization of the algorithm. Then, the corresponding prediction results are generated through the cooperation of the decoder of the convolutional neural network and the spatiotemporal self-attention transform network.

[0098] Figure 2 This is another flowchart of the video editing detection method based on the frequency-domain-aware spatiotemporal self-attention transformation network according to an embodiment of the present application.

[0099] like Figure 2 As shown in the figure, the video editing detection method based on frequency-domain-aware spatiotemporal self-attention transform network, when a series of frames are input, first the input image is serialized, and then two layers of linear mapping layers are used to generate corresponding codes from the spatiotemporal perspective. The codes are then input into the encoder of the self-attention transform network. The encoder is composed of 8 stacked self-attention transform network layers, and the structure of each self-attention transform network layer is as follows: Figure 2 As shown on the left, it contains two core modules: multi-head self-attention module and multi-layer perceptron, and the input of each core module will first perform layer regularization operation. At the same time, the extracted frequency domain perception feature map is sequentially down-sampled through the convolutional neural network, and combined with the feature map generated by the encoder up-sampling of the self-attention transformation network according to the size correspondence to generate the final prediction result map.

[0100] Figure 3 Schematic diagram of extracting frequency domain perception features in a video editing detection method based on a frequency domain perception spatiotemporal self-attention transformation network in an embodiment of the present application.

[0101] like Figure 3 As shown in the figure, the three rows on the left are the original video frame, the edited video frame, and the label map showing the edited part. After the edited video frame is transformed by DCT, the spectrum map is divided into different parts of three frequency bands through three filters, and then the required frequency domain perceptual feature map is obtained by IDCT transformation. Figure 3 ,It can be found that the feature maps of different frequency bands can effectively reveal the differences in the edited regions.

[0102] In order to implement the above-mentioned embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video editing detection method based on the frequency-domain-aware spatiotemporal self-attention transformation network of the above-mentioned embodiment is implemented.

[0103] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0105] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0106] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0107] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0108] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0110] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A video editing detection method based on frequency-domain-aware spatiotemporal self-attention transform network, characterized in that: The following steps are involved: Get the input video frame; Performing image serialization on the input video frame to generate a complete image serialization code, wherein the initialization sequence code is subjected to time domain linear mapping and space domain linear mapping, and the time domain linear mapping result and the space domain linear mapping result are directly added to form a complete image serialization code; Construct a spatiotemporal self-attention transformation network, wherein the spatiotemporal self-attention transformation network includes an encoder and a decoder, wherein the encoding process of the encoder is expressed as: Among them, LN(·) represents the layer regularization operation, is the output variable of the MSA layer, z l-1 and z l An encoded image representation representing the input and output; Inputting the complete image serialization code into the encoder for encoding, and downsampling the encoding result to generate an encoded image representation; progressively upsampling the encoded image representation using the decoder to obtain a first feature map combination; Perform frequency domain feature perception on the input video frame to obtain a frequency domain perception feature map, then downsample the frequency domain perception feature map for a preset number of times, and form a second feature map combination with all feature maps generated by the downsampling process, wherein the preset number of times is the same as the number of progressive upsampling, wherein discrete cosine transform is performed on the input video frame, and the image in the input video frame is converted into a spectrum map, and the spectrum map is divided into three parts according to the energy averaging principle: a low-frequency part, a medium-frequency part, and a high-frequency part, and an inverse discrete cosine transform is performed on the three parts of the spectrum map to obtain a low-frequency part, a medium-frequency part, and a high-frequency part corresponding to the input image, and components of different frequency bands are cascaded to form a final frequency domain perception feature; Feature maps of the same size in the first feature map combination and the second feature map combination are cascaded and fused to obtain a final prediction result.

2. The method according to claim 1, characterized in that The image serialization of the input video frame comprises the following steps: Cut each input image frame in the input video frame into small blocks, and then stretch the cut small blocks into vectors, wherein the small blocks are squares of a preset size; The vectors are concatenated to obtain the initialization sequence encoding of the image.

3. The method according to claim 1, characterized in that The encoder is composed of a stack of multiple self-attention transformation network layers, and each self-attention transformation network layer contains two core modules: a multi-head self-attention module and a multi-layer perceptron.

4. The method according to claim 1, characterized in that The downsampling is accomplished by extracting feature maps using a convolutional neural network.

5. The method according to claim 1, characterized in that The step of progressively upsampling the encoded image representation using the decoder comprises the following steps: Reassemble the encoded image representation output by the encoder into a square shape in sequence; The combined image is upsampled a fixed number of times using a convolutional neural network so that the feature map finally generated has the same size as the original image, and all feature maps generated by the upsampling process are combined into the first feature map combination.

6. The method according to claim 1, characterized in that It also includes training the spatiotemporal self-attention transformer network to generate a predictor, and the composite loss function used in the training process is expressed as: Among them, λ1 and λ2 are hyperparameters used to control the weights of Focal loss function and mIoU loss function. The Focal loss function is expressed as: in, represents the predicted pixel value, y represents the corresponding binary edit region label, α and γ represent the hyperparameters in the Focal loss function, The mIoU loss function is expressed as: in, Represents the predicted pixel value, y represents the corresponding binary editing area label, and ∈ is a very small number to ensure that there is no division by 0 error.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images