Video detection method and device based on implicit representation of generative model

Abstract

The invention discloses a video detection method and device based on implicit representation of a generative model, and the method comprises the steps: obtaining a to-be-detected video, carrying out the implicit feature extraction of the to-be-detected video through a video generation model, and obtaining the implicit representation; performing feature extraction on the to-be-detected video through the backbone network to obtain spatial-temporal features; performing feature fusion on the implicit representation and the spatial-temporal feature through a residual cross attention mechanism to obtain a fused feature; and outputting a detection result of the to-be-detected video through a classifier based on the fusion feature. According to the method, the inherent statistical characteristics of the video generation model and the spatial-temporal characteristics extracted by the backbone network are combined, so that the robustness for various disturbances is enhanced, and the generalization is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a video detection method and device based on implicit representation of a generative model. Background Art

[0002] With the continuous advancement of generative modeling technology, AI-generated content (AIGC) has rapidly expanded and continues to maintain its popularity. However, video generation algorithms can lead to some significant issues. For example, in the context of self-media, it is difficult to distinguish authenticity from fake videos, which may raise public concerns about security and identity privacy. Therefore, it is necessary to develop efficient, robust, and generalizable methods for detecting AI-generated videos to effectively mitigate the risks posed by deepfake technology.

[0003] In existing technologies, the generalization ability of the model can be improved by analyzing the inconsistencies in the temporal and spatial dimensions of the video. However, the above methods still have the problem of insufficient generalization when facing unknown generative models. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, one purpose of the present invention is to propose a video detection method based on implicit representation of a generative model. The method uses a video generation model to calculate the implicit representation of the content before and after reconstruction, and extracts the spatiotemporal features of the video to be detected through a backbone network. Then, the implicit representation and spatiotemporal features are integrated through a residual cross-attention mechanism to determine the detection result of the video. By combining the intrinsic statistical characteristics of the video generation model with the spatiotemporal features extracted by the backbone network, the robustness to various disturbances is enhanced and the generalization is improved.

[0006] Another object of the present invention is to provide a video detection device based on implicit representation of a generative model.

[0007] To achieve the above objectives, an embodiment of the present invention provides a video detection method based on implicit representation of a generative model, comprising:

[0008] Obtaining a video to be detected, and performing implicit feature extraction on the video to be detected using a video generation model to obtain an implicit representation;

[0009] Extracting features from the video to be detected through a backbone network to obtain spatiotemporal features;

[0010] The implicit representation and the spatiotemporal feature are fused through a residual cross attention mechanism to obtain a fused feature;

[0011] Based on the fusion features, a detection result of the video to be detected is outputted through a classifier.

[0012] The video detection method based on implicit representation of a generative model according to an embodiment of the present invention may also have the following additional technical features:

[0013] Furthermore, the video generation model includes a three-dimensional variational autoencoder, a diffusion module, a denoising module and a feature enhancement module;

[0014] The implicit feature extraction of the video to be detected by using a video generation model to obtain an implicit representation includes:

[0015] Mapping the video to be detected to an implicit variable space through the three-dimensional variational autoencoder to obtain a first implicit representation;

[0016] performing diffusion processing on the first implicit representation by the diffusion module to obtain a second implicit representation;

[0017] performing denoising processing on the second implicit representation by the denoising module to obtain a third implicit representation;

[0018] Performing feature enhancement processing on the first implicit representation and the third implicit representation respectively by the feature enhancement module to obtain enhanced corresponding first features and second features;

[0019] The first feature and the second feature are determined as implicit representations.

[0020] Furthermore, the backbone network includes a target backbone network and a feature alignment module;

[0021] The feature extraction of the video to be detected by the backbone network to obtain spatiotemporal features includes:

[0022] Extracting features from the video to be detected through the target backbone network to obtain a third feature;

[0023] The feature alignment module performs feature alignment on the third feature to obtain a spatiotemporal feature.

[0024] Furthermore, the feature fusion of the implicit representation and the spatiotemporal features is performed through the residual cross attention mechanism to obtain a fused feature, including: the feature fusion of the implicit representation and the spatiotemporal features is performed through multiple Transformer blocks to obtain a fused feature.

[0025] Furthermore, the Transformer block includes multiple layers of residual cross attention layers and multi-layer perceptron layers;

[0026] The step of fusing the implicit representation and the spatiotemporal features through multiple Transformer blocks to obtain fused features includes:

[0027] Linearly map the implicit representation and the output vector of the previous Transformer block to obtain the corresponding fourth feature;

[0028] Performing feature processing on the fourth feature and the spatiotemporal feature through a multi-layer residual cross attention layer and a multi-layer perceptron layer to obtain an output vector of the current Transformer block;

[0029] Repeat the above steps and determine the output vector of the last Transformer block as the fused feature.

[0030] Furthermore, the fourth feature and the spatiotemporal feature are subjected to feature processing by a multi-layer residual cross attention layer and a multi-layer perceptron layer to obtain an output vector of the current Transformer block, including:

[0031] Inputting the fourth feature and the spatiotemporal feature into each residual cross attention layer respectively to obtain a corresponding first output vector;

[0032] Performing a first process on the plurality of first output vectors to obtain a second output vector;

[0033] performing a second processing on the second output vector by a layer normalization operator to obtain a third output vector;

[0034] Performing a third process on the third output vector and an output vector of a previous Transformer block to obtain a fourth output vector;

[0035] Inputting the fourth output vector into the multilayer perceptron layer to obtain a fifth output vector;

[0036] performing a second processing on the fifth output vector by the layer normalization operator to obtain a sixth output vector;

[0037] Perform a third process on the sixth output vector and the fourth output vector to obtain an output vector of the current Transformer block.

[0038] To achieve the above-mentioned object, another embodiment of the present invention provides a video detection device based on implicit representation of a generative model, the device comprising:

[0039] An acquisition module is used to acquire a video to be detected and perform implicit feature extraction on the video to be detected through a video generation model to obtain an implicit representation;

[0040] A feature extraction module is used to extract features of the video to be detected through a backbone network to obtain spatiotemporal features;

[0041] A feature fusion module, configured to fuse the implicit representation and the spatiotemporal features through a residual cross-attention mechanism to obtain a fused feature;

[0042] The detection module is used to output the detection result of the video to be detected through a classifier based on the fusion feature.

[0043] The video detection method and device based on implicit representation of the generative model proposed in the present invention use the video generation model to calculate the implicit representation of the content before and after reconstruction, and extract the spatiotemporal features of the video to be detected through the backbone network. Then, the implicit representation and spatiotemporal features are integrated through the residual cross-attention mechanism to determine the detection result of the video. By combining the intrinsic statistical characteristics of the video generation model with the spatiotemporal features extracted by the backbone network, the robustness to various disturbances is enhanced and the generalization is improved.

[0044] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 Flowchart of a video detection method based on implicit representation of a generative model according to one embodiment of the present invention;

[0047] Figure 2 Schematic diagram of a video detection method based on implicit representation of a generative model according to an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a comparison result according to an embodiment of the present invention;

[0049] Figure 4 2 is a schematic structural diagram of a video detection device based on implicit representation of a generative model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0051] Currently, most generated video detection methods treat the detection of AI-generated videos as a simple discrimination problem. However, these methods are highly vulnerable to perturbed samples, which can cause the classifier to produce incorrect predictions. To address this issue, this paper leverages the powerful sample space modeling capabilities of generative models to assist in the detection task of generated videos.

[0052] Based on the above description, a video detection method and apparatus based on implicit representation of a generative model proposed in accordance with an embodiment of the present invention will be described with reference to the accompanying drawings.

[0053] First, a video detection method based on implicit representation of a generative model proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0054] Figure 1 Flowchart of a video detection method based on implicit representation of a generative model according to an embodiment of the present invention.

[0055] like Figure 1 As shown, the video detection method based on implicit representation of the generative model includes the following steps:

[0056] Step S1: Obtain a video to be detected, and perform implicit feature extraction on the video to be detected using a video generation model to obtain an implicit representation;

[0057] In one embodiment of the present invention, the video to be detected is obtained After that, the implicit feature extraction of the video to be detected can be performed through the video generation model to obtain the implicit representation. Where T0 is the number of frames of the video to be detected, H0 is the height of each frame image, and W0 is the width of each frame image.

[0058] In one embodiment of the present invention, the above-mentioned video generation model is trained, and the video generation model may include a three-dimensional variational autoencoder, a diffusion module, a denoising module and a feature enhancement module.

[0059] Furthermore, in one embodiment of the present invention, the method for extracting implicit features from a video to be detected by using a video generation model to obtain an implicit representation may include the following steps:

[0060] Step S11, mapping the video to be detected to an implicit variable space through a three-dimensional variational autoencoder to obtain a first implicit representation;

[0061] Step S12, performing diffusion processing on the first implicit representation through a diffusion module to obtain a second implicit representation;

[0062] Step S13, performing denoising processing on the second implicit representation by a denoising module to obtain a third implicit representation;

[0063] Step S14, performing feature enhancement processing on the first implicit representation and the third implicit representation respectively by a feature enhancement module to obtain the enhanced corresponding first feature and second feature;

[0064] Step S15: Determine the first feature and the second feature as implicit representations.

[0065] In one embodiment of the present invention, the three-dimensional variational autoencoder (3D VAE) can Mapped to the implicit variable space, we get the first implicit representation Z0∈R T×H×W×C Where T represents the length of the time dimension of the implicit representation, H and W correspond to the height and width dimensions of the implicit spatial representation, respectively, whose values ​​are determined by the downsampling rate of the encoder, and C is the number of implicit representation channels.

[0066] Furthermore, in one embodiment of the present invention, after obtaining the first implicit representation through the above steps, the diffusion module can perform a diffusion process on the first implicit representation in the implicit variable space. Specifically, in one embodiment of the present invention, the diffusion module can add noise through the diffusion noise formula to obtain the second implicit representation Z T ∈R T×H×W×C , where the diffusion noise formula is:

[0067]

[0068] Among them, x0=Z0 represents the initial implicit representation, x t is the implicit variable after the t-th step diffusion process, is the product factor of the noise scheduling parameter accumulated with the diffusion step number t, I is the unit covariance matrix. And, the diffusion process is gradually increased by The attenuation coefficient is used to transform the original data distribution into a Gaussian distribution.

[0069] Furthermore, in one embodiment of the present invention, the denoising module may perform denoising on the second implicit representation using a denoising formula to obtain a third implicit representation Z∈R T×H×W×C , where the denoising formula is:

[0070]

[0071] Among them, σ t is the noise standard deviation parameter, ∈ θ (x t ,t) is the noise component predicted by the denoising network,∈ t is standard Gaussian noise.

[0072] In one embodiment of the present invention, due to the symmetry of the diffusion module, Z0 and Z have the same dimension. In one embodiment of the present invention, the three-dimensional variational autoencoder and denoising module are CogVideoX models, and the network weights are kept frozen during training.

[0073] Furthermore, in one embodiment of the present invention, the feature enhancement module may be composed of a convolutional layer and a Transformer layer to expand the number of feature channels and enhance the discriminative ability of the representation. The convolutional layer and the Transformer layer may be the same as those in the prior art.

[0074] Furthermore, in one embodiment of the present invention, after the first implicit representation and the third implicit representation are subjected to feature enhancement processing respectively by the feature enhancement module with weight sharing, the enhanced first feature representation and the third implicit representation are output respectively. and the second feature

[0075] Step S2, extracting features from the video to be detected through the backbone network to obtain spatiotemporal features;

[0076] In one embodiment of the present invention, the backbone network is trained, and the backbone network may include a target backbone network and a feature alignment module.

[0077] Furthermore, in one embodiment of the present invention, the method for extracting features from a video to be detected through a backbone network to obtain spatiotemporal features may include the following steps:

[0078] Step S21, extracting features from the video to be detected through the target backbone network to obtain a third feature;

[0079] Step S22: align the third features using a feature alignment module to obtain spatiotemporal features.

[0080] In one embodiment of the present invention, before the backbone network is used to extract features from the video to be detected, the video to be detected I can be divided into N image blocks in the spatial dimension, where S represents the image block size, which can be set as needed. For example, the image block size S is set to 16.

[0081] Furthermore, in one embodiment of the present invention, the video to be detected is divided into image blocks, and after the video to be detected is lemmatized, the backbone network can be used to extract features of the video to be detected to obtain the third feature in In one embodiment of the present invention, the backbone network may be XCLIP in the prior art.

[0082] Furthermore, in one embodiment of the present invention, after obtaining the third feature through the above steps, it is necessary to align the third feature through a feature alignment module to facilitate subsequent cross-attention operations. In one embodiment of the present invention, the feature alignment module may be composed of a fully connected layer for calibrating the number of feature channels.

[0083] And, in one embodiment of the present invention, the third feature is aligned by the feature alignment module to obtain the spatiotemporal feature in,

[0084] Step S3: Fusing the implicit representation and spatiotemporal features through the residual cross attention mechanism to obtain fused features;

[0085] In one embodiment of the present invention, after obtaining the implicit representation and spatiotemporal features through the above steps, the implicit representation and spatiotemporal features can be fused based on the residual cross-attention layer through the residual cross-attention mechanism to obtain fused features.

[0086] In one embodiment of the present invention, the aforementioned feature fusion of the implicit representation and the spatiotemporal features through the residual cross-attention mechanism to obtain the fused features includes: fusing the implicit representation and the spatiotemporal features through multiple Transformer blocks to obtain the fused features. Furthermore, in one embodiment of the present invention, the aforementioned Transformer blocks may include multiple layers of residual cross-attention layers and multi-layer perceptron layers.

[0087] Specifically, in one embodiment of the present invention, the method for fusing implicit representations and spatiotemporal features through multiple Transformer blocks to obtain fused features may include the following steps:

[0088] Step S31, linearly mapping the implicit representation and the output vector of the previous Transformer block to obtain the corresponding fourth feature;

[0089] Step S32, performing feature processing on the fourth feature and the spatiotemporal feature through a multi-layer residual cross attention layer and a multi-layer perceptron layer to obtain an output vector of the current Transformer block;

[0090] Step S33: Repeat the above steps and determine the output vector of the last Transformer block as the fusion feature.

[0091] In one embodiment of the present invention, the above-mentioned multiple Transformer blocks are connected in series, that is, the output of the previous Transformer block is the input of the next Transformer block.

[0092] Furthermore, in one embodiment of the present invention, the method of performing feature processing on the fourth feature and the spatiotemporal feature through multiple layers of residual cross attention layers and multi-layer perceptron layers to obtain the output vector of the current Transformer block may include the following steps:

[0093] Step 1: Input the fourth feature and the spatiotemporal feature into each residual cross attention layer respectively to obtain the corresponding first output vector;

[0094] Step 2: performing a first processing on the plurality of first output vectors to obtain a second output vector;

[0095] Step 3: Perform a second processing on the second output vector using a layer normalization operator to obtain a third output vector;

[0096] Step 4: Perform a third process on the third output vector and the output vector of the previous Transformer block to obtain a fourth output vector;

[0097] Step 5: Input the fourth output vector into the multilayer perceptron layer to obtain a fifth output vector;

[0098] Step 6: Perform a second processing on the fifth output vector using a layer normalization operator to obtain a sixth output vector;

[0099] Step 7: Perform a third process on the sixth output vector and the fourth output vector to obtain the output vector of the current Transformer block.

[0100] In one embodiment of the present invention, the above-mentioned Transformer block may include two layers of residual cross attention layers and a multi-layer perceptron layer.

[0101] And, in one embodiment of the present invention, the implicit representation and the output vector of the previous Transformer block are linearly mapped to obtain the corresponding fourth feature and f Q , f K and f V Both are linear mappings, and l represents the index of the layer.

[0102] In one embodiment of the present invention, the fourth feature may be and Input the residual cross attention layer to get the corresponding first output vector The fourth feature K Z and V ZInput another residual cross attention layer to get the corresponding first output vector Among them, the spatiotemporal feature X Z and This will be processed by the residual crisscross attention module.

[0103] And, in one embodiment of the present invention, the first processing may be to perform subtraction processing on the first vectors, that is, the second output vector obtained by performing the first processing on multiple first output vectors is

[0104] Furthermore, in one embodiment of the present invention, the second output vector is processed by the layer normalization operator LN(·), and the third output vector obtained is

[0105] Furthermore, in one embodiment of the present invention, the third processing may be to add the third output vector and the output vector of the previous Transformer block. That is, by performing the third processing on the third output vector and the output vector of the previous Transformer block, the fourth output vector obtained is in, is the output vector of the previous Transformer block.

[0106] Furthermore, in one embodiment of the present invention, the multilayer perceptron layer may be an MLP neural network. Also, in one embodiment of the present invention, the fourth output vector is input into the multilayer perceptron layer, and the fifth output vector obtained may be

[0107] Furthermore, in one embodiment of the present invention, the sixth output vector obtained by performing the second processing on the fifth output vector through the layer normalization operator is

[0108] Furthermore, in one embodiment of the present invention, the third processing is performed on the sixth output vector and the fourth output vector, and the output vector of the current Transformer block obtained is

[0109] Step S4: Based on the fusion features, the detection result of the video to be detected is output through the classifier.

[0110] In one embodiment of the present invention, after obtaining the fusion features through the above steps, the detection result of the video to be detected can be output through a classifier based on the fusion features.

[0111] In one embodiment of the present invention, the above-mentioned classifier is trained and can be a multi-layer perceptron.

[0112] Furthermore, in one embodiment of the present invention, the detection result may be true or false. Specifically, in one embodiment of the present invention, if the detection result output by the classifier of the video to be detected is true, it means that the video to be detected is not generated and is a real video; if the detection result output by the classifier of the video to be detected is false, it means that the video to be detected is not generated and is a fake video.

[0113] Based on the above description, Figure 2 Schematic diagram of a video detection method based on implicit representation of a generative model proposed in an embodiment of the present invention. Figure 2 As shown, the input video obtains the first implicit representation Z0 through the three-dimensional variational self-encoder, and the first implicit obtains the second implicit representation Z through the diffusion process corresponding to the diffusion module T The second implicit representation is obtained by the denoising process corresponding to the denoising module to obtain the third implicit representation Z, and the first implicit representation Z0 and the third implicit representation Z are enhanced by the feature enhancement module to obtain the enhanced first feature and the second feature The input video passes through the target backbone network to obtain the third feature The third feature is aligned through the feature alignment module to obtain the spatiotemporal feature The first feature and the second feature and spatiotemporal characteristics The residual cross-attention module is used to perform feature fusion to obtain fused features, which are then passed through a classifier to output the detection results of the video to be detected. For a detailed introduction to feature fusion using the residual cross-attention module, please refer to the above embodiment.

[0114] In one embodiment of the present invention, Figure 3 FIG. 1 is a schematic diagram of a comparison result proposed in an embodiment of the present invention. Figure 3 As shown in the figure, the comparison results of t-distributed random neighbor embedding (t-SNE) show that compared with the existing video detection method, the video detection method proposed in the present invention can learn more discriminative features, thereby proving that the video generation model improves the learning performance of the model, enhances its robustness on different data types, and improves generalization.

[0115] According to the video detection method based on implicit representation of the generative model proposed in an embodiment of the present invention, the implicit representation of the content before and after reconstruction is calculated using the video generation model, and the spatiotemporal features of the video to be detected are extracted through the backbone network. Then, the implicit representation and spatiotemporal features are integrated through the residual cross-attention mechanism to determine the detection result of the video. By combining the intrinsic statistical characteristics of the video generation model with the spatiotemporal features extracted by the backbone network, the robustness to various disturbances is enhanced and the generalization is improved.

[0116] Next, a video detection device based on implicit representation of a generative model according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0117] Figure 4 2 is a schematic structural diagram of a video detection device based on implicit representation of a generative model according to an embodiment of the present invention.

[0118] like Figure 4 As shown, the video detection device 10 based on implicit representation of the generative model includes: an acquisition module 401, a feature extraction module 402, a feature fusion module 403 and a detection module 404, wherein,

[0119] An acquisition module 401 is used to acquire a video to be detected and perform implicit feature extraction on the video to be detected using a video generation model to obtain an implicit representation;

[0120] A feature extraction module 402 is used to extract features from the video to be detected through a backbone network to obtain spatiotemporal features;

[0121] A feature fusion module 403 is used to fuse the implicit representation and the spatiotemporal features through a residual cross-attention mechanism to obtain a fused feature;

[0122] The detection module 404 is configured to output a detection result of the video to be detected through a classifier based on the fusion features.

[0123] Furthermore, the video generation model includes a three-dimensional variational autoencoder, a diffusion module, a denoising module, and a feature enhancement module; the acquisition module 401 is specifically used to:

[0124] The video to be detected is mapped to the implicit variable space through a three-dimensional variational autoencoder to obtain the first implicit representation;

[0125] Diffusion processing is performed on the first implicit representation by a diffusion module to obtain a second implicit representation;

[0126] De-noising the second implicit representation through a denoising module to obtain a third implicit representation;

[0127] Performing feature enhancement processing on the first implicit representation and the third implicit representation respectively through a feature enhancement module to obtain enhanced first features and second features corresponding to the first features;

[0128] The first feature and the second feature are determined as implicit representations.

[0129] Furthermore, the backbone network includes a target backbone network and a feature alignment module; the feature extraction module 402 is specifically used to:

[0130] The target backbone network is used to extract features from the video to be detected and obtain the third feature;

[0131] The third feature is aligned through the feature alignment module to obtain the spatiotemporal feature.

[0132] Furthermore, the feature fusion module 403 is specifically configured to:

[0133] The implicit representation and spatiotemporal features are fused through multiple Transformer blocks to obtain fused features.

[0134] Furthermore, the Transformer block includes multiple layers of residual cross attention layers and multi-layer perceptron layers; the feature fusion module 403 is specifically used to:

[0135] Linearly map the implicit representation and the output vector of the previous Transformer block to obtain the corresponding fourth feature;

[0136] The fourth feature and the spatiotemporal feature are processed through multiple layers of residual cross attention layers and multi-layer perceptron layers to obtain the output vector of the current Transformer block;

[0137] Repeat the above steps and determine the output vector of the last Transformer block as the fused feature.

[0138] Furthermore, the feature fusion module 403 is further configured to:

[0139] Input the fourth feature and the spatiotemporal feature into each residual cross attention layer respectively to obtain the corresponding first output vector;

[0140] Performing a first process on the plurality of first output vectors to obtain a second output vector;

[0141] performing a second processing on the second output vector by a layer normalization operator to obtain a third output vector;

[0142] Performing a third process on the third output vector and the output vector of the previous Transformer block to obtain a fourth output vector;

[0143] Inputting the fourth output vector into the multilayer perceptron layer to obtain a fifth output vector;

[0144] performing a second processing on the fifth output vector using a layer normalization operator to obtain a sixth output vector;

[0145] The third processing is performed on the sixth output vector and the fourth output vector to obtain the output vector of the current Transformer block.

[0146] According to the video detection device based on implicit representation of the generative model proposed in an embodiment of the present invention, the implicit representation of the content before and after reconstruction is calculated using the video generation model, and the spatiotemporal features of the video to be detected are extracted through the backbone network. Then, the implicit representation and spatiotemporal features are integrated through the residual cross-attention mechanism to determine the detection result of the video. By combining the intrinsic statistical characteristics of the video generation model with the spatiotemporal features extracted by the backbone network, the robustness to various disturbances is enhanced and the generalization is improved.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0148] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0149] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A video detection method based on implicit representation of a generative model, characterized in that: The method comprises: Obtaining a video to be detected, and performing implicit feature extraction on the video to be detected using a video generation model to obtain an implicit representation; Extracting features from the video to be detected through a backbone network to obtain spatiotemporal features; The implicit representation and the spatiotemporal feature are fused through a residual cross attention mechanism to obtain a fused feature; Based on the fusion features, a detection result of the video to be detected is outputted through a classifier.

2. The method according to claim 1, characterized in that The video generation model includes a three-dimensional variational autoencoder, a diffusion module, a denoising module and a feature enhancement module; The implicit feature extraction of the video to be detected by using a video generation model to obtain an implicit representation includes: Mapping the video to be detected to an implicit variable space through the three-dimensional variational autoencoder to obtain a first implicit representation; performing diffusion processing on the first implicit representation by the diffusion module to obtain a second implicit representation; performing denoising processing on the second implicit representation by the denoising module to obtain a third implicit representation; Performing feature enhancement processing on the first implicit representation and the third implicit representation respectively by the feature enhancement module to obtain enhanced corresponding first features and second features; The first feature and the second feature are determined as implicit representations.

3. The method according to claim 1, characterized in that The backbone network includes a target backbone network and a feature alignment module; The feature extraction of the video to be detected by the backbone network to obtain spatiotemporal features includes: Extracting features from the video to be detected through the target backbone network to obtain a third feature; The feature alignment module performs feature alignment on the third feature to obtain a spatiotemporal feature.

4. The method according to claim 1, wherein The feature fusion of the implicit representation and the spatiotemporal features is performed through the residual cross attention mechanism to obtain the fused features, including: the feature fusion of the implicit representation and the spatiotemporal features is performed through multiple Transformer blocks to obtain the fused features.

5. The method according to claim 4, characterized in that The Transformer block includes multiple layers of residual cross attention layers and multi-layer perceptron layers; The step of fusing the implicit representation and the spatiotemporal features through multiple Transformer blocks to obtain fused features includes: Linearly map the implicit representation and the output vector of the previous Transformer block to obtain the corresponding fourth feature; Performing feature processing on the fourth feature and the spatiotemporal feature through a multi-layer residual cross attention layer and a multi-layer perceptron layer to obtain an output vector of the current Transformer block; Repeat the above steps and determine the output vector of the last Transformer block as the fused feature.

6. The method according to claim 4, characterized in that The step of performing feature processing on the fourth feature and the spatiotemporal feature through a multi-layer residual cross attention layer and a multi-layer perceptron layer to obtain an output vector of the current Transformer block includes: Inputting the fourth feature and the spatiotemporal feature into each residual cross attention layer respectively to obtain a corresponding first output vector; Performing a first process on the plurality of first output vectors to obtain a second output vector; performing a second processing on the second output vector by a layer normalization operator to obtain a third output vector; Performing a third process on the third output vector and an output vector of a previous Transformer block to obtain a fourth output vector; Inputting the fourth output vector into the multilayer perceptron layer to obtain a fifth output vector; performing a second processing on the fifth output vector by the layer normalization operator to obtain a sixth output vector; Perform a third process on the sixth output vector and the fourth output vector to obtain an output vector of the current Transformer block.

7. A video detection device based on implicit representation of a generative model, characterized in that: The device comprises: An acquisition module is used to acquire a video to be detected and perform implicit feature extraction on the video to be detected through a video generation model to obtain an implicit representation; A feature extraction module is used to extract features of the video to be detected through a backbone network to obtain spatiotemporal features; A feature fusion module, configured to fuse the implicit representation and the spatiotemporal features through a residual cross-attention mechanism to obtain a fused feature; The detection module is used to output the detection result of the video to be detected through a classifier based on the fusion feature.

8. The device according to claim 7, characterized in that The backbone network includes a target backbone network and a feature alignment module; the feature extraction module is specifically used to: Extracting features from the video to be detected through the target backbone network to obtain a third feature; The feature alignment module performs feature alignment on the third feature to obtain a spatiotemporal feature.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images