Video detection method and device

Abstract

The embodiment of the invention provides a video detection method and device, and relates to the technical field of data processing, and the method comprises the steps: obtaining an image feature of a face region in a video frame of a to-be-detected video; segmenting the obtained image features in the spatial dimension, taking the segmentation result as a feature sequence, segmenting the obtained image features in the time dimension, and taking the segmentation result as a feature sequence; for any feature value in any feature sequence, obtaining a self-attention feature representing the importance degree of the feature value in all feature values of the feature sequence, and obtaining a consistency feature representing the consistency degree between the feature value and the first feature value, the first feature value is other feature values except the feature value in the feature sequence; and detecting whether the to-be-detected video is a forged video or not based on the obtained self-attention feature and the consistency feature. By applying the scheme provided by the embodiment of the invention, the video of which the face region is tampered can be detected.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a video detection method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, a technology has emerged that can tamper with faces in videos. It is usually called AI (Artificial Intelligence) face-changing technology. The original intention of this technology is to replace the faces appearing in the video with fake faces that retain certain facial features. This allows the fake faces to have the ability to identify people while protecting the person's portrait privacy.

[0003] However, with the development of AI face-changing technology, the technology can also tamper with the faces of people appearing in the video to the faces of other people, and its abuse may cause great harm to people's privacy and security. For example, in some cases, after the face in the video is tampered with to the face of person A to obtain a forged video, the forged video can be used to pass the face video verification of the online lending platform, so as to borrow money and other behaviors in the identity of person A; for another example, the forged video can be used to pass the face video verification of the access control system, so as to pass the access control abnormally; for another example, the forged video can be sent to the relatives of person A, so as to commit fraud and other behaviors against the relatives of person A.

[0004] In view of the above situation, it is necessary to provide a video detection method to detect forged videos in which faces are tampered with. Summary of the invention

[0005] The purpose of the embodiment of the present application is to provide a video detection method and device to detect forged videos in which faces are tampered with. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a video detection method, the method comprising:

[0007] Obtaining image features of a face region in a video frame of a video to be detected;

[0008] The obtained image features are segmented in the spatial dimension, and the segmentation results are used as feature sequences, and the obtained image features are segmented in the time dimension, and the segmentation results are used as feature sequences;

[0009] For any feature value in any feature sequence, a self-attention feature characterizing the importance of the feature value in all feature values ​​in the feature sequence is obtained, and a consistency feature characterizing the consistency between the feature value and a target feature value is obtained, wherein the target feature value is: other feature values ​​in the feature sequence except the feature value;

[0010] Based on the obtained self-attention features and consistency features, it is detected whether the video to be detected is a forged video.

[0011] In a second aspect, an embodiment of the present application provides a video detection device, the device comprising:

[0012] The first feature acquisition module is used to obtain image features of the face area in the video frame of the video to be detected;

[0013] A feature segmentation module is used to segment the obtained image features in the spatial dimension and use the segmentation results as a feature sequence, and to segment the obtained image features in the time dimension and use the segmentation results as a feature sequence;

[0014] The second feature acquisition module is used to obtain, for any feature value in any feature sequence, a self-attention feature representing the importance of the feature value in all feature values ​​in the feature sequence, and obtain a consistency feature representing the consistency between the feature value and a target feature value, wherein the target feature value is: other feature values ​​in the feature sequence except the feature value;

[0015] The video detection module is used to detect whether the video to be detected is a forged video based on the obtained self-attention features and consistency features.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0017] Memory, used to store computer programs;

[0018] The processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0021] It can be seen from the above that when the solution provided by the embodiment of the present application is used for video detection, the image features of the face area in the video frame of the video to be detected are obtained, and the obtained image features are segmented in the spatial dimension and the time dimension respectively, and the segmentation results are used as a feature sequence. In this way, for any feature value in any feature sequence, a self-attention feature characterizing the importance of the feature value in all feature values ​​in the feature sequence can be obtained, and a consistency feature characterizing the degree of consistency between the feature value and the target feature value other than the feature value in the feature sequence can be obtained, so that whether the video to be detected is a forged video can be successfully detected based on the obtained self-attention feature and the consistency feature.

[0022] Among them, the above-mentioned feature sequence is obtained by segmenting the obtained image features in the spatial dimension and the temporal dimension respectively. Therefore, the above-mentioned feature sequence can reflect the feature changes of the face area in the video frame in the spatial dimension, and can reflect the feature changes of the face area in the video frame in the temporal dimension. Furthermore, when performing video detection based on the above-mentioned feature sequence, the feature changes of the face area in the spatial and temporal dimensions are comprehensively considered, so that the feature sequence is more sensitive to the mutation of the space-time dimension, and can comprehensively reflect the discontinuity, mutation, etc. of the face in the video frame of the video to be detected in the temporal and spatial dimensions, thereby improving the accuracy of video detection.

[0023] In addition, the self-attention feature characterizes the importance of each feature value in the feature sequence among all the feature values ​​in the feature sequence, and the consistency feature characterizes the consistency between each feature value in the feature sequence and the target feature value other than the feature value. Based on the above two features, the slight distortion of the face in the video frame of the video to be detected can be captured more sensitively in the two dimensions of feature importance and feature consistency, thereby further improving the accuracy of video detection.

[0024] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0026] Figure 1 A schematic diagram of a flow chart of a first video detection method provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a first feature segmentation method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a second feature segmentation method provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a time-space inconsistency comparison provided in an embodiment of the present application;

[0030] Figure 5 A feature comparison schematic diagram provided for an embodiment of the present application;

[0031] Figure 6 A schematic diagram of a flow chart of a second video detection method provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of a consistency feature acquisition process provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of a flow chart of a third video detection method provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of a self-attention feature acquisition process provided in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of a fourth video detection method provided in an embodiment of the present application;

[0036] Fig.11 A schematic diagram of a network structure provided for an embodiment of the present application;

[0037] Fig.12 A schematic diagram of a sample construction process provided in an embodiment of the present application;

[0038] Fig.13 A schematic diagram of a feature acquisition process provided in an embodiment of the present application;

[0039] Fig.14 A schematic diagram of the structure of a video detection device provided in an embodiment of the present application;

[0040] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0042] First, the execution subject of the solution provided in the embodiment of the present application is described.

[0043] The executor of the solution provided in the embodiment of the present application is: any electronic device with data processing, storage and other functions.

[0044] The video detection method provided in the embodiment of the present application is described in detail below.

[0045] See also Figure 1 , is a flow chart of the first video detection method provided in an embodiment of the present application, the method includes the following steps S101-S104.

[0046] Step S101: obtaining image features of a face region in a video frame of a video to be detected.

[0047] The video frame of the video to be detected may be each original video frame in the video to be detected, or may be a video frame containing a face area that is determined in advance from the video frames of the video to be detected.

[0048] Specifically, the image features of the face area in the video frame may be obtained in the following manner.

[0049] In one implementation, for each video frame, an object detection algorithm may be first used to identify the face region in the video frame of the video to be detected, thereby determining the video frame containing the face region, and then an object feature extraction algorithm may be used to extract the image features of the face region.

[0050] Among them, the above-mentioned object detection algorithm can be an algorithm for object detection based on the Fast R-CNN model, an algorithm for object detection based on the YoLo model, a DPM algorithm (Deformable Part Model), etc., and the above-mentioned object feature extraction algorithm can be an eigenface algorithm, a local binary pattern (Local Binary Patterns, LBP) algorithm, a convolutional neural network algorithm, a HOG (Histogram of Oriented Gradient, directional gradient histogram) algorithm, etc., which is not limited to this embodiment of the present application.

[0051] In another implementation, each video frame may be input into a pre-trained face detection model to obtain a face region output by the model, and then an object feature extraction algorithm may be used to extract image features of the face region.

[0052] The embodiments of the present application do not limit the specific architecture of the above-mentioned face detection model. For example, the above-mentioned face detection model can be a convolutional neural network model, etc.

[0053] Step S102: segmenting the obtained image features in the spatial dimension, and using the segmentation results as a feature sequence; segmenting the obtained image features in the temporal dimension, and using the segmentation results as a feature sequence.

[0054] In this step, the obtained image features can be segmented in two dimensions, namely, the spatial dimension and the temporal dimension. The segmentation results are all used as feature sequences. In order to distinguish them from feature sequences corresponding to different dimensions, the feature sequence obtained by segmenting the obtained image features in the spatial dimension can be called the first feature sequence, and the feature sequence obtained by segmenting the obtained image features in the temporal dimension can be called the second feature sequence.

[0055] Depending on the feature extraction algorithm used to extract features from the face area in the above step S101, the image features obtained can be high-dimensional vectors or feature maps. In order to facilitate the feature segmentation of this step, the image features in the form of high-dimensional vectors can be converted into image features in the form of feature maps based on matrix transformation.

[0056] First, the method of segmenting the obtained image features in the spatial dimension is explained.

[0057] The embodiment of the present application does not limit the specific method of segmenting the obtained image features in the spatial dimension. It only needs to ensure that the segmented features can reflect the differences in image features in different regions. The following is an illustration through examples.

[0058] In one implementation, for each obtained image feature, the image feature may be segmented by row in the horizontal direction, and the segmentation result is used as the first feature sequence.

[0059] Among them, the image feature can be segmented row by row in the horizontal direction, and the segmentation result is used as the first feature sequence; or the image feature can be segmented into a preset number of rows in the horizontal direction, and the segmentation result is used as the first feature sequence. The preset number of rows can be 2 rows, 3 rows, etc.

[0060] Combine the following Figure 2 Provide a more intuitive explanation.

[0061] See also Figure 2 , which is a schematic diagram of the first feature segmentation method provided in an embodiment of the present application.

[0062] Figure 2Each layer in the large cube shown represents an image feature. Taking the image feature shown in the first layer as an example, when the image feature is cut for the first time, the feature values ​​of the first two rows of the image feature can be cut in the horizontal direction to obtain the first feature sequence. Then, when the image feature is cut for the second time, the feature values ​​of the third and fourth rows of the image feature can be cut in the horizontal direction to obtain the first feature sequence, and so on, each first feature sequence can be obtained.

[0063] Since the image features are extracted from the face area in the video frame of the video to be detected, there is a corresponding relationship between the area in the obtained image features and the face area. Therefore, the image features are finely divided by rows in the horizontal direction, and the obtained first feature sequence can better reflect the feature changes of the face area in the horizontal direction.

[0064] In another implementation, for each obtained image feature, the feature image may be segmented by columns in the vertical direction, and the segmentation result is used as the first feature sequence.

[0065] Among them, the image feature can be segmented column by column in the vertical direction, and the segmentation result is used as the first feature sequence; or the image feature can be segmented into a preset number of columns in the vertical direction in sequence, and the segmentation result is used as the first feature sequence. The preset number of columns can be 2 rows, 3 rows, etc.

[0066] The segmentation method in this embodiment is similar to that in the previous embodiment, the only difference is that the horizontal segmentation by rows is replaced by the vertical segmentation by columns, which will not be further described here.

[0067] In this way, the image features are finely divided by columns in the vertical direction, and the obtained first feature sequence can better reflect the feature changes of the face area in the vertical direction.

[0068] Of course, the above two methods can also be combined, that is, the image features are segmented by rows in the horizontal direction, and the segmentation results are used as the first feature sequence, and the feature image is segmented by columns in the vertical direction, and the segmentation results are used as the first feature sequence. In this way, the image is finely segmented from the row and column dimensions respectively, and the segmentation results are used as the first feature sequence. The obtained first feature sequence can better reflect the feature changes of the face area in the horizontal direction, and can better reflect the feature changes of the face area in the vertical direction.

[0069] In another embodiment, a segmentation frame of a fixed size may be pre-set, and then the position of the segmentation frame may be changed in the obtained image features so that the segmentation frame traverses each region in the image features, and the feature values ​​within the segmentation frame after each position change are extracted as a first feature sequence.

[0070] Next, the method of segmenting the obtained image features in the time dimension is described.

[0071] Specifically, feature values ​​corresponding to the same region in the obtained image features may be determined, and a second feature sequence may be obtained based on the determined feature values.

[0072] The above-mentioned area can be a plurality of areas that jointly cover all areas in the image feature, and the size of the above-mentioned area can be determined based on the size of the image feature, which is not limited in the embodiment of the present application.

[0073] In one case, feature values ​​corresponding to the same position in the obtained image features may be determined, and a second feature sequence may be obtained based on the determined feature values.

[0074] Combine the following Figure 3 Provide a more intuitive explanation.

[0075] See also Figure 3 , which is a schematic diagram of the second feature segmentation method provided in an embodiment of the present application.

[0076] Figure 3 Each layer of the large cube shown in the figure represents an image feature. It can be seen that when the image features are first divided, the feature values ​​corresponding to the same position in each layer are obtained ( Figure 3 Then, when the second segmentation is performed, the feature value corresponding to the next position in the image features represented by each layer can be obtained to obtain the second feature sequence, and so on, each second feature sequence can be obtained.

[0077] Since the obtained image features are image features extracted from the face region of each video frame of the video to be detected, there is a corresponding relationship between the region in the obtained image features and the face region, so that the feature value corresponding to the same region in the obtained image features reflects the regional features of the same region in the face region of each video frame. Since each video frame is temporally related, the second feature sequence obtained based on the determined feature value can better reflect the feature changes of the face region of each video frame in the time dimension.

[0078] Step S103: For any feature value in any feature sequence, obtain a self-attention feature representing the importance of the feature value among all the feature values ​​in the feature sequence, and obtain a consistency feature representing the degree of consistency between the feature value and the target feature value.

[0079] The target feature value is: other feature values ​​in the feature sequence except the feature value.

[0080] In this step, the feature sequence includes the first feature sequence obtained by segmenting the obtained image features in the spatial dimension and the second feature sequence obtained by segmenting the obtained image features in the temporal dimension.

[0081] Since the above steps obtain a feature sequence reflecting the changes in the face area in the spatial dimension and a feature sequence reflecting the changes in the face area in the temporal dimension, in this step, the self-attention feature and consistency feature obtained also include features for the above two feature sequences.

[0082] The feature sequences involved in the processing in this step are described below.

[0083] For example, the above-mentioned self-attention feature and consistency feature can be obtained for any one or more feature values ​​in any one or more feature sequences, the above-mentioned self-attention feature and consistency feature can be obtained for each feature value in any one or more feature sequences, the above-mentioned self-attention feature and consistency feature can be obtained for any one or more feature values ​​in each feature sequence, and of course, the above-mentioned self-attention feature and consistency feature can also be obtained for each feature value in each feature sequence.

[0084] Specifically, when obtaining the above self-attention feature, the query vector, key vector and value vector of the feature value in the feature sequence can be first obtained, so that the self-attention feature can be obtained based on the above vectors. The specific implementation method is detailed in the subsequent Figure 8 Steps S804 to S805 in the illustrated embodiment are not described in detail here.

[0085] The following is an explanation of how to obtain the above consistency features.

[0086] Specifically, taking the consistency feature of a feature value in a feature sequence as an example, the predicted feature value of the feature value can be obtained based on the target feature value in the feature sequence, and then the consistency feature characterizing the degree of consistency between the feature value and the target feature value can be obtained based on the predicted feature value and the feature value.

[0087] The specific method of obtaining the predicted feature value is described in the following Figure 6 Steps S605 to S606 in the illustrated embodiment are not described in detail here.

[0088] It can be seen that the above-mentioned predicted eigenvalue is a eigenvalue obtained based on the target eigenvalue other than the eigenvalue, and the degree of consistency between the predicted eigenvalue and the target eigenvalue is relatively high. Therefore, based on the above-mentioned predicted eigenvalue, the degree of consistency between the eigenvalue and the target eigenvalue can be indirectly evaluated, so that the consistency feature characterizing the degree of consistency between the eigenvalue and the target eigenvalue can be accurately obtained.

[0089] Step S104: Based on the obtained self-attention features and consistency features, detect whether the video to be detected is a forged video.

[0090] In this step, the self-attention feature characterizes the importance of the feature value in the feature sequence among all the feature values ​​in the feature sequence, and the consistency feature characterizes the consistency between the feature value in the feature sequence and the target feature value other than the feature value. Based on the above two features, it is possible to detect whether the video to be detected is a forged video in two dimensions: feature importance and feature consistency.

[0091] Specifically, the following method may be used to detect whether the video to be detected is a forged video.

[0092] In one implementation, the obtained self-attention features and consistency features may be input into a pre-trained video detection model to obtain a forged video detection result output by the above model.

[0093] Among them, the above-mentioned video detection model can be a binary classification model obtained by taking sample self-attention features and sample consistency features as input and labeling whether the sample video corresponding to the above-mentioned features is a forged video.

[0094] In this way, if the video to be detected is a forged video, the obtained self-attention features and consistency features are input into the above-mentioned video detection model, and the above-mentioned model can output a result representing that the video is forged; if the video to be detected is not a forged video, the obtained self-attention features and consistency features are input into the above-mentioned video detection model, and the above-mentioned model can output a result representing that the video is not forged.

[0095] In another implementation, a first similarity between the self-attention feature and the baseline self-attention feature may be calculated, and a second similarity between the consistency feature and the baseline consistency feature may be calculated, and whether the video to be detected is a forged video may be detected based on the first similarity and the second similarity.

[0096] Among them, the above-mentioned baseline self-attention features can be obtained based on unforged videos.

[0097] Specifically, the self-attention features of multiple unforged videos can be obtained, and then the fusion features, average features, etc. of the above features can be obtained as the above-mentioned benchmark self-attention features.

[0098] The method for obtaining the above-mentioned baseline consistency features is similar to the method for obtaining the baseline self-attention features, which will not be repeated here.

[0099] After obtaining the first similarity and the second similarity, the following method may be used to detect whether the video to be detected is a forged video.

[0100] In one implementation, if the first similarity is greater than a first threshold or the second similarity is greater than a second threshold, it can be determined that the video to be detected is a forged video; otherwise, it is determined that the video to be detected is a non-forged video.

[0101] In another implementation, the sum of the first similarity and the second similarity may be calculated. If the sum is greater than a third threshold, the video to be detected may be determined to be a forged video; otherwise, the video to be detected may be determined to be a non-forged video.

[0102] The above-mentioned first threshold, second threshold and third threshold can be set by the staff based on experience and will not be elaborated here.

[0103] It can be seen from the above that when the solution provided by the embodiment of the present application is used for video detection, the image features of the face area in the video frame of the video to be detected are obtained, and the obtained image features are segmented in the spatial dimension and the time dimension respectively, and the segmentation results are used as a feature sequence. In this way, for any feature value in any feature sequence, a self-attention feature characterizing the importance of the feature value in all feature values ​​in the feature sequence can be obtained, and a consistency feature characterizing the degree of consistency between the feature value and the target feature value other than the feature value in the feature sequence can be obtained, so that whether the video to be detected is a forged video can be successfully detected based on the obtained self-attention feature and the consistency feature.

[0104] Among them, the above-mentioned feature sequence is obtained by segmenting the obtained image features in the spatial dimension and the temporal dimension respectively. Therefore, the above-mentioned feature sequence can reflect the feature changes of the face area in the video frame in the spatial dimension, and can reflect the feature changes of the face area in the video frame in the temporal dimension. Furthermore, when performing video detection based on the above-mentioned feature sequence, the feature changes of the face area in the spatial and temporal dimensions are comprehensively considered, so that the feature sequence is more sensitive to the mutation of the space-time dimension, and can comprehensively reflect the discontinuity, mutation, etc. of the face in the video frame of the video to be detected in the temporal and spatial dimensions, thereby improving the accuracy of video detection.

[0105] In addition, the self-attention feature characterizes the importance of each feature value in the feature sequence among all the feature values ​​in the feature sequence, and the consistency feature characterizes the consistency between each feature value in the feature sequence and the target feature value other than the feature value. Based on the above two features, the slight distortion of the face in the video frame of the video to be detected can be captured more sensitively in the two dimensions of feature importance and feature consistency, thereby further improving the accuracy of video detection.

[0106] Next, combine Figure 4 , further illustrating the beneficial effects of the embodiments of the present application.

[0107] See also Figure 4, which is a schematic diagram of a time-space inconsistency comparison provided in an embodiment of the present application.

[0108] Figure 4 The areas marked by ellipses in the first column of images are the mouth and nose areas in the real face images and the forged face images, respectively. The second column of images are stacked images obtained by stacking the elliptical areas in the real face images and the forged face images in the time dimension. It can be seen that compared with the stacked images of the real face images, the stacked images of the forged face images have more jitters and mutations in some areas (such as the areas indicated by ellipses in the stacked images). In other words, compared with the real face images, the forged face images have more jitters in the time dimension.

[0109] Obviously, compared with the original image, some areas of the forged face image also have jitter and mutation traces. In other words, compared with the real face image, the forged face image has more jitter in the spatial dimension.

[0110] In summary, compared with real face images, forged face images have more jitters and mutations in both time and space dimensions. Therefore, after obtaining the image features of the face area in the video frame of the video to be detected, the embodiment of the present application divides the obtained image features in the spatial dimension and the time dimension respectively to obtain a feature sequence. Since the feature sequence corresponds to the area in the video frame, the above feature sequence can reflect both the feature changes of the face area in the video frame in the spatial dimension and the feature changes of the face area in the video frame in the time dimension. When performing video detection based on the above feature sequence, the feature changes of the face area in the spatial and time dimensions are comprehensively considered, so that the features are more sensitive to mutations in the time and space dimensions, and can comprehensively capture the discontinuity, mutation, etc. of the face in the video frame of the video to be detected in the time and space dimensions, thereby improving the accuracy of video detection.

[0111] The following is an intuitive explanation of the differences between the self-attention features and consistency features corresponding to real face images and forged face images.

[0112] See also Figure 5 , which is a feature comparison schematic diagram provided in an embodiment of the present application.

[0113] Figure 5The first and third rows of images are images after the features corresponding to the real face images are visualized, and the second and fourth rows of images are images after the features corresponding to the forged face images are visualized. In addition, the first column of images is the first image after the self-attention features corresponding to the feature sequence obtained according to the spatial dimension are visualized, the second column of images is the second image after the self-attention features corresponding to the feature sequence obtained according to the time dimension are visualized, the third column of images is the third image after the consistency features corresponding to the feature sequence obtained according to the spatial dimension are visualized, the fourth column of images is the fourth image of the inconsistent area between the forged face image and the real face image, the fifth column of images is the fifth image after the consistency features corresponding to the feature sequence obtained according to the time dimension are visualized, and the sixth column of images is the sixth image obtained by stacking some areas in the fifth image in the time dimension.

[0114] It can be seen that compared with the real face images, the jitter, mutation, etc. of the first image to the sixth image corresponding to the forged face images are larger, that is, the self-attention features and consistency features corresponding to the forged face images are significantly different from those of the real face images.

[0115] exist Figure 1 On the basis of the embodiment shown, the predicted feature value of the feature value other than a certain feature value in the feature sequence can be obtained based on the query vector, key vector and value vector of the target feature value, so that the consistency feature characterizing the degree of consistency between the feature value and the target feature value can be obtained based on the predicted feature value and the feature value. In view of the above situation, the embodiment of the present application provides a second video detection method.

[0116] See also Figure 6 , is a flow chart of a second video detection method provided in an embodiment of the present application, the method includes the following steps S601-S608.

[0117] Step S601: Obtain image features of a face region in a video frame of a video to be detected.

[0118] Step S602: segmenting the obtained image features in the spatial dimension, and using the segmentation results as a feature sequence; segmenting the obtained image features in the temporal dimension, and using the segmentation results as a feature sequence.

[0119] The above steps S601 and S602 are the same as the above steps. Figure 1 In the illustrated embodiment, step S101 is the same as step S102 and will not be described in detail herein.

[0120] Step S603: Based on the preset query parameter matrix, key parameter matrix and value parameter matrix, a query vector, a key vector and a value vector of each feature value in the feature sequence are obtained.

[0121] Specifically, a matrix multiplication calculation may be performed on the query parameter matrix and each eigenvalue in the feature sequence, and the resulting new feature is the query vector of the eigenvalue, which may be denoted as qi.

[0122] For example, if the query parameter matrix is ​​W q , for the eigenvalue A in the feature sequence A i For example, its query vector qi = W q A i .

[0123] Similarly, the key vector ki of each eigenvalue in the feature sequence can be obtained by performing matrix multiplication on the key parameter matrix and each eigenvalue in the feature sequence, and the key vector vi of each eigenvalue in the feature sequence can be obtained by performing matrix multiplication on the value parameter matrix and each eigenvalue in the feature sequence.

[0124] For any eigenvalue in any feature sequence, the following steps S604-S607 are used to obtain a self-attention feature characterizing the importance of the eigenvalue among all the eigenvalues ​​in the feature sequence, and a consistency feature characterizing the degree of consistency between the eigenvalue and the target eigenvalue.

[0125] Step S604: Obtain a self-attention feature that characterizes the importance of the feature value among all the feature values ​​in the feature sequence.

[0126] Step S605: Obtain a first weight of each target feature value for the feature value based on the similarity between the query vector of the feature value and the key vector of the target feature value.

[0127] The similarity mentioned above may be the Euclidean distance, cosine distance, etc. between the query vector of the feature value and the key vector of the target feature value.

[0128] Specifically, after obtaining the above similarity, the first weight of each target feature value with respect to the feature value may be obtained in the following manner.

[0129] In one implementation, the similarity may be directly used as the first weight of the target feature value for the feature value.

[0130] In another case, the target eigenvalue can be obtained by the following expression for the eigenvalue A o The first weight W:

[0131]

[0132] Among them, s represents the sequence number of the eigenvalue in the feature sequence, which is a positive integer increasing from 1, e represents the natural base, and q o Denotes the eigenvalue A o The query vector kp The key vector representing the target feature value, k s A key vector representing any eigenvalue.

[0133] visible, Indicates the similarity between the key vector of the eigenvalue and the key vector of the target eigenvalue, The first weight W in the above expression actually represents the proportion of the similarity between the key vector of the eigenvalue and the key vector of each eigenvalue in the feature sequence in the above sum.

[0134] Step S606: According to the first weight of each target eigenvalue for the eigenvalue, weighted sum is performed on the value vectors of each target eigenvalue to obtain a predicted eigenvalue of the eigenvalue.

[0135] After obtaining the first weight of the target eigenvalue for the eigenvalue, weighted summation can be performed on the value vectors of each target eigenvalue according to the first weight to obtain the predicted eigenvalue of the eigenvalue.

[0136] In one case, the eigenvalue A can be obtained using the following expression: o The predicted feature value pre o :

[0137]

[0138] Among them, p and s represent the serial number of the eigenvalue, and p is not equal to the eigenvalue A o The serial number o, so p actually represents the characteristic sequence except the eigenvalue A o The number of the target eigenvalue other than v p A value vector representing the target eigenvalue. The meanings of other parameters have been explained above.

[0139] It can be seen that according to the above formula, except for the eigenvalue A o The value vector of the target eigenvalues ​​outside is weighted summed.

[0140] Step S607: According to the predicted eigenvalue and the eigenvalue, a consistency feature characterizing the degree of consistency between the eigenvalue and the target eigenvalue is obtained.

[0141] Specifically, the above consistency feature can be obtained in the following manner.

[0142] In one implementation, a difference between a predicted eigenvalue and a value vector of the eigenvalue may be obtained, and the difference may be used as a consistency feature representing the degree of consistency between the eigenvalue and the target eigenvalue.

[0143] When the predicted characteristic value is calculated according to the expression in step S607, the characteristic value Ao The corresponding consistency feature can be expressed as follows:

[0144]

[0145] Among them, c o represents the above consistency characteristics, v o Represents the eigenvalue A o The meanings of other parameters have been explained above.

[0146] The difference between the predicted eigenvalue and the eigenvalue can accurately measure the consistency between the eigenvalue and the predicted eigenvalue. Since the above-mentioned predicted eigenvalue is a eigenvalue obtained based on the target eigenvalue other than the eigenvalue, the consistency between the predicted eigenvalue and the target eigenvalue is relatively high. Therefore, the above-mentioned difference can also accurately measure the consistency between the eigenvalue and the target eigenvalue, so that a more accurate consistency feature can be obtained based on the above-mentioned difference.

[0147] Step S608: Based on the obtained self-attention features and consistency features, detect whether the video to be detected is a forged video.

[0148] The above step S608 is similar to the above Figure 1 Step S104 in the illustrated embodiment is the same and will not be described again here.

[0149] It can be seen that when the solution provided in the embodiment of the present application is applied to obtain the predicted eigenvalue of the eigenvalue, the first weight of each target eigenvalue for the eigenvalue is obtained based on the similarity between the query vector of the eigenvalue and the key vector of the target eigenvalue. It can be seen that the first weight is related to the key vector of each target eigenvalue, and then the value vectors of each target eigenvalue are weighted and summed according to the first weight of each target eigenvalue for the eigenvalue, and the weighted summation result is related to the value vector of each target eigenvalue, which is equivalent to comprehensively considering the key vector and value vector of each target eigenvalue except the eigenvalue to predict the eigenvalue, so that the predicted eigenvalue of the eigenvalue can be obtained more accurately.

[0150] Combine the following Figure 7 , the process of obtaining the consistency features shown in the above steps S605-607 is explained more intuitively.

[0151] See also Figure 7 , which is a schematic diagram of a consistency feature acquisition process provided in an embodiment of the present application.

[0152] Figure 7 Where A1-A5 represent the eigenvalues ​​in the feature sequence, qn, kn and vn represent the query vector, key vector and value vector of the eigenvalue An, respectively.

[0153] Taking the consistency feature of eigenvalue A1 as an example, first, the similarities between the query vector q1 of eigenvalue A1 and the key vectors k2-k5 of eigenvalues ​​A2-A5 are calculated respectively; based on the obtained similarity, the first weight of eigenvalue A2-A5 for feature A1 is determined, such as the first weight of eigenvalue A2 for eigenvalue A1 is determined based on the similarity of q1 and k2; then, the value vectors v2-v5 of eigenvalue A2-A5 are weightedly summed according to the first weights corresponding to eigenvalues ​​A2-A5 to obtain the weighted summation result; finally, the difference between the weighted summation result and the value vector v1 of feature A1 is calculated to obtain the consistency feature c1 that characterizes the degree of consistency between eigenvalue A1 and eigenvalue A2-A5.

[0154] The method for obtaining the consistency features of the remaining features A2-A5 is similar to the method for obtaining the consistency features of the feature value A1, and examples are not given one by one here.

[0155] exist Figure 1 Based on the embodiment shown, the self-attention feature of a certain feature value in the feature sequence can be obtained based on the query vector, key vector and value vector of each feature value in the feature sequence. In view of the above situation, the embodiment of the present application provides a third video detection method.

[0156] See also Figure 8 , is a flow chart of a third video detection method provided in an embodiment of the present application, the method includes the following steps S801-S807.

[0157] Step S801: Obtain image features of a face region in a video frame of a video to be detected.

[0158] Step S802: segmenting the obtained image features in the spatial dimension, and using the segmentation results as a feature sequence; segmenting the obtained image features in the temporal dimension, and using the segmentation results as a feature sequence.

[0159] Step S803: Based on the preset query parameter matrix, key parameter matrix and value parameter matrix, obtain the query vector, key vector and value vector of each feature value in the feature sequence.

[0160] For any feature value in any feature sequence, the following steps S804-S805 are used to obtain a self-attention feature characterizing the importance of the feature value among all the feature values ​​in the feature sequence, and a consistency feature characterizing the degree of consistency between the feature value and the target feature value.

[0161] Step S804: obtaining a second weight of each eigenvalue for the eigenvalue based on the similarity between the query vector of the eigenvalue and the key vector of each eigenvalue in the eigenvalue sequence.

[0162] This step is similar to the previous Figure 6 The method of obtaining the first weight in step S605 of the illustrated embodiment is similar, the only difference is that the target eigenvalue is replaced by each eigenvalue, which will not be repeated here.

[0163] Step S805: According to the second weight of each eigenvalue for the eigenvalue, weighted sum is performed on the value vectors of each eigenvalue to obtain a self-attention feature representing the importance of the eigenvalue among all the eigenvalues ​​in the feature sequence.

[0164] In one case, the eigenvalue A can be obtained using the following expression: o The self-attention feature y o :

[0165]

[0166] Among them, i represents the sequence number of the feature value in the feature sequence, which is a positive integer increasing from 1, and k i and v i They represent the key vector and value vector of the eigenvalues ​​respectively. The meanings of other parameters have been explained above.

[0167] In this way, according to the above formula, the value vector of each eigenvalue in the feature sequence can be weighted summed.

[0168] Step S806: Obtain a consistency feature that characterizes the degree of consistency between the feature value and the target feature value.

[0169] The specific implementation of this step is detailed in the above Figure 6 The illustrated embodiment includes steps S605 to S607.

[0170] Step S807: Based on the obtained self-attention features and consistency features, detect whether the video to be detected is a forged video.

[0171] The above step S807 is similar to the above step S807. Figure 1 Step S104 in the illustrated embodiment is the same and will not be described again here.

[0172] It can be seen that when the scheme provided in the embodiment of the present application is applied to obtain the self-attention feature that characterizes the importance of the eigenvalue in all the eigenvalues ​​of the feature sequence, the second weight of each eigenvalue for the eigenvalue is obtained based on the similarity between the query vector of the eigenvalue and the key vector of each eigenvalue in the feature sequence. It can be seen that the second weight is related to the key vector of each eigenvalue. Then, according to the second weight of each eigenvalue for the eigenvalue, the value vectors of each eigenvalue are weighted and summed, and the weighted summation result is related to the value vector of each eigenvalue. This is equivalent to comprehensively considering the key vector and value vector of each eigenvalue in the feature sequence to obtain the self-attention feature of the eigenvalue, so that the above self-attention feature can better characterize the importance of the eigenvalue in all the eigenvalues ​​of the feature sequence, that is, a more accurate self-attention feature can be obtained.

[0173] Combine the following Fig. 9 , a more intuitive explanation is given of the process of obtaining the self-attention features shown in the above steps S804-S805.

[0174] See also Fig. 9 , which is a schematic diagram of a self-attention feature acquisition process provided in an embodiment of the present application.

[0175] Fig. 9 Where A1-A5 represent the eigenvalues ​​in the feature sequence, qn, kn and vn represent the query vector, key vector and value vector of the eigenvalue An, respectively.

[0176] Taking the self-attention feature of eigenvalue A1 as an example, first, the similarities between the query vector q1 of eigenvalue A1 and the key vectors k1-k5 of each eigenvalue A1-A5 including eigenvalue A1 are calculated respectively; based on the obtained similarity, the second weights of eigenvalues ​​A1-A5 for feature A1 are determined, such as the second weight of eigenvalue A2 for eigenvalue A1 is determined based on the similarity of q1 and k2; then, the value vectors v1-v5 of eigenvalues ​​A1-A5 are weightedly summed according to the second weights corresponding to the eigenvalues ​​A1-A5 to obtain a weighted summation result, which is the self-attention feature y1 that characterizes the importance of eigenvalue A1 among all eigenvalues ​​in the feature sequence.

[0177] The method of obtaining the self-attention features of the remaining features A2-A5 is similar to the method of obtaining the self-attention features of the feature value A1, and will not be given examples one by one here.

[0178] exist Figure 1On the basis of the embodiment shown, each feature sequence can be input into a pre-trained spatiotemporal consistency network, and a self-attention feature characterizing the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and a consistency feature characterizing the degree of consistency between each feature value in each feature sequence and a target feature value, can be obtained based on the output features of the spatiotemporal consistency network. In view of the above situation, the embodiment of the present application provides a fourth video detection method.

[0179] See also Fig.10 , is a flow chart of the fourth video detection method provided in an embodiment of the present application, and the above method includes the following steps S1001-S1004.

[0180] Step S1001: Obtain image features of a face area in a video frame of a video to be detected.

[0181] Step S1002: segmenting the obtained image features in the spatial dimension, and using the segmentation results as a feature sequence; segmenting the obtained image features in the temporal dimension, and using the segmentation results as a feature sequence.

[0182] The above steps S1001 and S1002 are the same as the above steps S1002. Figure 1 In the illustrated embodiment, step S101 is the same as step S102 and will not be described in detail herein.

[0183] Step S1003: Input each feature sequence into a pre-trained spatiotemporal consistency network to obtain output features, and based on the output features, obtain self-attention features that characterize the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, as well as consistency features that characterize the degree of consistency between each feature value in each feature sequence and the target feature value.

[0184] Combine the following Fig.11 The network structure of the above-mentioned spatiotemporal consistency network is explained.

[0185] See also Fig.11 , is a schematic diagram of a network structure provided in an embodiment of the present application.

[0186] Depend on Fig.11 It can be seen that the spatiotemporal consistency network includes: the first normalization layer, the self-attention and consistency feature extraction layer, the second normalization layer and the multi-layer perceptron MLP layer.

[0187] Among them, the rectangle and ellipse in the self-attention and consistency feature extraction layers represent detection heads respectively, among which three rectangular detection heads are used for self-attention feature encoding, and three ellipse detection heads are used for consistency feature encoding.

[0188] The functions of each network layer in the spatiotemporal consistency network are explained below.

[0189] The first normalization layer is used to normalize the input feature sequences and output the normalized feature sequences.

[0190] Self-attention and consistency feature extraction layer: used to encode the normalized feature sequence and output the encoding result.

[0191] Specifically, the three rectangular detection heads of the self-attention and consistency feature extraction layer are used to perform self-attention feature encoding on the feature values ​​in the feature sequences of row, column and time dimensions respectively, and the three elliptical detection heads are used to perform consistency feature encoding on the feature values ​​in the feature sequences of row, column and time dimensions respectively. Therefore, the self-attention and consistency feature extraction layer can output 6 sets of encoding results.

[0192] The second normalization layer is used to normalize the encoding result and output the normalized encoding result.

[0193] MLP layer: used to perform feature mapping on the normalized encoding results to obtain output features.

[0194] Specifically, after the MLP layer performs feature mapping on the normalized encoding results of the feature values ​​in a feature sequence, the obtained output features can be used as self-attention features that characterize the importance of the feature value among all the feature values ​​in the feature sequence, as well as consistency features that characterize the degree of consistency between the feature value and the target feature value.

[0195] Step S1004: Based on the obtained self-attention features and consistency features, detect whether the video to be detected is a forged video.

[0196] The above step S1004 is the same as the above Figure 1 Step S104 in the illustrated embodiment is the same and will not be described again here.

[0197] It can be seen that the various network layers in the above-mentioned spatiotemporal consistency network can perform feature transformation on the input feature sequence through mutual cooperation, and finally obtain the output feature, so that based on the output feature, the self-attention feature that characterizes the importance of each feature value in each feature sequence among all the feature values ​​of the feature sequence, as well as the consistency feature that characterizes the degree of consistency between each feature value in each feature sequence and the target feature value can be conveniently and accurately obtained.

[0198] The training method of the above-mentioned spatiotemporal consistency network is explained below.

[0199] Specifically, sample image features of the face area in the video frame of the sample video can be obtained, and the preset network can be trained based on the sample image features to obtain self-attention features and consistency features output by the preset network, and the loss value of the preset network can be determined based on the self-attention features and the consistency features. Based on the loss value, the network parameters of the preset network can be adjusted to obtain a spatiotemporal consistency network.

[0200] Among them, the above training process is a self-supervised training process, and the above loss value L con It can be expressed by the following expression:

[0201]

[0202] Where P(i) represents the positive sample set containing sample image features corresponding to the real video, p represents any sample image feature in the positive sample set, that is, any positive sample, A(i) represents the entire sample set containing all sample image features, a represents any sample image feature in the entire sample set, that is, any sample, i∈real represents i is any sample image feature in the positive sample set, that is, any positive sample, F i Indicates the features corresponding to i, such as consistency, F p Indicates the features corresponding to p, such as consistency features, F a Represents the features corresponding to a, such as consistency features.

[0203] It can be seen that in the above formula, p∈P(i) exp(F i ·F p ) indicates that the sum of similarities between all positive samples is obtained by exp and summation, ∑ a∈A(i) exp(F i ·F a ) indicates that the sum of the similarities between all positive samples and all samples is obtained by exp and summation. The division of the two indicates the proportion of the similarity between positive samples to the similarity between positive samples and all samples. Therefore, the loss function L con The overall meaning is: make the similarity between positive samples account for as large a proportion of the similarity between positive samples and all samples as possible, so that based on this loss function, the features of positive samples can be clustered into one category and the features of negative samples can be distinguished from those of positive samples.

[0204] The positive sample set is a set of sample image features of the face area in the video frame of the video whose face area has not been tampered with, and the total sample set is a set of sample image features of all the samples.

[0205] Specifically, when adjusting the network parameters of the preset network based on the loss value, the network parameters can be iteratively adjusted based on the loss value until the preset training end condition is met to obtain a trained spatiotemporal consistency network. The preset training end condition can be that the loss value is less than a preset indicator, the number of training times is greater than or equal to a preset number of times, etc.

[0206] The construction method of the above sample video is explained below.

[0207] The key points of the face in the first sample video frame of the original video can be obtained. The method for obtaining the above key points can be implemented by using any face key point extraction algorithm, which is not limited in the embodiment of the present application. Then, based on the obtained key points, a real face image is captured from the first sample video frame, and then a sample video is obtained based on the real face image in the following manner:

[0208] In one implementation, disturbance may be added to a real face image to obtain a disturbed face image, and a sample video may be obtained based on the real face image and the disturbed face image.

[0209] For example, an image of a partial area in a real face and an image of a partial area in a disturbed face image may be spliced ​​to obtain a spliced ​​face image, and then a sample video containing each spliced ​​face image may be obtained.

[0210] In this way, after adding disturbance to the real face image, the perturbed face image obtained can better simulate the face image obtained after the real face is tampered with, and then based on the real face image and the perturbed face image, a sample video that is closer to the video in which the face area is tampered can be obtained.

[0211] In another implementation, a sample video in which the inter-frame arrangement order is different from the original arrangement order may be generated based on a real face image.

[0212] The original arrangement order is: the arrangement order of the first sample video frames to which the real face images belong in the original video.

[0213] For example, after obtaining the above-mentioned real face image, the arrangement order of the first sample video frame to which the real face image belongs in the original video can be reorganized, and a sample video containing the real face image can be obtained according to the shuffled order.

[0214] See also Fig.12 , which is a schematic diagram of a sample construction process provided in an embodiment of the present application.

[0215] in, Fig.12The first column of images are: images of the mouth and nose regions captured in sequence from real face images that are different from the original arrangement order, and the second column of images are: images of the mouth and nose regions captured in sequence from real face images that are consistent with the original arrangement order.

[0216] It can be seen that from the first row of mouth and nose area images compared to the second row of mouth and nose area images, the continuous change of the same area in adjacent images (such as the mouth area in the image) becomes abrupt. As a result, some areas of the face image in the sample video obtained based on the real face image with a different arrangement order from the original have larger jitter in the time dimension, which is consistent with the change of the face image in the time dimension in the forged video with the face area tampered with, and is conducive to obtaining a sample video that is closer to the video with the face area tampered with.

[0217] It can be seen that, on the one hand, the above-mentioned consistency feature reflects the degree of consistency between a certain area in the image and the area adjacent to the area, and there are obvious differences in the video frames of the forged face video and the video frames of the real face video. Therefore, the above-mentioned network can learn the consistency differences in the video frames of the forged face video and the video frames of the real face video during the training process; on the other hand, the above-mentioned self-attention feature reflects the importance of a certain area in the image and in the image, and there are obvious differences in the video frames of the forged face video and the video frames of the real face video. Therefore, the above-mentioned network can learn the importance differences in the video frames of the forged face video and the video frames of the real face video during the training process.

[0218] In summary, the above network can learn the consistency differences and importance differences between the video frames of the forged face video and the real face video, so that the subsequent features based on the output of the above network can more accurately measure the difference between the forged face video and the real face video, thereby more accurately detecting whether the video is a forged video.

[0219] In one embodiment of the present application, the above step S1003 can be implemented based on the following step A.

[0220] Step A: Input each feature sequence into the spatiotemporal consistency network that is cascaded in sequence to obtain the output features of the last spatiotemporal consistency network as the self-attention features that characterize the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and the consistency features that characterize the degree of consistency between each feature value in each feature sequence and the target feature value.

[0221] Combine the following Fig.13 The feature acquisition process shown in step A is explained more intuitively.

[0222] See also Fig.13 , which is a schematic diagram of a feature acquisition process provided in an embodiment of the present application.

[0223] Fig.13 The network architecture of the spatiotemporal consistency network can be found in Fig.11 .

[0224] It can be seen that spatiotemporal consistency network 1, spatiotemporal consistency network 2, ..., spatiotemporal consistency network N are connected in cascade sequence. After the feature sequence is input into the first spatiotemporal consistency network 1, the output features of the first spatiotemporal consistency network 1 can be used as the input of its next adjacent spatiotemporal consistency network 2, and so on, until the output features of the last spatiotemporal consistency network N are obtained as self-attention features characterizing the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and consistency features characterizing the degree of consistency between each feature value in each feature sequence and the target feature value.

[0225] In this way, by using multiple spatiotemporal consistency networks connected in cascade order, deeper feature mining of feature sequences can be performed, thereby obtaining more excellent self-attention features and consistency features with spatiotemporal expression capabilities.

[0226] In one embodiment of the present application, Fig.13 On the basis of the spatiotemporal consistency network shown in the figure, a feature extraction layer can be added before the first spatiotemporal consistency network, and a pooling and classification layer can be added after the last spatiotemporal consistency network to obtain a video detection model. In this way, after the video frame of the video to be detected is input into the video detection model, the feature extraction layer can extract features from the video frame to obtain each feature sequence, and the feature sequence is input into the spatiotemporal consistency network connected in cascade order to obtain the above self-attention features and consistency features output by the last spatiotemporal consistency network, and the obtained features are input into the pooling and classification layer to obtain the detection result of whether the video to be detected is a forged video.

[0227] Corresponding to the above-mentioned video detection method, an embodiment of the present application also provides a video detection device.

[0228] See also Fig.14 , is a schematic diagram of the structure of a video detection device provided in an embodiment of the present application, the device includes the following modules:

[0229] The first feature acquisition module 1401 is used to obtain image features of a face area in a video frame of a video to be detected;

[0230] The feature segmentation module 1402 is used to segment the obtained image features in the spatial dimension and use the segmentation results as a feature sequence, and segment the obtained image features in the time dimension and use the segmentation results as a feature sequence;

[0231] The second feature acquisition module 1403 is used to obtain, for any feature value in any feature sequence, a self-attention feature representing the importance of the feature value among all feature values ​​in the feature sequence, and obtain a consistency feature representing the consistency between the feature value and a target feature value, wherein the target feature value is: other feature values ​​in the feature sequence except the feature value;

[0232] The video detection module 1404 is used to detect whether the video to be detected is a forged video based on the obtained self-attention features and consistency features.

[0233] It can be seen from the above that when the solution provided by the embodiment of the present application is used for video detection, the image features of the face area in the video frame of the video to be detected are obtained, and the obtained image features are segmented in the spatial dimension and the time dimension respectively, and the segmentation results are used as a feature sequence. In this way, for any feature value in any feature sequence, a self-attention feature characterizing the importance of the feature value in all feature values ​​in the feature sequence can be obtained, and a consistency feature characterizing the degree of consistency between the feature value and the target feature value other than the feature value in the feature sequence can be obtained, so that whether the video to be detected is a forged video can be successfully detected based on the obtained self-attention feature and the consistency feature.

[0234] Among them, the above-mentioned feature sequence is obtained by segmenting the obtained image features in the spatial dimension and the temporal dimension respectively. Therefore, the above-mentioned feature sequence can reflect the feature changes of the face area in the video frame in the spatial dimension, and can reflect the feature changes of the face area in the video frame in the temporal dimension. Furthermore, when performing video detection based on the above-mentioned feature sequence, the feature changes of the face area in the spatial and temporal dimensions are comprehensively considered, so that the feature sequence is more sensitive to the mutation of the space-time dimension, and can comprehensively reflect the discontinuity, mutation, etc. of the face in the video frame of the video to be detected in the temporal and spatial dimensions, thereby improving the accuracy of video detection.

[0235] In addition, the self-attention feature characterizes the importance of each feature value in the feature sequence among all the feature values ​​in the feature sequence, and the consistency feature characterizes the consistency between each feature value in the feature sequence and the target feature value other than the feature value. Based on the above two features, the slight distortion of the face in the video frame of the video to be detected can be captured more sensitively in the two dimensions of feature importance and feature consistency, thereby further improving the accuracy of video detection.

[0236] In one embodiment of the present application, the second feature obtaining module 1403 includes:

[0237] A predicted feature value obtaining submodule is used to obtain a predicted feature value of any feature value in any feature sequence based on a target feature value in the feature sequence;

[0238] The first feature acquisition submodule is used to obtain a self-attention feature that characterizes the importance of the feature value among all the feature values ​​in the feature sequence, and to obtain a consistency feature that characterizes the degree of consistency between the feature value and the target feature value based on the predicted feature value and the feature value.

[0239] It can be seen that the above-mentioned predicted eigenvalue is a eigenvalue obtained based on the target eigenvalue other than the eigenvalue, and the degree of consistency between the predicted eigenvalue and the target eigenvalue is relatively high. Therefore, based on the above-mentioned predicted eigenvalue, the degree of consistency between the eigenvalue and the target eigenvalue can be indirectly evaluated, so that the consistency feature characterizing the degree of consistency between the eigenvalue and the target eigenvalue can be accurately obtained.

[0240] In one embodiment of the present application, after the feature segmentation module 1402, the following is further included:

[0241] A vector acquisition module, used to obtain a query vector, a key vector and a value vector of each feature value in the feature sequence based on a preset query parameter matrix, a key parameter matrix and a value parameter matrix;

[0242] The predicted feature value acquisition submodule is specifically used to obtain the first weight of each target feature value for the feature value based on the similarity between the query vector of the feature value and the key vector of the target feature value; according to the first weight of each target feature value for the feature value, the value vectors of each target feature value are weighted summed to obtain the predicted feature value of the feature value.

[0243] It can be seen that when the solution provided in the embodiment of the present application is applied to obtain the predicted eigenvalue of the eigenvalue, the first weight of each target eigenvalue for the eigenvalue is obtained based on the similarity between the query vector of the eigenvalue and the key vector of the target eigenvalue. It can be seen that the first weight is related to the key vector of each target eigenvalue. Then, according to the first weight of each target eigenvalue for the eigenvalue, the value vectors of each target eigenvalue are weighted and summed, and the weighted summation result is related to the value vector of each target eigenvalue, which is equivalent to comprehensively considering the key vectors and value vectors of each target eigenvalue except the eigenvalue to predict the eigenvalue, so that the predicted eigenvalue of the eigenvalue can be obtained more accurately.

[0244] In one embodiment of the present application, the predicted feature value obtaining submodule is specifically used to obtain the first weight of each target feature value for the feature value based on the similarity between the query vector of the feature value and the key vector of the target feature value; for each target feature value, the predicted feature value of the feature value is obtained according to the following expression: o : Among them, p and s represent the serial number of the eigenvalue, e represents the natural base, and qo The query vector representing the feature value, k p The key vector representing the target feature value, k s represents the key vector of any eigenvalue, v p A vector of values ​​representing the target eigenvalues, Indicates the first weight of the target eigenvalue for this eigenvalue.

[0245] In one embodiment of the present application, the first feature acquisition submodule is specifically used to obtain a self-attention feature that characterizes the importance of the feature value among all the feature values ​​in the feature sequence, and obtain the difference between the predicted feature value and the value vector of the feature value, and use the difference as a consistency feature that characterizes the degree of consistency between the feature value and the target feature value.

[0246] The difference between the predicted eigenvalue and the eigenvalue can accurately measure the consistency between the eigenvalue and the predicted eigenvalue. Since the above-mentioned predicted eigenvalue is a eigenvalue obtained based on the target eigenvalue other than the eigenvalue, the consistency between the predicted eigenvalue and the target eigenvalue is relatively high. Therefore, the above-mentioned difference can also accurately measure the consistency between the eigenvalue and the target eigenvalue, so that a more accurate consistency feature can be obtained based on the above-mentioned difference.

[0247] In one embodiment of the present application, after the feature segmentation module 1402, the following is further included:

[0248] A vector acquisition module, used to obtain a query vector, a key vector and a value vector of each feature value in the feature sequence based on a preset query parameter matrix, a key parameter matrix and a value parameter matrix;

[0249] The second feature acquisition module is specifically used to obtain a second weight for each eigenvalue based on the similarity between the query vector of the eigenvalue and the key vector of each eigenvalue in the feature sequence; according to the second weight of each eigenvalue, the value vectors of each eigenvalue are weighted summed to obtain a self-attention feature characterizing the importance of the eigenvalue among all eigenvalues ​​in the feature sequence, and to obtain a consistency feature characterizing the degree of consistency between the eigenvalue and the target eigenvalue.

[0250] It can be seen that when the scheme provided in the embodiment of the present application is applied to obtain the self-attention feature that characterizes the importance of the eigenvalue in all the eigenvalues ​​of the feature sequence, the second weight of each eigenvalue for the eigenvalue is obtained based on the similarity between the query vector of the eigenvalue and the key vector of each eigenvalue in the feature sequence. It can be seen that the second weight is related to the key vector of each eigenvalue. Then, according to the second weight of each eigenvalue for the eigenvalue, the value vectors of each eigenvalue are weighted and summed, and the weighted summation result is related to the value vector of each eigenvalue. This is equivalent to comprehensively considering the key vector and value vector of each eigenvalue in the feature sequence to obtain the self-attention feature of the eigenvalue, so that the above self-attention feature can better characterize the importance of the eigenvalue in all the eigenvalues ​​of the feature sequence, that is, a more accurate self-attention feature can be obtained.

[0251] In one embodiment of the present application, the feature segmentation module 1402 is specifically used to segment each image feature obtained by rows in the horizontal direction, and use the segmentation result as a feature sequence; for each image feature obtained, segment the feature image by columns in the vertical direction, and use the segmentation result as a feature sequence; segment the image features obtained in the time dimension, and use the segmentation result as a feature sequence.

[0252] In this way, the image is finely segmented from the row and column dimensions respectively, and the segmentation results are used as the feature sequence, so that the obtained feature sequence can better reflect the feature changes of the face area in the horizontal direction, and can better reflect the feature changes of the face area in the vertical direction.

[0253] In one embodiment of the present application, the feature segmentation module 1402 is specifically used to: segment the obtained image features in the spatial dimension and use the segmentation results as a feature sequence; determine the feature values ​​corresponding to the same area in the obtained image features, and obtain a feature sequence based on the determined feature values.

[0254] Since the obtained image features are image features extracted from the face region of each video frame of the video to be detected, there is a corresponding relationship between the region in the obtained image features and the face region, so that the feature value corresponding to the same region in the obtained image features reflects the regional features of the same region in the face region of each video frame. Since each video frame is temporally related, the feature sequence obtained based on the determined feature value can better reflect the feature changes of the face region of each video frame in the time dimension.

[0255] In one embodiment of the present application, the second feature obtaining module 1403 includes:

[0256] The second feature acquisition submodule is used to input each feature sequence into a pre-trained spatiotemporal consistency network to obtain output features, and based on the output features, obtain self-attention features that characterize the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and consistency features that characterize the degree of consistency between each feature value in each feature sequence and the target feature value, wherein the spatiotemporal consistency network includes: a first normalization layer, a self-attention and consistency feature extraction layer, a second normalization layer and a multi-layer perceptron MLP layer; the first normalization layer is used to normalize each input feature sequence and output a normalized feature sequence; the self-attention and consistency feature extraction layer is used to feature encode the normalized feature sequence and output the encoding result; the second normalization layer is used to normalize the encoding result and output the normalized encoding result; the MLP layer is used to feature map the normalized encoding result to obtain output features.

[0257] It can be seen that the various network layers in the above-mentioned spatiotemporal consistency network can perform feature transformation on the input feature sequence through mutual cooperation, and finally obtain the output features, so that based on the output features, the self-attention features that characterize the importance of each feature value in each feature sequence among all the feature values ​​of the feature sequence, as well as the consistency features that characterize the degree of consistency between each feature value in each feature sequence and the target feature value can be conveniently and accurately obtained.

[0258] In one embodiment of the present application, the second feature acquisition submodule is specifically used to input each feature sequence into a spatiotemporal consistency network that is cascaded in sequence to obtain the output features of the last spatiotemporal consistency network as a self-attention feature that characterizes the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and a consistency feature that characterizes the degree of consistency between each feature value in each feature sequence and the target feature value.

[0259] In this way, by using multiple spatiotemporal consistency networks connected in cascade order, deeper feature mining of feature sequences can be performed, thereby obtaining more excellent self-attention features and consistency features with spatiotemporal expression capabilities.

[0260] In one embodiment of the present application, the spatiotemporal consistency network is trained in the following manner:

[0261] Obtain sample image features of the face area in the video frame of the sample video, train a preset network based on the sample image features, and obtain self-attention features and consistency features output by the preset network; determine the loss value of the preset network based on the self-attention features and consistency features; adjust the network parameters of the preset network based on the loss value to obtain a spatiotemporal consistency network; wherein the sample video is generated by:

[0262] Obtain key points of a face in a first sample video frame of an original video, and capture a real face image from the first sample video frame based on the obtained key points; add disturbance to the real face image to obtain a disturbed face image, and obtain a sample video based on the real face image and the disturbed face image; and / or generate a sample video based on the real face image, whose frame arrangement order is different from the original arrangement order, wherein the original arrangement order is: the arrangement order of the first sample video frames to which each real face image belongs in the original video.

[0263] In this way, after adding disturbance to the real face image, the perturbed face image can better simulate the face image obtained after the real face is tampered with, and then based on the real face image and the perturbed face image, a sample video that is closer to the video with the face area tampered with can be obtained. In addition, the face image in the sample video obtained based on the real face image with a different arrangement order from the original has a larger jitter in the time dimension, which is consistent with the change of the face image in the time dimension in the forged video with the face area tampered with, which is conducive to obtaining a sample video that is closer to the video with the face area tampered with.

[0264] In one embodiment of the present application, the loss value L con Obtained according to the following expression: Where P(i) represents the positive sample set containing sample image features corresponding to the real video, p represents any sample image feature in the positive sample set, A(i) represents the entire sample set containing all sample image features, a represents any sample image feature in the entire sample set, i∈real represents i is any sample image feature in the positive sample set, F i represents the feature corresponding to i, F p represents the feature corresponding to p, F a Indicates the feature corresponding to a.

[0265] In the solutions provided in the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are all carried out under the premise of the user's knowledge and authorization, and comply with the relevant laws and regulations and do not violate public order and good morals.

[0266] It should be noted that the two-dimensional face images in this embodiment come from a public data set.

[0267] The present application also provides an electronic device, such as Fig.15 As shown, including:

[0268] Memory 1501, used for storing computer programs;

[0269] The processor 1502 is used to implement the above-mentioned video detection method when executing the program stored in the memory 1501.

[0270] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1502, the communication interface, and the memory 1501 communicate with each other via the communication bus.

[0271] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0272] The communication interface is used for communication between the above electronic device and other devices.

[0273] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0274] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0275] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video detection method is implemented.

[0276] In another embodiment provided in the present application, a computer program product including instructions is also provided, which enables the computer to execute the above-mentioned video detection method when it is run on a computer.

[0277] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or other media (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0278] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0279] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0280] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A video detection method, characterized in that: The method comprises: Obtaining image features of a face region in a video frame of a video to be detected; The obtained image features are segmented in the spatial dimension, and the segmentation results are used as feature sequences, and the obtained image features are segmented in the time dimension, and the segmentation results are used as feature sequences; For any feature value in any feature sequence, a self-attention feature characterizing the importance of the feature value in all feature values ​​in the feature sequence is obtained, and a consistency feature characterizing the consistency between the feature value and a target feature value is obtained, wherein the target feature value is: other feature values ​​in the feature sequence except the feature value; Based on the obtained self-attention features and consistency features, it is detected whether the video to be detected is a forged video.

2. The method according to claim 1, characterized in that The obtaining of a consistency feature characterizing the degree of consistency between the feature value and the target feature value includes: Based on the target feature value in the feature sequence, a predicted feature value of the feature value is obtained; Based on the predicted feature value and the feature value, a consistency feature characterizing the degree of consistency between the feature value and the target feature value is obtained.

3. The method according to claim 2, characterized in that After obtaining the feature sequence, it also includes: Based on the preset query parameter matrix, key parameter matrix and value parameter matrix, a query vector, a key vector and a value vector are obtained for each feature value in the feature sequence; The step of obtaining a predicted feature value of the feature value based on the target feature value in the feature sequence includes: Obtaining a first weight of each target eigenvalue for the eigenvalue based on a similarity between a query vector of the eigenvalue and a key vector of the target eigenvalue; According to the first weight of each target eigenvalue for the eigenvalue, a weighted sum is performed on the value vectors of each target eigenvalue to obtain a predicted eigenvalue of the eigenvalue.

4. The method according to claim 3, characterized in that The step of performing weighted summation on the value vectors of each target eigenvalue according to the first weight of the eigenvalue for each target eigenvalue to obtain the predicted eigenvalue of the eigenvalue includes: For each target feature value, the predicted feature value pre is obtained according to the following expression: o : Among them, p and s represent the serial number of the eigenvalue, e represents the natural base, and q o The query vector representing the feature value, k p The key vector representing the target feature value, k s represents the key vector of any eigenvalue, v p A vector of values ​​representing the target eigenvalues, Indicates the first weight of the target eigenvalue for this eigenvalue.

5. The method according to claim 3, characterized in that: The step of obtaining, based on the predicted feature value and the feature value, a consistency feature characterizing the degree of consistency between the feature value and the target feature value comprises: A difference between the predicted eigenvalue and the value vector of the eigenvalue is obtained, and the difference is used as a consistency feature characterizing the degree of consistency between the eigenvalue and the target eigenvalue.

6. The method according to claim 1, characterized in that After obtaining the feature sequence, it also includes: Based on the preset query parameter matrix, key parameter matrix and value parameter matrix, a query vector, a key vector and a value vector are obtained for each feature value in the feature sequence; The step of obtaining a self-attention feature representing the importance of the feature value among all feature values ​​of the feature sequence includes: Based on the similarity between the query vector of the feature value and the key vector of each feature value in the feature sequence, obtaining a second weight of each feature value for the feature value; According to the second weight of each eigenvalue for the eigenvalue, a weighted sum is performed on the value vectors of each eigenvalue to obtain a self-attention feature representing the importance of the eigenvalue in all eigenvalues ​​of the feature sequence.

7. The method according to any one of claims 1 to 6, characterized in that The step of segmenting the obtained image features in the spatial dimension and taking the segmentation results as a feature sequence includes: For each image feature obtained, the image feature is segmented by row in the horizontal direction, and the segmentation result is used as a feature sequence; For each image feature obtained, the feature image is segmented by columns in the vertical direction, and the segmentation result is used as a feature sequence.

8. The method according to any one of claims 1 to 6, characterized in that The step of segmenting the obtained image features in the time dimension and taking the segmentation results as a feature sequence includes: The feature values ​​corresponding to the same area in the obtained image features are determined, and a feature sequence is obtained based on the determined feature values.

9. The method according to any one of claims 1 to 6, characterized in that: The method of obtaining, for any feature value in any feature sequence, a self-attention feature representing the importance of the feature value in all feature values ​​of the feature sequence, and obtaining a consistency feature representing the degree of consistency between the feature value and a target feature value, includes: Input each feature sequence into a pre-trained spatiotemporal consistency network to obtain output features, and obtain a self-attention feature characterizing the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence based on the output features, and a consistency feature characterizing the consistency between each feature value in each feature sequence and a target feature value, wherein the spatiotemporal consistency network includes: a first normalization layer, a self-attention and consistency feature extraction layer, a second normalization layer, and a multi-layer perceptron MLP layer; The first normalization layer is used to normalize each input feature sequence and output a normalized feature sequence; The self-attention and consistency feature extraction layer is used to perform feature encoding on the normalized feature sequence and output the encoding result; The second normalization layer is used to perform normalization processing on the encoding result and output a normalized encoding result; The MLP layer is used to perform feature mapping on the normalized encoding result to obtain output features.

10. The method according to claim 9, characterized in that The step of inputting each feature sequence into a pre-trained spatiotemporal consistency network to obtain output features, and obtaining self-attention features characterizing the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence based on the output features, and consistency features characterizing the degree of consistency between each feature value in each feature sequence and a target feature value, including: Each feature sequence is input into a spatiotemporal consistency network that is cascaded in sequence to obtain the output features of the last spatiotemporal consistency network as a self-attention feature that characterizes the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, as well as a consistency feature that characterizes the degree of consistency between each feature value in each feature sequence and the target feature value.

11. The method according to claim 9, characterized in that The spatiotemporal consistency network is trained in the following way: Obtaining sample image features of a face region in a video frame of a sample video, training a preset network based on the sample image features, and obtaining self-attention features and consistency features output by the preset network; Determining a loss value of the preset network based on the self-attention feature and the consistency feature; Adjusting the network parameters of the preset network based on the loss value to obtain a spatiotemporal consistency network; The sample video is generated in the following way: Obtain key points of a human face in a first sample video frame of an original video, and capture a real human face image from the first sample video frame based on the obtained key points; Adding disturbance to the real face image to obtain a disturbed face image, and obtaining a sample video based on the real face image and the disturbed face image; and / or generating a sample video with an inter-frame arrangement order different from the original arrangement order based on the real face image, wherein the original arrangement order is: the arrangement order of the first sample video frames to which each real face image belongs in the original video.

12. The method according to claim 11, characterized in that The loss value L con Obtained according to the following expression: Where P(i) represents the positive sample set containing sample image features corresponding to the real video, p represents any sample image feature in the positive sample set, A(i) represents the entire sample set containing all sample image features, a represents any sample image feature in the entire sample set, i∈real represents i is any sample image feature in the positive sample set, F i represents the feature corresponding to i, F p represents the feature corresponding to p, F a Indicates the feature corresponding to a.

13. A video detection device, characterized in that: The device comprises: The first feature acquisition module is used to obtain image features of the face area in the video frame of the video to be detected; A feature segmentation module is used to segment the obtained image features in the spatial dimension and use the segmentation results as a feature sequence, and to segment the obtained image features in the time dimension and use the segmentation results as a feature sequence; The second feature acquisition module is used to obtain, for any feature value in any feature sequence, a self-attention feature representing the importance of the feature value in all feature values ​​in the feature sequence, and obtain a consistency feature representing the consistency between the feature value and a target feature value, wherein the target feature value is: other feature values ​​in the feature sequence except the feature value; The video detection module is used to detect whether the video to be detected is a forged video based on the obtained self-attention features and consistency features.

14. The device according to claim 13, characterized in that The second feature acquisition module includes: a predicted feature value acquisition submodule, which is used to obtain a predicted feature value of any feature value in any feature sequence based on a target feature value in the feature sequence; a first feature acquisition submodule, which is used to obtain a self-attention feature representing the importance of the feature value among all feature values ​​in the feature sequence, and obtain a consistency feature representing the consistency between the feature value and the target feature value based on the predicted feature value and the feature value; or After the feature segmentation module, it also includes: a vector acquisition module, which is used to obtain the query vector, key vector and value vector of each feature value in the feature sequence based on the preset query parameter matrix, key parameter matrix and value parameter matrix; the predicted feature value acquisition submodule is specifically used to obtain the first weight of each target feature value for the feature value based on the similarity between the query vector of the feature value and the key vector of the target feature value; according to the first weight of each target feature value for the feature value, the value vectors of each target feature value are weighted summed to obtain the predicted feature value of the feature value; or The predicted feature value obtaining submodule is specifically used to obtain the first weight of each target feature value for the feature value based on the similarity between the query vector of the feature value and the key vector of the target feature value; for each target feature value, the predicted feature value of the feature value is obtained according to the following expression: o : Among them, p and s represent the serial number of the eigenvalue, e represents the natural base, and q o The query vector representing the feature value, k p The key vector representing the target feature value, k s represents the key vector of any eigenvalue, v p A vector of values ​​representing the target eigenvalues, Represents the first weight of the target eigenvalue for this eigenvalue; or The first feature acquisition submodule is specifically used to obtain a self-attention feature that characterizes the importance of the feature value among all feature values ​​in the feature sequence, and obtain a difference between the predicted feature value and the value vector of the feature value, and use the difference as a consistency feature that characterizes the degree of consistency between the feature value and the target feature value; or After the feature segmentation module, it also includes: a vector acquisition module, which is used to obtain the query vector, key vector and value vector of each eigenvalue in the feature sequence based on the preset query parameter matrix, key parameter matrix and value parameter matrix; the second feature acquisition module is specifically used to obtain the second weight of each eigenvalue for the eigenvalue based on the similarity between the query vector of the eigenvalue and the key vector of each eigenvalue in the feature sequence; according to the second weight of each eigenvalue for the eigenvalue, the value vectors of each eigenvalue are weighted summed to obtain a self-attention feature that characterizes the importance of the eigenvalue in all eigenvalues ​​in the feature sequence, and obtain a consistency feature that characterizes the consistency between the eigenvalue and the target eigenvalue; or The feature segmentation module is specifically used to segment the image feature by rows in the horizontal direction for each obtained image feature, and use the segmentation result as a feature sequence; segment the feature image by columns in the vertical direction for each obtained image feature, and use the segmentation result as a feature sequence; segment the image feature in the time dimension, and use the segmentation result as a feature sequence; or The feature segmentation module is specifically used to: segment the obtained image features in the spatial dimension and use the segmentation results as a feature sequence; determine the feature values ​​corresponding to the same area in the obtained image features, and obtain a feature sequence based on the determined feature values; or The second feature acquisition module includes: a second feature acquisition submodule, which is used to input each feature sequence into a pre-trained spatiotemporal consistency network to obtain output features, and based on the output features, obtain a self-attention feature that characterizes the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and a consistency feature that characterizes the degree of consistency between each feature value in each feature sequence and a target feature value, wherein the spatiotemporal consistency network includes: a first normalization layer, a self-attention and consistency feature extraction layer, a second normalization layer and a multi-layer perceptron MLP layer; the first normalization layer is used to normalize each input feature sequence and output a normalized feature sequence; the self-attention and consistency feature extraction layer is used to feature encode the normalized feature sequence and output an encoding result; the second normalization layer is used to normalize the encoding result and output a normalized encoding result; the MLP layer is used to feature map the normalized encoding result to obtain an output feature; or The second feature acquisition submodule is specifically used to input each feature sequence into the spatiotemporal consistency network connected in cascade order, and obtain the output features of the last spatiotemporal consistency network as the self-attention features representing the importance of each feature value in each feature sequence among all feature values ​​in the feature sequence, and the consistency features representing the consistency between each feature value in each feature sequence and the target feature value; or The spatiotemporal consistency network is trained in the following manner: obtaining sample image features of the face area in the video frame of the sample video, training the preset network based on the sample image features, and obtaining the self-attention features and consistency features output by the preset network; determining the loss value of the preset network based on the self-attention features and consistency features; adjusting the network parameters of the preset network based on the loss value to obtain the spatiotemporal consistency network; wherein the sample video is generated in the following manner: obtaining the key points of the face in the first sample video frame of the original video, and capturing a real face image from the first sample video frame based on the obtained key points; adding disturbance to the real face image to obtain a disturbed face image, and obtaining a sample video based on the real face image and the disturbed face image; and / or generating a sample video whose frame arrangement order is different from the original arrangement order based on the real face image, wherein the original arrangement order is: the arrangement order of the first sample video frames to which each real face image belongs in the original video; or The loss value L con Obtained according to the following expression: Where P(i) represents the positive sample set containing sample image features corresponding to the real video, p represents any sample image feature in the positive sample set, A(i) represents the entire sample set containing all sample image features, a represents any sample image feature in the entire sample set, i∈real represents i is any sample image feature in the positive sample set, F i represents the feature corresponding to i, F p represents the feature corresponding to p, F a Indicates the feature corresponding to a.

15. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-12 when executing a program stored in a memory.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images