Method and device for identifying fake human video, electronic equipment and storage medium

Abstract

The application provides a method and device for identifying a fake human video, electronic equipment and a storage medium. The method comprises: performing frame extraction preprocessing on a target human video to be identified to obtain a plurality of groups of continuous frames; inputting the plurality of groups of continuous frames into an identification model to obtain an identification result output by the identification model; wherein the identification model is trained based on a training sample set composed of real human video samples and fake human video samples; the identification model comprises a space-time attention branch, an optical flow modulation module, an action guide branch and a classifier; the space-time attention branch is used for performing video segment-level feature extraction on the input plurality of groups of continuous frames based on a space-time attention mechanism to obtain first features; the optical flow modulation module is used for performing optical flow modulation on the input plurality of groups of continuous frames; the action guide branch is used for performing frame-by-frame feature extraction on each frame obtained by the optical flow modulation to obtain second features; and the classifier is used for outputting the identification result based on the first features and the second features.

Description

Technical Field

[0001] This invention relates to the field of deepfake technology, and in particular to a method, apparatus, electronic device, and storage medium for identifying forged human body videos. Background Technology

[0002] In recent years, with the rise of generative adversarial networks and diffusion models, deep generative methods have made significant progress. Their low cost, high customizability and scalability, and wide propagation make these generative models highly susceptible to malicious use for identity forgery. To mitigate the potentially detrimental effects of such fake identity forgery, researchers have proposed numerous related face forgery datasets and authentication models. However, human identity information is embedded in multiple modalities, not just the face.

[0003] For example, a class of human image animation methods based on Latent Diffusion Models (LDMs) has recently attracted widespread attention. These methods take a static human image indicating the person's identity as input and use a sequence of actions to drive the image and generate a corresponding video. Currently, this method is mainly applied to entertainment purposes such as dance production, but like face spoofing, it has the potential to be abused for malicious identity forgery. Therefore, improving the ability to identify forged human images is a pressing technical problem that the industry needs to solve. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for identifying fake human body videos, thereby improving the ability to identify fake human body videos.

[0005] In a first aspect, the present invention provides a method for identifying forged human body videos, comprising: The target human video to be identified is subjected to frame extraction preprocessing to obtain multiple sets of continuous frames; The multiple sets of consecutive frames are input into the discrimination model to obtain the discrimination result output by the discrimination model. The discrimination result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on a spatiotemporal attention mechanism to obtain a first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame-by-frame from each frame obtained by optical flow modulation to obtain a second feature. The classifier outputs the identification result based on the first and second features.

[0006] In some embodiments, the generation of the forged human video samples includes: Based on the pose estimation method, the motion pose information of the human body is extracted from each of the real human body video samples; Based on various human image animation methods, fake human video samples are generated using the identity information and action posture information of the human body in each of the real human video samples. The fake human video samples include fake human video samples with mismatched identity information and action posture information, as well as fake human video samples with matched identity information and action posture information.

[0007] In some embodiments, the generation of forged human video samples with mismatched identity information and gesture information includes: Each of the aforementioned real human video samples is paired up, and then for the same pair of real human video samples, mismatched combinations of identity information and action / pose information are performed to generate forged human video samples with mismatched identity information and action / pose information; or...

[0008] Each of the aforementioned real human video samples is numbered, and then mismatched combinations of identity information and action / pose information are performed in an orderly rotation according to the numbering to generate a forged human video sample with mismatched identity information and action / pose information; or...

[0009] Each real human video sample is subjected to a fully connected mismatched combination of identity information and action posture information to generate a fake human video sample with mismatched identity information and action posture information.

[0010] In some embodiments, the optical flow modulation of multiple sets of input consecutive frames includes: For each target frame in the multiple sets of consecutive frames, optical flow estimation is performed using the target frame and the next frame of the target frame as input to obtain the horizontal velocity estimate and vertical velocity estimate of each pixel in the target frame; Based on the estimated horizontal velocity and the estimated vertical velocity, the rate of each pixel in the target frame is determined; The color value of each pixel in the target frame is adjusted based on the rate of each pixel in the target frame.

[0011] In some embodiments, the step of performing frame-stripping preprocessing on the target human video to be identified to obtain multiple sets of consecutive frames includes: Multiple starting frames are selected at uniform intervals, and then frames are extracted from several consecutive frames after each starting frame based on the required number of frames for each group to obtain multiple groups of consecutive frames.

[0012] In some embodiments, the frame-sampling preprocessing method for training samples during the training of the discrimination model includes: Multiple start frames are randomly selected, and then frames are extracted from several consecutive frames following each start frame based on the required number of frames for each group.

[0013] Secondly, the present invention also provides a device for identifying forged human body videos, comprising: The preprocessing module is used to perform frame extraction preprocessing on the target human video to be identified, and obtain multiple sets of continuous frames; The identification module is used to input the multiple sets of consecutive frames into the identification model and obtain the identification result output by the identification model. The identification result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on a spatiotemporal attention mechanism to obtain a first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame-by-frame from each frame obtained by optical flow modulation to obtain a second feature. The classifier outputs the identification result based on the first and second features.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for identifying fake human videos as described in the first aspect above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying forged human body videos as described in the first aspect above.

[0016] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for identifying forged human body videos as described in the first aspect above.

[0017] The present invention provides a method, apparatus, electronic device, and storage medium for identifying fake human videos. After preprocessing the target human video by extracting frames, multiple sets of continuous frames are input into an identification model for identification. This model, on one hand, extracts video segment-level features from the input multiple sets of continuous frames based on a spatiotemporal attention mechanism to obtain a first feature; on the other hand, it modulates the input multiple sets of continuous frames with optical flow, and then extracts features frame-by-frame from each frame obtained by optical flow modulation to obtain a second feature. Finally, it combines the first and second features to output the identification result. This method effectively utilizes the spatiotemporal relationships and optical flow changes in the video to identify fake human videos, thereby greatly improving the model's identification capability and effectively filling the gap in the field of fake human identification. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the method for identifying fake human body videos provided by the present invention.

[0020] Figure 2 A schematic diagram of the internal processing of the identification model provided for the invention.

[0021] Figure 3 One of the example images for generating fake human body video samples provided for the invention.

[0022] Figure 4 Example image 2 of the generation of fake human body video samples provided for the invention.

[0023] Figure 5 This is a schematic diagram of the structure of the device for identifying fake human body videos provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The following is combined Figures 1-6 Embodiments of the present invention are described.

[0027] Figure 1 This is a flowchart illustrating the method for identifying forged human body videos provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101 and 102.

[0028] Step 101: Perform frame extraction preprocessing on the target human video to be identified to obtain multiple sets of continuous frames.

[0029] Step 102: Input multiple sets of consecutive frames into the discrimination model to obtain the discrimination result output by the discrimination model. The discrimination result is used to indicate whether the target human body video is a real human body video or a fake human body video.

[0030] The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The identification model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on the spatiotemporal attention mechanism to obtain the first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame by frame from each frame obtained by optical flow modulation to obtain the second feature. The classifier is used to output the identification result based on the first feature and the second feature.

[0031] Specifically, the present invention uses a trained identification model to identify fake human videos. Before inputting the model, the target human video to be identified needs to be preprocessed by frame extraction to obtain multiple sets of continuous frames.

[0032] In some embodiments, the target human video to be identified is subjected to frame extraction preprocessing to obtain multiple sets of continuous frames, including: selecting multiple starting frames at uniform intervals, and then extracting frames from several consecutive frames after each starting frame based on the required number of frames for each set to obtain multiple sets of continuous frames.

[0033] For example, assuming the uniform interval is set to 10 frames, the 1st, 11th, 21st, and 31st frames of the video can be selected as the starting frames. Then, frames are extracted according to the required number of frames for each group. Assuming the required number of frames for each group is 5 frames, the 1st to 5th frames can be selected as the first group of consecutive frames, the 11th to 15th frames as the second group of consecutive frames, the 21st to 25th frames as the third group of consecutive frames, the 31st to 35th frames as the fourth group of consecutive frames, and so on, thus obtaining multiple groups of consecutive frames.

[0034] It should be noted that in this invention, the processing of multiple sets of continuous frames obtained after frame extraction preprocessing into the discrimination model is consistent in both the training and inference stages, while the frame extraction preprocessing methods adopted in the training and inference stages may be different.

[0035] In some embodiments, the frame-sampling preprocessing method for training samples during the training process of the discrimination model includes: randomly selecting multiple starting frames, and then extracting frames from several consecutive frames after each starting frame based on the required number of frames for each group.

[0036] For example, a random sampling method conforming to a uniform distribution can be used to randomly select multiple starting frames. Then, frames are sampled according to the number of frames required for each group. Assuming that the randomly selected starting frames are the 3rd, 5th, 12th, 16th, etc., and the number of frames required for each group is set to 5, then frames 3 to 7 can be selected as the first group of consecutive frames, frames 5 to 9 as the second group of consecutive frames, frames 12 to 16 as the third group of consecutive frames, frames 16 to 20 as the fourth group of consecutive frames, and so on, thus obtaining multiple groups of consecutive frames.

[0037] It should be noted that the human body video described in this invention includes at least one central figure exhibiting obvious physical movements.

[0038] Figure 2 This is a schematic diagram of the internal processing of the identification model provided by the present invention, such as... Figure 2 As shown, the discrimination model provided by this invention includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The original video segment in the figure refers to multiple sets of continuous frames. On one hand, the original video segment is input into the spatiotemporal attention branch to extract the classification token at the video segment level, obtaining the first feature. On the other hand, the original video segment is first optically modulated by the optical flow modulation module to obtain the action guidance frame. Then, the action guidance frame is input into the action guidance branch for frame-by-frame feature extraction and combination (e.g., concatenating or adding the features extracted from each frame) to obtain the second feature. Finally, the first and second features are input into the classifier. The classifier balances the features obtained from the two branches in terms of parameters and combines them (e.g., concatenating or adding the features), and then performs binary classification to obtain the discrimination result.

[0039] Figure 2 In this context, F, C, H, and W refer to Frame, Channel, Height, and Width, respectively. Extra frames are used for optical flow modulation and can be the frame following the last frame in a group of consecutive frames, used to estimate the velocity of each pixel in the last frame of each group of consecutive frames.

[0040] In this invention, the spatiotemporal attention branch employs alternating temporal self-attention and spatial self-attention mechanisms. The temporal self-attention mechanism adopts a similar self-attention mechanism between patches at the same spatial location in each frame; the spatial self-attention mechanism adopts the classic visual Transformer (ViT) mechanism, which adaptively adjusts attention weights between patches in the same frame through queries, keys, and values.

[0041] The spatiotemporal attention branch captures the spatiotemporal relationships in a video, enabling more accurate extraction of video segment-level features. These features are crucial for identifying inconsistencies and anomalous behaviors in fake videos. Furthermore, the spatiotemporal attention mechanism can focus on key regions and time periods in a video, thereby more effectively extracting features related to forgery and improving the accuracy of model identification.

[0042] It should be noted that the specific network structure used in the spatiotemporal attention branch is not limited here. It can be an existing network structure, as long as it is based on the spatiotemporal attention mechanism to extract video segment-level features from multiple consecutive frames.

[0043] The action guidance branch can use any backbone network suitable for computer vision to extract and combine features frame by frame. This invention does not limit the specific network structure used in the action guidance branch.

[0044] In some embodiments, the classifier balances the features obtained from the two branches in terms of parameter quantity. This can be achieved by using a fully connected network to linearly project the features obtained from the two branches, thereby ensuring that the features are consistent in dimensionality.

[0045] In some embodiments, optical flow modulation is performed on multiple sets of consecutive input frames, including: For each target frame in multiple consecutive frames, optical flow estimation is performed using the target frame and the next frame as input to obtain the horizontal and vertical velocity estimates of each pixel in the target frame. Based on the horizontal and vertical velocity estimates, the velocity of each pixel in the target frame is determined. Adjust the color value of each pixel in the target frame based on the rate of each pixel in the target frame.

[0046] Specifically, optical flow estimation uses the smooth changes in color values ​​between adjacent frames to estimate the motion vector field of pixels between frames, analyzing motion in the video and tracking objects. This invention focuses on anomalous changes in color values ​​in pixel space through optical flow estimation, distinguishing between stationary and moving regions. This allows the subsequent backbone network to focus on small-scale moving regions, better improving model efficiency and discrimination accuracy. Figure 2For example, the horizontal and vertical velocities of each pixel (i.e., pixel point) in the first frame can be estimated using the first and second frames of the original video clip.

[0047] For each pixel, the estimated horizontal and vertical velocities are the horizontal and vertical components of its velocity vector. In some embodiments, the rate of each pixel in the target frame is determined based on the estimated horizontal and vertical velocities, which can be achieved by calculating the L2 norm of the velocity vector of each pixel as its rate.

[0048] After obtaining the velocity of each pixel in the target frame, there are several ways to adjust the color value of each pixel based on its velocity. For example, the velocity of each pixel can be used to determine a weight coefficient for each pixel, and then the color value of each pixel can be multiplied by its respective weight coefficient to adjust the color value of each pixel, thereby completing the optical flow modulation of the target frame. Figure 2 The action guide frame shown.

[0049] For example, for a target frame, the maximum and minimum values ​​of the rate of each pixel in the target frame can be counted and denoted as V. max and V min Assume the speed of a certain pixel is V. i Then the weight coefficient of this pixel can be (V max -V i ) / (V max -V min ).

[0050] For training the identification model, this invention uses a training sample set consisting of real human video samples and fake human video samples as training data. The real human video samples can be legally collected and sufficient publicly available video data, and the video data has at least one central figure showing obvious body movements in a single scene. The fake human video samples are generated by a deep generative model.

[0051] In some embodiments, the generation of forged human video samples includes: Based on pose estimation methods, human motion and pose information is extracted from real human video samples. Based on various human image animation methods, fake human video samples are generated using the identity information and action posture information of the human body in each real human video sample. The fake human video samples include fake human video samples with mismatched identity information and action posture information, as well as fake human video samples with matched identity information and action posture information.

[0052] Specifically, the pose estimation method used in this invention is not limited, as long as it can extract the human body's pose information from the video, such as... Figure 3 As shown, pose estimation method 1 can be used to extract the human body's motion and pose information in the video, pose estimation method 2 can be used to extract the human body's motion and pose information in the video, or other pose estimation methods can be used to extract the human body's motion and pose information in the video.

[0053] The identity information of a human being in a video can be represented by a static image of that human being. For example, a single frame of the video can be used as the identity information of the human being in the video.

[0054] Then, various human image animation methods can be employed to generate forged human video samples using the identity and pose information of the human body in each real human video sample. In this invention, the forged human video samples include both those with mismatched identity and pose information and those with matched identity and pose information. This allows the model to shift towards the domain of human forgery detection, thereby enabling the model to exhibit stronger adaptability when faced with unseen human forgery data and improving its generalization ability in identification.

[0055] A forged human video sample with a mismatch between identity information and pose information refers to a sample generated from real human video samples where the identity information and pose information are derived from different real human video samples. For example, a forged human video sample generated using human image animation methods, employing the identity information of a person in real human video sample A and the pose information of a person in real human video sample B, is an example of a forged human video sample with a mismatch between identity information and pose information. Figure 3 A set of samples in the lower right corner.

[0056] A forged human video sample whose identity information and pose information match refers to a forged human video sample whose identity information and pose information are derived from the same real human video sample. For example, a forged human video sample regenerated using human image animation methods based on the identity information and pose information of a person in real human video sample A is a forged human video sample whose identity information and pose information match. Figure 3 The sample in the upper right corner, although the identity information and action posture information of the fake human video sample match, is still a fake and different from real human video samples. Adding such samples to the training sample set to train the discrimination model is beneficial to further improve the model's discrimination ability.

[0057] In actual training, it can be like Figure 4 As illustrated, this method generates a batch of fake human video samples with mismatched identity and pose information, as well as fake human video samples with matched identity and pose information, thereby obtaining a sufficient number of fake human video samples.

[0058] It should be noted that the present invention does not limit the human body image animation method used, as long as it can generate human body video based on the human body's identity information and action posture information.

[0059] In some embodiments, the generation of forged human video samples with mismatched identity information and gesture information includes: (1) Pair each real human video sample in pairs, and then combine the mismatched identity information and action posture information for the same pair of real human video samples to generate fake human video samples with mismatched identity information and action posture information.

[0060] For example, assuming there are 10 real human video samples (this is just an example; the actual number of video samples during training is massive), they can be randomly divided into 5 pairs. The identity information and action / pose information of the human body in the same pair of real human video samples can be swapped. For example, a pair of real human video samples includes real human video sample A and real human video sample B. The identity information of the human body in real human video sample A can be combined with the action / pose information of the human body in real human video sample B to generate a fake human video sample. Similarly, the action / pose information of the human body in real human video sample A can be combined with the identity information of the human body in real human video sample B to generate a fake human video sample.

[0061] (2) Number each real human video sample, and then perform a mismatched combination of identity information and action posture information in an orderly rotation according to the number to generate a fake human video sample with mismatched identity information and action posture information.

[0062] For example, assuming there are 10 real human video samples, they can be numbered sequentially from 1# to 10#. The identity information of the person in sample 1# can be combined with the action and posture information of the person in sample 2# to generate a fake human video sample. Similarly, the identity information of the person in sample 2# can be combined with the action and posture information of the person in sample 3#, and so on. Finally, the identity information of the person in sample 10# can be combined with the action and posture information of the person in sample 1# to generate a fake human video sample, resulting in 10 fake human video samples.

[0063] (3) Perform a fully connected mismatch combination of identity information and action posture information on each real human video sample to generate a fake human video sample with mismatched identity information and action posture information.

[0064] For example, assuming there are 10 real human video samples, for any one real human video sample, the identity information of the human in the real human video sample can be combined with the action and posture information of the human in the other 9 real human video samples, that is, 9 fake human video samples can be generated, and finally 90 fake human video samples can be obtained.

[0065] The method for identifying fake human videos provided by this invention preprocesses the target human video by extracting frames, and then inputs multiple sets of continuous frames into an identification model for identification. This model extracts video segment-level features from the input multiple sets of continuous frames based on a spatiotemporal attention mechanism to obtain a first feature. Simultaneously, it modulates the input multiple sets of continuous frames with optical flow, and then extracts features frame-by-frame from each frame obtained by optical flow modulation to obtain a second feature. Finally, it combines the first and second features to output the identification result. This method effectively utilizes the spatiotemporal relationships and optical flow changes in the video to identify fake human videos, thereby greatly improving the model's identification capability and effectively filling the gap in the field of fake human identification.

[0066] The following describes the device for identifying fake human body videos provided by the present invention. The device for identifying fake human body videos described below can be referred to in correspondence with the method for identifying fake human body videos described above.

[0067] Figure 5 This is a schematic diagram of the structure of the device for identifying fake human body videos provided by the present invention, as shown below. Figure 5 As shown, the device includes: The preprocessing module 510 is used to perform frame extraction preprocessing on the target human video to be identified, and to obtain multiple sets of continuous frames. The identification module 520 is used to input multiple sets of continuous frames into the identification model and obtain the identification result output by the identification model. The identification result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The identification model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on the spatiotemporal attention mechanism to obtain the first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame by frame from each frame obtained by optical flow modulation to obtain the second feature. The classifier is used to output the identification result based on the first feature and the second feature.

[0068] In some embodiments, the generation of forged human video samples includes: Based on pose estimation methods, human motion and pose information is extracted from real human video samples. Based on various human image animation methods, fake human video samples are generated using the identity information and action posture information of the human body in each real human video sample. The fake human video samples include fake human video samples with mismatched identity information and action posture information, as well as fake human video samples with matched identity information and action posture information.

[0069] In some embodiments, the generation of forged human video samples with mismatched identity information and gesture information includes: Each real human video sample is paired up, and then for each pair of real human video samples, mismatched combinations of identity information and action / pose information are performed to generate fake human video samples with mismatched identity and action / pose information; or...

[0070] Each real human video sample is numbered, and then mismatched combinations of identity information and posture information are performed in an orderly rotation according to the number to generate fake human video samples with mismatched identity information and posture information; or...

[0071] The system performs a fully connected mismatched combination of identity information and action / pose information on each real human video sample to generate a fake human video sample with mismatched identity information and action / pose information.

[0072] In some embodiments, optical flow modulation is performed on multiple sets of consecutive input frames, including: For each target frame in multiple consecutive frames, optical flow estimation is performed using the target frame and the next frame as input to obtain the horizontal and vertical velocity estimates of each pixel in the target frame. Based on the horizontal and vertical velocity estimates, the velocity of each pixel in the target frame is determined. Adjust the color value of each pixel in the target frame based on the rate of each pixel in the target frame.

[0073] In some embodiments, the target human video to be identified undergoes frame-stripping preprocessing to obtain multiple sets of consecutive frames, including: Multiple starting frames are selected at uniform intervals, and then frames are extracted from several consecutive frames after each starting frame based on the required number of frames for each group, resulting in multiple groups of consecutive frames.

[0074] In some embodiments, the frame extraction preprocessing method for training samples during the training of the discrimination model includes: Multiple start frames are randomly selected, and then frames are extracted from several consecutive frames following each start frame based on the required number of frames for each group.

[0075] It should be noted that the above-mentioned device for identifying fake human videos provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0076] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the aforementioned method for identifying forged human videos, including: The target human video to be identified is subjected to frame extraction preprocessing to obtain multiple sets of continuous frames; Multiple sets of consecutive frames are input into the discrimination model to obtain the discrimination result output by the discrimination model. The discrimination result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The identification model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on the spatiotemporal attention mechanism to obtain the first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame by frame from each frame obtained by optical flow modulation to obtain the second feature. The classifier is used to output the identification result based on the first feature and the second feature.

[0077] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] It should be noted that the electronic device provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail.

[0079] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for identifying forged human body videos, including: The target human video to be identified is subjected to frame extraction preprocessing to obtain multiple sets of continuous frames; Multiple sets of consecutive frames are input into the discrimination model to obtain the discrimination result output by the discrimination model. The discrimination result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The identification model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on the spatiotemporal attention mechanism to obtain the first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame by frame from each frame obtained by optical flow modulation to obtain the second feature. The classifier is used to output the identification result based on the first feature and the second feature.

[0080] It should be noted that the non-transitory computer-readable storage medium provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0081] In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and which, when executed by a processor, can implement the above-mentioned method for identifying forged human body videos, including: The target human video to be identified is subjected to frame extraction preprocessing to obtain multiple sets of continuous frames; Multiple sets of consecutive frames are input into the discrimination model to obtain the discrimination result output by the discrimination model. The discrimination result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The identification model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on the spatiotemporal attention mechanism to obtain the first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame by frame from each frame obtained by optical flow modulation to obtain the second feature. The classifier is used to output the identification result based on the first feature and the second feature.

[0082] It should be noted that the computer program product provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying forged human body videos, characterized in that, include: The target human video to be identified is subjected to frame extraction preprocessing to obtain multiple sets of continuous frames; The multiple sets of consecutive frames are input into the discrimination model to obtain the discrimination result output by the discrimination model. The discrimination result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on a spatiotemporal attention mechanism to obtain a first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames. The action guidance branch is used to extract features frame-by-frame from each frame obtained by optical flow modulation to obtain a second feature. The classifier outputs an identification result based on the first and second features. The optical flow modulation includes: For each target frame in the multiple sets of consecutive frames, optical flow estimation is performed using the target frame and the next frame of the target frame as input to obtain the horizontal velocity estimate and vertical velocity estimate of each pixel in the target frame; The L2 norm of the velocity vector of each pixel is calculated based on the estimated horizontal and vertical velocities of each pixel in the target frame as the velocity of each pixel. The maximum and minimum velocities of each pixel in the target frame are counted. For a pixel with a velocity Vi, its weight coefficient is calculated as (Vmax-Vi) / (Vmax-Vmin); where Vmax and Vmin are the maximum and minimum velocities of each pixel in the target frame, respectively. The color value of each pixel in the target frame is multiplied by its respective weight coefficient, and the color value of each pixel is adjusted to obtain the action guidance frame. The spatiotemporal attention branch employs alternating temporal self-attention and spatial self-attention mechanisms.

2. The method for identifying forged human body videos according to claim 1, characterized in that, The methods for generating the forged human video samples include: Based on the pose estimation method, the motion pose information of the human body is extracted from each of the real human body video samples; Based on various human image animation methods, fake human video samples are generated using the identity information and action posture information of the human body in each of the real human video samples. The fake human video samples include fake human video samples with mismatched identity information and action posture information, as well as fake human video samples with matched identity information and action posture information.

3. The method for identifying forged human body videos according to claim 2, characterized in that, The methods for generating forged human video samples with mismatched identity information and action / pose information include: Each of the aforementioned real human video samples is paired up, and then for the same pair of real human video samples, mismatched combinations of identity information and action / pose information are performed to generate forged human video samples with mismatched identity information and action / pose information; or... Each of the aforementioned real human video samples is numbered, and then mismatched combinations of identity information and action / pose information are performed in an orderly rotation according to the numbering to generate a forged human video sample with mismatched identity information and action / pose information; or... Each real human video sample is subjected to a fully connected mismatched combination of identity information and action posture information to generate a fake human video sample with mismatched identity information and action posture information.

4. The method for identifying forged human body videos according to any one of claims 1 to 3, characterized in that, The target human video to be identified undergoes frame extraction preprocessing to obtain multiple sets of continuous frames, including: Multiple starting frames are selected at uniform intervals, and then frames are extracted from several consecutive frames after each starting frame based on the required number of frames for each group to obtain multiple groups of consecutive frames.

5. The method for identifying forged human body videos according to any one of claims 1 to 3, characterized in that, The frame extraction preprocessing method for training samples during the training process of the discrimination model includes: Multiple start frames are randomly selected, and then frames are extracted from several consecutive frames following each start frame based on the required number of frames for each group.

6. A device for identifying forged human body videos, employing the method for identifying forged human body videos as described in claim 1, characterized in that, include: The preprocessing module is used to perform frame extraction preprocessing on the target human video to be identified, and obtain multiple sets of continuous frames; The identification module is used to input the multiple sets of consecutive frames into the identification model and obtain the identification result output by the identification model. The identification result is used to indicate whether the target human body video is a real human body video or a fake human body video. The identification model is trained on a training sample set consisting of real human video samples and fake human video samples. The identification model includes a spatiotemporal attention branch, an optical flow modulation module, an action guidance branch, and a classifier. The spatiotemporal attention branch is used to extract video segment-level features from multiple sets of input continuous frames based on the spatiotemporal attention mechanism to obtain the first feature. The optical flow modulation module is used to perform optical flow modulation on multiple sets of input continuous frames; the action guidance branch is used to extract features frame by frame from each frame obtained by optical flow modulation to obtain a second feature; the classifier is used to output a discrimination result based on the first feature and the second feature. The spatiotemporal attention branch employs alternating temporal self-attention and spatial self-attention mechanisms.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for identifying fake human videos as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for identifying fake human videos as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for identifying fake human videos as described in any one of claims 1 to 5.

Images