Synthetic speaker video identity tracing method and device based on space-time causal depolarization

Abstract

The invention relates to the technical field of computer vision and machine learning, in particular to a synthetic speaker video identity tracing method and device based on space-time causal depolarization, and the method comprises the steps: obtaining at least one face key point sequence in a target synthetic speaker video, extracting at least one spatial-temporal feature from the at least one face key point sequence; extracting a driving feature and a target feature of the at least one spatial-temporal feature, and inputting the driving feature and the target feature into a preset time domain convolutional network based on a loss function of multi-target causal constraint optimization to generate dynamic identity embedding of the driving feature; and optimizing dynamic identity embedding by using a target antagonism causal verification mechanism, and generating an identity tracing result of the target synthetic speaker video based on the optimized dynamic identity embedding. Therefore, the problems that face key points are mixed and the accuracy and reliability of driving identity traceability are reduced due to appearance interference of the target identity and inherent errors of a generator in related technologies are solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and machine learning technology, and in particular to a method and apparatus for tracing the identity of synthetic speaking head videos based on spatiotemporal causal bias removal. Background Technology

[0002] In recent years, the rapid development of talking-head video generation technology has enabled anyone to synthesize highly realistic, real-time facial videos with flexible identities. While this technology is driving innovation in applications such as AR / VR and video conferencing, it also brings serious risks of privacy leaks and identity abuse. Malicious users can easily exploit the identities of others (target identities) to create false content, conduct impersonation attacks, and spread misinformation, posing a significant threat to social trust and individual rights.

[0003] Therefore, related technologies have proposed driver identity tracing for speaking head videos. The core objective of this task is to accurately identify the driver identity that drives facial movements in a given synthetic speaking head video, rather than the target identity presented superficially in the video. Completing this task is of great significance for tracing the source of malicious content, determining legal responsibility, and thus curbing identity abuse.

[0004] However, related technologies extract facial key points from video frames and directly manipulate the original facial key point sequences. These sequences suffer from two key types of confounding: static facial appearance interference from the target identity and inherent errors in the generator. These confounding factors undermine the model's ability to extract discriminative identity-driving features. Therefore, the facial key point confounding in speaking head video identity tracing methods suffers from reduced accuracy and reliability due to target identity appearance interference and inherent generator errors, necessitating a solution. Summary of the Invention

[0005] This application is based on the inventor's understanding and insights into the following issues: The proposed driver identity tracing method for speaking head videos differs from traditional model-based tracing tasks. Its core objective is to accurately identify the driving identity behind facial movements from a given synthetic speaking head video, rather than the target identity superficially presented in the video. Completing this task is of great significance for tracing the source of malicious content, determining legal responsibility, and thus curbing identity abuse. Previous studies extracted facial keypoints from video frames and directly manipulated the original facial keypoint sequences. These sequences suffer from two key types of confounding: static facial appearance interference from the target identity and errors inherent in the generator, which impair the model's ability to extract discriminative driver identity features.

[0006] like Figure 1As shown, the facial keypoint sequence at the top of the second column from the right is the generated result under the self-replay setting (the driving identity and the target identity are the same), eliminating interference from the target's appearance. However, the inherent reconstruction error of the generation model, as a confounding factor, causes the generated keypoint sequence to deviate from the real driving sequence, as shown in the generation error in the first column from the right, blurring the dynamic features of the driving identity. In contrast, the facial keypoint sequence at the bottom of the second column from the right corresponds to the cross-replay scenario (the driving identity and the target identity are different). In this case, the static facial geometry of the target identity (such as unique skeletal contours and facial feature distribution), as another stronger confounding factor, is inevitably superimposed on the driving dynamics, causing a serious deviation in the keypoint sequence, as shown in the target appearance in the first column from the right. Therefore, developing techniques that can decompose dynamic features and static interference bias and eliminate generation bias is crucial for accurately extracting the features of the driving identity.

[0007] This application provides a method and apparatus for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias correction, in order to solve the problems in related technologies, such as the mixing of facial key points due to interference from the appearance of the target identity and inherent errors of the generator, which reduces the accuracy and reliability of driving identity tracing.

[0008] The first aspect of this application provides a method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias correction, comprising the following steps: acquiring at least one facial keypoint sequence from a target synthetic speaking head video; extracting at least one spatiotemporal feature from the at least one facial keypoint sequence; extracting driving features and target features of the at least one spatiotemporal feature, and inputting the driving features and target features into a preset temporal convolutional network based on a loss function optimized by multi-objective causal constraints to generate a dynamic identity embedding of the driving features; optimizing the dynamic identity embedding using a target adversarial causal verification mechanism, and generating the identity tracing result of the target synthetic speaking head video based on the optimized dynamic identity embedding.

[0009] Optionally, in one embodiment of this application, obtaining at least one facial key point sequence in the target synthetic speaking head video includes: extracting at least one facial key point in each frame of the target synthetic speaking head video based on a pre-built key point detection model; and constructing the at least one facial key point sequence using the at least one facial key point.

[0010] Optionally, in one embodiment of this application, optimizing the dynamic identity embedding using a target adversarial causal verification mechanism includes: replacing the driving feature with a counterfactual driving feature based on the target adversarial causal verification mechanism; inputting the counterfactual driving feature and the target feature into the preset temporal convolutional network to generate a counterfactual dynamic identity embedding of the counterfactual driving feature; and constructing a spatiotemporal causal debiasing loss for the difference before and after intervention based on the difference between the counterfactual dynamic identity embedding and the dynamic identity embedding, so as to optimize the dynamic identity embedding using the spatiotemporal causal debiasing loss for the difference before and after intervention.

[0011] Optionally, in one embodiment of this application, the dynamic identity embedding is represented as:

[0012] in, Embedding for dynamic identities, For temporal convolutional networks, As a driving feature, The target feature.

[0013] Optionally, in one embodiment of this application, the loss function for the multi-objective causal constraint optimization is expressed as:

[0014]

[0015]

[0016]

[0017]

[0018] in, It is an exponential function. It is a measurement dimension A custom function for volatility. It is an average over the time dimension. It is the mean squared error loss function. It is a reconstruction decoder. These are the original facial key points. It is the Frobenius norm. Let Ctr represent the covariance matrix of the unbiased estimate, and let Ctr represent the centering along the channel dimension after flattening the non-feature channel dimensions. It is the loss of multi-objective causal constraint optimization, in addition, , , , , , , Both represent constant parameters.

[0019] A second aspect of this application provides an identity tracing device for synthetic speaking head videos based on spatiotemporal causal bias correction, comprising: an acquisition module for acquiring at least one facial key point sequence in a target synthetic speaking head video and extracting at least one spatiotemporal feature from the at least one facial key point sequence; an extraction module for extracting driving features and target features of the at least one spatiotemporal feature, and inputting the driving features and target features into a preset temporal convolutional network based on a loss function optimized by multi-objective causal constraints to generate a dynamic identity embedding of the driving features; and a generation module for optimizing the dynamic identity embedding using a target adversarial causal verification mechanism and generating an identity tracing result of the target synthetic speaking head video based on the optimized dynamic identity embedding.

[0020] Optionally, in one embodiment of this application, the acquisition module includes: an extraction unit, configured to extract at least one facial key point in each frame of the target synthetic speaking head video based on a pre-built key point detection model; and a construction unit, configured to construct the at least one facial key point sequence using the at least one facial key point.

[0021] Optionally, in one embodiment of this application, the generation module includes: a processing unit, configured to replace the driving feature with a counterfactual driving feature based on a target adversarial causal verification mechanism; a generation unit, configured to input the counterfactual driving feature and the target feature into the preset temporal convolutional network to generate a counterfactual dynamic identity embedding of the counterfactual driving feature; and an optimization unit, configured to construct a spatiotemporal causal debiasing loss of the difference before and after intervention based on the difference between the counterfactual dynamic identity embedding and the dynamic identity embedding, so as to optimize the dynamic identity embedding using the spatiotemporal causal debiasing loss of the difference before and after intervention.

[0022] Optionally, in one embodiment of this application, the dynamic identity embedding is represented as:

[0023] in, Embedding for dynamic identities, For temporal convolutional networks, As a driving feature, The target feature.

[0024] Optionally, in one embodiment of this application, the loss function for the multi-objective causal constraint optimization is expressed as:

[0025]

[0026]

[0027]

[0028]

[0029] in, It is an exponential function. It is a measurement dimension A custom function for volatility. It is an average over the time dimension. It is the mean squared error loss function. It is a reconstruction decoder. These are the original facial key points. It is the Frobenius norm. Let Ctr represent the covariance matrix of the unbiased estimate, and let Ctr represent the centering along the channel dimension after flattening the non-feature channel dimensions. It is the loss of multi-objective causal constraint optimization, in addition, , , , , , , Both represent constant parameters.

[0030] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias removal as described in the above embodiments.

[0031] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias removal.

[0032] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, is used to implement the above-described method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias removal.

[0033] This application embodiment can extract at least one spatiotemporal feature from at least one facial key point sequence obtained from a target synthetic speaking head video. Then, it extracts the driving and target features of the at least one spatiotemporal feature, and inputs the driving and target features into a preset temporal convolutional network based on a loss function optimized by multi-objective causal constraints. This generates a dynamic identity embedding of the driving feature, and optimizes the dynamic identity embedding using a target adversarial causal verification mechanism, thereby generating the identity tracing result of the target synthetic speaking head video, effectively improving the accuracy and reliability of driving identity tracing. This solves the problems in related technologies where facial key points are mixed due to interference from the target identity appearance and inherent errors in the generator, reducing the accuracy and reliability of driving identity tracing.

[0034] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0035] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A schematic diagram illustrating facial key point deviations in synthetic speaking head videos using related technologies; Figure 2 This is a flowchart of a synthetic speaking head video identity tracing method based on spatiotemporal causal bias removal provided in an embodiment of this application; Figure 3 This is a schematic diagram of the network structure for tracing the identity of a synthesized speaking head video in a specific embodiment of this application; Figure 4 This is a schematic diagram of a synthetic speaking head video identity tracing device based on spatiotemporal causal bias removal provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] The following describes, with reference to the accompanying drawings, a method and apparatus for identity tracing of synthetic speaking head videos based on spatiotemporal causal debiasing, according to embodiments of this application. Addressing the problems mentioned in the background art, such as the interference of target identity appearance and inherent errors in the generator leading to mixed facial key points, which reduces the accuracy and reliability of driving identity tracing, this application provides a method for identity tracing of synthetic speaking head videos based on spatiotemporal causal debiasing. In this method, at least one spatiotemporal feature is extracted from at least one facial key point sequence obtained from the target synthetic speaking head video. Then, driving features and target features of the at least one spatiotemporal feature are extracted. Based on a loss function optimized by multi-target causal constraints, the driving features and target features are input into a preset temporal convolutional network to generate a dynamic identity embedding of the driving features. The dynamic identity embedding is then optimized using a target adversarial causal verification mechanism, thereby generating the identity tracing result of the target synthetic speaking head video, effectively improving the accuracy and reliability of driving identity tracing. Thus, the problems of mixed facial key points caused by interference of target identity appearance and inherent errors in the generator, which reduce the accuracy and reliability of driving identity tracing, are solved in the related art.

[0038] Specifically, Figure 2 This is a flowchart illustrating a method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias removal, provided in an embodiment of this application.

[0039] like Figure 2 As shown, the method for tracing the identity of synthesized speaking head videos based on spatiotemporal causal bias correction includes the following steps: In step S201, at least one facial key point sequence is obtained from the target synthetic speaking head video, and at least one spatiotemporal feature is extracted from the at least one facial key point sequence.

[0040] In this embodiment of the application, the target synthetic speaking head video is a synthetic speaking head video that requires identity tracing; It is understood that embodiments of this application can obtain at least one facial key point sequence in the target synthetic speaking head video, for example, such as Figure 3 As shown, embodiments of this application can extract one or more spatiotemporal features from one or more facial key point sequences obtained from a synthesized speaking head video. For example, this application can use a three-dimensional convolutional network to perform preliminary feature extraction on one or more facial key point sequences to obtain spatiotemporal features, i.e., hidden layer features containing spatiotemporal semantics. This effectively improves the feasibility of tracing the identity of synthesized talking head videos.

[0041] In one embodiment of this application, obtaining at least one facial key point sequence in a target synthetic speaking head video includes: extracting at least one facial key point in each frame of the target synthetic speaking head video based on a pre-built key point detection model; and constructing at least one facial key point sequence using the at least one facial key point.

[0042] For example, such as Figure 3 As shown, the embodiments of this application can extract one or more facial key points from the synthetic speaking head video based on a pre-built key point detection model, and then form one or more facial key point sequences from the one or more facial key points, which effectively improves the accuracy of identity tracing of the synthetic speaking head video.

[0043] In step S202, at least one spatiotemporal feature is extracted as a driving feature and a target feature. Based on a loss function optimized by multi-objective causal constraints, the driving feature and the target feature are input into a preset temporal convolutional network to generate a dynamic identity embedding of the driving feature.

[0044] It is understood that embodiments of this application can extract at least one driving feature and target feature of a spatiotemporal feature, for example, such as Figure 3 As shown, embodiments of this application can employ a dual-branch encoder structure comprising a driver encoder and a target encoder. The driver encoder focuses on extracting temporal dynamic features that characterize unique motion patterns representing the driver's identity, while the target encoder captures temporally invariant features related to facial structure. For example, through a drive encoder and target encoder Extracting driving features from latent features that contain spatiotemporal semantics, i.e., spatiotemporal features. and target features As shown below:

[0045]

[0046] in, To drive the encoder, For the target encoder, As a driving feature, For target features, It is a spatiotemporal characteristic.

[0047] Next, based on the driving features and target features, the loss function for multi-objective causal constraint optimization is calculated, expressed as:

[0048]

[0049]

[0050]

[0051]

[0052] in, It is an exponential function. It is a measurement dimension A custom function for volatility. It is an average over the time dimension. It is the mean squared error loss function. It is a reconstruction decoder. These are the original facial key points. It is the Frobenius norm. It is the covariance matrix of the unbiased estimate, and Ctr represents the centering on the channel dimension after flattening the non-feature channel dimensions. It is the loss of multi-objective causal constraint optimization, where, , , , , , , Both represent constant parameters.

[0053] Furthermore, embodiments of this application can constrain its spatiotemporal characteristics through a loss function optimized by multi-objective causal constraints, and finally fuse the decomposed driving features and target features into a temporal convolutional network to generate a dynamic identity embedding. For example, embodiments of this application can connect driving features and target features, and then use a temporal convolutional network... Extracting dynamic identity embeddings Dynamic identity embedding is represented as:

[0054] in, Embedding for dynamic identities, For temporal convolutional networks, As a driving feature, The target feature.

[0055] In step S103, the dynamic identity embedding is optimized using the target adversarial causal verification mechanism, and the identity tracing result of the target synthetic speaking head video is generated based on the optimized dynamic identity embedding.

[0056] It is understood that the embodiments of this application can utilize the target adversarial causal verification mechanism in the following steps to optimize the dynamic identity embedding, so as to sever the causal relationship between facial key point confounding and driving features, and generate the identity tracing result of the target synthetic speaking head video based on the optimized dynamic identity embedding. Based on the spatiotemporal causal bias removal to maximize the difference before and after intervention, the quality of driving features can be quantitatively separated, thereby achieving accurate tracing of driving identity.

[0057] Therefore, the embodiments described in this application can solve the problem of motion and appearance entanglement in the original features, enabling clear separation of the dynamic patterns driving identity. Next, an adversarial causal verification mechanism is introduced to effectively suppress residual interference by severing the causal relationship between facial key point confounding and driving features. Finally, based on the spatiotemporal causal debiasing optimization strategy, the difference in model output before and after intervention is maximized to quantify the quality of driving features, forcing the model to focus on dynamic patterns that have key causal contributions to identity discrimination, enhancing the discriminativeness of dynamic identity embedding, and effectively improving the accuracy and reliability of driving identity tracing.

[0058] In one embodiment of this application, optimizing dynamic identity embedding using a target adversarial causal verification mechanism includes: replacing driving features with counterfactual driving features based on the target adversarial causal verification mechanism; inputting the counterfactual driving features and target features into a preset temporal convolutional network to generate a counterfactual dynamic identity embedding of the counterfactual driving features; and constructing a spatiotemporal causal debiasing loss based on the difference between the counterfactual dynamic identity embedding and the dynamic identity embedding, so as to optimize the dynamic identity embedding using the spatiotemporal causal debiasing loss based on the difference between the intervention and the intervention.

[0059] In some embodiments, such as Figure 3 As shown, embodiments of this application can introduce an adversarial causal verification mechanism to verify the original driving features. Replace with virtual counterfactual driving features Using temporal convolutional networks again Extracting counterfactual embeddings And use spatiotemporal causal bias-reduction loss based on the difference before and after intervention. Quantization and optimization are performed to drive feature quality, as shown below:

[0060]

[0061] in, Embed contrastive loss for the defined dynamic identity. It is a constant weight.

[0062] Furthermore, embodiments of this application can combine all losses to obtain the final loss function for end-to-end training:

[0063] in, It is a constant weight.

[0064] In other words, the embodiments of this application further sever the false causal association between confounding factors and driving features through an adversarial causal verification mechanism, and maximize the difference before and after intervention based on spatiotemporal causal bias removal, thereby achieving accurate tracing of driving identity. The entire network adopts an end-to-end training method to jointly optimize various losses, thereby achieving high-precision, robust, and interpretable accurate tracing of driving identity, effectively resisting identity confusion attacks in deepfakes.

[0065] The method for identity tracing of synthetic speaking head videos based on spatiotemporal causal bias correction proposed in this application extracts at least one spatiotemporal feature from at least one facial keypoint sequence obtained from the target synthetic speaking head video. Then, driving and target features of the at least one spatiotemporal feature are extracted. Based on a loss function optimized by multi-objective causal constraints, the driving and target features are input into a preset temporal convolutional network to generate a dynamic identity embedding of the driving feature. The dynamic identity embedding is then optimized using a target adversarial causal verification mechanism, thereby generating the identity tracing result of the target synthetic speaking head video, effectively improving the accuracy and reliability of driving identity tracing. This solves the problems in related technologies where facial keypoints are mixed due to interference from the target identity appearance and inherent errors in the generator, reducing the accuracy and reliability of driving identity tracing.

[0066] Secondly, referring to the accompanying drawings, a synthetic speaking head video identity tracing device based on spatiotemporal causal bias removal proposed according to an embodiment of this application is described.

[0067] Figure 4 This is a block diagram of a synthetic speech head video identity tracing device based on spatiotemporal causal bias removal according to an embodiment of this application.

[0068] like Figure 4 As shown, the synthetic speech head video identity tracing device 10 based on spatiotemporal causal bias correction includes: an acquisition module 100, an extraction module 200, and a generation module 300.

[0069] Specifically, the acquisition module 100 is used to acquire at least one facial key point sequence in the target synthetic speaking head video, and extract at least one spatiotemporal feature from the at least one facial key point sequence.

[0070] The extraction module 200 is used to extract at least one spatiotemporal feature driving feature and target feature, and based on a loss function optimized by multi-objective causal constraints, input the driving feature and target feature into a preset temporal convolutional network to generate a dynamic identity embedding of the driving feature.

[0071] The generation module 300 is used to optimize the dynamic identity embedding using the target adversarial causal verification mechanism, and generate the identity tracing result of the target synthetic speaking head video based on the optimized dynamic identity embedding.

[0072] Optionally, in one embodiment of this application, the acquisition module 100 includes an extraction unit and a construction unit.

[0073] The extraction unit is used to extract at least one facial key point from each frame of the target synthetic speaking head video based on a pre-built key point detection model.

[0074] A building unit for constructing at least one facial keypoint sequence using at least one facial keypoint.

[0075] Optionally, in one embodiment of this application, the generation module includes: a processing unit, a generation unit, and an optimization unit.

[0076] The processing unit is used to replace the driving features with counterfactual driving features based on the target adversarial causal verification mechanism.

[0077] The generation unit is used to input counterfactual driving features and target features into a preset temporal convolutional network to generate counterfactual dynamic identity embeddings of counterfactual driving features.

[0078] An optimization unit is used to construct a spatiotemporal causal debiasing loss based on the difference between counterfactual dynamic identity embedding and dynamic identity embedding, so as to optimize dynamic identity embedding by utilizing the spatiotemporal causal debiasing loss of the difference before and after intervention.

[0079] Optionally, in one embodiment of this application, dynamic identity embedding is represented as:

[0080] in, Embedding for dynamic identities, For temporal convolutional networks, As a driving feature, The target feature.

[0081] Optionally, in one embodiment of this application, the loss function for multi-objective causal constraint optimization is expressed as:

[0082]

[0083]

[0084]

[0085]

[0086] in, It is an exponential function. It is a measurement dimension A custom function for volatility. It is an average over the time dimension. It is the mean squared error loss function. It is a reconstruction decoder. These are the original facial key points. It is the Frobenius norm. Let Ctr represent the covariance matrix of the unbiased estimate, and let Ctr represent the centering along the channel dimension after flattening the non-feature channel dimensions. It is the loss of multi-objective causal constraint optimization, in addition, , , , , , , Both represent constant parameters.

[0087] It should be noted that the foregoing explanation of the embodiment of the synthetic speaking head video identity tracing method based on spatiotemporal causal bias also applies to the synthetic speaking head video identity tracing device based on spatiotemporal causal bias in this embodiment, and will not be repeated here.

[0088] The synthetic speaking head video identity tracing device based on spatiotemporal causal bias correction proposed in this application can extract at least one spatiotemporal feature from at least one facial key point sequence obtained from the target synthetic speaking head video. Then, it extracts the driving feature and target feature of the at least one spatiotemporal feature, and inputs the driving feature and target feature into a preset temporal convolutional network based on a loss function optimized by multi-objective causal constraints to generate a dynamic identity embedding of the driving feature. The dynamic identity embedding is then optimized using a target adversarial causal verification mechanism, thereby generating the identity tracing result of the target synthetic speaking head video, effectively improving the accuracy and reliability of driving identity tracing. This solves the problems in related technologies where facial key points are mixed due to interference from the target identity appearance and inherent errors in the generator, reducing the accuracy and reliability of driving identity tracing.

[0089] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0090] When the processor 502 executes the program, it implements the synthetic speech head video identity tracing method based on spatiotemporal causal bias removal provided in the above embodiments.

[0091] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.

[0092] The memory 501 is used to store computer programs that can run on the processor 502.

[0093] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0094] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0095] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0096] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0097] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias removal.

[0098] This embodiment also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias removal.

[0099] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0101] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0102] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0103] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0104] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0106] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for tracing the identity of synthesized speech-head videos based on spatiotemporal causal bias correction, characterized in that, Includes the following steps: Obtain at least one facial key point sequence from the target synthetic speaking head video, and extract at least one spatiotemporal feature from the at least one facial key point sequence; Extract the driving features and target features of at least one of the spatiotemporal features, and input the driving features and target features into a preset temporal convolutional network based on a loss function optimized by multi-objective causal constraints, so as to generate the dynamic identity embedding of the driving features; The dynamic identity embedding is optimized using a target adversarial causal verification mechanism, and the identity tracing result of the target synthetic speaking head video is generated based on the optimized dynamic identity embedding.

2. The method according to claim 1, characterized in that, The acquisition of at least one facial key point sequence in the target synthetic speaking head video includes: Based on a pre-built keypoint detection model, at least one facial keypoint is extracted from each frame of the target synthetic speaking head video; Construct the at least one facial key point sequence using the at least one facial key point.

3. The method according to claim 1, characterized in that, The optimization of the dynamic identity embedding using the target adversarial causal verification mechanism includes: Based on the target adversarial causal verification mechanism, the driving feature is replaced with a counterfactual driving feature; The counterfactual driving feature and the target feature are input into the preset temporal convolutional network to generate the counterfactual dynamic identity embedding of the counterfactual driving feature; Based on the difference between the counterfactual dynamic identity embedding and the dynamic identity embedding, a spatiotemporal causal debiasing loss of the difference before and after intervention is constructed, so as to optimize the dynamic identity embedding by utilizing the spatiotemporal causal debiasing loss of the difference before and after intervention.

4. The method according to claim 1, characterized in that, The dynamic identity embedding is represented as follows: in, Embedding for dynamic identities, For temporal convolutional networks, As a driving feature, For target features.

5. The method according to claim 1, characterized in that, The loss function for the multi-objective causal constraint optimization is expressed as: in, It is an exponential function. It is a measurement dimension A custom function for volatility. It is an average over the time dimension. It is the mean squared error loss function. It is a reconstruction decoder. These are the original facial key points. It is the Frobenius norm. It is the covariance matrix of the unbiased estimate, and Ctr represents the centering on the channel dimension after flattening the non-feature channel dimensions. It is the loss of multi-objective causal constraint optimization, in addition, , , , , , , Both represent constant parameters.

6. A device for tracing the identity of a synthetic speaking head video based on spatiotemporal causal bias correction, characterized in that, include: The acquisition module is used to acquire at least one facial key point sequence in the target synthetic speaking head video, and extract at least one spatiotemporal feature from the at least one facial key point sequence; An extraction module is used to extract the driving features and target features of the at least one spatiotemporal feature, and input the driving features and target features into a preset temporal convolutional network based on a loss function optimized by multi-objective causal constraints, so as to generate the dynamic identity embedding of the driving features; The generation module is used to optimize the dynamic identity embedding using a target adversarial causal verification mechanism, and generate the identity tracing result of the target synthetic speaking head video based on the optimized dynamic identity embedding.

7. The apparatus according to claim 6, characterized in that, The acquisition module includes: An extraction unit is used to extract at least one facial key point from each frame of the target synthetic speaking head video based on a pre-built key point detection model. A construction unit is used to construct the at least one facial key point sequence using the at least one facial key point.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method for tracing the identity of a synthetic speaking head video based on spatiotemporal causal debiasing as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for tracing the identity of synthetic speaking head videos based on spatiotemporal causal debiasing as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method for tracing the identity of synthetic speaking head videos based on spatiotemporal causal debiasing as described in any one of claims 1-5.

Images