Audio and video forgery detection method, device, equipment, medium and product
By extracting the visual identity consistency and auditory space-time characteristics of audio-visual sequences for alignment and fusion, combined with the audio-visual consistency characteristics, the problem of the existing technology being difficult to adapt to the iteration of forgery technology and identifying small tampered fragments is solved, and the efficiency of audio-visual sequences is achieved Identity characteristic changes analysis and detection.
Patent Information
- Application Number
- CN202411042157.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-26
AI Technical Summary
The existing deep learning methods for forging synthetic video detection technology is difficult to adapt to the iterative upgrade of forging technology, and its feature expression capabilities for small tampering fragments are limited, making it difficult to effectively identify identity jump defects in audio and video sequences.
By obtaining audio and video sequences, visual identity consistency features and auditory space-time features are extracted, and aligned and fused to generate audio and video consistency features. Combining visual identity consistency features forged detection, identifying identity feature changes and modality in audio and video sequences. Timing inconsistency.
It realizes adaptive detection of forgery technology with faster iteration and upgrade, and can identify tiny tampering fragments and identity jumps in audio and video sequences, improving the generalization and robustness of forgery detection.
Smart Images

Figure CN119028375A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of video vision technology, and in particular to an audio and video forgery detection method, device, equipment, medium and product. Background Art
[0002] In recent years, the use of deep learning to forge synthetic videos (DeepFake) has increased. As forgery methods become more diverse and more accurate, videos with both fake images and audio that are difficult to distinguish with the naked eye have appeared, posing a great challenge to anti-counterfeiting and authentication.
[0003] Most deep fake detection technologies use single-modal forgery detection technology for identification, such as visual forgery detection solutions or audio forgery detection solutions. However, this technical solution is difficult to adapt to the continuous iteration and upgrading of forgery technology. In addition, multi-modal forgery detection technology can also be used for identification. However, this technical solution has limited ability to express the characteristics of tiny tampered fragments. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a method, device, equipment, medium and product for detecting audio and video forgery.
[0005] In a first aspect, an embodiment of the present disclosure provides an audio and video forgery detection method, comprising:
[0006] Acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence;
[0007] Determining a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence;
[0008] Extracting features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and aligning and fusing the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features;
[0009] Based on the visual identity consistency feature and the audio-visual consistency feature, a forgery detection result of the audio and video sequence is determined.
[0010] In an optional implementation manner, determining the visual identity consistency feature based on the identity feature of each video clip in the video sequence includes:
[0011] Extracting facial identity features of each video frame in the video clip, and determining identity features of the video clip based on the facial identity features of each video frame;
[0012] Based on the similarity of the identity features between any two of the video clips, a visual identity consistency feature of the audio and video sequence is determined.
[0013] In an optional implementation, determining the visual identity consistency feature of the audio and video sequence based on the similarity of the identity features between any two of the video clips includes:
[0014] Determine the minimum similarity of the preset number;
[0015] Based on the difference between the target value and each of the minimum similarities, a visual identity consistency feature of the audio and video sequence is determined.
[0016] In an optional implementation manner, extracting features from the video sequence to obtain visual spatiotemporal features includes:
[0017] Extracting a facial visual feature map of each video frame in the video clip through a visual feature extraction network to obtain a first feature matrix;
[0018] The visual spatiotemporal features of the video clip are determined using the first feature matrix.
[0019] In an optional implementation manner, determining the visual spatiotemporal features of the video clip by using the first feature matrix includes:
[0020] Dividing the first feature matrix through a first timing modeling network to obtain a plurality of first matrix sub-blocks, and determining an embedded feature of each of the first matrix sub-blocks;
[0021] The embedded features are visually encoded by the encoding block in the first temporal modeling network to obtain local dynamic information of the face, and the local dynamic information of the face is aggregated to obtain the visual spatiotemporal features.
[0022] In an optional implementation manner, extracting features from the audio sequence to obtain auditory spatiotemporal features includes:
[0023] Extracting acoustic features of each audio segment in the audio sequence to obtain a second feature matrix;
[0024] dividing the second characteristic matrix to obtain a plurality of second matrix sub-blocks;
[0025] Each second matrix sub-block is processed by a second temporal modeling network, and the processing results are aggregated to obtain the auditory spatiotemporal features.
[0026] In an optional implementation manner, the fusing the auditory spatiotemporal features and the visual spatiotemporal features to obtain the audiovisual consistency features includes:
[0027] Determining an audiovisual connection relationship between the audio sequence and the video sequence;
[0028] The auditory spatiotemporal features and the visual spatiotemporal features are fused through the audiovisual connection relationship to obtain the audiovisual consistency features.
[0029] In an optional implementation, the audio-visual connection relationship includes: a segment of any modality in the audio and video sequence is associated with a previous or next segment of all modalities in time sequence, and a segment of any modality is associated with a simultaneous segment of other modalities; the modalities include a visual modality and an auditory modality.
[0030] In an optional implementation, the fusing of the auditory spatiotemporal features and the visual spatiotemporal features through the audio-visual connection relationship and the audio-visual consistency learning network to obtain the audio-visual consistency features includes:
[0031] Performing graph convolution calculation on the auditory spatiotemporal features and the auditory association relationship through a graph convolution neural network layer in an audio-visual consistency learning network to obtain a first output feature, and performing graph convolution calculation on the visual spatiotemporal features and the visual association relationship to obtain a second output feature; the auditory association relationship is used to indicate an audio segment with an association relationship, and the visual association relationship is used to indicate a video segment with an association relationship;
[0032] The first output feature, the second output feature and the cross-modal association relationship are subjected to attention fusion by the graph attention network layer in the audio-visual consistency learning network to obtain a third output feature; the cross-modal association relationship is used to indicate audio clips and video clips having an association relationship;
[0033] The audiovisual consistency feature is determined based on a fusion feature of the second output feature and the third output feature.
[0034] In an optional implementation, determining the forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature includes:
[0035] The visual identity consistency feature and the audiovisual consistency feature are fused through a fusion output network to obtain a discriminant feature, and the forgery detection result is determined based on the discriminant feature.
[0036] In an optional implementation manner, the method further includes:
[0037] Determining a first loss function based on the facial identity features of each target segment of each sample in the audio and video sequence sample set;
[0038] Determining a second loss function based on the discriminant feature of each of the samples;
[0039] Determining a third loss function based on the discriminant features and sample labels of each of the samples;
[0040] Based on the first loss function, the second loss function and the third loss function, the initial neural network is trained to obtain a target neural network after training. The target neural network includes a visual feature extraction network, a first temporal modeling network, a second temporal modeling network, an audio-visual consistency learning network and a fusion output network.
[0041] In a second aspect, an embodiment of the present disclosure provides an audio and video forgery detection device, comprising:
[0042] An acquisition unit, used to acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence;
[0043] A first determining unit is used to determine a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence;
[0044] A feature extraction unit, configured to extract features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and to align and fuse the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features;
[0045] The second determination unit is used to determine the forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature.
[0046] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.
[0047] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.
[0048] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, characterized in that the computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the first aspect, or the steps in any possible implementation of the first aspect.
[0049] The present disclosure provides a method, device, equipment, medium and product for detecting audio and video forgery. In an embodiment of the present disclosure, first, an audio and video sequence to be tested is obtained; the audio and video sequence includes an audio sequence and a video sequence; then, based on the identity features of each video clip in the video sequence, a visual identity consistency feature is determined; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence; next, feature extraction is performed on the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and the auditory spatiotemporal features and the visual spatiotemporal features are aligned and fused to obtain audiovisual consistency features; finally, based on the visual identity consistency features and the audiovisual consistency features, the forgery detection result of the audio and video sequence is determined.
[0050] In the above implementation, by combining audio sequences and video sequences for forgery detection, it can be applied to a wider range of forgery technologies, thereby meeting the needs of forgery technologies that require faster iteration and upgrade. By aligning and fusing the auditory spatiotemporal features of the audio sequence and the visual spatiotemporal features of the video sequence, and performing forgery detection using the aligned and fused audiovisual consistency features, it is possible to detect mismatched segments between the audio sequence and the video sequence, and this processing method can identify slightly tampered segments. By introducing the visual identity consistency features of each video segment for forgery detection, it is possible to perform identity feature change analysis on the audio and video sequence, thereby effectively capturing identity jump defects in the video.
[0051] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0053] Figure 1A flow chart of an audio and video forgery detection method provided by an embodiment of the present disclosure is shown;
[0054] Figure 2 A schematic diagram of the structure of a coding block of a ViT network provided by an embodiment of the present disclosure is shown;
[0055] Figure 3 A schematic diagram showing the structure of a timing diagram provided by an embodiment of the present disclosure is shown;
[0056] Figure 4 A schematic diagram showing the network principle of a heterogeneous graph neural network provided by an embodiment of the present disclosure is shown;
[0057] Figure 5 A flowchart of another audio and video forgery detection method provided by an embodiment of the present disclosure is shown;
[0058] Figure 6 A schematic diagram of an audio and video forgery detection device provided by an embodiment of the present disclosure is shown;
[0059] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0062] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.
[0063] Based on the above research, the present disclosure provides a method, device, equipment, medium and product for detecting audio and video forgery. In the embodiment of the present disclosure, first, the audio and video sequence to be tested is obtained; the audio and video sequence includes an audio sequence and a video sequence; then, based on the identity features of each video clip in the video sequence, the visual identity consistency feature is determined; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence; next, feature extraction is performed on the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and the auditory spatiotemporal features and the visual spatiotemporal features are aligned and fused to obtain audiovisual consistency features; finally, based on the visual identity consistency features and the audiovisual consistency features, the forgery detection result of the audio and video sequence is determined.
[0064] In the above implementation, by combining audio sequences and video sequences for forgery detection, it can be applied to a wider range of forgery technologies, thereby meeting the needs of forgery technologies that require faster iteration and upgrade. By aligning and fusing the auditory spatiotemporal features of the audio sequence and the visual spatiotemporal features of the video sequence, and performing forgery detection using the aligned and fused audiovisual consistency features, it is possible to detect mismatched segments between the audio sequence and the video sequence, and this processing method can identify slightly tampered segments. By introducing the visual identity consistency features of each video segment for forgery detection, it is possible to perform identity feature change analysis on the audio and video sequence, thereby effectively capturing identity jump defects in the video.
[0065] To facilitate understanding of this embodiment, a method for detecting audio and video forgery disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the method for detecting audio and video forgery provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities. In some possible implementations, the method for detecting audio and video forgery can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0066] See also Figure 1 FIG. 1 is a flowchart of an audio and video forgery detection method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S104, wherein:
[0067] S101: Acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence.
[0068] In the disclosed embodiment, the audio sequence includes the sound data of each frame in each audio segment, and the video sequence includes the face image of each frame in each video segment.
[0069] In specific implementation, an audio and video sequence X to be tested may be obtained, wherein if the audio and video sequence to be tested is a sample, the sample also carries a label indicating whether it is true or false, denoted as Y, and the value of the label is 0 or 1 respectively.
[0070] After obtaining the audio and video sequence to be tested, the audio and video sequence to be tested can be preprocessed to obtain an audio sequence and a video sequence. Here, a sliding window can be used to intercept n audio and video clips of a fixed length (in practice, the sliding window length is 240ms) from the audio and video sequence in time order to form the audio sequence to be tested A = {A1, A2..., A n} and video sequence V = {V1, V2..., V n}. Among them, A i represents the i-th audio segment in the audio sequence, Represents the i-th audio segment A i The sound data of the tth frame in V i represents the i-th video clip in the video sequence, Represents the i-th video segment V i The face image obtained by face detection and cropping of the t-th frame image, The length of is H, the width is W, and the number of channels is C=3. In practice, H=W=224.
[0071] S102: Determine a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence.
[0072] Here, the visual identity consistency feature can be a vector feature, which is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence, or to indicate the degree of discreteness of the identity features in the video sequence. The larger the value of the element in the vector, the greater the identity feature change between frames. The more elements with large values, the more identity jumps there are in the video, which is one of the important clues for distinguishing video forgery.
[0073] S103: extracting features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and aligning and fusing the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features.
[0074] In the disclosed embodiment, fine-grained auditory spatiotemporal features and visual spatiotemporal features can be extracted from the audio sequence A and the video sequence V obtained in the preprocessing step. Then, the audio clips and the video clips are regarded as nodes, and the connection relationship between the nodes of the same modality and the nodes across modalities is determined, so as to construct an audio-visual timing graph structure according to the connection relationship, and the auditory spatiotemporal features and the visual spatiotemporal features are aligned and fused based on the audio-visual timing graph structure to obtain discriminative audio-visual consistency features.
[0075] It should be noted that the execution order of the above steps S102 and S103 is not specifically limited here, and they can be executed simultaneously, or step S102 is executed first, or step S103 is executed first, whichever can be implemented.
[0076] S104: Determine a forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature.
[0077] In the above implementation, by combining audio sequences and video sequences for forgery detection, it can be applied to a wider range of forgery technologies, thereby meeting the needs of forgery technologies that require faster iteration and upgrade. By aligning and fusing the auditory spatiotemporal features of the audio sequence and the visual spatiotemporal features of the video sequence, and performing forgery detection using the aligned and fused audiovisual consistency features, it is possible to detect mismatched segments between the audio sequence and the video sequence, and this processing method can identify slightly tampered segments. By introducing the visual identity consistency features of each video segment for forgery detection, it is possible to perform identity feature change analysis on the audio and video sequence, thereby effectively capturing identity jump defects in the video.
[0078] In an optional implementation, the above step S102 determines the visual identity consistency feature based on the identity feature of each video clip in the video sequence, and specifically includes the following steps:
[0079] Step S11, extracting facial identity features of each video frame in the video clip, and determining identity features of the video clip based on the facial identity features of each video frame;
[0080] Step S12: determining the visual identity consistency feature of the audio and video sequence based on the similarity of the identity features between any two of the video clips.
[0081] In the disclosed embodiment, face identity features can be extracted from the video sequence V obtained in the preprocessing step. Here, for each video segment V i , we can use the face identity feature extraction network pre-trained on the face recognition dataset to extract each frame in the video clip Extracting facial identity features For example, in practice, the ResNet50 or ResNet100 model in ArcFace[1] can be used, and a fully connected layer can be added after the last convolutional layer to obtain a 512-dimensional face identity feature vector. In addition, other feature extraction networks that can extract face identity features, or other feature extraction networks that can replace the ResNet50 or ResNet100 model can also be used. Here, the structure of the feature extraction network is not specifically limited.
[0082] Next, you can convert the V i Multi-frame face identity features Use average pooling to fuse and obtain the clip-level identity feature I i .
[0083] The identity feature I corresponding to each video clip in the video sequence is obtained = {I1, I2..., I n}After that, the similarity of identity features between two clips is calculated to obtain an identity similarity matrix. Here, the similarity can be selected as cosine similarity. In addition, other similarities can also be selected, and the present disclosure does not make specific restrictions on this. For example, the value of the i-th row and j-th column in the identity similarity matrix is the identity feature I of the i-th video clip. i and the jth segment identity feature I j The cosine similarity (i.e., similarity) of is calculated as:
[0084] After the similarity is calculated, the visual identity consistency feature of the audio and video sequence can be determined based on the similarity, which specifically includes the following steps:
[0085] Step S121, determining a preset number of minimum similarities;
[0086] Step S122: determining the visual identity consistency feature of the audio and video sequence based on the difference between the target value and each minimum similarity.
[0087] In the disclosed embodiment, the elements of the identity similarity matrix can be arranged from small to large, the first m (preset number) similarities are selected, and the vector formed by these similarities is subtracted from the target value (e.g., 1), thereby obtaining the visual identity consistency feature Fid ={1-S1,1-S2...,1-S m This feature can indicate whether there is a significant change in the visual identity features of the same target in the video sequence. The larger the value of the element in the vector, the greater the identity feature change between frames. The more elements with large values, the more identity jumps there are in the video, which is one of the important clues for judging video forgery.
[0088] In an optional implementation, the above step S103 extracts features from the video sequence to obtain visual spatiotemporal features, which specifically includes the following steps:
[0089] Step S21, extracting a face visual feature map of each video frame in the video clip through a visual feature extraction network to obtain a first feature matrix;
[0090] Step S22: determining the visual spatiotemporal features of the video clip using the first feature matrix.
[0091] In the embodiment of the present disclosure, for each video segment V i , using the visual feature extraction network to extract each frame in the video clip Extracting facial visual feature maps Thus, the first feature matrix is obtained, where the video segment V i The corresponding feature matrix is given by express, The dimension is T×H v ×W v ×C v .
[0092] Here, the visual feature extraction network can use various convolutional neural networks. For example, in practice, the HRNet18 network pre-trained with the face key point detection data set is used to capture the face geometry and texture details. In addition, other time series modeling networks (such as LSTM) can be used instead. This disclosure does not make specific restrictions on this, and it is subject to what can be implemented.
[0093] In an optional implementation, the above step S22 determines the visual spatiotemporal features of the video clip by using the first feature matrix, and specifically includes the following steps:
[0094] Step S221, dividing the first feature matrix through a first sequential modeling network to obtain a plurality of first matrix sub-blocks, and determining an embedded feature of each of the first matrix sub-blocks;
[0095] Step S222: Visually encode the embedded features through the encoding block in the first temporal modeling network to obtain local dynamic information of the face, and aggregate the local dynamic information of the face to obtain the visual spatiotemporal features.
[0096] In the embodiment of the present disclosure, in order to better extract the local visual spatiotemporal information contained in the clip, Input to the first time series modeling network for processing, wherein the first time series modeling network may be a 3D-ViT (Vision Transformer) network. In addition, other time series modeling networks (such as LSTM) may be used instead, and the present disclosure does not make specific limitations on this, subject to what can be implemented.
[0097] In a specific implementation, the first characteristic matrix can be transformed into Divided into several dimensions P T ×P×P 3D sub-blocks (patch), that is, multiple first matrix sub-blocks, where P T , P represent the size of the matrix sub-block in the time dimension and in the space dimension respectively. Next, the embedding features (embedded features) of each first matrix sub-block are obtained by using the linear projection layer (LinearProjection) in the first temporal modeling network. After that, these embedded features are embedded into the encoding block of the first temporal modeling network (e.g., 3D-ViT network). Finally, the first temporal modeling network can output a one-dimensional vector, namely V i Corresponding segment-level visual spatiotemporal features
[0098] Here, the structure of the encoding block in the 3D-ViT network is as follows Figure 2 As shown in the figure, the multi-head attention mechanism can effectively extract the local dynamic information of the face in the video clip. Finally, the network output is aggregated into a one-dimensional vector through a maximum pooling layer, namely the visual spatiotemporal feature.
[0099] In an optional implementation, the above step S103 extracts features from the audio sequence to obtain auditory spatiotemporal features, including:
[0100] Step S31, extracting acoustic features of each audio segment in the audio sequence to obtain a second feature matrix;
[0101] Step S32, dividing the second characteristic matrix to obtain a plurality of second matrix sub-blocks;
[0102] Step S33: Process each second matrix sub-block through a second temporal modeling network, and aggregate the processing results to obtain the auditory spatiotemporal features.
[0103] In the embodiment of the present disclosure, for each audio segment A i , the corresponding acoustic features can be extracted to obtain a second feature matrix, wherein the acoustic features can be Mel Frequency Cepstral Coefficents (MFCC); in addition, other types of acoustic features can also be used, which are not specifically limited here. Next, the second feature matrix can be divided into several 2D sub-blocks (second matrix sub-blocks) and input into the second time series modeling network for processing. Finally, the output of the second time series modeling network is aggregated through a maximum pooling layer to obtain A i Corresponding segment-level auditory spatiotemporal features Here, the second temporal modeling network may be a ViT network, and the structure of its encoding block is the same as the figure.
[0104] Here, it should be noted that the encoding blocks in the above-mentioned visual spatiotemporal features and auditory spatiotemporal features extraction process can be replaced by other time series modeling networks (such as LSTM), and no specific limitation is made here.
[0105] In an optional implementation, the above step S103 fuses the auditory spatiotemporal features with the visual spatiotemporal features to obtain an audiovisual consistency feature, which specifically includes the following steps:
[0106] Step S41: determining an audio-visual connection relationship between the audio sequence and the video sequence.
[0107] Here, the audio-visual connection relationship may be used to indicate audio segments and / or video segments having an associated relationship.
[0108] Step S42: fusing the auditory spatiotemporal features and the visual spatiotemporal features through the audio-visual connection relationship to obtain the audio-visual consistency features.
[0109] In the embodiment of the present disclosure, given each audio segment A i and video clip V i , and the corresponding auditory spatiotemporal features and visual spatiotemporal features Construct an audio-visual timing graph G = {N, E, M} to represent the above audio-visual connection relationship through the audio-visual timing graph. Where N is the node set of the graph, which consists of all audio segments A1, A2..., A n and all video clips V1, V2..., V nIt consists of audio nodes and video nodes; one audio segment corresponds to one audio node, and one video segment corresponds to one video node. E is the edge set of the graph, which contains three types of edges, namely:
[0110] ① Audio edge (audio-audio edge): connects two audio nodes; ② Video edge (video-videoedge): connects two video nodes; ③ Audio-video edge (audio-video edge): connects an audio node and a video node.
[0111] Here, a connection relationship between nodes is defined, that is, an audio-visual connection relationship; through the audio-visual connection relationship, audio clips and / or video clips with an associated relationship can be indicated.
[0112] Specifically, the audio-visual connection relationship includes: the segment of any modality in the audio-video sequence is associated with the previous or next segment of all modalities in time sequence, and the segment of any modality is associated with the simultaneous segments of other modalities; the modalities include visual modalities and auditory modalities.
[0113] Specifically, a segment of any modality can be understood as an audio node, or a video node. For example, an audio node is associated with a previous audio node or a next audio node, and the audio node is associated with a previous video node or a next video node, and the audio node is associated with a video node at the same moment. For another example, a video node is associated with a previous video node or a next video node, and the video node is associated with a previous audio node or a next audio node, and the video node is associated with an audio node at the same moment.
[0114] That is to say, Figure 3 As shown, the audio node and the video node need to be connected to at least two nodes before and after them in time sequence (modeling the timing consistency within the modality), and each audio node is connected to at least the video node at the same moment (modeling the cross-modal timing consistency).
[0115] On this basis, the connection between the node and the more distant nodes can be increased to enhance the learning of long-term temporal dependency. For example, an audio node can be associated with the first two audio nodes or the last two audio nodes, and with the first two video nodes or the last two video nodes.
[0116] The above audio-visual connection relationship can be represented by the adjacency matrix M, where the visual subgraph, that is, the structure containing only video nodes and video edges, is represented by M v Represents; the auditory subgraph, that is, the structure containing only audio nodes, that is, audio edges, is represented by M aRepresented by; the cross-modal connection relationship is represented by M c express.
[0117] After the audiovisual connection relationship is determined in the manner described above, the auditory spatiotemporal features and the visual spatiotemporal features can be fused through the audiovisual connection relationship to obtain the audiovisual consistency feature. For example, the auditory spatiotemporal features and the visual spatiotemporal features can be fused through the audiovisual consistency learning network according to the audiovisual connection relationship to obtain the audiovisual consistency feature.
[0118] In the above implementation, by fusing auditory spatiotemporal features with visual spatiotemporal features, the learning ability of multimodal and multi-scale spatiotemporal potential connections can be enhanced, and the visual and auditory intra-modal temporal inconsistencies in forged videos can be captured.
[0119] In the disclosed embodiment, the audiovisual consistency learning network includes a graph convolutional neural network (GraphConvolution Network, GCN) layer and a graph attention layer (GraphAttentional Layer, GAT) layer. On this basis, the above steps fuse the auditory spatiotemporal features and the visual spatiotemporal features through the audiovisual connection relationship and the audiovisual consistency learning network in the target neural network to obtain the audiovisual consistency features, specifically including the following steps:
[0120] Step S421, performing graph convolution calculation on the auditory spatiotemporal features and the auditory association relationship through a graph convolutional neural network layer in an audio-visual consistency learning network to obtain a first output feature, and performing graph convolution calculation on the visual spatiotemporal features and the visual association relationship to obtain a second output feature; the auditory association relationship is used to indicate an audio segment with an association relationship, and the visual association relationship is used to indicate a video segment with an association relationship;
[0121] Step S422, performing attention fusion on the first output feature, the second output feature, and the cross-modal association relationship through the graph attention network layer in the audio-visual consistency learning network to obtain a third output feature; the cross-modal association relationship is used to indicate audio clips and video clips with an association relationship;
[0122] Step S423: determining the audiovisual consistency feature based on the fusion feature of the second output feature and the third output feature.
[0123] The audio-visual consistency learning network is based on the feature matrix of visual nodes and the feature matrix of the auditory node As input, its core module consists of Figure 4 As shown in the figure, it consists of a graph convolutional neural network layer GCN and a graph attention network layer GAT.
[0124] Both GCN and GAT can aggregate and update the graph node features F according to the adjacency matrix M. This process is expressed as: F' = GCN (M, F) = σ (MFW). Among them, σ represents the activation function and W represents the weight of the network layer. GAT adds the attention coefficient α for information fusion between nodes on this basis. ij =softmax(σ(a T [WF i ||WF j ]), represents the attention coefficient between the i-th node and the j-th node connected to it. || represents the concatenation operation, and a represents the attention kernel of GAT. Therefore, the feature output of the i-th node through the GAT layer can be expressed as: Among them, N i Represents the set of all nodes connected to the i-th node.
[0125] like Figure 4 As shown in the figure, the core module of the audiovisual consistency learning network is divided into two branches: visual and auditory. First, the feature matrix and auditory association relationship (i.e., auditory subgraph) of the input auditory node are subjected to graph convolution calculation to obtain the output feature of the auditory branch, i.e., the first output feature. Then, the feature matrix and visual association relationship (i.e., visual subgraph) of the visual node are subjected to graph convolution calculation to obtain the second output feature. Among them, the weights of the GCN of the visual and auditory branches are shared so as to map the audiovisual spatiotemporal information to the same feature space.
[0126] Finally, the first output feature, the second output feature, and the cross-modal adjacency matrix (i.e., the cross-modal association relationship) are input into GAT for attention fusion to obtain the third output feature, and its output is superimposed with the second output feature of the visual GCN layer to obtain the output feature of the visual branch.
[0127] The audio-visual consistency learning network contains multiple Figure 4 As shown in the module, multiple modules can be stacked in multiple layers, and finally the visual branch output of the last module is passed through a fully connected layer to obtain the audio-visual consistency feature F av .
[0128] In an optional implementation, the above step S104 determines the forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature, and specifically includes the following steps:
[0129] The visual identity consistency feature and the audiovisual consistency feature are fused through a fusion output network to obtain a discriminant feature, and the forgery detection result is determined based on the discriminant feature.
[0130] Here, the fusion output network can be selected as a fully connected layer, or other networks that can achieve the fusion of visual identity consistency features and audio-visual consistency features.
[0131] In the disclosed embodiment, the visual identity consistency features and the audiovisual consistency features may be processed and then input into a fusion output network for fusion processing; wherein the processing includes addition, multiplication, concatenation, etc.
[0132] For example, the visual identity consistency feature F id , audio-visual consistency feature F av At the same time, it is sent to the fully connected layer for fusion, thereby outputting the discriminant feature: F cls =FC([F id ||F av ]), and then get the predicted label based on the discriminant feature That is, falsifying test results.
[0133] In an optional embodiment, the method further comprises the following steps:
[0134] Firstly, based on the facial identity features of each target segment of each sample in the audio and video sequence sample set, a first loss function is determined;
[0135] Secondly, determining a second loss function based on the discriminant features of each of the samples;
[0136] Next, based on the discriminant features and sample labels of each sample, the third loss function is determined;
[0137] Finally, based on the first loss function, the second loss function and the third loss function, the initial neural network is trained to obtain a target neural network after training. The target neural network includes a visual feature extraction network, a first temporal modeling network, a second temporal modeling network, an audiovisual consistency learning network and a fusion output network.
[0138] Here, the target neural network includes a visual feature extraction network, a first temporal modeling network, a second temporal modeling network, an audiovisual consistency learning network, and a fusion output network. Among them, the visual feature extraction network is a network used to extract the visual feature map of the face of each video frame in the video clip in the above-mentioned implementation mode, the first temporal modeling network is a network that divides the first feature matrix in the above-mentioned embodiment to obtain multiple first matrix sub-blocks, and determines the embedded features of each of the first matrix sub-blocks, and visually encodes the embedded features to obtain local dynamic information of the face, and aggregates the local dynamic information of the face to obtain the network of the visual spatiotemporal features, the second temporal modeling network is a network that processes each second matrix sub-block and aggregates the processing results to obtain the auditory spatiotemporal features, and the audiovisual consistency learning network includes a graph convolutional neural network layer and a graph attention network layer. The fusion output network is a network that fuses the visual identity consistency features and the audiovisual consistency features to obtain the discriminant features.
[0139] In the embodiment of the present disclosure, the target neural network can be trained in the following manner. First, the first loss function, i.e., the loss function L of the identity consistency branch, can be determined based on the face identity features of each target segment of each sample in the audio and video sequence sample set. id ; Then, a second loss function is determined based on the discriminant features of each sample, wherein the second loss function may be contrast loss, or other loss functions that can replace contrast loss; and a third loss function is determined based on the discriminant features and sample labels of each sample, for example, binary cross entropy loss, or other loss functions that can replace binary cross entropy loss.
[0140] The optimization goal of network model training is to minimize (including but not limited to) the above loss function. After the above loss function meets the requirements, the target neural network can be trained.
[0141] Here, the loss function L of the identity consistency branch is id It can be expressed as:
[0142]
[0143] Among them, N true Represents the number of real samples in a batch, n represents the number of audio and video clips sampled by each sample, i, j∈N true represents the sample numbers with different identities in a batch, p,q∈n represents two different fragment numbers in a sample, τ is the balance coefficient, Represents the face identity feature of the pth segment of the ith sample. This loss can make the identity features of samples with different identities discrete from each other, and the identity features of samples with the same identity as aggregated as possible, so that the model can capture the slight perturbation caused by identity jumps in forged videos, and the visual identity consistency features learned for real and forged samples have more significant differences.
[0144] Contrastive loss L con It can be expressed as:
[0145]
[0146] Where N is the size of the entire batch, Sim(·,·) refers to the cosine similarity, are the discriminant features of the i-th and j-th samples in the batch, respectively, and α is the balance coefficient. This loss function can increase the inter-class difference of the discriminant features output by the classifier.
[0147] Binary cross entropy loss L ce It can be expressed as:
[0148]
[0149] Among them, Y i represents the discriminant feature, Represents a sample label. For example, the sample label can be 1 or 0. A sample label of 1 indicates that the sample is a forged sample, and a sample label of 0 indicates that the sample is not a forged sample.
[0150] Based on the above first loss function, second loss function and third loss function, the final training loss can be obtained, which is expressed as: L = δ1L id +δ2L con +δ3L ce , where δ1, δ2, and δ3 are loss weights that can be adjusted during the training process. At this point, the initial neural network can be trained using the final training loss, and the target neural network is obtained after training.
[0151] like Figure 5 The flowchart of an optional audio and video forgery detection method is shown below. Figure 5 The above process is introduced. Figure 5 As shown, the audio and video forgery detection method specifically includes the following process:
[0152] An audio and video sequence to be tested is obtained, and a plurality of audio clips and a plurality of video clips are intercepted through a sliding window; wherein the plurality of audio clips constitute an audio sequence, and the plurality of video clips constitute a video sequence.
[0153] The face identity features of each video frame in the video clip are extracted through an identity feature extraction network, and the face identity features are fused frame by frame to obtain the identity features at the clip level; based on the similarity of the identity features between any two of the video clips, the visual identity consistency features of the audio and video sequence are determined.
[0154] The face visual feature map of each video frame in the video clip is extracted through the visual feature extraction network, and the face visual feature map is processed through the first temporal modeling network to obtain visual spatiotemporal features.
[0155] Extract the sound features of the audio clip, such as the Mel-frequency cepstral coefficient features, and perform speech encoding on the Mel-frequency cepstral coefficient features through the Transformer attention mechanism network to obtain auditory spatiotemporal features.
[0156] An audio-visual timing graph is determined, and based on the audio-visual timing graph and the audio-visual consistency learning network, the auditory spatiotemporal features and the visual spatiotemporal features are fused to obtain an audio-visual consistency feature. Finally, based on the visual identity consistency feature and the audio-visual consistency feature, a forgery detection result of the audio and video sequence is determined.
[0157] It can be seen from the above description that the technical solution disclosed in the present invention has the following advantages:
[0158] An identity consistency learning method based on identity similarity is proposed, which can effectively capture the identity jump phenomenon, an important clue for distinguishing video forgeries, and improve the reliability and robustness of identifying forgeries that strictly replicate lip movements but lack identity features.
[0159] An audiovisual consistency learning method based on audiovisual timing graph is proposed, and the Transformer attention mechanism is used to learn fine-grained visual and auditory local spatiotemporal information. The audiovisual timing graph structure is explicitly used to define the intra-modal temporal relationship and cross-modal alignment relationship, capturing the visual and auditory intra-modal temporal inconsistency and audiovisual cross-modal temporal inconsistency in forged videos, effectively enhancing the learning ability of multi-modal and multi-scale spatiotemporal potential connections.
[0160] A network framework and training optimization strategy for joint modeling of identity consistency and audio-visual consistency were proposed. It can utilize the principle that "there is a trade-off between accurately synthesizing lip movements with deep fake technology and retaining the target person's identity information", and simultaneously extract the two major clues of identity jump + facial dynamics and voice mismatch in the fake video for judgment, effectively improving the generalization and robustness of audio and video forgery detection.
[0161] The advantages of the technical solution disclosed in the present invention are: ① It explicitly uses graph structures to define the intra-modal temporal relationship and the cross-modal alignment relationship, and captures the visual and auditory intra-modal temporal inconsistency and audio-visual cross-modal temporal inconsistency in forged videos; ② It adopts a joint modeling strategy of identity consistency and audio-visual consistency, and simultaneously extracts the two major clues of identity jump + facial dynamics and voice mismatch in forged videos for judgment, thereby improving the generalization and robustness of audio and video forgery detection.
[0162] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0163] Based on the same inventive concept, an audio and video forgery detection device corresponding to the audio and video forgery detection method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned audio and video forgery detection method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0164] Reference Figure 6 FIG. 1 is a schematic diagram of an audio and video forgery detection device provided by an embodiment of the present disclosure, wherein the device comprises: an acquisition unit 10, a first determination unit 20, a feature extraction unit 30 and a second determination unit 40; wherein:
[0165] An acquisition unit, used to acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence;
[0166] A first determining unit is used to determine a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence;
[0167] A feature extraction unit, configured to extract features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and to align and fuse the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features;
[0168] The second determination unit is used to determine the forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature.
[0169] In a possible implementation manner, the first determining unit is further configured to:
[0170] Extracting facial identity features of each video frame in the video clip, and determining identity features of the video clip based on the facial identity features of each video frame;
[0171] Based on the similarity of the identity features between any two of the video clips, a visual identity consistency feature of the audio and video sequence is determined.
[0172] In a possible implementation manner, the first determining unit is further configured to:
[0173] Determine the minimum similarity of the preset number;
[0174] Based on the difference between the target value and each of the minimum similarities, a visual identity consistency feature of the audio and video sequence is determined.
[0175] In a possible implementation manner, the feature extraction unit is further used for:
[0176] Extracting a facial visual feature map of each video frame in the video clip through a visual feature extraction network to obtain a first feature matrix;
[0177] The visual spatiotemporal features of the video clip are determined using the first feature matrix.
[0178] In a possible implementation manner, the feature extraction unit is further used for:
[0179] Dividing the first feature matrix through a first timing modeling network to obtain a plurality of first matrix sub-blocks, and determining an embedded feature of each of the first matrix sub-blocks;
[0180] The embedded features are visually encoded by the encoding block in the first temporal modeling network to obtain local dynamic information of the face, and the local dynamic information of the face is aggregated to obtain the visual spatiotemporal features.
[0181] In a possible implementation manner, the feature extraction unit is further used for:
[0182] Extracting acoustic features of each audio segment in the audio sequence to obtain a second feature matrix;
[0183] dividing the second characteristic matrix to obtain a plurality of second matrix sub-blocks;
[0184] Each second matrix sub-block is processed by a second temporal modeling network, and the processing results are aggregated to obtain the auditory spatiotemporal features.
[0185] In a possible implementation manner, the feature extraction unit is further used for:
[0186] Determining an audiovisual connection relationship between the audio sequence and the video sequence;
[0187] The auditory spatiotemporal features and the visual spatiotemporal features are fused through the audiovisual connection relationship to obtain the audiovisual consistency features.
[0188] In a possible implementation, the audio-visual connection relationship includes: a segment of any modality in the audio and video sequence is associated with a previous or next segment of all modalities in time sequence, and a segment of any modality is associated with simultaneous segments of other modalities; the modalities include visual modalities and auditory modalities.
[0189] In a possible implementation manner, the feature extraction unit is further used for:
[0190] Performing graph convolution calculation on the auditory spatiotemporal features and the auditory association relationship through a graph convolution neural network layer in an audio-visual consistency learning network to obtain a first output feature, and performing graph convolution calculation on the visual spatiotemporal features and the visual association relationship to obtain a second output feature; the auditory association relationship is used to indicate an audio segment with an association relationship, and the visual association relationship is used to indicate a video segment with an association relationship;
[0191] The first output feature, the second output feature and the cross-modal association relationship are subjected to attention fusion by the graph attention network layer in the audio-visual consistency learning network to obtain a third output feature; the cross-modal association relationship is used to indicate audio clips and video clips having an association relationship;
[0192] The audiovisual consistency feature is determined based on a fusion feature of the second output feature and the third output feature.
[0193] In a possible implementation manner, the second determining unit is further configured to:
[0194] The visual identity consistency feature and the audiovisual consistency feature are fused through a fusion output network to obtain a discriminant feature, and the forgery detection result is determined based on the discriminant feature.
[0195] In a possible implementation manner, the device is also used for:
[0196] Determining a first loss function based on the facial identity features of each target segment of each sample in the audio and video sequence sample set;
[0197] Determining a second loss function based on the discriminant feature of each of the samples;
[0198] Determining a third loss function based on the discriminant features and sample labels of each of the samples;
[0199] Based on the first loss function, the second loss function and the third loss function, the initial neural network is trained to obtain a target neural network after training. The target neural network includes a visual feature extraction network, a first temporal modeling network, a second temporal modeling network, an audio-visual consistency learning network and a fusion output network.
[0200] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0201] Corresponds to Figure 1 The audio and video forgery detection method in the present disclosure also provides an electronic device 700, such as Figure 7 FIG. 7 is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present disclosure, including:
[0202] Processor 71, memory 72, and bus 73; memory 72 is used to store execution instructions, including internal memory 721 and external memory 722; the internal memory 721 is also called internal memory, which is used to temporarily store the operation data in the processor 71 and the data exchanged with the external memory 722 such as a hard disk. The processor 71 exchanges data with the external memory 722 through the internal memory 721. When the electronic device 700 is running, the processor 71 communicates with the memory 72 through the bus 73, so that the processor 71 executes the following instructions:
[0203] Acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence;
[0204] Determining a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence;
[0205] Extracting features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and aligning and fusing the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features;
[0206] Based on the visual identity consistency feature and the audio-visual consistency feature, a forgery detection result of the audio and video sequence is determined.
[0207] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the audio and video forgery detection method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0208] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the audio and video forgery detection method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0209] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0210] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0212] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0213] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0214] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A method for detecting audio and video forgery, It is characterized in that include: Acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence; Determining a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence; Extracting features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and aligning and fusing the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features; Based on the visual identity consistency feature and the audio-visual consistency feature, a forgery detection result of the audio and video sequence is determined.
2. The method according to claim 1, It is characterized in that The determining of visual identity consistency features based on the identity features of each video clip in the video sequence includes: Extracting facial identity features of each video frame in the video clip, and determining identity features of the video clip based on the facial identity features of each video frame; Based on the similarity of the identity features between any two of the video clips, a visual identity consistency feature of the audio and video sequence is determined.
3. The method according to claim 2, It is characterized in that The determining of the visual identity consistency feature of the audio and video sequence based on the similarity of the identity features between any two of the video clips includes: Determine the minimum similarity of the preset number; Based on the difference between the target value and each of the minimum similarities, a visual identity consistency feature of the audio and video sequence is determined.
4. The method according to claim 1, It is characterized in that The step of extracting features from the video sequence to obtain visual spatiotemporal features includes: Extracting a facial visual feature map of each video frame in the video clip through a visual feature extraction network to obtain a first feature matrix; The visual spatiotemporal features of the video clip are determined using the first feature matrix.
5. The method according to claim 4, It is characterized in that The determining the visual spatiotemporal features of the video clip by using the first feature matrix includes: Dividing the first feature matrix through a first timing modeling network to obtain a plurality of first matrix sub-blocks, and determining an embedded feature of each of the first matrix sub-blocks; The embedded features are visually encoded by the encoding block in the first temporal modeling network to obtain local dynamic information of the face, and the local dynamic information of the face is aggregated to obtain the visual spatiotemporal features.
6. The method according to claim 1, It is characterized in that The step of extracting features from the audio sequence to obtain auditory spatiotemporal features includes: Extracting acoustic features of each audio segment in the audio sequence to obtain a second feature matrix; dividing the second characteristic matrix to obtain a plurality of second matrix sub-blocks; Each second matrix sub-block is processed by a second temporal modeling network, and the processing results are aggregated to obtain the auditory spatiotemporal features.
7. The method according to claim 1, It is characterized in that The fusing of the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features includes: Determining an audiovisual connection relationship between the audio sequence and the video sequence; The auditory spatiotemporal features and the visual spatiotemporal features are fused through the audiovisual connection relationship to obtain the audiovisual consistency features.
8. The method according to claim 7, It is characterized in that The audio-visual connection relationship includes: the clips of any modality in the audio and video sequence are associated with the previous or next clips of all modalities in time sequence, and the clips of any modality are associated with the simultaneous clips of other modalities; the modalities include visual modalities and auditory modalities.
9. The method according to claim 7, It is characterized in that The step of fusing the auditory spatiotemporal features and the visual spatiotemporal features through the audiovisual connection relationship to obtain the audiovisual consistency features includes: Performing graph convolution calculation on the auditory spatiotemporal features and the auditory association relationship through a graph convolution neural network layer in an audio-visual consistency learning network to obtain a first output feature, and performing graph convolution calculation on the visual spatiotemporal features and the visual association relationship to obtain a second output feature; the auditory association relationship is used to indicate an audio segment with an association relationship, and the visual association relationship is used to indicate a video segment with an association relationship; The first output feature, the second output feature and the cross-modal association relationship are subjected to attention fusion by the graph attention network layer in the audio-visual consistency learning network to obtain a third output feature; the cross-modal association relationship is used to indicate audio clips and video clips having an association relationship; The audiovisual consistency feature is determined based on a fusion feature of the second output feature and the third output feature.
10. The method according to claim 1, It is characterized in that The determining the forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature includes: The visual identity consistency feature and the audiovisual consistency feature are fused through a fusion output network to obtain a discriminant feature, and the forgery detection result is determined based on the discriminant feature.
11. The method according to claim 1, It is characterized in that The method further comprises: Determining a first loss function based on the facial identity features of each target segment of each sample in the audio and video sequence sample set; Determining a second loss function based on the discriminant feature of each of the samples; Determining a third loss function based on the discriminant features and sample labels of each of the samples; Based on the first loss function, the second loss function and the third loss function, the initial neural network is trained to obtain a target neural network after training. The target neural network includes a visual feature extraction network, a first temporal modeling network, a second temporal modeling network, an audio-visual consistency learning network and a fusion output network.
12. An audio and video forgery detection device, It is characterized in that include: An acquisition unit, used to acquire an audio and video sequence to be tested; the audio and video sequence includes an audio sequence and a video sequence; A first determining unit is used to determine a visual identity consistency feature based on the identity features of each video clip in the video sequence; wherein the visual identity consistency feature is used to indicate whether there is a significant change in the visual identity features of the same target in the video sequence; A feature extraction unit, configured to extract features from the audio sequence and the video sequence respectively to obtain auditory spatiotemporal features and visual spatiotemporal features, and to align and fuse the auditory spatiotemporal features and the visual spatiotemporal features to obtain audiovisual consistency features; The second determination unit is used to determine the forgery detection result of the audio and video sequence based on the visual identity consistency feature and the audio-visual consistency feature.
13. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the audio and video forgery detection method as described in any one of claims 1 to 11 are performed.
14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio and video forgery detection method according to any one of claims 1 to 11 are executed.
15. A computer program product, It is characterized in that The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the audio and video forgery detection method according to any one of claims 1 to 11.
Citation Information
Cited By
Multi-modal speaker authentic identification model training method and electronic equipment
CN120148521A
Method for training multi-modal speaker verification model and electronic device
CN120148521B
Deepfake content detection method and electronic apparatus
TWI921096B
