Audio-visual authenticity identification method, system and device and storage medium

Abstract

The invention discloses an audiovisual authenticity identification method, system and device and a storage medium, and relates to the technical field of digital security, the method comprises the following steps: constructing a first annotation data set and a second annotation data set according to multi-modal data, using the first annotation data set to pre-train an audiovisual authenticity identification model, and using the second annotation data set to pre-train the audiovisual authenticity identification model; the audio-visual authenticity identification model comprises an image modal sub-model and an audio modal sub-model; applying the pre-trained audio-visual authenticity identification model to an audio-visual authenticity identification task, performing fine adjustment on the audio-visual authenticity identification model by using a second annotation data set, and optimizing the audio-visual authenticity identification model after fine adjustment; and inputting to-be-identified multi-modal data into the optimized audio-visual authenticity identification model, and performing authenticity identification. By using image data, video data, audio data and other data in various modes, the model can comprehensively judge the authenticity of the digital content from multiple angles, so that the identification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data security, and in particular to an audio-visual authenticity identification method, system, device and storage medium. Background Art

[0002] With the rapid development of digital technology, the authenticity of digital content has become increasingly important. Digital content forgery detection is an important part of the effective supervision and governance of digital content generated by deep synthesis technology. At present, forgery detection technology mainly includes single-modal authentication method and multi-modal authentication method.

[0003] Single-modal authentication methods mainly detect single modalities such as images and audio. Although they have fast processing speed, they cannot detect the authenticity of multiple modalities at the same time when processing multimodal data, resulting in limited scene generalization performance, obvious data dependence problems, and difficulty in maintaining good generalization performance in large-scale data. Multimodal authentication methods attempt to combine multiple modalities for detection, but when faced with large-scale and diverse data, the generalization ability of the model is insufficient, and the accuracy and robustness in complex scenarios are limited. In addition, although transfer learning and multi-task learning have solved the problem of insufficient data to a certain extent, the negative transfer phenomenon and the difficulty in selecting pre-trained models limit their application. In addition, the existing detection models have poor interpretability, the detection results are difficult to explain, and the forensic results are difficult to align with human cognition, making evidence extraction difficult. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, system, device and storage medium for audio-visual authenticity identification. By using data of multiple modalities such as image data, video data and audio data, the model can comprehensively judge the authenticity of digital content from multiple angles, thereby improving the accuracy of identification. Pre-training is performed using a large-scale first annotated data set to learn general feature representations and task knowledge; then fine-tuning is performed on a second annotated data set for a specific task to further optimize the model parameters so that the model can better adapt to specific tasks.

[0005] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions: According to a first aspect of an embodiment of the present application, a method for distinguishing the authenticity of audio and video is provided, the method comprising: Constructing a first annotated data set and a second annotated data set according to the multimodal data, wherein the first annotated data set and the second annotated data set include real samples and forged samples of the multimodal data; the multimodal data includes image data, video data and audio data; Pre-training an audiovisual authenticity identification model using a first annotated data set, wherein the audiovisual authenticity identification model includes an image modality sub-model and an audio modality sub-model; Applying the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, fine-tuning the audio-visual authenticity identification model using the second annotated data set, and optimizing the fine-tuned audio-visual authenticity identification model; The multimodal data to be identified is input into the optimized audio-visual authenticity identification model to perform authenticity identification.

[0006] Optionally, the step of inputting the multimodal data to be identified into the optimized audio-visual authenticity identification model to perform authenticity identification includes: Inputting the multimodal data to be identified into the optimized audio-visual authenticity identification model, and extracting image features and / or audio features through the image modality sub-model and the audio modality sub-model; The image features and / or audio features are fused and weighted through the feature extraction layer to obtain the target authenticity identification result.

[0007] Optionally, the image features and / or audio features are fused and weighted through a feature extraction layer according to the following formula to obtain a target authenticity identification result: The image features are , the audio features are , σ is the sigmoid function, which is used to map the fusion result to the interval [0,1]. and is the fusion weight.

[0008] Optionally, the pre-training of the audiovisual authenticity identification model using the first annotated data set includes: The audiovisual authenticity identification model is pre-trained using the first annotated data set with the goal of minimizing the total loss function; wherein the total loss function L(θ) is expressed as: in, are the parameters of the model, and are the loss functions for image and audio modalities respectively, and is the weight balancing factor between the image modality and the audio modality.

[0009] Optionally, the image modality loss function LI Calculate according to the following formula: =- ( log( )+(1- )log(1- )) in, is the true label of the image modality sample, is the probability predicted by the model, is the number of image modality samples; The audio modality loss function Calculate according to the following formula: =- ( log( )+(1- )log(1- )) in, is the true label of the audio modality sample, is the probability predicted by the model, 𝑀 is the number of audio modal samples.

[0010] Optionally, fine-tuning the audiovisual authenticity identification model using the second annotated data set includes: Using the second annotated data set, the audio-visual authenticity identification model is fine-tuned with the goal of minimizing the loss function of the target task; wherein the loss function of the audio-visual authenticity identification task is Ltask : in, and They are the loss functions of the image modality and audio modality in the audio-visual authenticity identification task respectively.

[0011] Optionally, the fine-tuned audiovisual authenticity identification model is optimized, including: The fine-tuned audio-visual authenticity identification model is optimized using the gradient descent method according to the following formula to update the model parameters; in, is the learning rate, is the gradient operator, L task is the loss function for the audio-visual authenticity identification task.

[0012] According to a second aspect of an embodiment of the present application, a system for distinguishing the authenticity of audio and video is provided, the system comprising: A labeling data set construction module, used to construct a first labeling data set and a second labeling data set according to multimodal data, wherein the first labeling data set and the second labeling data set include real samples and forged samples of the multimodal data; the multimodal data includes image data, video data and audio data; A pre-training module, used to pre-train an audio-visual authenticity identification model using a first annotated data set, wherein the audio-visual authenticity identification model includes an image modality sub-model and an audio modality sub-model; A fine-tuning optimization module, used to apply the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, fine-tune the audio-visual authenticity identification model using the second annotated data set, and optimize the fine-tuned audio-visual authenticity identification model; The application module is used to input the multimodal data to be identified into the optimized audio-visual authenticity identification model to perform authenticity identification.

[0013] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0014] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0015] In summary, the embodiments of the present application provide a method, system, device and storage medium for audio-visual authenticity identification. By constructing a first annotated data set and a second annotated data set according to multimodal data, the first annotated data set and the second annotated data set include real samples and forged samples of multimodal data; the multimodal data include image data, video data and audio data; using the first annotated data set to pre-train the audio-visual authenticity identification model, the audio-visual authenticity identification model includes an image modality sub-model and an audio modality sub-model; applying the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, using the second annotated data set to fine-tune the audio-visual authenticity identification model, and optimizing the fine-tuned audio-visual authenticity identification model; inputting the multimodal data to be identified into the optimized audio-visual authenticity identification model for authenticity identification. By using data of multiple modalities such as image data, video data and audio data, the model can comprehensively judge the authenticity of digital content from multiple angles, thereby improving the accuracy of identification. Pre-training is performed using a large-scale first labeled dataset to learn general feature representations and task knowledge; then fine-tuning is performed on a second labeled dataset for a specific task to further optimize the model parameters so that the model can better adapt to specific tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0017] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0018] Figure 1 A flowchart of an audio-visual authenticity identification method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an audio-visual authenticity identification model provided in an embodiment of the present application; Figure 3 A flow chart of transfer learning pre-training and fine-tuning provided in an embodiment of the present application; Figure 4 A schematic diagram of the result fusion provided in the embodiment of the present application; Figure 5 A flow chart of model evaluation provided for an embodiment of the present application; Figure 6 A schematic diagram of an audio-visual authenticity identification system provided in an embodiment of the present application; Figure 7 A structural diagram of an electronic device provided in an embodiment of the present application is shown; Figure 8 A diagram showing a computer-readable storage medium provided by an embodiment of the present application.

[0019] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0022] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0023] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0025] Figure 1 A method for distinguishing the authenticity of audiovisual content provided by an embodiment of the present application is shown, and the method includes: Step 101: constructing a first annotated data set and a second annotated data set according to multimodal data, wherein the first annotated data set and the second annotated data set include real samples and forged samples of the multimodal data; the multimodal data includes image data, video data and audio data; Step 102: pre-training an audiovisual authenticity identification model using a first annotated data set, wherein the audiovisual authenticity identification model includes an image modality sub-model and an audio modality sub-model; Step 103: applying the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, fine-tuning the audio-visual authenticity identification model using the second annotated data set, and optimizing the fine-tuned audio-visual authenticity identification model; Step 104: input the multimodal data to be identified into the optimized audio-visual authenticity identification model to perform authenticity identification.

[0026] In a possible implementation, in step 102, using the first annotated data set to pre-train the audiovisual authenticity identification model includes: The audiovisual authenticity identification model is pre-trained using the first annotated data set with the goal of minimizing the total loss function; wherein the total loss function L(θ) is expressed as: in, are the parameters of the model, and are the loss functions for image and audio modalities respectively, and is the weight balancing factor between the image modality and the audio modality.

[0027] The pre-training phase uses a large-scale first-labeled dataset to enable the model to learn universal feature representations, thereby improving the model's generalization capabilities on different data distributions and tasks. For example, the universal features learned by the pre-trained model on large-scale image and audio data can be effectively applied to specific image or audio authenticity identification tasks to improve the ability to identify forged images or audio. By defining the total loss function, the model can balance the importance of the image modality and the audio modality, thereby achieving effective fusion of multimodal data. For example, in a video authenticity identification task that contains images and audio, the image modality loss function LI And the audio modality loss function LA Weigh the prediction accuracy of image and audio modalities separately, and the weight balance factor λI and λA Used to adjust the contribution of both.

[0028] In a possible implementation, in step 102, the image modality loss function LI Calculate according to the following formula: =- ( log( )+(1- )log(1- )) in, is the true label of the image modality sample, is the probability predicted by the model, is the number of image modality samples; The audio modality loss function Calculate according to the following formula: =- ( log( )+(1- )log(1- )) in, is the true label of the audio modality sample, is the probability predicted by the model, 𝑀 is the number of audio modal samples.

[0029] By using multimodal data (images, audio) for pre-training, the model can learn feature representations in different modalities, thereby more comprehensively judging the authenticity of digital content. For example, in video authenticity identification, the image modality can detect details and textures in the picture, and the audio modality can analyze the naturalness and coherence of the sound. The combination of the two can more comprehensively reveal traces of forgery.

[0030] In a possible implementation, in step 103, fine-tuning the audiovisual authenticity identification model using the second annotated data set includes: Using the second annotated data set, the audio-visual authenticity identification model is fine-tuned with the goal of minimizing the loss function of the target task; wherein the loss function of the audio-visual authenticity identification task is Ltask : in, and They are the loss functions of the image modality and audio modality in the audio-visual authenticity identification task respectively.

[0031] Through transfer learning, the pre-trained model learns general knowledge on a large-scale dataset and then fine-tunes it on a small-scale second annotated dataset, greatly reducing the reliance on a large amount of annotated data. For example, in the task of audio authenticity identification, only a small amount of annotated audio samples are needed to fine-tune the pre-trained model to adapt it to a specific audio authenticity identification scenario.

[0032] In a possible implementation, in step 103, the fine-tuned audiovisual authenticity identification model is optimized, including: The fine-tuned audio-visual authenticity identification model is optimized using the gradient descent method according to the following formula to update the model parameters; in, is the learning rate, is the gradient operator, L task is the loss function for the audio-visual authenticity identification task.

[0033] By optimizing the fine-tuned model using the gradient descent method, the model parameters can be precisely adjusted according to the loss function of the target task. Using the gradient descent method (such as the Adam optimizer) can efficiently update the model parameters, ensuring that the model converges quickly during the fine-tuning phase and further improving performance.

[0034] Suppose that during the fine-tuning phase, the model needs to identify a sample containing a forged video. The forged video may contain the following features: Image modality: The image contains forged traces such as blurred edges and inconsistent textures. Audio modality: The audio contains abnormal features such as frequency mutations and unnatural speech speed. Through fine-tuning and optimization, the model can better identify these features and generate the final authenticity identification result through weighted fusion. For example, if the weight of the image modality is λI Higher, the model will pay more attention to the artifacts in the image; if the weight of the audio modality is λA Higher, the model pays more attention to unusual features in the audio.

[0035] In a possible implementation, in step 104, inputting the multimodal data to be identified into the optimized audiovisual authenticity identification model to perform authenticity identification includes: The multimodal data to be identified is input into the optimized audio-visual authenticity identification model, and image features and / or audio features are extracted through the image modality sub-model and the audio modality sub-model; the image features and / or audio features are fused and weighted through the feature extraction layer to obtain the target authenticity identification result.

[0036] The image modality sub-model and the audio modality sub-model are used to extract image features and audio features respectively. This separate extraction method ensures that the model can capture potential traces of forged content from different angles. The image features and audio features are fused through the feature extraction layer, and the contributions of different modalities are adjusted in a weighted manner. This fusion method can make full use of multimodal information and avoid possible misjudgments of a single modality. At the same time, through weighted processing, the model can dynamically adjust the importance of each modality according to the needs of specific tasks.

[0037] In a possible implementation, the image features and / or audio features are fused and weighted through a feature extraction layer according to the following formula to obtain a target authenticity identification result: The image features are , the audio features are , σ is the sigmoid function, which is used to map the fusion result to the interval [0,1]. and is the fusion weight.

[0038] The sigmoid function is used to map the fusion result to the interval [0,1], so that the final authenticity identification result has clear interpretability (e.g., close to 1 means true, close to 0 means forged). By fusing image features and audio features, the model can make comprehensive use of multimodal information, thereby improving the accuracy of authenticity identification. By assigning weights to features of different modalities, the model can adjust the contribution of each modality according to the needs of specific tasks, further improving the reliability of identification. Adaptive weight adjustment mechanisms can also be applied. For example, the weights are dynamically adjusted according to the feature distribution of the input data. If the confidence of the image feature is high (e.g., the traces of image forgery are obvious), the weight of the image modality is automatically increased; conversely, if the audio feature is more reliable, the weight of the audio modality is increased. Adjust the weight according to the needs of different tasks. For example, in the task of video authenticity identification, if the focus of the task is to detect image forgery, the weight of the image modality can be increased; if the focus of the task is to detect audio forgery, the weight of the audio modality can be increased.

[0039] It should be noted that in addition to linear weighted fusion, other fusion methods can also be used in the embodiments of the present application, such as multi-layer perceptron (MLP) fusion, attention mechanism fusion, etc. For example, the importance of different modal features in different scenarios can be dynamically adjusted through the attention mechanism, so that the model can more flexibly deal with complex counterfeiting methods.

[0040] Suppose there is a video to be identified, in which the image part has obvious signs of forgery (such as blurred edges and inconsistent textures), while the audio part is relatively normal. Through this technical solution: the image modality sub-model extracts image features, finds that there are signs of forgery in the image, and outputs the image feature vector yI. The audio modality sub-model extracts audio features, finds that there are no obvious abnormalities in the audio part, and outputs the audio feature vector yA. The feature extraction layer performs weighted fusion of image features and audio features according to preset weights wI and wA to obtain a weighted feature vector. Finally, the fusion result is mapped to the [0,1] interval through the sigmoid function to obtain the authenticity identification result y. If the weight wI of the image modality is higher, the model will be more inclined to believe that the video is forged.

[0041] Suppose there is a forged video, in which the image part has signs of blurred edges and inconsistent textures, and the audio part has abnormalities such as sudden frequency changes and unnatural speech speed. Through this technical solution, the model learns the common features of images and audio in the pre-training stage, and adjusts it for the video authenticity identification task in the fine-tuning stage. Finally, the model extracts image features and audio features through the image modality sub-model and the audio modality sub-model respectively, and generates the final authenticity identification result through weighted fusion. If the weight of the image modality is higher, the model will pay more attention to the forgery traces in the image; if the weight of the audio modality is higher, the model will pay more attention to the abnormal features in the audio.

[0042] It should be noted that in the embodiments of the present application, in addition to images and audio, text modalities (such as video subtitles, audio transcriptions, etc.) can also be introduced. For example, in a video, the subtitle content may be inconsistent with the audio or image content, and this inconsistency can serve as a clue of forgery. By adding text modality features yT , and introduce the corresponding weights w , which can further enrich the model input and improve the accuracy of authenticity identification.

[0043] The audiovisual authenticity identification method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0044] Aspect 1: Data collection and preprocessing Step 1: Collect a large amount of image and audio data, including real and forged samples. Image data can come from public datasets such as Celeb-DF, DFDC, etc.; audio data can come from datasets such as LibriSpeech. Ensure the diversity and representativeness of the data, covering different scenarios, different forgery methods, and different quality levels.

[0045] Step 2: Data cleaning: remove noisy data and invalid samples.

[0046] Step 3: Normalization: Normalize the data to the same scale, for example, normalize image pixel values ​​to the [0,1] interval and audio signals to the [-1,1] interval.

[0047] Step 4: Data enhancement: Enhance image data through operations such as rotation, scaling, and cropping; enhance audio data through operations such as time stretching and frequency masking to increase data diversity.

[0048] Second aspect: Construction of audio-visual authenticity identification model Build an audio-visual authenticity identification model, which includes multiple tasks, such as image authenticity identification task, audio authenticity identification task, etc. Figure 2 The schematic diagram of the structure of the audiovisual authenticity identification model provided by the embodiment of the present application is shown. The model is an audiovisual authenticity identification model, which is designed to process data in two modalities, image and audio, and perform authenticity identification.

[0049] Part 1: Model Structure 1. Assume the input of image mode is , the input of the audio mode is The structure of the model can be expressed as: in, is the image modality sub-model, is the audio modal sub-model.

[0050] 2. Image modality sub-model: Use convolutional neural network (CNN) as the basic network structure, specifically ResNet50. ResNet50 is a deep residual network that can effectively process image data and extract image features. The output of this sub-model is the image authenticity identification result.

[0051] 3. Audio modality sub-model: Use recurrent neural network (RNN) as the basic network structure, specifically LSTM (Long Short-Term Memory Network). LSTM can effectively process sequence data, such as audio signals, and extract audio features. The output of this sub-model is the audio authenticity identification result.

[0052] 4. Feature extraction layer: This layer receives the output results of the image modality sub-model and the audio modality sub-model, namely the image authenticity identification results and the audio authenticity identification results. The function of the feature extraction layer is to fuse the features of the two modalities and extract a more comprehensive feature representation.

[0053] 5. Multimodal fusion result: The output of the feature extraction layer is the multimodal fusion result, which integrates the information of both image and audio modalities and can perform authenticity identification more accurately.

[0054] The present invention can comprehensively judge the authenticity of digital content from multiple angles by fusing data of multiple modes, such as images, audio, etc. Compared with the single-mode identification method, multimodal fusion can capture feature information of more dimensions, thereby significantly improving the accuracy of identification. For example, in video authenticity identification, the image mode can detect the details and textures in the picture, and the audio mode can analyze the naturalness and coherence of the sound. The combination of the two can more comprehensively reveal the traces of forgery.

[0055] Part II: Loss Function The loss function of the model consists of the image modality loss function and the audio modality loss function, and the total loss function is obtained by weighted summation.

[0056] 1. Image modality loss function: =- ( log( )+(1- )log(1- )) in, is the true label, is the probability predicted by the model, is the number of samples. This is a binary cross entropy loss function that measures the difference between the model's predicted probability and the true label.

[0057] 2. Audio modality loss function: =- ( log( )+(1- )log(1- )) in, is the true label, is the probability predicted by the model, is the number of samples. This is also a binary cross entropy loss function, which is used to measure the difference between the model's predicted probability and the true label.

[0058] 3. Total loss function of the audio-visual authenticity identification model It can be expressed as: in, and are the loss functions for image and audio modalities respectively, and is a weight balancing factor used to adjust the importance of different tasks. The total loss function comprehensively considers the image modality loss and the audio modality loss through weighted summation, thereby optimizing the entire audio-visual authenticity identification model.

[0059] This audio-visual authenticity identification model processes image and audio data separately and performs feature fusion, which can perform more comprehensive authenticity identification. The loss function is designed to take into account the characteristics of different modalities and adjust the importance of different tasks through weight balancing factors, thereby achieving more accurate multi-modal fusion results.

[0060] The third aspect: transfer learning application Transfer learning is a machine learning technique that improves the learning efficiency and performance of the target task by applying the knowledge learned from one task (source task) to another related task (target task). The application of transfer learning makes it possible to effectively train and fine-tune the model with a small amount of labeled data. In practical applications, obtaining a large amount of labeled data is often costly, time-consuming and laborious. The present invention greatly reduces the dependence on a large amount of labeled data and reduces the cost of data collection and labeling by learning general knowledge on a large-scale pre-training dataset and then fine-tuning it on a small amount of labeled target dataset. For example, in the task of audio authenticity identification, only a small amount of labeled audio samples are needed to fine-tune the pre-trained model to adapt it to a specific audio authenticity identification scenario.

[0061] In the multimodal digital authenticity identification task, Figure 3The flowchart of transfer learning pre-training and fine-tuning provided by the embodiment of the present application is shown, which specifically includes the following steps: Phase 1: Pre-training: Pre-train the audio-visual authenticity identification model on a large-scale annotated dataset. The goal of pre-training is to minimize the total loss function L, where θ is the parameter of the model. Through pre-training, the model learns common feature representations and task knowledge, which helps the model generalize better in subsequent tasks.

[0062] Pre-training the audio-visual authenticity identification model on large-scale annotated datasets allows the model to learn common feature representations and task knowledge. These datasets usually contain rich samples and labels, which helps the model learn more comprehensive feature representations. The goal of pre-training is to minimize the total loss function : in, are the parameters of the model.

[0063] The general feature representations learned by the audio-visual authentication model during the pre-training stage have strong generalization capabilities. These feature representations are not only applicable to specific authentication tasks, but also maintain effectiveness in data of different modalities and different authenticity situations. For example, the image features learned by the pre-trained model can effectively identify different types of forged images, including simply edited images and complex deep forged images.

[0064] The second stage: learn common features and task knowledge to obtain a pre-trained model.

[0065] During the pre-training process, the general features and task knowledge learned by the model can be transferred to other related tasks to improve the learning efficiency of the target task. The audio-visual authenticity identification model learns multiple related tasks simultaneously during the pre-training stage, which can more efficiently utilize the information in the data. Sharing feature representations between different tasks allows models to learn from and complement each other during the learning process, improving the efficiency of data utilization. For example, in multi-task learning of image and audio modalities, the texture features learned by the image modality can assist the audio modality in better understanding the authenticity of the audio content, and vice versa.

[0066] After pre-training, the model already has certain feature extraction and task understanding capabilities, which can serve as the basis for transfer learning. Using transfer learning, the general feature representation and task knowledge learned by the model in the pre-training stage can be transferred to specific multimodal digital authenticity identification tasks. This enables the model to better adapt and identify authenticity when faced with new and unseen data, further improving accuracy. For example, general features such as edge detection and texture recognition learned by the pre-trained model on large-scale image data can be effectively applied to specific image authenticity identification tasks to improve the ability to identify forged images.

[0067] Stage 3: Fine-tune using a small amount of labeled data; Migrate the pre-trained model to the specific multimodal digital authenticity identification task and fine-tune the model using a small amount of labeled data. The goal of fine-tuning is to minimize the loss function of the specific task. : in, θ are the parameters of the model, is the loss function for a specific task: in, and They are the loss functions for specific tasks for image and audio modalities respectively. λ I and λ A is a weight balancing factor used to adjust the importance of tasks in different modalities.

[0068] During fine-tuning, the model is adjusted based on the data and objectives of a specific task to improve performance on that task. Since pre-training has provided good feature representation and task knowledge, only a small amount of labeled data is needed to achieve good performance during the fine-tuning phase. Through fine-tuning, the performance of the model on a specific task is optimized and it can better complete the multimodal digital authenticity identification task. Ultimately, the pre-trained and fine-tuned model can be applied to the multimodal digital authenticity identification task to achieve authenticity identification of multimodal data such as images and audio.

[0069] Pre-trained models in transfer learning can be reused, avoiding the high computational cost of training models from scratch. Pre-trained models have learned rich feature representations and task knowledge on large-scale datasets and can be directly applied to new tasks with only a small amount of fine-tuning. This greatly reduces training time and computing resource consumption. For example, a pre-trained ResNet50 model can be quickly migrated to the image authenticity identification task, and only the last few layers need to be fine-tuned to achieve good performance. The audio-visual authenticity identification model can handle multiple related tasks at the same time, improving computational efficiency. During the training process, multiple tasks can be performed in parallel, sharing the computational results of the feature extraction layer, reducing repeated calculations. For example, in multi-task learning of image and audio modalities, the feature extraction layer can provide feature representations for both the image authenticity identification task and the audio authenticity identification task, improving overall computational efficiency.

[0070] The fourth aspect: model optimization and fusion Figure 3 The result fusion schematic diagram provided in the embodiment of the present application is shown, which specifically includes the following steps: Step 1: Adjust the weight balance factor: By adjusting the weight balancing factor λ I and λ A , which can optimize the performance of the audio-visual authenticity identification model. These factors are used to balance the importance of tasks in different modalities, thereby improving the performance of the model on specific tasks.

[0071] Step 2: Model optimization: Use the gradient descent method for optimization. Specifically, you can use the Adam optimizer. The Adam optimizer combines the advantages of momentum and adaptive learning rate to update model parameters more effectively.

[0072] During fine-tuning, by adjusting the weight balancing factor and The performance of the audio-visual authenticity identification model can be optimized by other means. The specific optimization method can be the gradient descent method, such as the Adam optimizer: in, is the learning rate, is the gradient operator, L task is the loss function for a specific task.

[0073] Step 3: Result fusion: Result fusion is to integrate the identification results of different modalities to obtain the final authenticity identification result.

[0074] Assume that the identification result of the image modality is , the identification result of the audio modality is , the fused result y can be expressed as: Among them, σ is the sigmoid function, which is used to map the fusion result to the [0,1] interval. and Fusion weights can be determined by cross-validation or other methods. and Determined by cross-validation and other methods. Cross-validation is a commonly used model evaluation method that can help select the optimal model parameters. Determine the fusion weights by cross-validation and To ensure the accuracy and reliability of the fusion results.

[0075] Through model optimization and result fusion, the performance of multimodal digital authenticity identification tasks can be significantly improved. Model optimization optimizes the performance of the audio-visual authenticity identification model by adjusting the weight balance factor and using the gradient descent method. Result fusion obtains the final authenticity identification result by weighted fusion of the identification results of different modalities. This method can make full use of information from different modalities to improve the accuracy and robustness of identification. Through multimodal fusion, the present invention can visualize the features of different modalities to help users understand the decision-making process of the model more intuitively. For example, the traces of forgery in the image modality (such as blurred edges and inconsistent textures) and the abnormal features in the audio modality (such as sudden frequency changes and unnatural speech speed) can be visualized, so that users can clearly see how the model integrates these features to judge the authenticity of digital content.

[0076] Aspect 5: Model evaluation and application Model evaluation is an important step to verify the accuracy and generalization ability of the optimized model on the test dataset. Figure 4 The model evaluation flow chart provided in the embodiment of the present application is shown, which specifically includes: Phase 1: Model Evaluation Use the test data set to evaluate the optimized model to verify its accuracy and generalization ability. Evaluation indicators can include accuracy: Accuracy measures the proportion of correct predictions made by the model.

[0077] Precision measures the proportion of samples that are actually positive among those predicted by the model: Recall measures the proportion of samples that are actually positive that are correctly predicted as positive by the model: The F1 score is the harmonic mean of precision and recall, taking both into account: Among them, TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example.

[0078] By calculating the above evaluation indicators, the accuracy and generalization ability of the model are verified to ensure the performance of the model in practical applications.

[0079] Phase 2: Model Application Apply the trained model to actual multimodal digital authenticity identification scenarios, such as identifying fake video and audio content on the Internet. On social media platforms, automatically detect and identify fake video, image, and audio content to prevent the spread of false information. For example, by real-time monitoring of user-uploaded content, quickly identify whether there are traces of forgery. And, identify whether digital media content has been tampered with or forged to protect the integrity and authenticity of copyrighted content. For example, in the process of digital media distribution, use the authenticity identification model to ensure that the content has not been illegally tampered with. And, detect abnormal behavior or forged content in surveillance videos to improve the reliability of security monitoring. For example, by analyzing the image and audio features in the surveillance video, determine whether there are forged surveillance images. And, ensure the authenticity of online teaching videos to prevent malicious tampering of teaching content. For example, in online course platforms, use the authenticity identification model to ensure that teaching videos have not been tampered with to ensure teaching quality.

[0080] The specific application steps are as follows: Step 1: Data input: Input the multimodal data to be identified (such as images and audio) into the model.

[0081] Step 2: Result output: The model outputs the fused authenticity identification result y.

[0082] Step 3: Decision-making: Based on The value of is used to determine the authenticity of the data. , it is judged as real data; otherwise, it is judged as forged data.

[0083] Through the above technical scheme, the present invention can fully utilize the advantages of transfer learning, improve the accuracy and efficiency of multimodal digital authenticity identification, while reducing the dependence on a large amount of labeled data and enhancing the generalization ability of the model.

[0084] Through model evaluation and application, the effectiveness and reliability of the model in actual scenarios can be ensured. Model evaluation verifies the accuracy and generalization ability of the model by calculating indicators such as accuracy, precision, recall and F1 score. Model application applies the trained model to the actual multimodal digital authenticity identification task, inputs the multimodal data to be identified, outputs the fused authenticity identification results, and makes decisions based on the results. This method can improve the efficiency and accuracy of authenticity identification of digital content, and help prevent and combat the spread of false information.

[0085] The present invention can extract and display specific evidence during the identification process, and these evidences can serve as a reliable basis for subsequent operations (such as legal proceedings). For example, in image authenticity identification, the forged areas and forged features in the image can be extracted and displayed; in audio authenticity identification, the abnormal paragraphs and feature parameters in the audio can be extracted and displayed. These evidences not only improve the interpretability of the model, but also enhance the user's trust in the identification results.

[0086] In summary, the embodiment of the present application provides an audio-visual authenticity identification method, by constructing a first annotated data set and a second annotated data set according to multimodal data, wherein the first annotated data set and the second annotated data set include real samples and forged samples of multimodal data; the multimodal data include image data, video data and audio data; the first annotated data set is used to pre-train an audio-visual authenticity identification model, and the audio-visual authenticity identification model includes an image modality sub-model and an audio modality sub-model; the pre-trained audio-visual authenticity identification model is applied to the audio-visual authenticity identification task, the audio-visual authenticity identification model is fine-tuned using the second annotated data set, and the fine-tuned audio-visual authenticity identification model is optimized; the multimodal data to be identified is input into the optimized audio-visual authenticity identification model to perform authenticity identification. By using data of multiple modalities such as image data, video data and audio data, the model can comprehensively judge the authenticity of digital content from multiple angles, thereby improving the accuracy of identification. Pre-training is performed using a large-scale first labeled dataset to learn general feature representations and task knowledge; then fine-tuning is performed on a second labeled dataset for a specific task to further optimize the model parameters so that the model can better adapt to specific tasks.

[0087] Based on the same technical concept, the embodiment of the present application also provides an audio-visual authenticity identification system, such as Figure 6 As shown, the system comprises: The annotated data set construction module 601 is used to construct a first annotated data set and a second annotated data set according to the multimodal data, wherein the first annotated data set and the second annotated data set include real samples and forged samples of the multimodal data; the multimodal data includes image data, video data and audio data; A pre-training module 602, configured to pre-train an audio-visual authenticity identification model using a first annotated data set, wherein the audio-visual authenticity identification model includes an image modality sub-model and an audio modality sub-model; A fine-tuning and optimization module 603 is used to apply the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, fine-tune the audio-visual authenticity identification model using the second annotated data set, and optimize the fine-tuned audio-visual authenticity identification model; The application module 604 is used to input the multimodal data to be identified into the optimized audio-visual authenticity identification model to perform authenticity identification.

[0088] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 7 , which shows an electronic device diagram provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.

[0089] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0090] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0091] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0092] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0093] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 8 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0094] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for distinguishing the authenticity of audio and video, characterized in that: The method comprises: Constructing a first annotated data set and a second annotated data set according to the multimodal data, wherein the first annotated data set and the second annotated data set include real samples and forged samples of the multimodal data; the multimodal data includes image data, video data and audio data; Pre-training an audiovisual authenticity identification model using a first annotated data set, wherein the audiovisual authenticity identification model includes an image modality sub-model and an audio modality sub-model; Applying the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, fine-tuning the audio-visual authenticity identification model using the second annotated data set, and optimizing the fine-tuned audio-visual authenticity identification model; The multimodal data to be identified is input into the optimized audio-visual authenticity identification model to perform authenticity identification.

2. The method according to claim 1, characterized in that The step of inputting the multimodal data to be identified into the optimized audio-visual authenticity identification model to perform authenticity identification includes: Inputting the multimodal data to be identified into the optimized audio-visual authenticity identification model, and extracting image features and / or audio features through the image modality sub-model and the audio modality sub-model; The image features and / or audio features are fused and weighted through the feature extraction layer to obtain the target authenticity identification result.

3. The method according to claim 2, characterized in that The image features and / or audio features are fused and weighted through the feature extraction layer according to the following formula to obtain the target authenticity identification result: The image features are , the audio features are , σ is the sigmoid function, which is used to map the fusion result to the interval [0,1]. and is the fusion weight.

4. The method according to claim 1, characterized in that The pre-training of the audio-visual authenticity identification model using the first labeled data set includes: The audiovisual authenticity identification model is pre-trained using the first annotated data set with the goal of minimizing the total loss function; wherein the total loss function L(θ) is expressed as: in, are the parameters of the model, and are the loss functions for image and audio modalities respectively, and is the weight balancing factor between the image modality and the audio modality.

5. The method according to claim 4, characterized in that The image modality loss function LI Calculate according to the following formula: =- ( log( )+(1- )log(1- )) in, is the true label of the image modality sample, is the probability predicted by the model, is the number of image modality samples; The audio modality loss function Calculate according to the following formula: =- ( log( )+(1- )log(1- )) in, is the true label of the audio modality sample, is the probability predicted by the model, is the number of audio modal samples.

6. The method according to claim 4, characterized in that The fine-tuning of the audio-visual authenticity identification model using the second annotated data set includes: Using the second annotated data set, the audio-visual authenticity identification model is fine-tuned with the goal of minimizing the loss function of the target task; wherein the loss function of the audio-visual authenticity identification task is Ltask : in, and They are the loss functions of the image modality and audio modality in the audio-visual authenticity identification task respectively.

7. The method according to claim 1, characterized in that The fine-tuned audio-visual authenticity identification model is optimized, including: The fine-tuned audio-visual authenticity identification model is optimized using the gradient descent method according to the following formula to update the model parameters; in, is the learning rate, is the gradient operator, L task is the loss function for the audio-visual authenticity identification task.

8. An audio-visual authenticity identification system, characterized in that: The system comprises: A labeling data set construction module, used to construct a first labeling data set and a second labeling data set according to multimodal data, wherein the first labeling data set and the second labeling data set include real samples and forged samples of the multimodal data; the multimodal data includes image data, video data and audio data; A pre-training module, used to pre-train an audio-visual authenticity identification model using a first annotated data set, wherein the audio-visual authenticity identification model includes an image modality sub-model and an audio modality sub-model; A fine-tuning optimization module, used to apply the pre-trained audio-visual authenticity identification model to the audio-visual authenticity identification task, fine-tune the audio-visual authenticity identification model using the second annotated data set, and optimize the fine-tuned audio-visual authenticity identification model; The application module is used to input the multimodal data to be identified into the optimized audio-visual authenticity identification model to perform authenticity identification.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1-7.

Images