Integrated text, picture, audio and video multi-dimensional authenticity identification method

Abstract

The invention relates to the technical field of big data processing, in particular to an integrated text, picture, audio and video multi-dimensional authenticity identification method. Multi-dimensional data processing is carried out based on deep learning, an information technology and big data analysis; the method is suitable for efficient and accurate authenticity identification of multi-dimensional data. Through comprehensive application of multiple technical means such as deep learning, image processing, audio analysis and the like, the authenticity of multimedia data is comprehensively, efficiently and accurately identified, and powerful support is provided for information authenticity verification. The authenticity of the multimedia content can be evaluated more comprehensively by integrating information of multiple dimensions for comprehensive analysis and identification. And the advanced algorithm and model are adopted to quickly process and analyze mass data, so that the authenticity identification efficiency is improved. A visual and understandable identification result and a detailed identification report are provided, the user is helped to better understand the identification result, and the reliability and safety of information acquisition are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a method for multi-dimensional authenticity identification integrating text, pictures, audio and video. Multi-dimensional data processing is performed based on deep learning, information technology and big data analysis; a method for efficient and accurate authenticity identification of multi-dimensional data. Background Art

[0002] With the rapid development of Internet technology and multimedia technology, the speed of information dissemination has accelerated, but it has also brought about the problem of the proliferation of false information. Traditional authenticity identification methods are mostly targeted at a single data type, making it difficult to fully evaluate the authenticity of information. In addition, the increasingly sophisticated counterfeiting technology makes it difficult to effectively deal with single-dimensional identification methods. Therefore, developing a method and system that can integrate multiple data types and conduct multi-dimensional and in-depth analysis of the authenticity of information is of great significance to improving the ability to detect information authenticity and maintaining information security. Summary of the invention

[0003] The present invention aims to provide a method for multi-dimensional authenticity identification that integrates text, pictures, audio and video. By comprehensively using various technical means such as deep learning, image processing, audio analysis, etc., it can achieve comprehensive, efficient and accurate authenticity identification of multimedia data, providing strong support for information authenticity verification.

[0004] Data collection: Collect text, images, audio and video data to be authenticated from various sources, such as the Internet, social media, surveillance cameras, etc.

[0005] Data preprocessing: Clean, denoise, unify formats, label data, and perform other preprocessing operations on the collected data to ensure data quality and consistency and prepare for subsequent analysis.

[0006] Feature extraction: Feature extraction is divided into text feature extraction, image feature extraction, audio feature extraction and video feature extraction.

[0007] Text feature extraction: TF-IDF technology is used to extract the feature values ​​of text keywords; Image feature extraction: Convolutional neural network (CNN) is used to extract image features such as texture, color and shape; Audio feature extraction: Mel frequency cepstral coefficient (MFCC) algorithm is used to extract audio features such as timbre, rhythm and melody; Video feature extraction: 3D CNN (C3D) model is used to extract motion information, key frame features and video stream characteristics of video frames.

[0008] Feature fusion: Multimodal-Long Short-Term Memory (MM-LSTM) networks are used to fuse features from different media types, and principal component analysis (PCA) is used to reduce dimensionality.

[0009] Authenticity Identification: Based on the deep learning model (Transformer), the fused features are trained and authenticity is determined.

[0010] Result output: Display the identification results in an intuitive and easy-to-understand manner, including authenticity labels, identification basis and suggestions, and provide a detailed identification report.

[0011] Beneficial effects of the present invention: 1. Significantly improve the accuracy of identification: By integrating information from multiple dimensions for comprehensive analysis and identification, the authenticity of multimedia content can be more comprehensively evaluated. 2. Enhance identification efficiency: Using advanced algorithms and models to quickly process and analyze massive amounts of data, the efficiency of authenticity identification is improved. 3. Improve user experience: Provide intuitive and easy-to-understand identification results and detailed identification reports to help users better understand the identification results and enhance the reliability and security of information acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0013] Main innovation description - Multi-dimensional data fusion: Innovatively propose a multi-dimensional information fusion system that integrates multiple data types, significantly improving the accuracy and comprehensiveness of information authenticity identification. Application of deep learning model: Use pre-trained deep learning models for in-depth analysis to improve identification efficiency and accuracy. Comprehensive judgment strategy: Formulate a scientific comprehensive judgment strategy, combining multi-dimensional feature analysis results and logical reasoning to achieve a comprehensive evaluation of complex information content.

[0014] The present invention solves the following technical problems: Limitations of single-dimensional authentication: Traditional authentication methods often focus on only one dimension among text, images, audio and video, ignoring the correlation and complementarity between different dimensions.

[0015] Improvement of multimedia content tampering technology: With the continuous development of image processing, audio editing and video synthesis technology, multimedia content tampering has become easier and more covert, and traditional authentication methods are difficult to effectively identify.

[0016] Challenges in processing massive amounts of data: Internet multimedia content data is huge and growing rapidly. How to accurately and quickly process the data and authenticate its authenticity is an urgent problem that needs to be solved.

[0017] In view of the above technical problems, the technical effects achieved by the invention include: Improve identification accuracy: By integrating information from multiple dimensions for comprehensive analysis and identification, the authenticity of multimedia content can be evaluated more comprehensively.

[0018] Enhanced authentication efficiency: Using advanced algorithms and models to quickly process and analyze massive amounts of data improves the efficiency of authentication. At the same time, by optimizing the use of algorithms and hardware resources, it reduces computing and time costs.

[0019] Improve user experience: Users can use the system to quickly and conveniently obtain the authenticity verification results of multimedia content, improving the reliability and security of information acquisition. In addition, detailed verification reports and explanations can be provided to help users better understand the verification results.

[0020] Promote the healthy development of information: By improving the ability to identify the authenticity of information, the public's trust in information can be enhanced and the effective dissemination and use of information can be promoted.

[0021] Take a piece of information published by a news website containing pictures, videos, audio and text as an example, the technical solution is organized and implemented as follows: 1. Data Collection Use the requests library, BeautifulSoup data parsing library, etc. to obtain the web page's image, video, audio, and text data, that is, the data to be identified.

[0022] 2) Data Preprocessing 1. Text preprocessing: remove HTML tags and stop words, and perform word segmentation and stem extraction.

[0023] 2. Image preprocessing: adjust the image size to a uniform size and perform normalization and other processing.

[0024] 3. Audio preprocessing: convert to a unified sampling rate and format, and perform noise suppression and other processing.

[0025] 4. Video preprocessing: extract video key frames, convert them into grayscale images, and resize them to a uniform size.

[0026] 3) Feature extraction According to the different media types in the data, corresponding features are extracted according to the corresponding media types.

[0027] Text feature extraction: Initialize the TfidfVectorizer object and set parameters (such as stop word list, word segmentation method, etc.). Then use the .fit_transform() method to convert the preprocessed text data into a TF-IDF (term frequency-inverse document frequency) feature matrix. The implementation process is as follows.

[0028] Initialize the TfidfVectorizer object and set parameters (e.g. stop words, word segmentation, etc.).

[0029] Vectorizer = TfidfVectorizer("Parameter Settings") Use the .fit_transform() method to transform the text data into a TF-IDF feature matrix.

[0030] X = vectorizer.fit_transform("text variable") X is the TF-IDF feature matrix obtained by converting the original text, and each column represents the feature of a word or phrase.

[0031] Image feature extraction: Use the convolutional layer and pooling layer in the convolutional neural network (CNN) to extract image features. In the convolutional neural network, the convolutional layers are usually stacked, and each layer uses a different convolution kernel to extract different features. If I is the input image and k is the convolution kernel, the convolution process is as follows.

[0032] K(m,n) Where (i, j) represents the position of the feature map, m and n represent the dimensions of the convolution kernel, and the convolution kernel is used to capture the local dependencies and spatial hierarchical structures in the input image. However, the convolution kernel is relatively small, and the image is still very large after the convolution process. The pooling layer is needed to reduce the dimensionality of the data, retain the main feature information of the image, and filter out the interference information of the image.

[0033] +1 Among them, the input size is i, the output size is o, the kernel size is k, and the stride is s.

[0034] Here, the image extraction uses the VGG model structure, removes the top fully connected layer, and only retains the convolution layer and pooling layer. Then, the preprocessed image input is sent to the VGG model to obtain the feature map through forward propagation.

[0035] Audio feature extraction: The audio features are extracted using the Mel-frequency cepstral coefficient (MFCC) algorithm. The calculation of MFCC includes pre-emphasis, framing, windowing, fast Fourier transform, Mel-frequency conversion, discrete cosine transform, and dynamic feature process. The specific implementation is as follows.

[0036] Pre-emphasis: Use a high-pass filter to highlight the high-frequency characteristics of the speech signal, making the signal spectrum flat and keeping the same signal-to-noise ratio in the entire frequency band from low frequency to high frequency. After the audio signal x[n] is pre-emphasized (α=0.97), the energy of the obtained y[n] in the high-frequency part will be higher than that of the original signal, which is particularly important for subsequent feature extraction.

[0037] y[n]=x[n]-α·x[n-1] Framing: The audio signal is divided into a series of overlapping short-time frames, each frame is 30ms long (i.e., for audio with a sampling rate of 16kHz, the frame length is about 480 sample points), and the frame shift (the overlapping part of adjacent frames) is usually half of the frame length or less (i.e., 15ms or about 240 sample points) to ensure a smooth transition between frames.

[0038] Windowing: To reduce the discontinuity at both ends of the frame and obtain a higher quality spectrum, a Hamming window is used to smooth the signal, where s[n] is the framed signal. is the window function, is the windowed signal.

[0039] Fast Fourier Transform (FFT): Perform FFT on each frame of the windowed signal to convert from the time domain to the frequency domain. is the frequency domain signal, N is the frame length, is the frequency index.

[0040] Mel frequency conversion: The spectrum obtained by FFT is passed through a set of Mel-scale triangular filters. The output of each filter is the weighted sum of the spectrum energy within the frequency range covered by the filter. That is, the filter output is the weighted average of all FFT amplitude values ​​within the covered frequency range. The weight is determined by the frequency response of the filter to ensure that the filters are equidistant on the Mel scale. Among them, is the output energy of the mth filter, is the frequency response of the mth filter.

[0041] Logarithmic operation: Perform logarithmic operation on the output energy of the Mel filter bank to simulate the nonlinear perception of sound intensity by the human ear: Discrete Cosine Transform (DCT): To further decorrelate and compress the feature vector, the result of the logarithmic operation is subjected to DCT to obtain the MFCC coefficients, where is the nth MFCC coefficient, and M is the number of Mel filter banks.

[0042] Dynamic feature extraction: Extract the first-order and second-order differences of MFCC coefficients as dynamic features to capture the changing characteristics of audio signals over time. and are the second-order difference and first-order difference of the nth MFCC coefficient respectively.

[0043] In summary, the MFCC algorithm converts the audio signal into a set of coefficients that can represent its sound characteristics through a series of mathematical changes and signal processing steps. The coefficients are used as feature input in tasks such as speech recognition.

[0044] Here, the steps of framing, windowing, FFT, Mel frequency conversion, logarithmic operation and DCT are implemented by the audio processing library (librosa).

[0045] Video feature extraction: 3D CNN (C3D) is used to implement video feature extraction. C3D is a pre-trained model. The specific implementation is as follows.

[0046] This process is similar to image feature extraction, only the input data is three-dimensional.

[0047] Feature extraction: The preprocessed video key frames are passed through a 3D convolution kernel (3x3x3) and a ReLU activation function to extract video features.

[0048] in represents the input video frame, f represents the output feature map, is the convolution kernel, b is the model bias, is the convolution operation, Represents the ReLU activation function.

[0049] Use the maximum pooling layer to reduce the size of the feature map. The pooling kernel p(3D) is the pooling kernel, and p is the feature map after pooling. Use the fully connected layer to convert the pooled feature map into a one-dimensional vector. Where p is the pooled feature map, y represents the feature vector, is the weight of the fully connected layer, and b is the bias of the fully connected layer.

[0050] The feature vector is normalized using L2 regularization. Where ||.||2 represents the L2 norm.

[0051] IV) Feature Fusion The multimodal-long short-term memory network (MM-LSTM) is used to achieve the fusion of text, image, video and audio features. First, the data of each modality is encoded by LSTM to generate the corresponding hidden state and output. For the input sequence of modality i, the hidden state of LSTM and output It can be expressed as, where W is the weight matrix, b is the bias, and f is the activation function.

[0052] After modality encoding, the attention mechanism is used to fuse the information of different modalities and set the attention weight of each modality. , where v is used to calculate the attention weight vector.

[0053] The weighted sum is used to obtain the fused features.

[0054] To reduce the computational complexity, the principal component analysis (PCA) method is used to reduce the dimension of the fused features.

[0055] The fused feature data is standardized to zero mean and unit variance, where is the mean, Represents standard deviation.

[0056] Calculate the covariance matrix of the standardized feature data. Where n represents the number of samples, represents the covariance matrix, Represents the standardized feature data.

[0057] Compute the eigenvalues ​​and eigenvectors of the covariance matrix. represents the covariance matrix, represents the eigenvalue, represents a feature vector.

[0058] Select the k eigenvectors with the largest eigenvalues ​​as principal components.

[0059] Use the principal component matrix for dimensionality reduction, where U represents the principal component matrix and Y represents the feature data after dimensionality reduction.

[0060] 5) Authenticity Identification Based on the fourth step, the fused features are converted into the input format of the Transformer model, and the Transformer model is used for authenticity identification. The Transformer model can effectively process sequence data. The process of authenticity identification is as follows.

[0061] 1. Convert the fused feature data into the input format of the Transformer model, which is a sequence format.

[0062] 2. Build a Transformer model, where the encoder is responsible for processing the input sequence and the decoder is responsible for generating the output sequence. h_i represents the i-th encoder output, and y_i represents the i-th decoder output.

[0063] 3. Train the Transformer model using labeled datasets (real data and fake data).

[0064] 4. Use the trained Transformer model to authenticate the input sequence, where y_pred is the prediction result.

[0065] VI) Model Evaluation Based on the fifth step, the Transformer model trained with the labeled dataset is evaluated using the following common indicators.

[0066] Here, common concepts of indicators are defined: TP (True Positive): Predicts the correct positive class. The sample is predicted to be positive. FN (False Negative): Predicts the wrong negative class, the sample is a positive class but is predicted as a negative class FP (False Positive): The wrong positive class is predicted, and the sample is a negative class but is predicted as a positive class. TN (True Negative): Predicts the correct negative class. The sample is a negative class and is predicted as a negative class. 1. Accuracy: The ratio of samples correctly predicted by the calculation model to the total number of samples. It indicates the percentage of correct predictions, where TP is a true positive, TN is a true negative, FP is a false positive, and FN is a false negative.

[0067] 2. Precision: It indicates the percentage of correctly predicted positive classes to predicted positive classes.

[0068] 3. Recall: It indicates the percentage of correctly predicted positive classes to actual positive classes.

[0069] 4. F1 value: Comprehensively consider the precision and recall rate, and calculate the weighted harmonic mean of Precision and Recall.

[0070] 5. AUC curve: The AUC curve is the area under the ROC curve. The ROC curve is a curve drawn with the false positive rate (FPR) as the horizontal axis and the true positive rate (TPR) as the vertical axis. The FPR and TPR are calculated in the following way: In the ROC curve, the closer the curve is to the upper left corner, the better the model performance is. The curve shifts toward the lower left corner, which means the model performance is worse. The area of ​​the AUC curve represents the area under the ROC curve. The value range is 0-1. The closer the AUC is to 1, the better the model performance is. AUC less than 0.5 means the model performance is poor.

[0071] Analyze the above indicators to evaluate the overall performance of the model. According to the evaluation results, continuously adjust the model parameters until the model is close to the ideal state.

[0072] VII) Results display It displays the identification results, confidence level and possible forgery traces analysis, supports result export function, and allows users to save the identification report in PDF and Word format for easy sharing and archiving.

[0073] It should be noted that the above technical route is a general description. In the specific implementation process, model optimization, parameter adjustment, model processing, etc. may be adjusted according to the specific technical solution, application scenario and performance requirements. In addition, due to the confidentiality of patented technology, the specific implementation details may not be available from public channels.

[0074] Technical points 1. Integrated multi-dimensional data processing system: Build a system architecture that can process multiple data types such as text, images, audio and video.

[0075] 2. Efficient feature extraction and fusion method: A feature extraction and fusion method based on deep learning, image processing, audio analysis and other technical means is proposed.

[0076] 3. Advanced authenticity identification algorithm: The Transformer model is used for authenticity identification, which effectively improves the accuracy and efficiency of identification.

[0077] Glossary 1. Multi-dimensional authenticity identification: Combining multiple media such as text, pictures, audio and video, and using a variety of technical means to conduct comprehensive and in-depth identification of the authenticity and non-tampering of the content of these media.

[0078] Deep learning: A branch of machine learning that can learn the inherent patterns and representation levels of data in the same way the brain does.

[0079] 3. Feature fusion: Merge or combine features from different feature extraction methods or models to improve the performance or effect of the model.

[0080] 4. TF-IDF (Term Frequency-Inverse Document Frequency): A common weighting technique used in information retrieval and text mining that combines term frequency (TF) and inverse document frequency (IDF) to assess the importance of a term to a document set or one of the documents in a corpus.

[0081] 5.3D CNN (C3D): A convolutional neural network that processes 3D input data. The input data is rich in information and can learn convolutional neural networks in both time and space dimensions to give more accurate results.

Claims

1. A multi-dimensional authenticity identification method integrating text, pictures, audio and video, characterized in that The steps include: S1. Data Collection Use the requests library and BeautifulSoup library to parse the data to obtain the web page's pictures, videos, audios, and text data. The above data is the data to be identified. S2. Data preprocessing The text data is preprocessed by removing HTML tags and stop words, and performing word segmentation and stem extraction; the image data is preprocessed by adjusting the image size to a uniform size and performing normalization. The audio is converted to a uniform sampling rate and format, and pre-processed for noise suppression; Extract video key frames, convert them into grayscale images, and resize them to a uniform size for preprocessing; S3. Feature extraction According to different media types in the data, corresponding features are extracted according to the corresponding media types; Initialize the TfidfVectorizer object and set parameters, such as the stop word list and word segmentation method; then use the .fit_transform() method to convert the preprocessed text data into a TF-IDF feature matrix; Initialize the TfidfVectorizer object and set the parameters: Vectorizer = TfidfVectorizer("Parameter Settings") Use the .fit_transform() method to transform the text data into a TF-IDF feature matrix: X = vectorizer.fit_transform ("text variable") X is the TF-IDF feature matrix obtained by converting the original text, and each column represents the feature of a word or phrase; 2. Image feature extraction: Use the convolutional layer and pooling layer in the convolutional neural network CNN to extract image features. In the convolutional neural network, the convolutional layers are usually stacked, and each layer uses a different convolution kernel to extract different features; If I is the input image and k is the convolution kernel, the convolution process is as follows; ·K(m,n) Where (i, j) represents the position of the feature map, m and n represent the dimensions of the convolution kernel, and the convolution kernel is used to capture the local dependency and spatial hierarchy in the input image. However, the convolution kernel is relatively small, and the image is still very large after the convolution process. The pooling layer is needed to reduce the dimensionality of the data, retain the main feature information of the image, and filter out the interference information of the image. +1 Among them, the input size is i, the output size is o, the kernel size is k, and the stride is s; Here, the image extraction uses the VGG model structure, removes the top fully connected layer, and only retains the convolution layer and pooling layer. Then, the preprocessed image input is sent to the VGG model to obtain the feature map through forward propagation; 3. Audio feature extraction: Use the Mel frequency cepstral coefficient MFCC algorithm to extract audio features. The calculation of MFCC includes pre-emphasis, framing, windowing, fast Fourier transform, Mel frequency conversion, discrete cosine transform, and dynamic feature process: Pre-emphasis: Use a high-pass filter to highlight the high-frequency characteristics of the speech signal, make the signal spectrum flat, and keep the same signal-to-noise ratio in the entire frequency band from low frequency to high frequency. The audio signal x[n] is processed with the pre-emphasis weight α; y[n]=x[n]-α·x[n-1] Framing: Divide the audio signal into a series of overlapping short-time frames, with the frame shift usually within half of the frame length; Windowing: To reduce the discontinuity at both ends of the frame and obtain a higher quality spectrum, a Hamming window is used to smooth the signal, where s[n] is the framed signal. is the window function, is the windowed signal; Fast Fourier Transform FFT: Perform FFT on each frame of the windowed signal to convert from the time domain to the frequency domain; is the frequency domain signal, N is the frame length, is the frequency index; Mel frequency conversion: The spectrum obtained by FFT is passed through a set of Mel-scale triangular filters; the output of each filter is the weighted sum of the spectrum energy within the frequency range covered by the filter; that is, the filter output is the weighted average of all FFT amplitude values ​​within the covered frequency, and the weight is determined by the frequency response of the filter to ensure that the filters are equidistant on the Mel scale; where, is the output energy of the mth filter, is the frequency response of the mth filter; Logarithmic operation: Perform logarithmic operation on the output energy of the Mel filter bank to simulate the nonlinear perception of sound intensity by the human ear: Discrete Cosine Transform DCT: Further decorrelation and compression of feature vectors, DCT of the result obtained by logarithmic operation to obtain MFCC coefficients, where is the nth MFCC coefficient, M is the number of Mel filter banks; Dynamic feature extraction: Extract the first-order and second-order differences of MFCC coefficients as dynamic features to capture the changing characteristics of audio signals over time. and They are the second-order difference and first-order difference of the nth MFCC coefficient respectively; The MFCC algorithm converts the audio signal into a set of coefficients that can represent its sound characteristics through mathematical changes and signal processing steps. The coefficients are used as feature input in tasks such as speech recognition. Here, the steps of framing, windowing, FFT, Mel frequency conversion, logarithmic operation and DCT are implemented by the audio processing library librosa; 4. Video feature extraction: Use 3D CNN to implement video feature extraction. C3D is a pre-trained model. This process is similar to image feature extraction, except that the input data is three-dimensional. Feature extraction: The preprocessed video key frames are passed through a 3D convolution kernel (3x3x3) and a ReLU activation function to extract video features; in represents the input video frame, f represents the output feature map, is the convolution kernel, b is the model bias, is the convolution operation, ReLU activation function. Use the maximum pooling layer to reduce the size of the feature map; the pooling kernel p(3D) is the pooling kernel, and p is the feature map after pooling; Use a fully connected layer to convert the pooled feature map into a one-dimensional vector; where p is the pooled feature map, y represents the feature vector, is the weight of the fully connected layer, and b is the bias of the fully connected layer; The feature vector is normalized using L2 regularization, where ||.||2 represents the L2 norm; S4 Feature Fusion The multimodal-long short-term memory network MM-LSTM is used to realize the fusion of text, picture, video and audio features. First, the data of each modality is encoded by LSTM to generate the corresponding hidden state and output. For the input sequence of modality i, the hidden state of LSTM and output Represents, where W is the weight matrix, b is the bias, and f is the activation function; After modality encoding, the attention mechanism is used to fuse the information of different modalities and set the attention weight of each modality. , where v is used to calculate the attention weight vector; Use weighted summation to obtain the fused features; To reduce the computational complexity, the principal component analysis (PCA) method is used to reduce the dimension of the fused features. The fused feature data is standardized to zero mean and unit variance, where is the mean, represents standard deviation; Calculate the covariance matrix of the standardized feature data; Where n represents the number of samples, represents the covariance matrix, Represents the standardized feature data; Compute the eigenvalues ​​and eigenvectors of the covariance matrix; in represents the covariance matrix, represents the eigenvalue, represents the feature vector; Select the k eigenvectors with the largest eigenvalues ​​as principal components; Use the principal component matrix for dimensionality reduction, where U represents the principal component matrix and Y represents the feature data after dimensionality reduction; 。 2. According to claim 1, a multi-dimensional authenticity identification method integrating text, pictures, audio and video is characterized in that Also includes S5 Authenticity Authentication According to S4, the fused features are converted into the input format of the Transformer model, and the Transformer model is used for authenticity identification. The Transformer model can effectively process sequence data. The authenticity identification process is as follows:

1. Convert the fused feature data into the input format of the Transformer model, which is a sequence format; 2. Build a Transformer model, where the encoder is responsible for processing the input sequence and the decoder is responsible for generating the output sequence; h_i represents the i-th encoder output, and y_i represents the i-th decoder output; 3. Train the Transformer model using labeled real data and fake data datasets; 4. Use the trained Transformer model to identify the authenticity of the input sequence, where y_pred is the prediction result; 。 3. According to claim 1, a multi-dimensional authenticity identification method integrating text, pictures, audio and video is characterized in that Also includes S6. Model Evaluation According to S5, the Transformer model trained with the labeled dataset is evaluated using the following metrics; TP (True Positive): predicts the correct positive class, the sample is predicted as the positive class; FN (False Negative): The wrong negative class is predicted, and the sample is a positive class but is predicted as a negative class; FP (False Positive): The wrong positive class is predicted, and the sample is a negative class but is predicted as a positive class; TN (True Negative): predicts the correct negative class, and the sample is predicted as a negative class; Accuracy: The ratio of samples predicted correctly by the calculation model to the total number of samples; it represents the percentage of correct predictions, where TP is a true positive, TN is a true negative, FP is a false positive, and FN is a false negative. Precision: indicates the percentage of correctly predicted positive classes in the predicted positive classes; Recall: indicates the percentage of correctly predicted positive classes to actual positive classes; F1 value: Comprehensively consider the precision and recall rate, and calculate the weighted harmonic mean of Precision and Recall; AUC curve: The AUC curve is the area under the ROC curve. The ROC curve is a curve drawn with the false positive rate FPR as the horizontal axis and the true positive rate TPR as the vertical axis. The FPR and TPR are calculated in the following way: 。

Images