Method and system for identifying forged audio

Abstract

The invention relates to the technical field of audio identification, and discloses a forged audio identification method and system, and the method comprises the steps: carrying out the framing and windowing of original audio data, and obtaining frame sequence audio data; performing short-time Fourier transform on the frame sequence audio data to obtain frequency domain representation data and an initial acoustic feature vector; performing depth feature reconstruction on the initial acoustic feature vector, and performing multi-modal feature fusion on the reconstructed feature tensor and the frame sequence audio data to obtain a joint identification feature set; performing time sequence context analysis on the joint identification feature set, and capturing a feature dependency relationship of the frame sequence audio data to obtain a context enhanced feature sequence; quantifying the cosine similarity between the context enhanced feature sequence and the real voiceprint template, and analyzing the feature mutation amplitude of the context enhanced feature sequence to obtain dual identification measurement; performing comprehensive scoring on the dual identification measurement to obtain an audio identification conclusion; according to the invention, the efficiency of counterfeit audio identification can be improved.

Description

Technical Field

[0001] This invention relates to the field of audio identification technology, and in particular to a method and system for identifying counterfeit audio. Background Technology

[0002] Against the backdrop of the rapid development of artificial intelligence and audio synthesis technology, the methods for generating fake audio are becoming increasingly sophisticated. The similarity between the generated audio and real audio at the auditory level is constantly improving, which poses a severe challenge to audio fake detection technology. Existing fake audio identification methods mostly rely on single acoustic feature extraction and matching, which makes it difficult to capture the subtle generation traces hidden in fake audio. The depth and dimension of feature mining are insufficient, which leads to the easy occurrence of feature omissions during the identification process and cannot effectively meet the identification needs of various new types of fake audio.

[0003] Traditional audio forgery detection technology lacks in-depth analysis of audio temporal features, making it difficult to capture long-distance feature dependencies between audio frames. Furthermore, its method of determining the identification metric is relatively simple, relying solely on simple similarity threshold comparisons to arrive at identification conclusions. When faced with audio data where features are in ambiguous ranges, the identification accuracy drops significantly, and the overall identification efficiency and accuracy are insufficient to meet the high requirements of practical applications. In complex audio application scenarios, it is prone to misjudgment and missed judgment. Therefore, how to improve the efficiency of forged audio identification has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a method and system for identifying counterfeit audio, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for identifying counterfeit audio, comprising: S1. Perform frame-segmentation and windowing processing on the user's original audio data to obtain the user's frame sequence audio data; S2. Perform a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extract the Mel frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector; S3. Perform deep feature reconstruction on the initial acoustic feature vector to obtain the reconstructed feature tensor of the user, and perform multimodal feature fusion on the reconstructed feature tensor and the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user; S4. Perform temporal context analysis on the joint identification feature set to capture the feature dependencies of the frame sequence audio data over a long distance range, and obtain the user's context-enhanced feature sequence. S5. Quantify the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyze the feature mutation amplitude between adjacent frames in the context-enhanced feature sequence to obtain the user's dual identification metric; S6. Perform a comprehensive score on the dual authentication metrics to obtain the audio authentication conclusion for the user.

[0006] In a preferred embodiment, the step of performing frame-segmentation and windowing processing on the user's original audio data to obtain the user's frame sequence audio data includes: Collect the user's raw audio data, and perform pre-emphasis processing on the raw audio data to obtain the user's emphasized audio data; According to the preset frame length, the stressed audio data is uniformly divided into frames to obtain the user's framed audio data. The audio data in the segmented audio data is subjected to windowing and weighting processing to obtain the user's frame sequence audio data.

[0007] In a preferred embodiment, the step of performing a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extracting Mel-frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector, includes: A short-time Fourier transform is performed on the audio data in the frame sequence audio data to map the audio data from the time domain to the frequency domain, thereby obtaining the user's frequency domain representation data; A nonlinear transformation at the Mel frequency scale is performed on the spectral amplitude values ​​in the frequency domain representation data to obtain the Mel spectral energy distribution data of the user. Take the logarithm of the Mel spectrum energy distribution data to obtain the logarithmic Mel spectrum data of the user; The log-Mel spectrum data is subjected to discrete cosine transform to remove the correlation between features of different dimensions, thereby obtaining the user's initial acoustic feature vector.

[0008] In a preferred embodiment, the step of performing deep feature reconstruction on the initial acoustic feature vector to obtain the user's reconstructed feature tensor includes: The initial acoustic feature vector is mapped into a feature space to obtain the extended feature representation of the user. Residual information is captured on the extended feature representation, and subtle fluctuation components associated with audio generation traces are extracted from the extended feature representation to obtain the user's residual feature components; The residual feature components are differentially combined with the main feature components in the extended feature representation to obtain the user's reconstructed feature tensor.

[0009] In a preferred embodiment, the step of fusing the reconstructed feature tensor with the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user includes: The user's pitch period trajectory features are obtained by performing pitch period detection on the frame sequence audio data; Linear prediction analysis is performed on the frame sequence audio data to obtain the formant trajectory features of the user; The pitch period trajectory features and the formant trajectory features are aligned, and the time scales of the pitch period trajectory features and the formant trajectory features are unified to the same frame index correspondence as the reconstructed feature tensor to obtain the aligned physiological feature sequence of the user. The aligned physiological feature sequence is fused with the reconstructed feature tensor to obtain the joint identification feature set of the user.

[0010] In a preferred embodiment, the step of performing temporal context analysis on the joint identification feature set to capture the feature dependencies of the frame sequence audio data over a long distance to obtain the user's context-enhanced feature sequence includes: Arrange the feature vectors in the joint identification feature set in chronological order to obtain the user's original feature sequence matrix; The original feature sequence matrix is ​​sliced ​​using a sliding window to obtain the user's local feature window matrix; Perform cross-frame correlation analysis on the local feature window matrix to identify the degree of mutual influence between the feature vectors of each frame within the local feature window, and obtain the user's inter-frame correlation weight matrix. Based on the inter-frame correlation weight matrix, the feature vectors within the local feature window are weighted and aggregated to obtain the user's context-aware feature vector. The context-aware feature vectors are combined according to the original time sequence to obtain the context-enhanced feature sequence of the user.

[0011] In a preferred embodiment, quantifying the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyzing the amplitude of feature abrupt changes between adjacent frames in the context-enhanced feature sequence to obtain the user's dual-identification metric, includes: The frame feature vectors in the context enhancement feature sequence are compared with the user's real voiceprint template using cosine similarity analysis to obtain the user's instantaneous similarity value. The instantaneous similarity values ​​are aggregated by mean to obtain the global similarity metric for the user; Perform Euclidean distance analysis on the feature vectors of adjacent frames in the context-enhanced feature sequence to obtain the feature mutation magnitude value of the user, and integrate the feature mutation magnitude value to obtain the global mutation metric value of the user; The global similarity metric and the global mutation metric are nonlinearly fused to obtain the user's dual authentication metric.

[0012] In a preferred embodiment, the formula for calculating the dual discrimination metric is: ; in, This indicates the dual discrimination metric. This represents the global similarity metric value. This represents the global mutation metric. This represents the preset first weighting coefficient. This represents the preset similarity threshold. This represents the preset nonlinear transformation steepness coefficient. Represents the natural constant. This represents the hyperbolic tangent function.

[0013] In a preferred embodiment, the step of comprehensively scoring the dual authentication metrics to obtain the user's audio authentication conclusion includes: Confidence interval estimation is performed on the user's historical audio data to obtain the user's historical confidence distribution interval; The dual authentication metric is compared with the upper and lower thresholds of the historical confidence distribution interval. If the dual authentication metric is greater than or equal to the upper limit of the historical confidence distribution interval, the original audio data is determined to be real audio. If the dual authentication metric is less than or equal to the lower limit of the historical confidence distribution interval, the original audio data is determined to be fake audio. If the dual discrimination metric is within the historical confidence distribution interval, then the silence segment distribution features and energy envelope features in the frame sequence audio data are extracted to obtain the user's auxiliary discrimination feature vector; The auxiliary identification feature vector is compared with the pre-stored real audio auxiliary feature template to obtain the user's auxiliary confidence increment. The auxiliary confidence increment is then accumulated and fused with the dual identification metric to obtain the user's corrected dual identification metric. The corrected dual authentication metric is then compared with the upper and lower thresholds of the historical confidence distribution interval to obtain the audio authentication conclusion for the user.

[0014] To address the above problems, the present invention also provides a fake audio identification system, the system comprising: The audio framing module is used to perform frame-framing and windowing processing on the user's original audio data to obtain the user's frame sequence audio data. The data processing module is used to perform a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extract the Mel frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector. The feature fusion module is used to perform deep feature reconstruction on the initial acoustic feature vector to obtain the reconstructed feature tensor of the user, and to perform multimodal feature fusion on the reconstructed feature tensor and the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user; The feature analysis module is used to perform temporal context analysis on the joint identification feature set, capture the feature dependencies of the frame sequence audio data over a long distance, and obtain the user's context-enhanced feature sequence. The dual authentication module is used to quantify the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyze the feature mutation amplitude between adjacent frames in the context-enhanced feature sequence to obtain the user's dual authentication metric. The comprehensive analysis module is used to comprehensively score the dual identification metrics to obtain the audio identification conclusion of the user.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves a significant improvement in identification performance through multi-stage technical optimization. From raw audio processing to feature extraction, refined operations are employed. Combining deep feature reconstruction and multimodal feature fusion, it integrates acoustic features, physiological features, and other multi-dimensional information, giving the joint identification feature set a more comprehensive audio representation capability. Simultaneously, it captures long-distance feature dependencies through temporal context analysis and constructs dual identification metrics using cosine similarity and feature mutation amplitude. This multi-dimensional feature analysis and identification index design significantly improves the accuracy of audio feature recognition, enabling more sensitive capture and identification of the feature differences in forged audio.

[0016] 2. This invention further enhances the identification effect through a comprehensive scoring mechanism. It makes a preliminary judgment by combining historical confidence distribution intervals and completes a secondary correction by extracting auxiliary identification features from audio in ambiguous intervals. This achieves layer-by-layer verification of the identification results, effectively reducing the probability of false positives and false negatives. The corresponding identification system, through modular functional design, efficiently connects various technical links, ensuring the implementation efficiency of the identification method. While improving the accuracy of forged audio identification, it also achieves efficient progress in the identification process, allowing the overall identification technology to have better performance in practical applications. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for identifying counterfeit audio according to an embodiment of the present invention. Figure 2 A functional block diagram of a fake audio identification system provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for identifying spoofed audio. The execution subject of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for identifying spoofed audio can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a method for identifying counterfeit audio according to an embodiment of the present invention. In this embodiment, the method for identifying counterfeit audio includes: S1. Perform frame-segmentation and windowing processing on the user's original audio data to obtain the user's frame sequence audio data; In this embodiment of the invention, the step of performing frame-segmentation and windowing processing on the user's original audio data to obtain the user's frame sequence audio data includes: Collect the user's raw audio data, and perform pre-emphasis processing on the raw audio data to obtain the user's emphasized audio data; According to the preset frame length, the stressed audio data is uniformly divided into frames to obtain the user's framed audio data. The audio data in the segmented audio data is subjected to windowing and weighting processing to obtain the user's frame sequence audio data.

[0021] After collecting the user's original audio data, pre-emphasis processing is performed on the original audio data. By increasing the signal strength of the high-frequency part of the audio, the high-frequency attenuation problem that occurs during the propagation of the audio is compensated, and the signal correction of the original audio data is completed to obtain the corresponding emphasized audio data.

[0022] The stressed audio data is uniformly segmented according to a fixed frame length, cutting the continuous stressed audio data into multiple audio data segments of the same length, so that the originally continuous audio signal is transformed into a discrete segmented frame form, and the corresponding segmented audio data is obtained.

[0023] Each audio data segment in the framed audio data is subjected to windowing and weighting processing. A corresponding window function weight value is assigned to each framed audio data segment. The edge parts of the framed audio data segments are smoothed to avoid the spectral leakage problem caused by the framing operation, and the corresponding frame sequence audio data is obtained.

[0024] The beneficial effects are that the standardized signal preprocessing of the original audio data is completed through step-by-step processing of pre-emphasis, uniform framing, and windowing weighting. This not only specifically compensates for the high-frequency attenuation problem during audio propagation, but also effectively avoids the spectral leakage problem caused by framing operations. The processed frame sequence audio data completely retains all kinds of effective acoustic features of the original audio, providing a high-quality and standardized audio data foundation for subsequent frequency domain transformation and acoustic feature extraction. This ensures the accuracy of operation and the effectiveness of feature extraction in all subsequent audio identification-related processing steps from the data source.

[0025] S2. Perform a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extract the Mel frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector; In this embodiment of the invention, the step of performing a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extracting Mel-frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector, includes: A short-time Fourier transform is performed on the audio data in the frame sequence audio data to map the audio data from the time domain to the frequency domain, thereby obtaining the user's frequency domain representation data; A nonlinear transformation at the Mel frequency scale is performed on the spectral amplitude values ​​in the frequency domain representation data to obtain the Mel spectral energy distribution data of the user. Take the logarithm of the Mel spectrum energy distribution data to obtain the logarithmic Mel spectrum data of the user; The log-Mel spectrum data is subjected to discrete cosine transform to remove the correlation between features of different dimensions, thereby obtaining the user's initial acoustic feature vector.

[0026] The short-time Fourier transform is performed on each segment of audio data in the frame sequence audio data in sequence, which is originally in the time domain and is mapped to the frequency domain. The audio signal features in the time domain are transformed into the spectral features in the frequency domain, and the corresponding frequency domain representation data is obtained.

[0027] A nonlinear transformation at the Mel frequency scale is performed on all spectral amplitude values ​​contained in the frequency domain representation data. The distribution characteristics of the spectral amplitude values ​​are adjusted according to the correspondence between Mel frequency and linear frequency to make the spectral amplitude values ​​fit the auditory perception characteristics of the human ear, thus obtaining the corresponding Mel spectral energy distribution data.

[0028] Logarithms are calculated for all data values ​​in the Mel spectrum energy distribution data, and the energy values ​​of the Mel spectrum are compressed to reduce the difference between energy values ​​while highlighting the detailed features of the spectrum, thus obtaining the corresponding logarithmic Mel spectrum data.

[0029] The discrete cosine transform is performed on the log-Mel spectrum data, and the data is orthogonally transformed to remove the correlation information between features of different dimensions in the log-Mel spectrum data, thereby eliminating the correlation between features and obtaining the corresponding initial acoustic feature vector.

[0030] The beneficial effects are that the audio data is converted from the time domain to the frequency domain by short-time Fourier transform, and the spectral features are extracted by combining the nonlinear transformation of the Mel frequency scale with the characteristics of human hearing. After layers of optimization by logarithmic processing and discrete cosine transform, the core acoustic features in the frame sequence audio data are effectively extracted. The resulting initial acoustic feature vector eliminates redundant feature association information, and the feature dimensions are clear and can accurately represent the acoustic properties of the audio. This provides accurate and effective basic feature data for subsequent deep feature reconstruction and multimodal feature fusion, and improves the pertinence and reliability of subsequent feature processing steps.

[0031] S3. Perform deep feature reconstruction on the initial acoustic feature vector to obtain the reconstructed feature tensor of the user, and perform multimodal feature fusion on the reconstructed feature tensor and the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user; In this embodiment of the invention, the step of performing deep feature reconstruction on the initial acoustic feature vector to obtain the user's reconstructed feature tensor includes: The initial acoustic feature vector is mapped into a feature space to obtain the extended feature representation of the user. Residual information is captured on the extended feature representation, and subtle fluctuation components associated with audio generation traces are extracted from the extended feature representation to obtain the user's residual feature components; The residual feature components are differentially combined with the main feature components in the extended feature representation to obtain the user's reconstructed feature tensor.

[0032] The step of fusing the reconstructed feature tensor with the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user includes: The user's pitch period trajectory features are obtained by performing pitch period detection on the frame sequence audio data; Linear prediction analysis is performed on the frame sequence audio data to obtain the formant trajectory features of the user; The pitch period trajectory features and the formant trajectory features are aligned, and the time scales of the pitch period trajectory features and the formant trajectory features are unified to the same frame index correspondence as the reconstructed feature tensor to obtain the aligned physiological feature sequence of the user. The aligned physiological feature sequence is fused with the reconstructed feature tensor to obtain the joint identification feature set of the user.

[0033] The initial acoustic feature vector is mapped to a higher-dimensional feature space, thereby enriching the expression dimension and representation information of the features and obtaining the corresponding extended feature representation.

[0034] Residual information is captured from the extended feature representation, and frame-by-frame subtle changes in the signal data in the extended feature representation are detected. Subtle fluctuation components associated with audio generation traces are screened out, and these components are extracted and integrated separately to obtain the corresponding residual feature components.

[0035] The main feature component is separated from the extended feature representation. The residual feature component is then combined with the main feature component in a differentiated manner. The components are then hierarchically divided and fused according to the attributes and representational meaning of the features to obtain the corresponding reconstructed feature tensor.

[0036] Pitch period detection is performed on the frame sequence audio data. The pitch period value in the frame sequence audio data is identified frame by frame. The pitch period values ​​of each frame are arranged in chronological order to form a continuous pitch period trajectory feature.

[0037] Linear predictive analysis is performed on frame sequence audio data. By constructing a linear predictive model to fit the signal variation law of the frame sequence audio data, the frequency and amplitude information of the formants are extracted from the fitting results. The formant information of each frame is integrated in time order to obtain the corresponding formant trajectory features.

[0038] Based on the frame index of the reconstructed feature tensor, feature alignment processing is performed on the pitch period trajectory feature and formant trajectory feature. The time scale of the two features is adjusted so that each feature point of the pitch period trajectory feature and formant trajectory feature can form a one-to-one correspondence with the frame index of the reconstructed feature tensor, thus obtaining the corresponding aligned physiological feature sequence.

[0039] The aligned physiological feature sequence and the reconstructed feature tensor are spliced ​​together and fused. The feature dimensions of the aligned physiological feature sequence and the feature dimensions of the reconstructed feature tensor are expanded and spliced ​​together. All representational information of the two types of features is integrated to obtain the corresponding joint discriminative feature set.

[0040] The beneficial effects are that the initial acoustic feature vector is deeply reconstructed through feature space mapping and residual information capture, allowing the reconstructed feature tensor to accurately capture hidden generation trace features in the audio. At the same time, the physiological acoustic features of the audio are extracted through pitch period detection and linear prediction analysis. After feature alignment, the multimodal features are accurately spliced ​​and fused. The resulting joint identification feature set integrates deep reconstruction features and physiological trajectory features. The feature dimensions are comprehensive and the representation information is accurate, which effectively makes up for the representation defects of single features. It provides a feature foundation with rich dimensions and strong correlation for subsequent temporal context analysis, and improves the comprehensiveness and effectiveness of the fake audio identification features.

[0041] S4. Perform temporal context analysis on the joint identification feature set to capture the feature dependencies of the frame sequence audio data over a long distance range, and obtain the user's context-enhanced feature sequence. In this embodiment of the invention, the step of performing temporal context analysis on the joint identification feature set to capture the feature dependencies of the frame sequence audio data over a long distance to obtain the user's context-enhanced feature sequence includes: Arrange the feature vectors in the joint identification feature set in chronological order to obtain the user's original feature sequence matrix; The original feature sequence matrix is ​​sliced ​​using a sliding window to obtain the user's local feature window matrix; Perform cross-frame correlation analysis on the local feature window matrix to identify the degree of mutual influence between the feature vectors of each frame within the local feature window, and obtain the user's inter-frame correlation weight matrix. Based on the inter-frame correlation weight matrix, the feature vectors within the local feature window are weighted and aggregated to obtain the user's context-aware feature vector. The context-aware feature vectors are combined according to the original time sequence to obtain the context-enhanced feature sequence of the user.

[0042] All feature vectors in the joint identification feature set are arranged in chronological order according to the audio sequence, and the discrete feature vectors are integrated into a regular matrix form to obtain the corresponding original feature sequence matrix.

[0043] The original feature sequence matrix is ​​sliced ​​using a fixed window length and sliding step size, and the original feature sequence matrix is ​​cut into multiple local matrix blocks containing feature vectors of consecutive frames, thus obtaining the corresponding local feature window matrix.

[0044] Cross-frame correlation analysis is performed on all frame feature vectors within each local feature window matrix. The degree of feature correlation between any two frame feature vectors is calculated. Based on the magnitude of the correlation, the interaction relationship between the feature vectors of each frame is assigned a corresponding numerical label, and the corresponding inter-frame correlation weight matrix is ​​obtained.

[0045] Using the values ​​in the inter-frame correlation weight matrix as the weights of each frame feature vector, a weighted summation is performed on all frame feature vectors within each local feature window matrix. The calculated comprehensive feature vector is then used as the feature representation corresponding to that window, resulting in the corresponding context-aware feature vector.

[0046] All context-aware feature vectors obtained through the local feature window matrix are combined sequentially according to their corresponding audio time order, integrating the discrete context-aware feature vectors into a continuous feature sequence, thus obtaining the corresponding context-enhanced feature sequence.

[0047] The beneficial effects are as follows: through multi-step processing of temporal context analysis, feature dependencies over a long range are mined from the joint identification feature set of frame sequence audio data. By using sliding window slicing and cross-frame correlation analysis, the degree of mutual influence between frames is accurately identified. Then, through weighted aggregation and ordered combination, a context-enhanced feature sequence is obtained. This feature sequence not only retains the core representation information of the joint identification features, but also incorporates the temporal correlation features between frames, allowing the audio features to fully reflect the changing rules of the temporal context. This effectively makes up for the lack of temporal correlation in single-frame features, and provides basic data that is more in line with the actual temporal features of the audio for the subsequent acquisition of dual identification measures. This improves the efficiency of using temporal features and the accuracy of identification in the process of identifying fake audio.

[0048] S5. Quantify the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyze the feature mutation amplitude between adjacent frames in the context-enhanced feature sequence to obtain the user's dual identification metric; In this embodiment of the invention, quantifying the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyzing the amplitude of feature abrupt changes between adjacent frames in the context-enhanced feature sequence to obtain the user's dual-identification metric, includes: The frame feature vectors in the context enhancement feature sequence are compared with the user's real voiceprint template using cosine similarity analysis to obtain the user's instantaneous similarity value. The instantaneous similarity values ​​are aggregated by mean to obtain the global similarity metric for the user; Perform Euclidean distance analysis on the feature vectors of adjacent frames in the context-enhanced feature sequence to obtain the feature mutation magnitude value of the user, and integrate the feature mutation magnitude value to obtain the global mutation metric value of the user; The global similarity metric and the global mutation metric are nonlinearly fused to obtain the user's dual authentication metric.

[0049] The formula for calculating the dual discrimination metric is as follows: ; in, This indicates the dual discrimination metric. This represents the global similarity metric value. This represents the global mutation metric. This represents the preset first weighting coefficient. This represents the preset similarity threshold. This represents the preset nonlinear transformation steepness coefficient. Represents the natural constant. This represents the hyperbolic tangent function.

[0050] Each frame feature vector in the context-enhanced feature sequence is subjected to cosine similarity analysis with the user's real voiceprint template. The cosine value of the angle between the vectors is used to characterize the similarity between each frame feature vector and the real voiceprint template. Each calculation result corresponds to the instantaneous similarity value of a frame.

[0051] The instantaneous similarity values ​​corresponding to all frames in the context-enhanced feature sequence are summed, and then the sum is divided by the total number of instantaneous similarity values ​​to complete the mean aggregation of all instantaneous similarity values, thus obtaining the corresponding global similarity metric.

[0052] The feature vectors of two adjacent frames in the context enhancement feature sequence are selected sequentially for Euclidean distance analysis. The Euclidean distance between the vectors is used to characterize the degree of change of the feature vectors of adjacent frames. Each calculation result corresponds to a set of feature mutation amplitude values ​​of adjacent frames. All feature mutation amplitude values ​​are integrated and calculated as a whole to obtain the corresponding global mutation metric value.

[0053] The global similarity metric and the global mutation metric are nonlinearly fused. The two types of metrics are fused according to their respective characterization characteristics, and the identification information of the two types of metrics is integrated to form a comprehensive metric index that has both similarity and mutation characterization, thus obtaining the corresponding dual identification metric.

[0054] The global similarity metric is generated by performing cosine similarity analysis between the frame feature vectors in the context-enhanced feature sequence and the user's real voiceprint template to obtain the instantaneous similarity value, and then aggregating the mean of the instantaneous similarity values. The global mutation metric is generated by performing Euclidean distance analysis between the feature vectors of adjacent frames in the context-enhanced feature sequence to obtain the feature mutation amplitude value, and then integrating all feature mutation amplitude values. The preset first weight coefficient is a value set in advance by humans based on the actual needs of the fake audio identification scenario. The similarity benchmark threshold is a value set in advance by humans based on the similarity judgment standard when identifying fake audio. The nonlinear transformation steepness coefficient is a value set in advance by humans based on the curve change requirements of the nonlinear transformation. The natural constant is a fixed constant in mathematics, and the hyperbolic tangent function is a fixed function form in mathematics. Both are general mathematical elements that are directly applied to the calculation.

[0055] The global similarity metric is calculated by substituting it into the logistic function. This calculation involves multiplying the global similarity metric by the difference between the global similarity metric and the similarity benchmark threshold using a nonlinear transformation steepness coefficient, then raising the result to the negative exponent of the natural constant, adding one to the exponent, taking the reciprocal, and multiplying the reciprocal by a preset first weighting coefficient to obtain the first part of the calculation result. The global mutation metric is then calculated using the hyperbolic tangent function. This involves substituting the global mutation metric into the hyperbolic tangent function formula, obtaining the corresponding function result through exponential and arithmetic operations, subtracting the preset first weighting coefficient from one, and multiplying the difference by the hyperbolic tangent function result to obtain the second part of the calculation result. The first and second parts of the calculation result are then added together to obtain the dual discrimination metric. This calculation process achieves nonlinear fusion of the global similarity metric and the global mutation metric, allowing the two dimensions of discrimination indicators to be combined into a unified metric according to the set weights, thereby comprehensively reflecting the authenticity characteristics of the audio.

[0056] The larger the global similarity metric value, the closer the result after calculation using the logistic function is to one. The larger the first part of the result after multiplying by the preset first weight coefficient, the greater the final double-identification metric value. Conversely, the smaller the global similarity metric value, the closer the result after calculation using the logistic function is to zero. The smaller the first part of the result after multiplying by the preset first weight coefficient, the smaller the final double-identification metric value. Similarly, the larger the global mutation metric value, the closer the result after calculation using the hyperbolic tangent function is to one. The larger the second part of the result after multiplying by the difference between one and the preset first weight coefficient, the greater the final double-identification metric value. Conversely, the smaller the global mutation metric value, the closer the result after calculation using the hyperbolic tangent function is to negative one. The smaller the second part of the result after multiplying by the difference between one and the preset first weight coefficient, the smaller the final double-identification metric value. The value of the dual identification metric will also decrease accordingly; the larger the preset first weight coefficient, the higher the proportion of the global similarity metric in the calculation of the dual identification metric, and the lower the proportion of the global mutation metric; the smaller the preset first weight coefficient, the higher the proportion of the global mutation metric in the calculation of the dual identification metric, and the lower the proportion of the global similarity metric; the larger the nonlinear transformation steepness coefficient, the steeper the change of the logistic function curve, and a small change in the global similarity metric will bring about a large change in the first part of the calculation result; the smaller the nonlinear transformation steepness coefficient, the smoother the change of the logistic function curve, and the smoother the change in the first part of the calculation result brought about by the change in the global similarity metric will be; the similarity benchmark threshold serves as a reference standard for the global similarity metric, and its value setting directly affects the result of the difference between the global similarity metric and the threshold, and thus affects the first part of the calculation result through the calculation of the logistic function, ultimately changing the trend of the dual identification metric value.

[0057] The beneficial effects are that cosine similarity analysis and Euclidean distance analysis are used to measure the context-enhanced feature sequence from the two dimensions of similarity and abrupt change, respectively. The global metric obtained by mean aggregation and integration can comprehensively reflect the matching degree between audio and real voiceprint and the feature stability between audio frames. Then, the two core metrics are integrated into a dual identification metric through nonlinear fusion. This metric takes into account both the voiceprint matching features and temporal abrupt change features of the audio. It provides a comprehensive and accurate measurement basis for audio identification from two key dimensions, effectively improving the comprehensiveness and accuracy of the identification metric, and laying a reliable measurement foundation for subsequent comprehensive scoring to draw identification conclusions.

[0058] S6. Perform a comprehensive score on the dual authentication metrics to obtain the audio authentication conclusion for the user.

[0059] In this embodiment of the invention, the step of comprehensively scoring the dual authentication metrics to obtain the user's audio authentication conclusion includes: Confidence interval estimation is performed on the user's historical audio data to obtain the user's historical confidence distribution interval; The dual authentication metric is compared with the upper and lower thresholds of the historical confidence distribution interval. If the dual authentication metric is greater than or equal to the upper limit of the historical confidence distribution interval, the original audio data is determined to be real audio. If the dual authentication metric is less than or equal to the lower limit of the historical confidence distribution interval, the original audio data is determined to be fake audio. If the dual discrimination metric is within the historical confidence distribution interval, then the silence segment distribution features and energy envelope features in the frame sequence audio data are extracted to obtain the user's auxiliary discrimination feature vector; The auxiliary identification feature vector is compared with the pre-stored real audio auxiliary feature template to obtain the user's auxiliary confidence increment. The auxiliary confidence increment is then accumulated and fused with the dual identification metric to obtain the user's corrected dual identification metric. The corrected dual authentication metric is then compared with the upper and lower thresholds of the historical confidence distribution interval to obtain the audio authentication conclusion for the user.

[0060] Confidence interval estimation is performed on the user's historical audio data, statistical analysis is conducted on the distribution pattern of the discrimination metric values ​​corresponding to the historical audio data, the reasonable range of the value distribution is determined, and the corresponding historical confidence distribution interval is obtained.

[0061] The obtained dual authentication metric values ​​are compared with the upper and lower thresholds of the historical confidence distribution interval. When the dual authentication metric value is greater than or equal to the upper threshold of the historical confidence distribution interval, the original audio data is directly determined to be real audio. When the dual authentication metric value is less than or equal to the lower threshold of the historical confidence distribution interval, the original audio data is directly determined to be fake audio.

[0062] When the value of the dual discrimination metric is between the upper and lower thresholds of the historical confidence distribution interval, silence segment detection and energy analysis are performed on the frame sequence audio data. The distribution features such as the position and duration of silence segments in the frame sequence audio data are extracted. At the same time, the energy change trend of the audio signal in the time dimension, i.e., the energy envelope feature, is extracted. The two types of features are integrated into a vector form to obtain the corresponding auxiliary discrimination feature vector.

[0063] The extracted auxiliary identification feature vector is compared with the pre-stored real audio auxiliary feature template in the system. The corresponding auxiliary confidence increment is calculated based on the comparison result. The auxiliary confidence increment is then numerically accumulated and fused with the original dual identification metric to obtain the corresponding corrected dual identification metric.

[0064] The calculated corrected dual authentication metric is compared again with the upper and lower thresholds of the historical confidence distribution interval. Based on the comparison results, the final audio authentication conclusion is determined as whether the user's original audio data is real or fake audio.

[0065] The beneficial effects are as follows: by estimating the confidence interval of users' historical audio data to determine the benchmark range for identification, and combining the threshold comparison of dual identification measures to achieve a preliminary determination of the authenticity of the audio, for audio data in the fuzzy range, auxiliary features of the distribution of silent segments and energy envelope are further extracted and similarity comparison is completed. After correcting the dual identification measures by auxiliary confidence increment, the threshold comparison is performed again to obtain the final conclusion. This comprehensive scoring method realizes hierarchical audio identification and judgment. It not only relies on historical data to ensure the personalization and accuracy of the judgment benchmark, but also achieves accurate identification of audio in the fuzzy range through the supplement of auxiliary features. It effectively avoids the misjudgment problem caused by single measure judgment and greatly improves the overall accuracy of fake audio identification and the reliability of the judgment results.

[0066] like Figure 2 The diagram shown is a functional block diagram of a fake audio identification system provided in an embodiment of the present invention.

[0067] The forged audio identification system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the forged audio identification system 100 may include an audio framing module 101, a data processing module 102, a feature fusion module 103, a feature analysis module 104, a dual identification module 105, and a comprehensive judgment module 106. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0068] In this embodiment, the functions of each module / unit are as follows: The audio framing module 101 is used to perform framing and windowing processing on the user's original audio data to obtain the user's frame sequence audio data. The data processing module 102 is used to perform a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extract the Mel frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector. The feature fusion module 103 is used to perform deep feature reconstruction on the initial acoustic feature vector to obtain the reconstructed feature tensor of the user, and to perform multimodal feature fusion on the reconstructed feature tensor and the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user; The feature analysis module 104 is used to perform temporal context analysis on the joint identification feature set, capture the feature dependencies of the frame sequence audio data over a long distance, and obtain the user's context-enhanced feature sequence. The dual authentication module 105 is used to quantify the cosine similarity between the context enhancement feature sequence and the user's real voiceprint template, and analyze the feature mutation amplitude between adjacent frames in the context enhancement feature sequence to obtain the user's dual authentication metric. The comprehensive analysis module 106 is used to perform a comprehensive score on the dual identification metric to obtain the user's audio identification conclusion.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0070] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0072] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0073] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for identifying counterfeit audio, characterized in that, The method includes: S1. Perform frame-segmentation and windowing processing on the user's original audio data to obtain the user's frame sequence audio data; S2. Perform a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extract the Mel frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector; S3. Perform deep feature reconstruction on the initial acoustic feature vector to obtain the reconstructed feature tensor of the user, and perform multimodal feature fusion on the reconstructed feature tensor and the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user; S4. Perform temporal context analysis on the joint identification feature set to capture the feature dependencies of the frame sequence audio data over a long distance range, and obtain the user's context-enhanced feature sequence. S5. Quantify the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyze the feature mutation amplitude between adjacent frames in the context-enhanced feature sequence to obtain the user's dual identification metric; S6. Perform a comprehensive score on the dual authentication metrics to obtain the audio authentication conclusion for the user.

2. The method for identifying counterfeit audio as described in claim 1, characterized in that, The step of performing frame-segmentation and windowing processing on the user's original audio data to obtain the user's frame sequence audio data includes: Collect the user's raw audio data, and perform pre-emphasis processing on the raw audio data to obtain the user's emphasized audio data; According to the preset frame length, the stressed audio data is uniformly divided into frames to obtain the user's framed audio data. The audio data in the segmented audio data is subjected to windowing and weighting processing to obtain the user's frame sequence audio data.

3. The method for identifying counterfeit audio as described in claim 1, characterized in that, The step of performing a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extracting Mel-frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector, includes: A short-time Fourier transform is performed on the audio data in the frame sequence audio data to map the audio data from the time domain to the frequency domain, thereby obtaining the user's frequency domain representation data; A nonlinear transformation at the Mel frequency scale is performed on the spectral amplitude values ​​in the frequency domain representation data to obtain the Mel spectral energy distribution data of the user. Take the logarithm of the Mel spectrum energy distribution data to obtain the logarithmic Mel spectrum data of the user; The log-Mel spectrum data is subjected to discrete cosine transform to remove the correlation between features of different dimensions, thereby obtaining the user's initial acoustic feature vector.

4. The method for identifying counterfeit audio as described in claim 1, characterized in that, The step of performing deep feature reconstruction on the initial acoustic feature vector to obtain the user's reconstructed feature tensor includes: The initial acoustic feature vector is mapped into a feature space to obtain the extended feature representation of the user. Residual information is captured on the extended feature representation, and subtle fluctuation components associated with audio generation traces are extracted from the extended feature representation to obtain the user's residual feature components; The residual feature components are differentially combined with the main feature components in the extended feature representation to obtain the user's reconstructed feature tensor.

5. The method for identifying counterfeit audio as described in claim 1, characterized in that, The step of fusing the reconstructed feature tensor with the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user includes: The user's pitch period trajectory features are obtained by performing pitch period detection on the frame sequence audio data; Linear prediction analysis is performed on the frame sequence audio data to obtain the formant trajectory features of the user; The pitch period trajectory features and the formant trajectory features are aligned, and the time scales of the pitch period trajectory features and the formant trajectory features are unified to the same frame index correspondence as the reconstructed feature tensor to obtain the aligned physiological feature sequence of the user. The aligned physiological feature sequence is fused with the reconstructed feature tensor to obtain the joint identification feature set of the user.

6. The method for identifying counterfeit audio as described in claim 1, characterized in that, The step of performing temporal context analysis on the joint identification feature set to capture the feature dependencies of the frame sequence audio data over a long distance, and obtaining the user's context-enhanced feature sequence, includes: Arrange the feature vectors in the joint identification feature set in chronological order to obtain the user's original feature sequence matrix; The original feature sequence matrix is ​​sliced ​​using a sliding window to obtain the user's local feature window matrix; Perform cross-frame correlation analysis on the local feature window matrix to identify the degree of mutual influence between the feature vectors of each frame within the local feature window, and obtain the user's inter-frame correlation weight matrix. Based on the inter-frame correlation weight matrix, the feature vectors within the local feature window are weighted and aggregated to obtain the user's context-aware feature vector. The context-aware feature vectors are combined according to the original time sequence to obtain the context-enhanced feature sequence of the user.

7. The method for identifying counterfeit audio as described in claim 1, characterized in that, The process of quantifying the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyzing the amplitude of feature abrupt changes between adjacent frames in the context-enhanced feature sequence to obtain the user's dual-discrimination metric includes: The frame feature vectors in the context enhancement feature sequence are compared with the user's real voiceprint template using cosine similarity analysis to obtain the user's instantaneous similarity value. The instantaneous similarity values ​​are aggregated by mean to obtain the global similarity metric for the user; Perform Euclidean distance analysis on the feature vectors of adjacent frames in the context-enhanced feature sequence to obtain the feature mutation magnitude value of the user, and integrate the feature mutation magnitude value to obtain the global mutation metric value of the user; The global similarity metric and the global mutation metric are nonlinearly fused to obtain the user's dual authentication metric.

8. The method for identifying counterfeit audio as described in claim 7, characterized in that, The formula for calculating the dual discrimination metric is as follows: ； in, This indicates the dual discrimination metric. This represents the global similarity metric value. This represents the global mutation metric. This represents the preset first weighting coefficient. This represents the preset similarity threshold. This represents the preset nonlinear transformation steepness coefficient. Represents the natural constant. This represents the hyperbolic tangent function.

9. The method for identifying counterfeit audio as described in claim 1, characterized in that, The process of comprehensively scoring the dual authentication metrics to obtain the user's audio authentication conclusion includes: Confidence interval estimation is performed on the user's historical audio data to obtain the user's historical confidence distribution interval; The dual authentication metric is compared with the upper and lower thresholds of the historical confidence distribution interval. If the dual authentication metric is greater than or equal to the upper limit of the historical confidence distribution interval, the original audio data is determined to be real audio. If the dual authentication metric is less than or equal to the lower limit of the historical confidence distribution interval, the original audio data is determined to be fake audio. If the dual discrimination metric is within the historical confidence distribution interval, then the silence segment distribution features and energy envelope features in the frame sequence audio data are extracted to obtain the user's auxiliary discrimination feature vector; The auxiliary identification feature vector is compared with the pre-stored real audio auxiliary feature template to obtain the user's auxiliary confidence increment. The auxiliary confidence increment is then accumulated and fused with the dual identification metric to obtain the user's corrected dual identification metric. The corrected dual authentication metric is then compared with the upper and lower thresholds of the historical confidence distribution interval to obtain the audio authentication conclusion for the user.

10. A system for identifying counterfeit audio, characterized in that, The system for implementing the fake audio identification method according to claim 1 includes: The audio framing module is used to perform frame-framing and windowing processing on the user's original audio data to obtain the user's frame sequence audio data. The data processing module is used to perform a short-time Fourier transform on the frame sequence audio data to obtain the user's frequency domain representation data, and extract the Mel frequency cepstral coefficients from the frequency domain representation data to obtain the user's initial acoustic feature vector. The feature fusion module is used to perform deep feature reconstruction on the initial acoustic feature vector to obtain the reconstructed feature tensor of the user, and to perform multimodal feature fusion on the reconstructed feature tensor and the pitch period trajectory features and formant trajectory features in the frame sequence audio data to obtain the joint identification feature set of the user; The feature analysis module is used to perform temporal context analysis on the joint identification feature set, capture the feature dependencies of the frame sequence audio data over a long distance, and obtain the user's context-enhanced feature sequence. The dual authentication module is used to quantify the cosine similarity between the context-enhanced feature sequence and the user's real voiceprint template, and analyze the feature mutation amplitude between adjacent frames in the context-enhanced feature sequence to obtain the user's dual authentication metric. The comprehensive analysis module is used to comprehensively score the dual identification metrics to obtain the audio identification conclusion of the user.

Images