Speech synthesis detection method based on multiple modes

Abstract

The invention relates to the technical field of speech synthesis detection, in particular to a speech synthesis detection method based on multiple modes, which comprises the following steps: extracting key audio detection features and key video detection features of a speech audio signal and a video mode signal at different signal points; dividing the voice audio signal and the video modal signal into a plurality of audio events and video events based on the key audio detection features and the key video detection features; calculating to obtain uncertainty between the audio event and the video event, constructing to obtain a cross-modal anchor point diagram, and extracting an alignment relationship between anchor points; a cross-modal speech synthesis detection model is used for detection to obtain the forgery probability of the audio events in the anchor points and the forgery synthesis probability of the speech audio signals; therefore, joint recognition of local audio event abnormity and global modal inconsistency is realized, so that whether the voice signal is forged synthesis or not is accurately judged, and the accuracy and robustness of voice synthesis detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis detection, and in particular relates to a speech synthesis detection method based on multi-modal. BACKGROUND

[0002] With the continuous maturity of speech synthesis technology, fake audio generation technology and cross-modal deep forgery technology in algorithm, computing power and data scale, current end-to-end speech cloning and high-fidelity speech synthesis models can generate highly realistic speech content in a very short time according to a small amount of samples. They show very close performance to real speech in timbre, speech rate, emotion and rhythm, thereby significantly weakening the effectiveness of traditional anti-forgery methods that rely on acoustic statistical features, rhythm rhythm patterns, short-time spectrum differences or hand-crafted forgery indicators.

[0003] In actual propagation scenarios, such deep forged speech usually appears synchronously with the corresponding video screen, forming audio-video joint forgery content, but the differences in sampling rate, cross-frame processing and encoding method between the speech modality and the video modality will lead to the existence of non-strict synchronization between them, which is manifested as local time drift, unstable information alignment or key event mismatch. The above factors make it difficult for technologies that rely solely on a single acoustic modality to achieve stable and reliable forgery segment recognition under complex conditions such as multi-modal timing misalignment, local forgery being masked by video action, and different modal forgery signals enhancing each other, resulting in a significant decline in detection accuracy and generalization ability.

[0004] In existing research, CN115171725B proposes a self-supervised anti-speech synthesis attack method, which constructs speech data training samples, uses a pre-trained model for self-supervised training, uses multiple synthetic speech algorithms to expand the anti-attack data set, and based on the fusion of the gating recurrent unit and the attention module, a synthesis detection model is used to improve the utilization efficiency and generalization ability of unannotated data. This scheme jointly optimizes the pre-trained model and the synthesis detection model through two-stage training, which can improve the detection accuracy in multiple synthetic speech algorithm scenarios. However, this scheme mainly carries out detection training for the acoustic modality, does not jointly model the speech generation process and the physical synchronization features such as lip movement and facial behavior of the video modality, and cannot perform joint detection with the help of multi-modal information. SUMMARY

[0005] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, this invention proposes a multimodal speech synthesis detection method. Existing speech forgery detection relies solely on the audio modality, failing to identify subtle anomalies caused by cross-frame splicing, fine-grained forgery, or deep forgery generation. This application simultaneously introduces video modal signals to achieve cross-modal consistency comparison, improving the ability to identify complex forgeries. Existing methods cannot represent the temporal relationships, spatial semantics, and motion changes of audio and video events using graph structures. This application constructs a cross-modal anchor graph, using audio and video events as anchors and uncertainty as edge relationships, forming a learnable cross-modal structured representation, thereby capturing the fine-grained dependencies between the two modalities. Introducing graph representation learning based on anchor point temporal drift allows learning the true alignment between different events, effectively identifying offset patterns caused by forgery. Through a cross-modal speech synthesis detection model, the forgery probability of audio events and the global description vector are jointly input into the final discriminant layer, achieving accurate inference of the forgery synthesis probability of the entire speech segment. This can locate specific suspicious audio events and improve the generalization detection effect under different scenarios and audio quality conditions.

[0006] To achieve the above objectives, this invention proposes a multimodal speech synthesis detection method, which includes the following steps: acquiring a speech audio signal and its corresponding associated video modal signal; extracting key audio detection features of the speech audio signal at different signal points, and extracting key video detection features of the video modal signal at different signal points, wherein the key audio detection features include short-time energy and fundamental frequency, and the key video detection features include lip opening and closing event detection results and lip width; dividing the speech audio signal into multiple audio events based on the key audio detection features, and dividing the video modal signal into multiple video events based on the key video detection features; calculating the uncertainty between the audio events and the video events, and constructing a cross-modal anchor graph by using the audio events and the video events as anchor points and the uncertainty between the anchor points as edge relationships; performing graph representation learning based on anchor point time drift on the cross-modal anchor graph to obtain the alignment relationship between the anchor points; inputting the alignment relationship between the anchor points into a pre-trained cross-modal speech synthesis detection model to obtain the forgery probability of the audio events in the anchor points and the forgery synthesis probability of the speech audio signal.

[0007] The multimodal speech synthesis detection method proposed in this embodiment of the invention has the advantage that, based on multimodal detection data, it can achieve robust synthesized speech detection under non-strict temporal synchronization conditions, thereby improving the detection accuracy of AI synthesized speech.

[0008] In addition, the multimodal speech synthesis detection method proposed in the above embodiments of the present invention may also have the following additional technical features: Optionally, the key video detection features of the video modality signal at different signal points are extracted, including: performing face key point detection on the signal values of the video modality signal at different signal points by using a face key point detector to obtain a contour point set of the upper lip and the lower lip; calculating a lip opening and closing height according to an absolute value of a difference between a midpoint coordinate of the contour points of the upper lip and a midpoint coordinate of the contour points of the lower lip; generating a lip opening and closing event detection result according to a change value of adjacent lip opening and closing heights; and generating a lip width according to a coordinate difference value of the horizontal direction coordinates of the left contour points of the lip and the horizontal direction coordinates of the right contour points of the lip.

[0009] Optionally, the speech audio signal is divided into a plurality of audio events according to the key audio detection features, including: According to the key audio detection features, an audio event segmentation determination result of the speech audio signal at different signal points is generated, wherein a determination formula of the audio event segmentation determination result is:

[0010] wherein, represents the speech audio signal the audio event segmentation determination result at the nth signal point, represents a conditional discriminant, if the formula in the conditional discriminant is true, the conditional discriminant outputs True, otherwise the conditional discriminant outputs False, represents the speech audio signal the short-time energy at the nth signal point, represents the speech audio signal the average value of the short-time energies at the N signal points, represents the speech audio signal the standard deviation of the short-time energies at the N signal points, represents an audio event control parameter, represents the speech audio signal the fundamental frequency at the nth signal point, represents the speech audio signal the fundamental frequency at the (n-1)th signal point, represents the speech audio signal the change value of the fundamental frequency at the nth signal point, represents a preset fundamental frequency change threshold value, represents a logical AND operator; N represents the signal length of the speech audio signal . When both sides of the logical AND operator output True, then output is True, indicating that the nth signal point of the speech audio signal is an audio event segmentation point; if Output is not True, indicating that the nth signal point of the speech audio signal is not a speech event segmentation point. According to the audio event segmentation point in the speech audio signal, the speech audio signal is divided into a plurality of mutually non-overlapping audio signal segments, and each audio signal segment is taken as an independent audio event, and the confidence of the audio event is calculated.

[0011] Optionally, according to the key video detection feature, the video modality signal is divided into a plurality of video events, comprising: According to the key video detection feature, a video event segmentation decision result of the video modality signal at different signal points is generated, wherein the decision formula of the video event segmentation decision result is:

[0012] wherein, indicates the video modality signal at the mth signal point, indicates the video modality signal at the mth signal point, is 1, indicating that the lip opening and closing event is detected, is 0, indicating that the lip opening and closing event is not detected, indicates the video modality signal at the mth signal point, indicates the video modality signal at the (m-1)th signal point, indicates a preset lip width change threshold, and M indicates the signal length of the video modality signal . When and are both output as True, output is True, indicating that the mth signal point of the video modality signal is a video event segmentation point; if output is not True, indicating that the mth signal point of the video modality signal is not a video event segmentation point. According to the video event segmentation point in the video modality signal, the video modality signal is divided into a plurality of mutually non-overlapping video signal segments, and each video signal segment is taken as an independent video event, and the confidence of the video event is calculated.

[0013] Optionally, calculating the uncertainty between the audio event and the video event includes: obtaining the confidence levels corresponding to the audio event and the video event; converting the audio event and the video event into spectral sequences respectively, and calculating the similarity between the spectral sequences corresponding to the audio event and the video event as the multimodal event similarity between the audio event and the video event; converting the multimodal event similarity between the audio event and the video event into uncertainty based on the confidence levels corresponding to the audio event and the video event, wherein the lower the uncertainty, the higher the degree of event matching between the audio event and the video event, and the audio event and the video event describe similar events.

[0014] Optionally, the video events are converted into spectral sequences, including: extracting the lip opening height and lip width at each signal point in the video event and concatenating them as feature vectors of the video event; mapping the feature vectors into S pseudo-Mel energies using a nonlinear mapping method; and concatenating the S pseudo-Mel energies as the spectral sequence of the video event.

[0015] Optionally, the structure of the cross-modal anchor point diagram is as follows:

[0016]

[0017]

[0018] in, Represents a cross-modal anchor point diagram. Represents cross-modal anchor point diagram The set of anchor points in the middle, These are, in order, audio event-anchor set and video event-anchor set. Represents audio event-anchor set The h-th anchor point in the sequence, where H represents the number of audio events. Represents a video event-anchor set The first in There are several anchor points, where G represents the number of video events. Represents cross-modal anchor point diagram The set of edge relations in the middle, Indicates anchor point The edge relationships between them.

[0019] Optionally, the cross-modal anchor point graph is subjected to anchor point time drift-based graph representation learning to obtain alignment relationships between the anchor points, including: obtaining event timestamps corresponding to the anchor points, wherein the event timestamp is the median of the timestamps of the continuous signal points corresponding to the anchor points; according to the event timestamps, anchor point time offsets between different anchor points are calculated; and according to the anchor point time offsets between the different anchor points and the edge relationships, alignment relationships between the anchor points are generated based on an attention mechanism.

[0020] Optionally, the cross-modal speech synthesis detection model includes a cross-modal alignment encoder, an audio event authenticity verification network, a global consistency aggregation network, and a speech synthesis discrimination network, wherein the cross-modal alignment encoder includes a normalization layer and an encoding layer, the audio event authenticity verification network includes a fully connected layer and a discrimination layer, the global consistency aggregation network includes a weighting module and a global consistency conversion module, and the speech synthesis discrimination network includes a concatenation layer and a global discrimination layer.

[0021] Optionally, the alignment relationships between the anchor points are input into a pre-trained cross-modal speech synthesis detection model to obtain a forgery probability of an audio event in the anchor points and a forgery synthesis probability of a speech audio signal, including: the cross-modal speech synthesis detection model receives the alignment relationships between the anchor points, the normalization layer in the cross-modal alignment encoder converts the alignment relationships between the anchor points into an alignment relationship matrix of H rows and G columns, wherein each row in the alignment relationship matrix represents the alignment relationship between an anchor point as an audio event and G anchor points as video events, and the encoding layer sequentially convolves and encodes each row in the alignment relationship matrix to obtain cross-modal encoding features of each anchor point as an audio event; the fully connected layer in the audio event authenticity verification network extracts hidden states in the cross-modal encoding features, and the discrimination layer based on a double-layer activation function converts the hidden states into a forgery probability as a forgery probability of an audio event associated with the hidden states; the weighting module in the global consistency aggregation network takes the forgery probability as a weight to weight the cross-modal encoding features corresponding to the audio events, and the global consistency conversion module projects the weighting result to obtain a global description vector; the concatenation layer in the speech synthesis discrimination network concatenates the global description vector and the forgery probabilities corresponding to the audio events, and the final discrimination layer based on a fully connected layer structure discriminates the concatenation result to output the forgery synthesis probability of the speech audio signal. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a multi-modal-based speech synthesis detection method according to an embodiment of the present application; Figure 2 A schematic diagram of a cross-modal anchor point graph according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] Embodiments of the present application are described below in the following detailed description and in conjunction with the following figures, wherein like numbers refer to like elements throughout. The embodiments described below are merely exemplary and are not intended to limit the present application, except as provided in the appended claims.

[0024] For a better understanding of the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0025] For a better understanding of the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0026] Reference Figure 1 The multi-modal based speech synthesis detection method of the embodiments of the present application includes the following steps: S101, acquiring a speech audio signal and its corresponding associated video modal signal.

[0027] That is, the speech audio signal and its associated video modal signal are collected, wherein the two signals can be collected from different collection links or independent devices, and thus there is no strict sampling clock synchronization relationship. The video modal signal can be directly provided by the speech audio signal provider. The video modal signal is in MP4 format, and the signal values of the video modal signal at different signal points are in the form of video frames.

[0028] Specifically, the signal points in the speech audio signal and the video modal signal are the discrete encoding numbers of the signals, wherein the representation form of the speech audio signal is:

[0029]

[0030] wherein, represents the speech audio signal, represents the video modal signal, represents the speech audio signal at the N signal points, wherein the speech audio signal has N signal points corresponding to timestamps in sequence, respectively , and N represents the signal length of the speech audio signal , represents the video modal signal signal values of the M signal points of the video modality signal are sequentially respectively M represents a signal length of the video modality signal represents a signal value of the n-th signal point of the speech audio signal represents a signal value of the m-th signal point of the video modality signal

[0031] S102, extracting key audio detection features of the speech audio signal at different signal points, and extracting key video detection features of the video modality signal at different signal points, wherein the key audio detection features include short-time energy and fundamental frequency, and the key video detection features include lip opening and closing event detection results and lip width.

[0032] Specifically, the extraction formula of the key audio detection features of the speech audio signal at different signal points is as follows:

[0033]

[0034]

[0035] wherein, represents a key audio detection feature of the n-th signal point of the speech audio signal are sequentially respectively short-time energy and fundamental frequency in the key audio detection feature represents a signal value of the k-th signal point of the speech audio signal L represents a signal time window length, and L is set to 4, represents a window function, represents a corresponding window function value, represents a fundamental frequency period corresponding to an autocorrelation function of a signal value sequence represents a fundamental frequency period range, represents a minimum fundamental frequency period (which can be set to 2), represents a maximum fundamental frequency period (which can be set to 10), represents extracting in such that reaches a maximum as the fundamental frequency of the n-th signal point of the speech audio signal

[0036] ​​​​​​​Optionally, the window function is a Hamming window function. represents a cosine function.

[0037] As an embodiment, the key video detection features of the video modality signal at different signal points are extracted, including: using a face key point detector to perform face key point detection on the signal values of the video modality signal at different signal points to obtain a contour point set of the upper lip and the lower lip; calculating a lip opening height according to an absolute value of a difference between a midpoint coordinate of the contour points of the upper lip and a midpoint coordinate of the contour points of the lower lip; generating a lip opening event detection result according to a change value of adjacent lip opening heights; and generating a lip width according to a coordinate difference value of the horizontal direction coordinates of the left contour point of the lip and the horizontal direction coordinates of the right contour point of the lip.

[0038] Optionally, the face key point detector uses a MediaPipe Face Mesh model, which outputs 468 fine-grained face key points, facilitating direct acquisition of lip contours and mandibular positions.

[0039] Specifically, the video modality signal The lip opening event detection result at the mth signal point is:

[0040] wherein, represents the video modality signal The lip opening event detection result at the mth signal point, represents the video modality signal The lip opening height at the mth signal point, represents a lip opening height detection threshold, is 1, indicating that a lip opening event is detected, is 0, indicating that no lip opening event is detected.

[0041] Optionally, the setting is the maximum value in the absolute values of the differences between the midpoint coordinates of the contour points of the upper lip and the midpoint coordinates of the contour points of the lower lip of the M collected signal points.

[0042] Specifically, the video modality signal The lip width calculation formula at the mth signal point is:

[0043] wherein, represents the video modality signal The lip width at the mth signal point, represents the video modality signal ​A left contour point of the lip at the mth signal point in a horizontal direction coordinate, representing the video modal signal A right contour point of the lip at the mth signal point in a horizontal direction coordinate, representing the calculation of the absolute value.

[0044] It should be noted that based on the short-time energy and fundamental frequency extraction formula, the present application can stably capture the energy mutation and periodic acoustic structure of speech at a micro-scale time point, accurately identify whether speech exists in the speech audio signal, when it starts and the change of the sound intensity, and solve the problem that pure reliance on endpoint detection cannot stably process weak sound, gas sound and the like. At the same time, the present application extracts the upper and lower lip contours by using a face key point detection model, constructs the lip opening height and lip width indicators, obtains the key features that can directly reflect the dynamic behavior of the mouth shape, so that the video modal can still provide clear pronunciation action information in the case of silent, partially blocked speech or noise interference scene, and the lip opening event detection result constructed based on the change of the lip opening height makes the video modal have the ability to express discrete event characteristics, which can be mutually verified with the short-time energy peak value of the audio modal, thereby improving the stability and robustness of cross-modal event alignment.

[0045] S103, dividing the speech audio signal into a plurality of audio events according to the key audio detection feature, and dividing the video modal signal into a plurality of video events according to the key video detection feature.

[0046] As an embodiment, dividing the speech audio signal into a plurality of audio events according to the key audio detection feature includes: generating an audio event segmentation determination result of the speech audio signal at different signal points according to the key audio detection feature, wherein the determination formula of the audio event segmentation determination result is:

[0047] wherein, representing the speech audio signal the audio event segmentation determination result at the nth signal point, representing the conditional discriminant, if the formula in the conditional discriminant is true, the output of the conditional discriminant is True, otherwise the output of the conditional discriminant is False, representing the speech audio signal the short-time energy at the nth signal point, representing the speech audio signal the average of the short-time energy at N signal points, representing the speech audio signal the standard deviation of the short-time energy at N signal points, representing the audio event control parameter, representing the speech audio signal The fundamental frequency at the nth signal point Represents speech audio signals The fundamental frequency at the (n-1)th signal point Represents speech audio signals The fundamental frequency change at the nth signal point This represents the preset fundamental frequency variation threshold. This represents the logical AND operator; N represents the audio signal. The signal length; When both sides of the logical AND operator output True, then An output of True indicates a speech audio signal. The nth signal point is used as the audio event segmentation point; if If the output is not True, it indicates a speech audio signal. The nth signal point is not used as an audio event segmentation point; Based on the audio event segmentation points in the speech audio signal, the speech audio signal is divided into multiple non-overlapping audio signal segments, and each audio signal segment is treated as an independent audio event, and the confidence level of the audio event is calculated.

[0048] Specifically, if the audio event segmentation points are respectively the first... and the There are 10 signal points, among which... , The audio events are then divided as follows: The key audio detection features of the audio event at each signal point are obtained, and the confidence level of the audio event is calculated. The formula for calculating the confidence level is:

[0049]

[0050]

[0051] in, Indicates audio events Confidence level, Indicates audio events Middle start end The confidence coefficient, Indicates audio events Middle end The confidence coefficient, Indicates audio events The short-time energy mean at all signal points. Indicates audio events The fundamental frequency mean at all signal points in the signal. representing audio events standard deviation of short-time energy at all signal points, all represent audio confidence control coefficients, set in turn 0.6 and 0.4, representing an activation function, the selected activation function is a Sigmoid function, used to control the confidence between 0 and 1.

[0052] As an embodiment, the video modal signal is divided into multiple video events according to the key video detection features, including: according to the key video detection features, the video event segmentation determination result of the video modal signal at different signal points is generated, wherein the determination formula of the video event segmentation determination result is:

[0053] wherein, representing the video modal signal video event segmentation determination result at the mth signal point, representing the video modal signal lip opening and closing event detection result at the mth signal point, 1 represents that the lip opening and closing event is detected, 0 represents that the lip opening and closing event is not detected, representing the video modal signal lip width at the mth signal point, representing the video modal signal lip width at the m-1th signal point, representing a preset lip width change threshold, M represents the signal length of the video modal signal ; when and are both output as True, then output as True, indicating that the mth signal point of the video modal signal is a video event segmentation point; if output is not True, indicating that the mth signal point of the video modal signal is not a video event segmentation point; According to the video event segmentation points in the video modal signal, the video modal signal is divided into multiple mutually non-overlapping video signal segments, and each video signal segment is taken as an independent video event, and the confidence of the video event is calculated.

[0054] Specifically, if the video event segmentation points are the first and the second signal points in turn, wherein , The video events are then divided as follows: The key video detection features of the video event at each signal point are obtained, and the confidence level of the video event is calculated. The formula for calculating the confidence level is:

[0055]

[0056]

[0057] in, Indicates video event Confidence level, Indicates audio events Middle start end The confidence coefficient, Indicates audio events Middle end The confidence coefficient, Indicates video event The mean of lip opening and closing event detection results at all signal points. Indicates video event The mean lip width at all signal points. Indicates video event The standard deviation of lip opening and closing event detection results at all signal points. All represent video confidence control coefficients, set The values ​​are 0.3 and 0.7 respectively. This represents the activation function.

[0058] S104. Calculate the uncertainty between audio events and video events, and use the audio events and video events as anchor points and the uncertainty between the anchor points as edge relationships to construct a cross-modal anchor point graph.

[0059] As one embodiment, calculating the uncertainty between audio events and video events includes: obtaining the confidence levels corresponding to the audio events and video events; converting the audio events and video events into spectral sequences respectively, and calculating the similarity between the spectral sequences corresponding to the audio events and video events as the multimodal event similarity between the audio events and video events; converting the multimodal event similarity between the audio events and video events into uncertainty based on the confidence levels corresponding to the audio events and video events, wherein the lower the uncertainty, the higher the degree of event matching between the audio events and video events, and the audio events and video events describe similar events.

[0060] As a specific embodiment, audio events The procedure of converting into a spectrum sequence is as follows: performing short-time Fourier transform on the audio event and calculating a power spectrum, wherein the audio event The corresponding power spectrum is:

[0061] wherein, denotes the power spectrum of the audio event at the frequency index k, and j denotes an imaginary unit, , denotes an exponential function with a natural constant as a base; mapping the power spectrum to S mel-frequency bands as a spectrum sequence:

[0062]

[0063] wherein, denotes the spectrum sequence of the audio event corresponding to the spectrum sequence, denotes a mel-energy obtained by mapping the power spectrum to the Sth mel-frequency band (S can be set as 10), denotes the Sth mel-filter, denotes a control parameter, and is set as 0.1, denotes taking a logarithm.

[0064] As a specific embodiment, converting a video event into a spectrum sequence includes: extracting and splicing the lip opening height and the lip width at each signal point in the video event as a feature vector of the video event; mapping the feature vector into S pseudo-mel energies by using a nonlinear mapping manner; and splicing the S pseudo-mel energies as a spectrum sequence of the video event.

[0065] Specifically, the mapping formula of the pseudo-mel energy is:

[0066] wherein, denotes the Sth pseudo-mel energy obtained by mapping the feature vector F, denotes a mapping weight parameter of the Sth nonlinear mapping function, denotes a mapping constant of the Sth nonlinear mapping function.

[0067] Optionally, the cosine similarity algorithm is used to calculate the similarity between the audio event and the video event corresponding to the spectrum sequence, wherein the cosine similarity-based spectrum similarity metric can robustly reflect the shape consistency between events, can weaken the influence of event length difference and local offset, and can obtain stable cross-modal event similarity. Based on the confidence of the audio event and the video event corresponding to the audio event and the video event, the multi-modal event similarity between the audio event and the video event is converted into uncertainty, wherein the lower the uncertainty, the higher the event matching degree between the audio event and the video event, and the audio event and the video event describe similar events.

[0068] It should be noted that the present application converts the audio event into a spectrum sequence based on the Mel energy, which can effectively compress the high-dimensional power spectrum and strengthen the discriminability of the energy distribution of the speech, so that the event-level speech feature has a stable frequency domain expression; at the same time, the pseudo-spectrum domain is constructed by the lip opening height and the lip width feature, the pseudo-Mel energy is generated by using nonlinear mapping, so that the video modality forms comparability with the audio modality in the frequency band structure, and the cross-modal spectrum alignment is realized.

[0069] Specifically, the uncertainty conversion formula between the audio event and the video event is:

[0070] wherein, indicates the uncertainty between the hth audio event and the indicates the uncertainty between the hth audio event and the indicates the multi-modal event similarity between the hth audio event and the indicates the confidence of the hth audio event, indicates the confidence of the hth video event, indicates the confidence of the hth video event, indicates the confidence of the hth video event, indicates the confidence of the hth video event, indicates the confidence of the hth video event,

[0071] It should be noted that by introducing the confidence of the event, the similarity is converted into uncertainty by logarithmic weighting, a joint evaluation mechanism based on event reliability and cross-modal consistency is realized, false matching caused by low-confidence events is suppressed, and the matching strength of real speech events and real lip events is enhanced, and the speech-lip synchronization relationship between the two is accurately reflected.

[0072] As an embodiment, the structure of the cross-modal anchor point graph is:

[0073]

[0074]

[0075] in, Represents a cross-modal anchor point diagram. Represents cross-modal anchor point diagram The set of anchor points in the middle, These are, in order, audio event-anchor set and video event-anchor set. Represents audio event-anchor set The h-th anchor point in the sequence, where H represents the number of audio events. Represents a video event-anchor set The first in There are several anchor points, where G represents the number of video events. Represents cross-modal anchor point diagram The set of edge relations in the middle, Indicates anchor point The edge relationships between them.

[0076] S105, perform graph representation learning based on anchor time drift on the cross-modal anchor graph to obtain the alignment relationship between anchors.

[0077] As an example, graph representation learning based on anchor time drift is performed on the cross-modal anchor graph to obtain the alignment relationship between anchors, including: obtaining the event timestamps corresponding to the anchors, wherein the event timestamps are the median timestamps of the continuous signal points corresponding to the anchors; calculating the anchor time offset between different anchors based on the event timestamps; and generating the alignment relationship between anchors based on the anchor time offsets between different anchors and the edge relationships, using an attention mechanism.

[0078] Specifically, anchor point The time offset between the anchor points is: R

[0079] in, Indicates anchor point Anchor point time offset between Indicates the prior drift standard deviation. This represents the prior drift mean. anchor point The corresponding event timestamp, Indicates anchor point The corresponding event timestamps are obtained by simultaneously acquiring multiple sets of audio and video modal signals, and calculating the mean and standard deviation of the timestamp deviations between the two types of signals, which are used as the prior drift mean and prior drift standard deviation, respectively. This represents an exponential function with the natural constant as its base.

[0080] Anchor point The alignment relationship between the anchor points is generated by a formula:

[0081] wherein, represents the anchor point between the anchor points.

[0082] It should be noted that, by defining the event timestamp of each anchor point as the median of the continuous signal point timestamps, the application can effectively eliminate local noise within the event, boundary ambiguity and detection jitter, so that the event timing position has stability, and based on the timestamp deviation of multiple sets of speech audio signals and video modal signals, the prior drift mean and the prior drift standard deviation are quantified in advance, so that a statistical prior reflecting the real voice synchronization offset law is formed, and a Gaussian type anchor time offset function is used to reflect the natural physiological delay between acoustic generation and facial muscle movement, which can maintain matching robustness in the presence of natural mouth lag, pronunciation advance, frame rate difference and the like. Synchronization introduces attention mechanism to construct adaptive weight , which can enhance the coupling strength of real speech events and real mouth events, suppress common artifacts such as voice-mouth asynchronization, mouth lag, audio event missing in AI synthesized speech, and improve discrimination accuracy.

[0083] S106, input the alignment relationship between the anchor points into the pre-trained cross-modal speech synthesis detection model to obtain the forgery probability of the audio events in the anchor points and the forgery synthesis probability of the speech audio signals.

[0084] As an embodiment, the cross-modal speech synthesis detection model includes a cross-modal alignment encoder, an audio event authentication network, a global consistency aggregation network, and a speech synthesis discrimination network, wherein the cross-modal alignment encoder includes a standardization layer and an encoding layer, the audio event authentication network includes a fully connected layer and a discrimination layer, the global consistency aggregation network includes a weighting module and a global consistency conversion module, and the speech synthesis discrimination network includes a splicing layer and a global discrimination layer.

[0085] As an embodiment, the alignment relationship between the anchor points is input into a pre-trained cross-modal speech synthesis detection model to obtain a forgery probability of an audio event in the anchor points and a forged synthesis probability of a speech audio signal, including: adopting the cross-modal speech synthesis detection model to receive the alignment relationship between the anchor points, a standardization layer in a cross-modal alignment encoder converts the alignment relationship between the anchor points into an alignment relationship matrix of H rows and G columns, wherein each row in the alignment relationship matrix represents the alignment relationship between an anchor point as an audio event and G anchor points as video events, and an encoding layer sequentially performs convolutional encoding on each row in the alignment relationship matrix to obtain a cross-modal encoding feature of each anchor point as an audio event; a fully connected layer in an audio event forgery judgment network extracts a hidden state in the cross-modal encoding feature, and adopts a discrimination layer based on a double-layer activation function to convert the hidden state into a forgery probability as the forgery probability of the audio event associated with the hidden state; a weighting module in a global consistency aggregation network takes the forgery probability as a weight to weight the cross-modal encoding feature corresponding to the audio event, and a global consistency conversion module projects the weighting result to obtain a global description vector; a splicing layer in a speech synthesis discrimination network splices the global description vector and the forgery probability corresponding to each audio event, and adopts a final discrimination layer based on a fully connected layer structure to discriminate the splicing result to output the forged synthesis probability of the speech audio signal.

[0086] Specifically, the hth row in the alignment relationship matrix is the cross-modal encoding feature of the anchor point and the G anchor points as video events, wherein the convolutional encoding formula of the encoding layer is:

[0087] wherein, represents the cross-modal encoding feature of the anchor point , represents a trainable encoding weight matrix, represents a trainable encoding bias matrix.

[0088] Specifically, the cross-modal encoding feature is converted into a forgery probability , and the formula is:

[0089] wherein, represents a hidden state in the cross-modal encoding feature , represents a ReLU activation function, represents an activation function, and the selected activation function is a Sigmoid function, represents a trainable hidden layer weight parameter, represents a trainable hidden layer bias parameter, denote trainable discriminative layer weight parameters, denote trainable discriminative layer bias parameters. Optionally, the ReLU activation function is replaced by a Leaky ReLU activation function or a GELU activation function.

[0090] Specifically, the formula for the global consistency conversion module to project the weighted result is:

[0091] wherein, denote global description vectors, denote weighted results, denote trainable projection weight parameters, denote trainable bias parameters, denote hyperbolic tangent functions.

[0092] As an embodiment of the present application, by collecting the alignment relationship between multiple groups of anchor points, and the real / fake probability (0 or 1, 1 indicating that the audio event is a fake signal segment) of each anchor point as an audio event and the real / fake synthesis probability (0 or 1, 1 indicating that the speech audio signal is a fake signal) of the speech audio signal, a loss function is constructed to minimize the difference between the real / fake probability / synthesis probability and the fake probability of the audio event and the fake synthesis probability of the speech audio signal output by the model, and a gradient descent algorithm or Adam optimizer is used to train and optimize the trainable parameters in the model.

[0093] Referring to Figure 2 Fig. 1 shows a schematic diagram of a cross-modal anchor point graph provided by an embodiment of the present application, wherein E1_1, E1_2 and E1_3 are anchor points corresponding to audio events, and there is an edge relationship between the anchor points E2_1 and E2_2 corresponding to video events, the edge relationship representing the uncertainty between events, which is comprehensively measured by the confidence and similarity of the events, wherein the higher the edge relationship, the lower the event matching degree between the audio events and the video events.

[0094] In summary, according to the multi-modal based speech synthesis detection method of the present application, through the synchronous judgment based on the short-time energy mean value and the standard deviation and the pitch transition amplitude, the real starting and ending points of the sound can be effectively identified, and the problem of weak voice missing detection or noise false detection caused by relying on energy alone can be avoided. After the audio event segmentation, by calculating the confidence coefficient at the starting end and the ending end of the audio event respectively and taking the average, combining the normalized confidence expression constructed by the short-time energy mean value, the pitch mean value and the standard deviation in the audio event, the authenticity, continuity and speech stability of each audio event can be quantified, the reliability evaluation of the event level is strengthened, and the Sigmoid activation function is used to make the confidence stable in the range of 0 to 1, which is convenient for subsequent cross-modal alignment analysis. At the same time, by constructing the cross-modal speech synthesis detection model driven by cross-modal alignment, the fine-grained consistency modeling between audio events and video events is realized, and the accuracy and robustness of the fake speech detection are significantly improved. Specifically, the cross-modal alignment encoder uses the standardization layer to convert the cross-modal correspondence relationship at the anchor point level into an alignment relationship matrix of HxG, so that the model can capture the local offset features of the cross-modal events in time, semantics and motion trajectory; the encoding layer extracts the cross-modal encoding features row by row to enhance the coupling expression ability between the cross-modal; the audio event authentication network maps the cross-modal encoding features to the audio event level fake probability based on the double-layer activation structure, realizes the fake mode revelation at the event granularity, and can effectively identify local inconsistency, micro-fake and cross-frame discontinuity and other hidden abnormalities; the global consistency aggregation network takes the event fake probability as the adaptive weight, emphasizes the contribution of key suspicious events to the global discrimination, generates a stable global description vector through the projection structure, and thus realizes the unified extraction of the fake features from the event level to the global level; the final discrimination layer fuses the global description and the event fake probability to realize high-precision speech fake synthesis probability prediction, and finally realizes multi-level and multi-scale cross-modal consistency authentication, which has the advantages of interpretability, real-time performance and generalization ability.

[0095] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROMs, optical memory, etc.) having computer usable program code embodied thereon.

[0096] The present application is described in reference to the flowchart and / or block diagram of the method, apparatus (system) and computer program product according to an embodiment of the present application. It is understood that each flow and / or block in the flowchart and / or block diagram, and a combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or a combination of flows and / or blocks.

[0097] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or a combination of flows and / or blocks.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or a combination of flows and / or blocks.

[0099] It should be noted that any references made in the claims to an "apparatus" or "means" should not be construed to cover the corresponding structures only. The phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. The use of "including" and "comprising" and variations thereof is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms "connected," "coupled," and "pathway" are used broadly and encompass both direct and indirect connections, couplings and pathways.

[0100] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such variations and modifications as fall within the scope of the application.

[0101] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the application. Accordingly, it is intended that all such modifications and changes be included in the present application only as far as they come to be within the scope of the claims and their equivalents.

[0102] In the description of the present application, it is to be understood that the terms "first", "second", "third" and the like, merely mean different instances of the same thing, and do not imply relative importance or a number of the indicated technical features. Thus, a feature defined with "first", "second", "third" can include one or more of the features. In the description of the present application, the meaning of "plurality" is two or more, unless specifically defined otherwise.

[0103] In the present application, unless specifically defined and limited otherwise, the terms "mounting", "connection", "connecting", "fixed", and the like, should be interpreted broadly, for example, can be fixed connection, can be detachable connection, or integral; can be mechanical connection, can be electrical connection; can be direct connection, can be indirect connection through intermediate medium, can be internal communication of two elements or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0104] In the present application, unless specifically defined and limited otherwise, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through intermediate medium. Moreover, the first feature "on", "above" and "on the surface of" the second feature can be that the first feature is directly above or obliquely above the second feature, or only means that the horizontal height of the first feature is higher than that of the second feature. The first feature "under", "below" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or only means that the horizontal height of the first feature is less than that of the second feature.

[0105] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0106] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that variations, modifications, substitutions and changes can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A multimodal speech synthesis detection method, characterized in that, Includes the following steps: Acquire the speech audio signal and its corresponding associated video modal signal; Extract key audio detection features of the speech audio signal at different signal points, and extract key video detection features of the video modal signal at different signal points, wherein the key audio detection features include short-time energy and fundamental frequency, and the key video detection features include lip opening and closing event detection results and lip width; The speech audio signal is divided into multiple audio events based on the key audio detection features, and the video modal signal is divided into multiple video events based on the key video detection features; Calculate the uncertainty between the audio event and the video event, and construct a cross-modal anchor point graph by using the audio event and the video event as anchor points and the uncertainty between the anchor points as edge relationships; The cross-modal anchor point graph is subjected to graph representation learning based on anchor point time drift to obtain the alignment relationship between anchor points; The alignment relationship between the anchor points is input into a pre-trained cross-modal speech synthesis detection model to obtain the forgery probability of audio events in the anchor points and the forgery synthesis probability of speech audio signals.

2. The speech synthesis detection method based on multimodal processing as described in claim 1, characterized in that, Extracting key video detection features of the video modal signal at different signal points, including: A facial key point detector is used to detect facial key points on the signal values ​​of the video modal signal at different signal points to obtain the contour point set of the upper lip and lower lip; The lip opening height is calculated based on the absolute value of the difference between the midpoint coordinates of the upper lip contour and the midpoint coordinates of the lower lip contour. Based on the change in the height of adjacent lip opening and closing, generate lip opening and closing event detection results; The lip width is generated based on the difference between the horizontal coordinates of the left and right contour points of the lip.

3. The speech synthesis detection method based on multimodal processing as described in claim 1, characterized in that, The speech audio signal is divided into multiple audio events based on the key audio detection features, including: Based on the key audio detection features, audio event segmentation determination results of the speech audio signal at different signal points are generated, wherein the determination formula for the audio event segmentation determination results is as follows: in, Represents speech audio signals The audio event segmentation determination result at the nth signal point. This represents a conditional statement. If any expression in the conditional statement is true, the output of the conditional statement is True; otherwise, the output of the conditional statement is False. Represents speech audio signals The short-time energy at the nth signal point, Represents speech audio signals The short-time energy mean of N signal points Represents speech audio signals The short-time energy standard deviation at N signal points Indicates audio event control parameters, Represents speech audio signals The fundamental frequency at the nth signal point Represents speech audio signals The fundamental frequency at the (n-1)th signal point Represents speech audio signals The fundamental frequency change at the nth signal point This represents the preset fundamental frequency variation threshold. This represents the logical AND operator; N represents the audio signal. The signal length; When both sides of the logical AND operator output True, then An output of True indicates a speech audio signal. The nth signal point is used as the audio event segmentation point; if If the output is not True, it indicates a speech audio signal. The nth signal point is not used as an audio event segmentation point; Based on the audio event segmentation points in the speech audio signal, the speech audio signal is divided into multiple non-overlapping audio signal segments, and each audio signal segment is treated as an independent audio event, and the confidence level of the audio event is calculated.

4. The speech synthesis detection method based on multimodal processing as described in claim 1, characterized in that, The video modal signal is divided into multiple video events based on the key video detection features, including: Based on the key video detection features, video event segmentation determination results are generated for video modal signals at different signal points. The determination formula for the video event segmentation results is as follows: in, Represents video modal signal The video event segmentation determination result at the m-th signal point. Represents video modal signal The lip opening and closing event detection result at the m-th signal point A value of 1 indicates that a lip opening / closing event was detected. A value of 0 indicates that no lip opening / closing event was detected. Represents video modal signal The lip width at the m-th signal point, Represents video modal signal The lip width at the (m-1)th signal point, This represents the preset threshold for lip width variation, and M represents the video modal signal. The signal length; when as well as If all outputs are True, then An output of True indicates a video modal signal. The m-th signal point is used as the video event segmentation point; if If the output is not True, it indicates a video modal signal. The m-th signal point is not used as a video event segmentation point; Based on the video event segmentation points in the video modal signal, the video modal signal is divided into multiple non-overlapping video signal segments, and each video signal segment is treated as an independent video event. The confidence level of the video event is then calculated.

5. The speech synthesis detection method based on multimodal processing as described in claim 2, characterized in that, Calculating the uncertainty between the audio event and the video event includes: Obtain the confidence scores corresponding to the audio events and the video events; The audio events and video events are converted into spectral sequences, and the similarity between the spectral sequences corresponding to the audio events and video events is calculated as the multimodal event similarity between the audio events and video events. Based on the confidence levels corresponding to the audio and video events, the multimodal event similarity between the audio and video events is converted into uncertainty. The lower the uncertainty, the higher the degree of event matching between the audio and video events, indicating that the audio and video events describe similar events.

6. The multimodal speech synthesis detection method as described in claim 5, characterized in that, The video events are converted into spectral sequences, including: Extract the lip opening and closing height and lip width at each signal point in the video event and concatenate them to obtain the feature vector of the video event; The feature vector is mapped to S pseudo-Mel energies using a nonlinear mapping method; The S pseudo-Mel energy values ​​are spliced ​​together to form the spectral sequence of the video event.

7. The speech synthesis detection method based on multimodal processing as described in claim 1, characterized in that, The structure of the cross-modal anchor point diagram is as follows: in, Represents a cross-modal anchor point diagram. Represents cross-modal anchor point diagram The set of anchor points in the middle, These are, in order, audio event-anchor set and video event-anchor set. Represents audio event-anchor set The h-th anchor point in the sequence, where H represents the number of audio events. Represents a video event-anchor set The first in There are several anchor points, where G represents the number of video events. Represents cross-modal anchor point diagram The set of edge relations in the middle, Indicates anchor point The edge relationships between them.

8. The speech synthesis detection method based on multimodal processing as described in claim 1, characterized in that, The cross-modal anchor graph is subjected to graph representation learning based on anchor time drift to obtain the alignment relationship between anchors, including: Obtain the event timestamp corresponding to the anchor point, where the event timestamp is the median of the timestamps of the consecutive signal points corresponding to the anchor point; Based on the event timestamps, the anchor point time offset between different anchor points is calculated; Based on the anchor point time offset and edge relationship between the different anchor points, an alignment relationship between anchor points is generated using an attention mechanism.

9. The speech synthesis detection method based on multimodal processing as described in claim 1, characterized in that, The cross-modal speech synthesis detection model includes a cross-modal alignment encoder, an audio event authentication network, a global consistency aggregation network, and a speech synthesis discriminant network. The cross-modal alignment encoder includes a normalization layer and an encoding layer, the audio event authentication network includes a fully connected layer and a discriminant layer, the global consistency aggregation network includes a weighting module and a global consistency transformation module, and the speech synthesis discriminant network includes a splicing layer and a global discriminant layer.

10. The multimodal speech synthesis detection method as described in claim 9, characterized in that, The alignment relationship between the anchor points is input into a pre-trained cross-modal speech synthesis detection model to obtain the forgery probability of audio events in the anchor points and the forgery synthesis probability of speech audio signals, including: A cross-modal speech synthesis detection model is used to receive the alignment relationship between anchor points. The normalization layer in the cross-modal alignment encoder converts the alignment relationship between anchor points into an H-row G-column alignment relationship matrix. Each row in the alignment relationship matrix represents the alignment relationship between the anchor point as an audio event and G anchor points as video events. The coding layer sequentially performs convolutional coding on each row in the alignment relationship matrix to obtain the cross-modal coding feature of each anchor point as an audio event. The fully connected layer in the audio event authentication network extracts the hidden state from the cross-modal coding features, and a discriminative layer based on a two-layer activation function is used to convert the hidden state into a forgery probability, which is used as the forgery probability of the audio event associated with the hidden state. The weighting module in the global consistency aggregation network uses the forgery probability as weight to weight the cross-modal encoded features corresponding to the audio event. The global consistency transformation module projects the weighting result to obtain the global description vector. The concatenation layer in the speech synthesis discriminant network concatenates the global description vector and the forgery probability corresponding to each audio event. The final discriminant layer, based on a fully connected layer structure, discriminates the concatenation result and outputs the forgery synthesis probability of the speech audio signal.

Images