Voice payment interaction method and system based on cloud sound

Abstract

This invention relates to the field of intelligent voice interaction technology, providing a voice payment interaction method and system based on cloud speakers. The method uses a multi-microphone array to collect ambient sound and three-dimensionally locate the user's coordinates. Based on the coordinates, it determines whether the user is within a preset payment area and facing the cloud speaker. If these conditions are met, voice commands and voiceprint features are extracted. Voiceprint verification is performed, and the system simultaneously analyzes for the presence of recording / playback attack features. If attack features are detected, an enhanced verification mode is triggered, collecting a randomly recited verification code from the user for dual comparison of voiceprint and content. Upon successful verification, payment is executed. This invention confines payment to a physical space and confirms the user's active intent through spatial positioning and orientation determination. Combined with multi-dimensional attack feature detection and a random verification code mechanism, it effectively resists recording / playback attacks. Simultaneously, beamforming technology ensures voice quality in noisy environments, significantly improving the security and reliability of cloud speaker voice payment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent voice interaction technology, and in particular relates to a voice payment interaction method and system based on cloud speakers. Background Technology

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, voice interaction has become one of the mainstream control methods for smart devices, and voice payment, as a new payment method, is gradually penetrating into daily life and business scenarios. Cloud speakers, as smart audio devices with cloud connectivity, are demonstrating enormous application potential in homes, supermarkets, convenience stores, vending machines, and other scenarios due to their ubiquitous deployment convenience and natural human-computer interaction. Users can complete operations such as purchasing goods and paying bills simply through voice commands, greatly improving payment efficiency and user experience, aligning with the future trend of contactless payment.

[0003] However, existing technical solutions have significant technical problems in practical applications. Simple voiceprint verification cannot effectively distinguish between genuine user voice and malicious recordings. Attackers can bypass identity verification simply by pre-recording legitimate user payment instructions and playing them near the device, resulting in a persistently high risk of payment fraud. Furthermore, this solution cannot confirm whether the user is within a reasonable physical payment area, nor can it determine whether the user has the subjective intent to pay. This makes the device susceptible to being mistakenly triggered by conversations of other unrelated individuals in the area, or to being used by others to exploit the user's location without their knowledge. In addition, in noisy commercial environments, background noise and multi-source interference severely affect the quality of voice acquisition, leading to a significant decrease in voiceprint recognition accuracy, an increase in payment failure rates, and a degraded user experience. These technical deficiencies collectively restrict the security and reliability of cloud-based voice payment in real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a voice payment interaction method based on cloud audio, which aims to solve the technical problems existing in the prior art as identified in the background art.

[0005] This invention is implemented as follows: a voice payment interaction method based on cloud audio, the method comprising:

[0006] The cloud speaker uses a built-in multi-microphone array to collect ambient sound signals in real time, and performs three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information.

[0007] Based on the three-dimensional coordinate information, it is determined whether the user is located within the preset payment area and the user's current orientation, and whether the prerequisite for this payment transaction has been met. If it has been met, the user's voice commands and voiceprint features are extracted from the ambient sound signal.

[0008] Speech recognition and voiceprint verification are performed, and the ambient sound signal is analyzed to determine if there are any recording replay attack features. If the recording replay attack features are detected, an enhanced verification mode is triggered, the user's voice signal is collected, and voiceprint matching and content comparison are performed. If the enhanced verification is successful, the payment operation is executed; otherwise, the payment is rejected and a third prompt message is output.

[0009] As a further aspect of the present invention, the step of acquiring ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and performing three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information specifically includes:

[0010] By using a built-in microphone array to synchronously acquire ambient sound signals, and performing analog-to-digital conversion and bandpass filtering preprocessing on each analog signal, multiple digital audio streams are obtained.

[0011] The multi-channel digital audio stream is processed by frame segmentation. The generalized cross-correlation function between each pair of signals is calculated in the frequency domain. The arrival time difference of the sound source to each microphone relative to the reference microphone is obtained through peak detection, and a set of arrival time difference values ​​are obtained.

[0012] Based on the known three-dimensional spatial coordinates and arrival time difference of each microphone, a hyperboloid equation system with the sound source position as the unknown is constructed to calculate the three-dimensional spatial coordinates of the sound source relative to the cloud speaker, and obtain the user's three-dimensional coordinate information.

[0013] Based on the user's three-dimensional coordinate information, the multiple digital audio streams are weighted and summed to enhance the signal-to-noise ratio of the speech signal in the direction indicated by the user's three-dimensional coordinate information, thereby obtaining an enhanced speech signal.

[0014] As a further aspect of the present invention, determining whether the user is within a preset payment area specifically includes:

[0015] Obtain the preset payment area parameters of the cloud speaker. The preset payment area parameters are spatial geometric bodies defined by coordinate range thresholds in a three-dimensional coordinate system established with the geometric center of the cloud speaker as the origin, including the minimum value of the X-axis, the maximum value of the X-axis, the minimum value of the Y-axis, the maximum value of the Y-axis, the minimum value of the Z-axis, and the maximum value of the Z-axis.

[0016] The X, Y, and Z coordinates of the user's three-dimensional coordinate information are compared item by item with the corresponding minimum, maximum, minimum, maximum, minimum, maximum, minimum, and maximum values ​​of the X, Y, Z axes in the preset payment area parameters.

[0017] If all the comparison results are yes, that is, the X, Y and Z coordinates all fall within the corresponding preset payment area parameter coordinate range threshold, then the user is determined to be within the preset payment area; if any comparison result is no, then the user is determined to be outside the preset payment area, the payment process is suspended and the first prompt message is broadcast.

[0018] As a further embodiment of the present invention, determining the user's current orientation based on the three-dimensional coordinate information specifically includes:

[0019] The cloud speaker's built-in camera captures images within the current field of view, performs deep learning-based face detection on the images, locates rectangular regions containing faces, and obtains face region images.

[0020] Facial key point detection is performed on the face region image to identify the pixel coordinates of the key points. Combined with the camera's intrinsic parameter matrix and distortion coefficients, the pitch angle, yaw angle, and roll angle representing the user's head orientation are obtained.

[0021] The absolute values ​​of the pitch angle, yaw angle, and roll angle obtained by the solution are compared with the preset angle threshold range. If all of them exceed the angle threshold range, it is determined that the user is facing the cloud speaker. Otherwise, it is determined that the user is not facing the cloud speaker, the payment process is suspended, and a second prompt message is output.

[0022] As a further aspect of the present invention, the extraction of the user's voice commands and voiceprint features from the ambient sound signal specifically includes:

[0023] Speech activity detection is performed on the enhanced speech signal, the start and end points of speech segments are marked from the continuous speech signal stream, and the effective speech segments containing the user's speech are extracted.

[0024] The effective speech segment is divided into frames, and a Hamming window is applied to each frame of signal to obtain the windowed speech frame sequence.

[0025] Each windowed speech frame is processed and spliced ​​to obtain the speakerprint feature vector.

[0026] The similarity between the voiceprint feature vector and the preset voiceprint template is calculated. At the same time, the effective speech segments are converted into text sequences, payment information is parsed, and the user's voice command parsing results are obtained.

[0027] As a further aspect of the present invention, the step of performing speech recognition and voiceprint verification, and simultaneously analyzing whether there are recording replay attack features in the environmental sound signal, specifically includes:

[0028] If the similarity calculation result exceeds the preset voiceprint threshold, the voiceprint verification is deemed to have passed; otherwise, the voiceprint verification is deemed to have failed.

[0029] The enhanced speech signal is subjected to spectral analysis. If any analytical feature in the spectral analysis exceeds a preset threshold, it is determined that there are characteristics of a recording replay attack.

[0030] As a further aspect of the present invention, the triggering of the enhanced verification mode, which involves collecting the user's voice signal and performing voiceprint matching and content comparison, specifically includes:

[0031] After determining that there are characteristics of a recording replay attack, a set of 4-digit random numbers is generated as a random verification code. The random verification code is then bound to the current session identifier and broadcast through the cloud speaker.

[0032] The system collects the user's repeated voice signal through a multi-microphone array and simultaneously identifies the voiceprint feature vector. It then performs content recognition matching with a random number string. If both the content recognition and voiceprint match, the content comparison is deemed successful; otherwise, the enhanced verification is deemed unsuccessful, the payment process is suspended, and a third prompt message is broadcast.

[0033] Another object of the present invention is to provide a voice payment interaction system based on cloud audio, the system comprising:

[0034] The sound signal acquisition and positioning module is used to acquire ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and to perform three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information.

[0035] The payment prerequisite judgment and feature extraction module is used to determine whether the user is located in the preset payment area and the user's current orientation based on the three-dimensional coordinate information, and to determine whether the prerequisite for this payment transaction has been met. If it has been met, the user's voice command and voiceprint features are extracted from the ambient sound signal.

[0036] The speech recognition and security verification module is used to perform speech recognition and voiceprint verification, and simultaneously analyze whether there are recording replay attack features in the environmental sound signal. If the recording replay attack features are detected, the enhanced verification mode is triggered, the user's voice signal is collected and voiceprint matching and content comparison are performed. If the enhanced verification is passed, the payment operation is executed; otherwise, the payment is rejected and a third prompt message is output.

[0037] The beneficial effects of this invention are:

[0038] The voice payment interaction method and system based on cloud speakers provided by this invention realizes three-dimensional spatial positioning of users through a multi-microphone array, restricts payment behavior to a preset physical area, isolates remote voice attacks and false triggering outside the area from a spatial dimension, and ensures that only users who enter the payment area can initiate the payment process.

[0039] By combining facial orientation detection to confirm that the user is actively facing the device, it effectively distinguishes between subjective payment intent and voice manipulation, resisting position-borrowing attacks and ensuring that payment operations truly reflect the user's intentions. Building upon voiceprint verification, it simultaneously performs multi-dimensional recording playback attack detection based on spectral flatness, phase consistency, and background noise repetition patterns. It accurately identifies various playback attacks from the frequency, phase, and time domains, addressing the core security vulnerability of simple voiceprint verification's inability to distinguish between real people and recordings.

[0040] When attack signatures are detected, a random CAPTCHA is automatically triggered for enhanced verification, completely eliminating the possibility of pre-recorded voice attacks through real-time interaction, forming a three-layer proactive defense system of voiceprint verification, attack signature detection, and random CAPTCHA. Simultaneously, beamforming technology based on user location results effectively enhances the signal-to-noise ratio of speech in the target direction, ensuring speech quality in noisy environments and improving the accuracy of voiceprint recognition and command parsing.

[0041] This invention organically combines physical space perception, user posture confirmation, biometric verification, and liveness attack detection to construct a progressive security chain from the environment to the user and then to transaction information. While maintaining a natural voice interaction experience, it significantly improves the anti-attack capability and environmental adaptability of cloud audio voice payment, laying a reliable foundation for the large-scale commercial use of voice payment technology. Attached Figure Description

[0042] Figure 1 A flowchart illustrating a voice payment interaction method based on cloud audio provided in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart for three-dimensional spatial positioning of a user who is making a sound, provided in an embodiment of the present invention.

[0044] Figure 3 A flowchart for determining whether the prerequisites for this payment transaction have been met, provided in an embodiment of the present invention;

[0045] Figure 4 This is a flowchart of speech recognition and voiceprint verification provided in an embodiment of the present invention;

[0046] Figure 5 This is a structural block diagram of a voice payment interaction system based on cloud audio provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] Figure 1A flowchart of a voice payment interaction method based on cloud audio provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0049] S100 collects ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and performs three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information.

[0050] First, an array of at least four microphones is used to synchronously acquire ambient sound. Each analog signal undergoes analog-to-digital conversion and bandpass filtering preprocessing to obtain multiple digital audio streams. These digital audio streams are then framed, with each frame lasting between 20 and 40 milliseconds. The generalized cross-correlation function between each pair of signals is calculated in the frequency domain. Peak detection is used to obtain the time difference between the sound source and the reference microphone. Based on the known three-dimensional spatial coordinates of each microphone and the time difference, a hyperboloid equation system with the sound source location as the unknown is constructed. The least squares method is used to solve for the user's three-dimensional spatial coordinates relative to the cloud speaker. Finally, based on the obtained user three-dimensional coordinate information, the multiple digital audio streams are weighted and summed to form a pickup beam in the user's direction, significantly enhancing the signal-to-noise ratio of the speech signal in that direction, resulting in an enhanced speech signal for subsequent steps.

[0051] S200: Based on the three-dimensional coordinate information, determine whether the user is located within the preset payment area and the user's current orientation, and determine whether the prerequisite for this payment transaction has been met. If it has been met, extract the user's voice commands and voiceprint features from the ambient sound signal.

[0052] The system determines whether the user is within a preset payment area. This determination process involves obtaining payment area parameters in a pre-established three-dimensional coordinate system with the geometric center of the cloud speaker as the origin. These parameters describe a spatial geometry using coordinate range thresholds, including minimum and maximum X-axis values, minimum and maximum Y-axis values, minimum and maximum Z-axis values, and maximum and minimum Z-axis values. The X, Y, and Z coordinates of the user's three-dimensional coordinate information obtained in step S100 are compared with these threshold values ​​one by one. Only when all coordinate values ​​fall within the corresponding coordinate range thresholds is the user determined to be within the preset payment area; if any value exceeds the range, the payment process is paused and the first prompt message is played. Assuming the user is within the payment area, the system further determines the user's current orientation. Images within the current field of view are captured using the cloud speaker's built-in camera. Deep learning-based face detection is performed on the images to locate the rectangular region containing the face, resulting in a face region image.

[0053] Facial landmark detection is performed on the face region image to identify the pixel coordinates of the landmarks. Combined with the camera's intrinsic parameter matrix and distortion coefficients, the pitch, yaw, and roll angles representing the user's head orientation are calculated. The absolute values ​​of these angles are compared with a preset angle threshold range. If all angles are within the threshold range, the user is determined to be facing the cloud speaker; otherwise, the user is determined not to be facing the cloud speaker, the payment process is paused, and a second prompt message is output.

[0054] The payment transaction is considered complete only when the user meets both conditions: being within the payment area and facing the cloud speaker. Subsequently, the user's voice commands and voiceprint features are extracted from the ambient sound signal.

[0055] The extraction process includes: performing speech activity detection on the enhanced speech signal obtained in step S100, marking the start and end points of speech segments from the continuous speech signal stream, and extracting effective speech segments containing user speech; performing framing operations on the effective speech segments, with each frame having a length of 25 milliseconds and a frame shift of 10 milliseconds, applying a Hamming window to each frame signal to obtain a windowed speech frame sequence; performing a fast Fourier transform on each windowed speech frame to obtain the spectrum, calculating the square of the spectrum amplitude to obtain the power spectrum, filtering the power spectrum through a Mel filter bank, taking the logarithm of the filtering result and performing a discrete cosine transform to obtain the Mel frequency cepstral coefficients of each frame, and concatenating the Mel frequency cepstral coefficients of multiple consecutive frames to form a voiceprint feature vector; simultaneously inputting the effective speech segments into an automatic speech recognition engine, converting the speech into a text sequence, and parsing the payment intention, payment amount, and payee information from the text sequence through a natural language understanding module to obtain the user's voice command parsing result.

[0056] S300 performs voice recognition and voiceprint verification, and simultaneously analyzes whether there are recording replay attack features in the environmental sound signal. If the recording replay attack features are detected, the enhanced verification mode is triggered, the user's voice signal is collected and voiceprint matching and content comparison are performed. If the enhanced verification is passed, the payment operation is executed; otherwise, the payment is rejected and a third prompt message is output.

[0057] This step involves voiceprint verification. The extracted voiceprint feature vector is compared with a preset voiceprint template for similarity calculation. If the similarity exceeds a preset voiceprint threshold, the voiceprint verification is considered successful; otherwise, it is considered a failure. Simultaneously, spectral analysis is performed on the enhanced speech signal to detect the presence of recording / playback attack features. This spectral analysis includes three dimensions: calculating the spectral flatness in the 4kHz to 8kHz range to detect the presence of missing harmonic structures; analyzing the phase consistency of the speech signal to detect the presence of phase discontinuities; and extracting the steady-state background noise portion to analyze the presence of repetitive patterns. If any of these analyzed features exceeds a preset threshold, the presence of recording / playback attack features is determined.

[0058] Upon detecting signs of a recording replay attack, the system triggers an enhanced verification mode, generating a 4-digit random number string as a random verification code. This random verification code is then bound to the current session identifier and broadcast via a cloud speaker. Subsequently, a multi-microphone array is used to collect the user's repeated voice signal, simultaneously identifying the voiceprint feature vector of the repeated speech and performing content recognition matching with the random number string. Only when both content recognition and voiceprint matching are successful is the enhanced verification considered successful, and the payment operation is executed; otherwise, the enhanced verification is considered unsuccessful, the payment process is suspended, and a third prompt message is broadcast.

[0059] like Figure 2 As shown, the process of acquiring ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and performing three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information specifically includes:

[0060] The S110 uses a built-in microphone array to synchronously collect ambient sound signals, performs analog-to-digital conversion and bandpass filtering preprocessing on each analog signal to obtain multiple digital audio streams.

[0061] S120, perform frame segmentation processing on the multi-channel digital audio stream, with each frame lasting 20-40 milliseconds, calculate the generalized cross-correlation function between each pair of signals in the frequency domain, obtain the arrival time difference of the sound source to each microphone relative to the reference microphone through peak detection, and obtain a set of arrival time difference values.

[0062] S130, based on the known three-dimensional spatial coordinates and arrival time difference of each microphone, constructs a hyperboloid equation system with the sound source position as the unknown, and uses the least squares method to solve and calculate the three-dimensional spatial coordinates of the sound source relative to the cloud speaker, thereby obtaining the user's three-dimensional coordinate information.

[0063] The offline payment scenario of cloud speakers is an open environment, which has problems such as environmental noise, multi-user interference, and off-site malicious attacks. A single microphone cannot distinguish the spatial location of the sound source. However, three-dimensional positioning based on the time difference of arrival and microphone coordinates is the only technical means that can accurately determine the physical spatial location of the user making the sound, and can clearly distinguish the spatial attributes of legitimate users and interference / attack sources.

[0064] Specifically:

[0065] Select the 0th microphone in the microphone array as the reference microphone, and for the ... One microphone ( , (where the total number of microphones is ), the corresponding hyperboloid equation is:

[0066] ;

[0067] Define the three-dimensional coordinate vector of the sound source to be solved as follows: The objective function for minimizing the sum of squared residuals is:

[0068] ;

[0069] The residual function of the single equation is:

[0070] ;

[0071] The Gauss-Newton method is used for nonlinear least squares iterative solution, and the iterative formula is as follows:

[0072] ;

[0073] in:

[0074] The three-dimensional spatial coordinates of the user (sound source) relative to the cloud speaker coordinate system are the unknowns to be solved.

[0075] Indicates the first The known three-dimensional spatial coordinates of a microphone in the cloud audio coordinate system;

[0076] This represents the known three-dimensional spatial coordinates of the reference microphone (0th microphone) in the cloud audio coordinate system;

[0077] Indicates the first The three-dimensional coordinate vector of each microphone. ;

[0078] This represents the three-dimensional coordinate vector of the reference microphone. ;

[0079] The speed of sound in air is taken as 340 m / s at normal temperature and pressure.

[0080] This indicates that the sound emitted by the sound source reaches the th The time difference between each microphone and the reference microphone is calculated using the generalized cross-correlation function.

[0081] This indicates the total number of microphones in the built-in microphone array of the cloud speaker;

[0082] This indicates the number of iterations in the Gauss-Newton iteration;

[0083] Indicates the first The estimated coordinates of the sound source obtained in the next iteration;

[0084] Indicates the first The Jacobian matrix of the residual function relative to the sound source coordinates at the next iteration;

[0085] Indicates the first The residual vector formed by the residuals corresponding to all microphones at the next iteration.

[0086] S140, based on the user's three-dimensional coordinate information, the multiple digital audio streams are weighted and summed to form a pickup beam in the direction indicated by the user's three-dimensional coordinate information, thereby enhancing the signal-to-noise ratio of the speech signal in that direction and obtaining an enhanced speech signal.

[0087] like Figure 3 As shown, determining whether a user is within a preset payment area specifically includes:

[0088] S210, obtain the preset payment area parameters of the cloud speaker. The preset payment area parameters are spatial geometric bodies defined by coordinate range thresholds in a three-dimensional coordinate system established with the geometric center of the cloud speaker as the origin, including the minimum value of the X-axis, the maximum value of the X-axis, the minimum value of the Y-axis, the maximum value of the Y-axis, the minimum value of the Z-axis, and the maximum value of the Z-axis.

[0089] The legitimate scenario for offline cloud speaker payment is when a user initiates payment within the limited physical space of the merchant's cashier. The preset payment area essentially defines the physical boundary of the legitimate payment space, clarifying that only users within this space are qualified to initiate the payment process. At the same time, it filters out invalid commands and malicious attacks from outside the area, reduces invalid system calculations, and balances the convenience and security of payment.

[0090] S220, compare the X, Y, and Z coordinates in the user's three-dimensional coordinate information with the corresponding minimum, maximum, minimum, maximum, minimum, maximum, minimum, and maximum values ​​of the X-axis, Y-axis, Z-axis, and Z-axis in the preset payment area parameters item by item;

[0091] S230. If all the comparison results are yes, that is, the X, Y and Z coordinates all fall within the corresponding preset payment area parameter coordinate range threshold, then the user is determined to be within the preset payment area; if any comparison result is no, then the user is determined to be outside the preset payment area, the payment process is paused and the first prompt message is broadcast.

[0092] Furthermore, determining the user's current orientation based on the three-dimensional coordinate information specifically includes:

[0093] The S240 uses the built-in camera of the cloud speaker to capture images within the current field of view, performs face detection based on deep learning on the images, locates the rectangular region containing the face, and obtains the face region image.

[0094] S250 performs facial landmark detection on the face region image, identifies the pixel coordinates of the landmarks, and combines the camera's intrinsic parameter matrix and distortion coefficients to solve for the pitch angle, yaw angle and roll angle representing the user's head orientation.

[0095] Specifically:

[0096] First, the pixel coordinates of the detected facial key points are converted into normalized imaging plane coordinates, and then distortion correction is performed.

[0097] ;

[0098] ;

[0099] in .

[0100] For the The projection relationship between the key points of an individual's face, in 3D world coordinates and 2D normalized coordinates, is as follows:

[0101] ;

[0102] The optimization objective of the PnP algorithm is to minimize the reprojection error.

[0103] ;

[0104] Using the ZYX rotation sequence, the relationship between the rotation matrix and the attitude angle is as follows:

[0105] ;

[0106] The fundamental rotation matrix is:

[0107] ;

[0108] ;

[0109] ;

[0110] The final formula for calculating the head pose angle is:

[0111] ;

[0112] in:

[0113] This represents the original pixel coordinates obtained from facial landmark detection.

[0114] The pixel coordinates representing the camera principal point (the intersection of the optical axis and the imaging plane) are derived from the camera intrinsic parameter matrix;

[0115] This represents the normalized focal length of the camera in the x-axis and y-axis directions, derived from the camera intrinsic parameter matrix;

[0116] Normalized imaging plane coordinates representing distortion;

[0117] Represents the normalized imaging plane coordinates after distortion correction;

[0118] This represents the radial distortion coefficient of the camera;

[0119] This represents the tangential distortion coefficient of the camera;

[0120] Indicates the first The normalized coordinate vectors after distortion removal at key points ;

[0121] Indicates the first The known 3D coordinates of individual facial key points in the world coordinate system of a standard 3D face model.

[0122] Indicates the first The Z-axis depth value of each key point in the camera coordinate system;

[0123] Represents a 3×3 rotation matrix that describes the rotation transformation from the world coordinate system to the camera coordinate system;

[0124] This represents a 3×1 translation vector, describing the translation transformation from the world coordinate system to the camera coordinate system;

[0125] This indicates taking the first two elements of the vector;

[0126] This represents the total number of facial landmarks used for solving the problem;

[0127] The pitch angle of the head describes the angle at which the head rotates up and down.

[0128] The yaw angle represents the angle at which the head turns left or right.

[0129] The roll angle of the head describes the angle at which the head tilts to the left or right.

[0130] Representing the rotation matrix The Middle line, number The elements of the column.

[0131] S260: Compare the absolute values ​​of the pitch angle, yaw angle and roll angle obtained by the solution with the preset angle threshold range. If all of them exceed the angle threshold range, it is determined that the user is facing the cloud speaker. Otherwise, it is determined that the user is not facing the cloud speaker, the payment process is suspended and the second prompt message is output.

[0132] Location alone can only confirm that a user is within the payment area; it cannot confirm whether the user has the subjective intention to initiate the payment, nor can it defend against location-borrowing attacks. The core purpose of orientation determination is to confirm that the user is actively facing the device to initiate the payment, rather than being passively manipulated by others, while filtering out accidental triggers from irrelevant personnel within the area, further strengthening the verification of real-person active verification of payment operations.

[0133] Furthermore, the extraction of the user's voice commands and voiceprint features from the ambient sound signal specifically includes:

[0134] S270 performs speech activity detection on the enhanced speech signal, marks the start and end points of speech segments from the continuous speech signal stream, and extracts the valid speech segments containing the user's speech.

[0135] S280, perform framing operation on the effective speech segment, with each frame length being 25 milliseconds and a frame shift of 10 milliseconds, and apply Hamming window to each frame signal to obtain the windowed speech frame sequence;

[0136] S290, perform a fast Fourier transform on each windowed speech frame to obtain the spectrum, calculate the square of the spectrum amplitude to obtain the power spectrum, filter the power spectrum through a Mel filter bank, take the logarithm of the filtering result and perform a discrete cosine transform to obtain the Mel frequency cepstral coefficients of each frame, and concatenate the Mel frequency cepstral coefficients of multiple consecutive frames to form a voiceprint feature vector.

[0137] Specifically:

[0138] Mel frequency to linear frequency conversion:

[0139] ;

[0140] Mel filter bank frequency response formula:

[0141] The Mel filter bank contains a total of The triangular filter, the first Filters ( The frequency response of ) is:

[0142] ;

[0143] Filter bank energy calculation:

[0144] ;

[0145] Discrete Cosine Transform:

[0146] Performing a DCT transform on the logarithmic energy yields the Mel frequency cepstral coefficients:

[0147] ;

[0148] in:

[0149] This represents the linear frequency, measured in Hz.

[0150] This represents the Mel frequency, measured in Mel.

[0151] This indicates the total number of filters in a Mel filter bank;

[0152] Indicates the sequence number of the Mel filter;

[0153] Represents the frequency index after the Fast Fourier Transform (FFT) of the speech frame;

[0154] Indicates the first The FFT frequency index corresponding to the center frequency of each filter;

[0155] Indicates the number of points in the FFT transform of the speech frame;

[0156] This indicates the first frame of a single-frame speech signal after FFT. Power spectrum values ​​at each frequency point;

[0157] Indicates the first The output energy of a Mel filter;

[0158] Indicates the first Meyer frequency cepstral coefficients (MFCC);

[0159] This indicates the order of the final extracted MFCC coefficients;

[0160] Represents the natural logarithm operation.

[0161] S2100 calculates the similarity between the voiceprint feature vector and the preset voiceprint template, and inputs the effective speech segment into the automatic speech recognition engine to convert the speech into a text sequence. Then, it uses the natural language understanding module to parse the payment intention, payment amount, and payee information from the text sequence to obtain the user's voice command parsing result.

[0162] like Figure 4 As shown, the process of performing speech recognition and voiceprint verification, and simultaneously analyzing whether there are recording replay attack features in the environmental sound signal, specifically includes:

[0163] S310: For the similarity calculation result, if the similarity exceeds the preset voiceprint threshold, the voiceprint verification is deemed to have passed; otherwise, the voiceprint verification is deemed to have failed.

[0164] S320 simultaneously performs spectral analysis on the enhanced speech signal, including: calculating the spectral flatness in the range of 4kHz to 8kHz and detecting whether there are missing harmonic structures; analyzing the phase consistency of the speech signal and detecting whether there are phase discontinuity features; extracting the steady-state background noise component and analyzing whether there are repetitive patterns.

[0165] If any analytical feature in the spectrum analysis exceeds a preset threshold, it is determined that there is a recording replay attack feature; otherwise, it is determined that there is no recording replay attack feature.

[0166] Replay attacks are the most prevalent attack method in voice payment. Attackers can easily bypass simple voiceprint verification by recording and playing back the voice of legitimate users. However, there are fundamental differences between real-time human voice and recorded / replayed voice in terms of spectral structure, phase continuity, and background noise characteristics. By detecting these three types of features, replay attacks can be accurately identified, thus compensating for the security shortcomings of voiceprint verification.

[0167] Table 1. Attack Performance and Security Role of Each Feature

[0168] Detection features Typical manifestations of a recording replay attack Safety enhancement role 4kHz~8kHz spectral flatness Ordinary recording / playback devices have limited high-frequency response capabilities, resulting in a lack of high-frequency harmonic structure and an abnormally flat spectrum in recorded and played-back audio. In contrast, real-time human speech has rich harmonic structure in the high-frequency range, exhibiting extremely low spectrum flatness. It can identify replayable audio recorded by the vast majority of ordinary devices, covering the most common attack tools such as mobile phones and speakers, and distinguish between real human voices and replayable audios in the frequency domain. Phase consistency Real human speech has continuous phase between adjacent frames, conforming to the physical laws of sound production; however, recorded speech, after sampling, compression, and digital-to-analog conversion, exhibits severe phase distortion and discontinuity between adjacent frames. This characteristic is even more pronounced in compressed audio playback. This technology identifies high-quality replayable speech that has been compressed, edited, and processed. Even if frequency domain features are forged, phase distortion cannot be completely eliminated, thus constructing a second line of defense against attacks from the phase dimension. Background noise repeating pattern The background noise of real-time human speech is random and non-periodic; however, the background noise of recorded speech is the fixed noise from the recording process, which will show obvious periodic repetition patterns when played back. This method identifies pre-recorded audio attacks that loop through multiple frames. Even if the frequency and phase features of a single frame are forged, the repetitive background noise patterns caused by looping cannot be eliminated, thus constructing a third line of defense against attacks from the time domain.

[0169] The three types of features cover the core vulnerabilities of all mainstream recording playback attacks from the three dimensions of frequency domain, phase, and time domain, forming a multi-dimensional and complementary attack detection system. It can be implemented without additional hardware, is compatible with the hardware conditions of cloud speakers, and completely solves the core security vulnerability that simple voiceprint verification cannot distinguish between real people and high-quality recordings, greatly reducing the fraud risk of voice payment.

[0170] Specifically, regarding spectral flatness:

[0171] For the 4kHz~8kHz frequency band, the formula for calculating spectral flatness (SFM) is:

[0172] ;

[0173] in:

[0174] This represents the FFT frequency index corresponding to a 4kHz frequency.

[0175] This represents the FFT frequency index corresponding to an 8kHz frequency.

[0176] Indicates the first speech frame after FFT The spectral amplitude value of each frequency point;

[0177] It represents the geometric product of the spectral amplitudes of all frequency points within the 4kHz~8kHz frequency band;

[0178] This indicates the flatness of the spectrum, measured in dB. A higher value indicates a flatter spectrum.

[0179] For phase consistency indices:

[0180] Phase difference between adjacent frames ;

[0181] Phase discontinuity ;

[0182] in:

[0183] Indicates the first Frame of speech signal Phase value at each frequency point;

[0184] Indicates the first Frame of speech signal Phase value at each frequency point;

[0185] Indicates the first The linear frequency corresponding to each frequency point is expressed in Hz.

[0186] The time interval between adjacent speech frames is equal to the duration corresponding to the frame shift, and the unit is seconds (s).

[0187] Indicates the first Frame number Phase compensation difference at each frequency point;

[0188] This represents the total number of valid frequency points in the FFT transform;

[0189] Indicates the total number of speech frames involved in the analysis;

[0190] This represents the phase consistency index. The larger the value, the higher the degree of phase discontinuity and the greater the possibility of a replay attack.

[0191] For repeating pattern detection:

[0192] 1. Calculation of normalized autocorrelation function

[0193] Extracting steady-state background noise segments from speech signals Calculate its normalized autocorrelation function:

[0194] ;

[0195] 2. Rules for Determining Repeating Patterns

[0196] Traverse all delays The number of peaks in the autocorrelation function whose amplitude exceeds a preset threshold is counted. If the number of peaks is ≥2 and the peak interval is periodic, then a background noise repetition pattern is determined to exist.

[0197] in:

[0198] This represents the sequence of sampling points for the extracted steady-state background noise signal;

[0199] This represents the time delay in autocorrelation calculations, expressed in the number of sampling points.

[0200] This represents the total number of sampling points in the background noise segment;

[0201] Indicates the delay amount as The normalized autocorrelation function value at time t, with a range of [-1, 1].

[0202] Furthermore, the triggering of the enhanced verification mode, which involves collecting the user's voice signal and performing voiceprint matching and content comparison, specifically includes:

[0203] After determining that there are characteristics of a recording replay attack, the S330 generates a set of 4-digit random numbers as a random verification code, and binds the random verification code with the current session identifier and broadcasts it through the cloud speaker.

[0204] Spectrum feature detection has a certain probability of false positives (e.g., when there is excessive environmental noise or poor user voice quality). Furthermore, high-fidelity audio playback attacks can potentially bypass spectrum feature detection. Random CAPTCHA mechanisms are the ultimate defense against audio playback attacks, fundamentally eliminating the possibility of pre-recorded voice attacks while balancing security and user experience, preventing legitimate users from being mistakenly blocked.

[0205] S340 collects the user's repeated voice signal through a multi-microphone array and simultaneously identifies the voiceprint feature vector, and performs content recognition matching with a random number string. If the content recognition matches and the voiceprint matches, the content comparison is deemed successful; otherwise, the enhanced verification is deemed to have failed, the payment process is suspended, and a third prompt message is broadcast.

[0206] Figure 5 The structural block diagram of the voice payment interaction system based on cloud audio provided in the embodiments of the present invention is as follows: Figure 5 As shown, the system includes:

[0207] The sound signal acquisition and positioning module 100 is used to acquire ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and to perform three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information.

[0208] The payment prerequisite judgment and feature extraction module 200 is used to determine whether the user is located in the preset payment area and the user's current orientation based on the three-dimensional coordinate information, and to determine whether the prerequisite for this payment transaction has been met. If it has been met, the user's voice command and voiceprint features are extracted from the environmental sound signal.

[0209] The voice recognition and security verification module 300 is used to perform voice recognition and voiceprint verification, and simultaneously analyze whether there are recording replay attack features in the environmental sound signal. If the recording replay attack features are detected, the enhanced verification mode is triggered, the user's voice signal is collected and voiceprint matching and content comparison are performed. If the enhanced verification is passed, the payment operation is executed; otherwise, the payment is rejected and a third prompt message is output.

[0210] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0212] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A voice payment interaction method based on cloud speakers, characterized in that, The method includes: The cloud speaker uses a built-in multi-microphone array to collect ambient sound signals in real time, and performs three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information. Based on the three-dimensional coordinate information, it is determined whether the user is located within the preset payment area and the user's current orientation, and whether the prerequisite for this payment transaction has been met. If it has been met, the user's voice commands and voiceprint features are extracted from the ambient sound signal. Speech recognition and voiceprint verification are performed, and the ambient sound signal is analyzed to determine if there are any recording replay attack features. If the recording replay attack features are detected, an enhanced verification mode is triggered, the user's voice signal is collected, and voiceprint matching and content comparison are performed. If the enhanced verification is successful, the payment operation is executed; otherwise, the payment is rejected and a third prompt message is output.

2. The method according to claim 1, characterized in that, The process of acquiring ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and performing three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information specifically includes: By using a built-in microphone array to synchronously acquire ambient sound signals, and performing analog-to-digital conversion and bandpass filtering preprocessing on each analog signal, multiple digital audio streams are obtained. The multi-channel digital audio stream is processed by frame segmentation. The generalized cross-correlation function between each pair of signals is calculated in the frequency domain. The arrival time difference of the sound source to each microphone relative to the reference microphone is obtained through peak detection, and a set of arrival time difference values ​​are obtained. Based on the known three-dimensional spatial coordinates and arrival time difference of each microphone, a hyperboloid equation system with the sound source position as the unknown is constructed to calculate the three-dimensional spatial coordinates of the sound source relative to the cloud speaker, and obtain the user's three-dimensional coordinate information. Based on the user's three-dimensional coordinate information, the multiple digital audio streams are weighted and summed to enhance the signal-to-noise ratio of the speech signal in the direction indicated by the user's three-dimensional coordinate information, thereby obtaining an enhanced speech signal.

3. The method according to claim 2, characterized in that, The determination of whether the user is within the preset payment area specifically includes: Obtain the preset payment area parameters of the cloud speaker. The preset payment area parameters are spatial geometric bodies defined by coordinate range thresholds in a three-dimensional coordinate system established with the geometric center of the cloud speaker as the origin, including the minimum value of the X-axis, the maximum value of the X-axis, the minimum value of the Y-axis, the maximum value of the Y-axis, the minimum value of the Z-axis, and the maximum value of the Z-axis. The X, Y, and Z coordinates of the user's three-dimensional coordinate information are compared item by item with the corresponding minimum, maximum, minimum, maximum, minimum, maximum, minimum, and maximum values ​​of the X, Y, Z axes in the preset payment area parameters. If all the comparison results are yes, that is, the X, Y and Z coordinates all fall within the corresponding preset payment area parameter coordinate range threshold, then the user is determined to be within the preset payment area; if any comparison result is no, then the user is determined to be outside the preset payment area, the payment process is suspended and the first prompt message is broadcast.

4. The method according to claim 3, characterized in that, The step of determining the user's current orientation based on the three-dimensional coordinate information specifically includes: The cloud speaker's built-in camera captures images within the current field of view, performs deep learning-based face detection on the images, locates rectangular regions containing faces, and obtains face region images. Facial key point detection is performed on the face region image to identify the pixel coordinates of the key points. Combined with the camera's intrinsic parameter matrix and distortion coefficients, the pitch angle, yaw angle, and roll angle representing the user's head orientation are obtained. The absolute values ​​of the pitch angle, yaw angle, and roll angle obtained by the solution are compared with the preset angle threshold range. If all of them exceed the angle threshold range, it is determined that the user is facing the cloud speaker. Otherwise, it is determined that the user is not facing the cloud speaker, the payment process is suspended, and a second prompt message is output.

5. The method according to claim 4, characterized in that, The extraction of the user's voice commands and voiceprint features from the ambient sound signal specifically includes: Speech activity detection is performed on the enhanced speech signal, the start and end points of speech segments are marked from the continuous speech signal stream, and the effective speech segments containing the user's speech are extracted. The effective speech segment is divided into frames, and a Hamming window is applied to each frame of signal to obtain the windowed speech frame sequence. Each windowed speech frame is processed and spliced ​​to obtain the speakerprint feature vector. The similarity between the voiceprint feature vector and the preset voiceprint template is calculated. At the same time, the effective speech segments are converted into text sequences, payment information is parsed, and the user's voice command parsing results are obtained.

6. The method according to claim 5, characterized in that, The specific steps of performing speech recognition and voiceprint verification, and simultaneously analyzing whether there are recording replay attack features in the environmental sound signal, include: If the similarity calculation result exceeds the preset voiceprint threshold, the voiceprint verification is deemed to have passed; otherwise, the voiceprint verification is deemed to have failed. The enhanced speech signal is subjected to spectral analysis. If any analytical feature in the spectral analysis exceeds a preset threshold, it is determined that there are characteristics of a recording replay attack.

7. The method according to claim 6, characterized in that, The triggering of the enhanced verification mode, which involves collecting the user's voice signal and performing voiceprint matching and content comparison, specifically includes: After determining that there are characteristics of a recording replay attack, a set of 4-digit random numbers is generated as a random verification code. The random verification code is then bound to the current session identifier and broadcast through the cloud speaker. The system collects the user's repeated voice signal through a multi-microphone array and simultaneously identifies the voiceprint feature vector. It then performs content recognition matching with a random number string. If both the content recognition and voiceprint match, the content comparison is deemed successful; otherwise, the enhanced verification is deemed unsuccessful, the payment process is suspended, and a third prompt message is broadcast.

8. A voice payment interaction system based on cloud speakers, characterized in that: The system includes: The sound signal acquisition and positioning module is used to acquire ambient sound signals in real time through the multi-microphone array built into the cloud speaker, and to perform three-dimensional spatial positioning of the user based on the ambient sound signals to obtain the user's three-dimensional coordinate information. The payment prerequisite judgment and feature extraction module is used to determine whether the user is located in the preset payment area and the user's current orientation based on the three-dimensional coordinate information, and to determine whether the prerequisite for this payment transaction has been met. If it has been met, the user's voice command and voiceprint features are extracted from the ambient sound signal. The speech recognition and security verification module is used to perform speech recognition and voiceprint verification, and simultaneously analyze whether there are recording replay attack features in the environmental sound signal. If the recording replay attack features are detected, the enhanced verification mode is triggered, the user's voice signal is collected and voiceprint matching and content comparison are performed. If the enhanced verification is passed, the payment operation is executed; otherwise, the payment is rejected and a third prompt message is output.

Images