Identity identification method and device, computer device and storage medium

Abstract

The application relates to an identity recognition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a to-be-recognized signal, the to-be-recognized signal comprising a millimeter wave signal and an audio signal; performing living body feature detection based on the millimeter wave signal and the audio signal to obtain a living body millimeter wave signal and a living body audio signal; performing feature fusion based on the living body millimeter wave signal and the living body audio signal to obtain a living body voice signal fusion response graph; and performing identity recognition based on the living body voice signal fusion response graph. That is to say, in the face of malicious spoofing attacks in a voice identity recognition and authentication scene, voiceprint perception data on an audio modality and living body data of a human body skin surface are extracted, a response graph for measuring data information quality is constructed, identity recognition is realized, malicious spoofing attacks are resisted, and the accuracy of voiceprint recognition and authentication is improved.

Description

Technical Field

[0001] This application relates to the field of voiceprint authentication technology, and in particular to an identity recognition method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of voiceprint authentication technology, it has become a widely recognized method for identifying trusted users, freeing them from the constraints of remembering passwords and using auxiliary gestures. Commands and verification can be unified simply through voice, gaining widespread attention and being applied in various sectors of society due to its convenience and user-friendliness. Applications include smart speakers, financial systems, and access control. High robustness and high security are fundamental performance requirements for corresponding voice control systems. Traditional pure voiceprint authentication systems typically employ single-modal voiceprint authentication algorithms. However, due to the inherent propagation properties of sound waves, they are easily susceptible to malicious spoofing attacks. Besides easily manipulated replay attacks and imitation attacks, attackers may also launch adversarial attacks, causing the pure voiceprint authentication system to mistakenly identify the adversarial signals as registered users in the user database. Furthermore, the accuracy of authentication decreases under the influence of environmental noise.

[0003] In existing technologies, researchers have proposed multimodal voiceprint authentication systems to address the risks and shortcomings of pure voiceprint authentication systems. Multimodal voiceprint authentication systems integrate additional complementary modalities, such as gestures, Wi-Fi, images, and electrocardiograms, to fuse multiple user features for identity verification. However, these methods only consider unimodal attacks targeting speech, ignoring multimodal attacks targeting speech and other modalities. In most multimodal voiceprint authentication systems, the additional complementary modalities are developed and utilized independently, ignoring the internal correlations related to speech. When encountering interference such as motion or changes in lighting, the overall system authentication performance degrades. Furthermore, recognition that fuses gestures or electrocardiograms with speech requires the user to perform specific actions, significantly limiting the user experience.

[0004] Therefore, there is an urgent need in related technologies for a way to resist malicious deception attacks and improve authentication accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide an identity verification method, apparatus, computer device, and computer-readable storage medium that can resist malicious deception attacks and improve authentication accuracy in response to the above-mentioned technical problems.

[0006] Firstly, this application provides an identity verification method. The method includes:

[0007] Acquire the signal to be identified, which includes millimeter-wave signals and audio signals;

[0008] Liveness feature detection is performed based on the millimeter-wave signal and audio signal to obtain live millimeter-wave signal and live audio signal;

[0009] Based on the feature fusion of the live millimeter wave signal and the live audio signal, a live speech signal fusion response map is obtained;

[0010] Identity recognition is performed based on the fusion response map of the live speech signal.

[0011] Optionally, in one embodiment of this application, the step of performing liveness detection based on the millimeter-wave signal and the audio signal to obtain the liveness millimeter-wave signal and the liveness audio signal includes:

[0012] Speech activity detection is performed on the millimeter-wave signal and audio signal to obtain millimeter-wave signal and audio signal with speech activity;

[0013] The millimeter-wave signal and audio signal containing voice activity are subjected to noise reduction processing to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal.

[0014] Optionally, in one embodiment of this application, the step of performing voice activity detection on the millimeter-wave speech signal and audio signal to obtain millimeter-wave speech signal and audio signal with voice activity includes:

[0015] The millimeter-wave signal and the audio signal are sampled to obtain a millimeter-wave sampled signal and an audio sampled signal;

[0016] Obtain the phase of the millimeter-wave sampling signal and determine the phase difference between millimeter-wave sampling signals of the same frequency;

[0017] Low-pass filtering is performed based on the phase difference and audio sampling signal to obtain millimeter-wave speech and audio signals with speech activity.

[0018] Optionally, in one embodiment of this application, the step of performing noise reduction processing on the millimeter-wave signal and audio signal containing voice activity to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal includes:

[0019] The millimeter-wave signal and the audio signal are decomposed to obtain millimeter-wave sub-signals and audio sub-signals;

[0020] The correlation between the millimeter-wave sub-signal and the audio sub-signal is calculated, and the millimeter-wave sub-signal and the audio sub-signal are then filtered based on the correlation.

[0021] The filtered millimeter-wave sub-signal and audio sub-signal are recombined to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal.

[0022] Optionally, in one embodiment of this application, the step of performing liveness detection based on the millimeter-wave signal and the audio signal to obtain the liveness millimeter-wave signal and the liveness audio signal includes:

[0023] Millimeter-wave liveness features and audio liveness features are extracted based on the millimeter-wave signal and audio signal, respectively;

[0024] A similarity coefficient is calculated based on the millimeter-wave liveness feature and the audio liveness feature, and a dual-mode reference signal is generated based on the similarity coefficient.

[0025] The dual-modal reference signal is input into the classification model to obtain the live millimeter-wave signal and the live audio signal, wherein the classification model is trained using a standard dataset.

[0026] Optionally, in one embodiment of this application, the step of fusing features based on the live millimeter-wave signal and the live audio signal to obtain a live speech signal fusion response map includes:

[0027] Millimeter-wave response map and audio response map are generated based on the live millimeter-wave signal and the live audio signal, respectively;

[0028] By fusing the millimeter-wave response map and the audio response map, a live speech signal fusion response map is obtained.

[0029] Optionally, in one embodiment of this application, the identity recognition based on the liveness speech signal fusion response map includes:

[0030] The fused response map of the live speech signal is input into the identity recognition network to obtain the user identity label, wherein the identity recognition network includes a channel attention module and a spatial attention module.

[0031] Secondly, this application also provides an identity verification device. The device includes:

[0032] A signal acquisition module is used to acquire a signal to be identified, which includes millimeter wave signals and audio signals.

[0033] The liveness detection module is used to perform liveness detection based on the millimeter-wave signal and the audio signal to obtain the liveness millimeter-wave signal and the liveness audio signal.

[0034] The feature fusion module is used to perform feature fusion based on the live millimeter-wave signal and the live audio signal to obtain a live speech signal fusion response map;

[0035] An identity recognition module is used to perform identity recognition based on the fusion response map of the live voice signal.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the steps of the methods described in the various embodiments above.

[0037] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the various embodiments above.

[0038] The aforementioned identity recognition method, apparatus, computer equipment, and storage medium first acquire a signal to be recognized, which includes millimeter-wave signals and audio signals. Then, based on the millimeter-wave signals and audio signals, liveness detection is performed to obtain live millimeter-wave signals and live audio signals. Next, feature fusion is performed based on the live millimeter-wave signals and live audio signals to obtain a live speech signal fusion response map. Finally, identity recognition is performed based on the live speech signal fusion response map. In other words, to prevent malicious spoofing attacks in voice identity recognition and authentication scenarios, by extracting voiceprint perception data from the audio modality and wirelessly sensing liveness data from the human skin surface, a response map measuring the quality of data information is constructed to achieve identity recognition, resisting malicious spoofing attacks, and simultaneously improving the accuracy of voiceprint recognition and authentication. Attached Figure Description

[0039] Figure 1 This is a diagram illustrating the application environment of an identity recognition method in one embodiment;

[0040] Figure 2 This is a flowchart illustrating an identity recognition method in one embodiment;

[0041] Figure 3 This is a flowchart illustrating the noise reduction process in one embodiment;

[0042] Figure 4 This is a schematic diagram of a live speech signal fusion response obtained in one embodiment;

[0043] Figure 5 This is a flowchart illustrating the specific implementation steps of the identity recognition method in one embodiment;

[0044] Figure 6 This is a structural block diagram of an identity recognition device in one embodiment;

[0045] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] The identity recognition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0048] In one embodiment, such as Figure 2 As shown, an identity recognition method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0049] S201: Acquire the signal to be identified, which includes millimeter wave signals and audio signals.

[0050] In this embodiment, firstly, the signal to be identified is acquired. This signal includes millimeter-wave signals and audio signals. The millimeter-wave signal is acquired by a millimeter-wave radar, and the audio signal is acquired by a microphone. Both the millimeter-wave radar and the microphone can receive the actual voice signal and other unrelated signals. Specifically, for the acquisition of the millimeter-wave signal, the millimeter-wave radar receives the echo signal, normalizes and scales the echo signal to obtain a frequency-modulated continuous wave signal, then processes it using a distance-fast Fourier transform (FFT) to obtain the distance information of the sound-emitting object. Then, a statistically ordered constant false alarm rate (CFAR) detection technique is used to detect the object, and the FFT points of the reflecting object are used to extract the phase, ultimately forming the millimeter-wave signal. After the millimeter-wave signal and audio signal are acquired, they are divided into 3-second frames with 50% overlap between consecutive frames, and then normalized and scaled to unify the value range of the millimeter-wave signal and audio signal.

[0051] S203: Perform liveness detection based on the millimeter wave signal and audio signal to obtain live millimeter wave signal and live audio signal.

[0052] In this embodiment, after obtaining the millimeter wave signal and the audio signal, liveness feature extraction is performed on the millimeter wave signal and the audio signal respectively. The liveness feature of the millimeter wave signal is related to the biological tissue characteristics of human skin, and the liveness feature of the audio signal is related to the acoustic organ characteristics. Based on the liveness feature, liveness feature detection is performed to identify whether the millimeter wave signal and the audio signal both come from real live signals, remove fake signals or data streams, and retain the live millimeter wave signal and the live audio signal.

[0053] S205: Based on the feature fusion of the live millimeter wave signal and the live audio signal, a live speech signal fusion response map is obtained.

[0054] After performing liveness detection on millimeter-wave and audio signals, it is necessary to extract human-specific features for identity recognition. In existing cross-modal recognition systems, fusion techniques merely merge and process the data from two modalities in parallel, often ignoring irrelevant and discordant information between heterogeneous modalities. Furthermore, they fail to fully utilize the correlation between cross-modal features to enhance fused features, and often suffer from interference noise overwhelming user identity information in the multimodal features, leading to reduced recognition accuracy. Therefore, in this embodiment, a correlation filter is used to process the live millimeter-wave and live audio signals to generate a response map that measures feature quality. The correlation filter tracks and learns the effective features of the target through frame streams. The learning target can be the spectral energy trajectory of the live audio signal or the live millimeter-wave signal. The region of interest is modeled, and a response map is generated to track temporal and spatial changes in the target. Then, fusion features are generated based on the response map to obtain an enhanced live speech signal fusion response map.

[0055] S207: Perform identity recognition based on the live voice signal fusion response map.

[0056] Traditional fusion mechanisms, such as voting mechanisms, select the better result from the outputs of multiple independent systems. While voting mechanisms seem convenient, compensating for information loss in one mode, they often fail to recognize users remotely in noisy environments when calling for identity recognition based on voting mechanisms. This is because long-distance propagation leads to a decrease in the audio signal-to-noise ratio, and ubiquitous multipath noise further masks useful information in millimeter-wave signals. In such cases, a simple voting mechanism cannot provide a practical voice identity recognition application. Therefore, this application employs an attention mechanism, widely developed and utilized in deep neural networks (DNNs) to improve the network's learning and representation capabilities. Specifically, an attention residual network is used to identify users by absorbing fusion features from the fusion response map of the live speech signal across modalities, ultimately outputting the user's identity.

[0057] In the aforementioned identity recognition method, firstly, a signal to be identified is acquired, which includes a millimeter-wave signal and an audio signal. Then, liveness detection is performed based on the millimeter-wave signal and the audio signal to obtain a live millimeter-wave signal and a live audio signal. Next, feature fusion is performed based on the live millimeter-wave signal and the live audio signal to obtain a live speech signal fusion response map. Finally, identity recognition is performed based on the live speech signal fusion response map. In other words, to prevent malicious spoofing attacks in voice identity recognition and authentication scenarios, by extracting voiceprint perception data from the audio modality and wirelessly sensing liveness data from the human skin surface, a response map measuring the quality of data information is constructed to achieve identity recognition, resisting malicious spoofing attacks, and simultaneously improving the accuracy of voiceprint recognition authentication.

[0058] In one embodiment of this application, the step of performing liveness feature detection based on the millimeter-wave signal and the audio signal to obtain the liveness millimeter-wave signal and the liveness audio signal includes:

[0059] S301: Perform speech activity detection on the millimeter wave signal and audio signal to obtain millimeter wave signal and audio signal with speech activity.

[0060] S303: Perform noise reduction processing on the millimeter wave signal and audio signal containing voice activity to obtain a noise-reduced millimeter wave signal and a noise-reduced audio signal.

[0061] In one embodiment of this application, the acquired millimeter-wave signal and audio signal may include those related to user voice activity, those unrelated to user voice activity, environmental noise, body movement, etc. Therefore, preliminary processing can be performed to remove irrelevant interference. First, voice activity detection is performed on the millimeter-wave signal and audio signal. Utilizing the difference between noise and voice signals in the frequency domain, coherent demodulation is used to obtain the millimeter-wave signal and audio signal containing voice activity. Then, the millimeter-wave signal and audio signal containing voice activity are subjected to noise reduction processing by utilizing the common characteristics of millimeter-wave mode and voice mode in perceiving voice activity, removing irrelevant interference to obtain the denoised millimeter-wave signal and denoised audio signal.

[0062] In this embodiment, by performing voice activity detection on the millimeter-wave signal and the audio signal, the millimeter-wave signal and the audio signal containing voice activity are obtained. The millimeter-wave signal and the audio signal containing voice activity are then subjected to noise reduction processing to obtain the noise-reduced millimeter-wave signal and the noise-reduced audio signal. This can ensure the high quality of the input signal source, select the sub-signal related to the voice activity and remove irrelevant noise components to improve noise immunity.

[0063] In one embodiment of this application, the step of performing voice activity detection on the millimeter-wave speech signal and audio signal to obtain millimeter-wave speech signal and audio signal with voice activity includes:

[0064] S401: Sample the millimeter wave signal and the audio signal to obtain the millimeter wave sampling signal and the audio sampling signal.

[0065] S403: Obtain the phase of the millimeter-wave sampling signal and determine the phase difference between millimeter-wave sampling signals of the same frequency.

[0066] S405: Based on the phase difference and the audio sampling signal, perform low-pass filtering to obtain millimeter-wave speech signal and audio signal with speech activity.

[0067] In one embodiment of this application, the millimeter-wave signal and the audio signal are first sampled separately. The preprocessed millimeter-wave signal is upsampled to 16kHz using linear interpolation, and the preprocessed audio signal is downsampled to 16kHz using signal decimation, resulting in a millimeter-wave sampled signal and an audio sampled signal v(n). Then, a fast Fourier transform is used to obtain the phase φ(n) of the millimeter-wave sampled signal. Based on the phase, the phase difference Δφ(n) between millimeter-wave sampled signals of the same frequency is calculated, as shown in the following formula.

[0068] Δφ(n)=φ(n)-φ(n-1)

[0069] Subsequently, due to the frequency correlation between the phase difference and the audio sampling signal, indicating a correlation with speech activity, the phase difference Δφ(n) and the audio sampling signal v(n) are multiplied and then low-pass filtered using a low-pass frequency filter. The cutoff frequency of the low-pass frequency filter is set to 300Hz, meaning only phase differences Δφ(n) and audio sampling signals v(n) below the cutoff frequency pass through. If the phase difference Δφ(n) and the audio sampling signal v(n) have the same or similar frequency components, a low-frequency energy peak is obtained. If the spectral entropy of this low-frequency energy peak is greater than a given threshold, it indicates that vocal cord vibration is simultaneously recorded by both the phase difference Δφ(n) and the audio sampling signal v(n), indicating that speech activity has occurred. If the spectral entropy of this low-frequency energy peak is less than the given threshold, it indicates that the received signal segment does not contain speech activity, and the signal is discarded without further processing. The given threshold is determined experimentally and can be selected, for example, 0.835. Even if noise corrupts the audio signal or the millimeter-wave signal, or worse, both, coherent demodulation remains effective due to the difference between noise and speech signals in the frequency domain.

[0070] In this embodiment, by sampling the millimeter-wave signal and the audio signal, millimeter-wave sampling signal and audio sampling signal are obtained, the phase of the millimeter-wave sampling signal is obtained, the phase difference between millimeter-wave sampling signals of the same frequency is determined, and low-pass filtering is performed based on the phase difference and the audio sampling signal to obtain millimeter-wave voice signal and audio signal related to the user's voice activity, ensuring high quality of the received signal.

[0071] In one embodiment of this application, the step of performing noise reduction processing on the millimeter-wave signal and audio signal containing voice activity to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal includes:

[0072] S501: Decompose the millimeter-wave signal and the audio signal to obtain the millimeter-wave sub-signal and the audio sub-signal.

[0073] S503: Calculate the correlation between the millimeter-wave sub-signal and the audio sub-signal, and filter the millimeter-wave sub-signal and the audio sub-signal based on the correlation.

[0074] S505: Based on the screened millimeter-wave sub-signal and audio sub-signal, recombine to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal.

[0075] In one embodiment of this application, such as Figure 3As shown, firstly, the millimeter-wave signal and audio signal are decomposed using Fast Independent Component Analysis (FastICA) and Dual-Tree Complex Wavelet Transform (DTCWT) respectively, resulting in a series of millimeter-wave sub-signals and audio sub-signals. Then, the correlation between the millimeter-wave sub-signals and audio sub-signals is calculated. Optionally, the Pearson correlation coefficient is used to assess the correlation; the Pearson correlation coefficient is calculated by dividing the covariance of two variables by the product of their standard deviations. A correlation matrix is ​​constructed based on the Pearson correlation coefficient. This correlation matrix contains cross-modal information, where a higher coefficient indicates a stronger correlation between the corresponding sub-signals. A strong correlation means that the millimeter-wave sub-signals contain speech vibrations, while the audio sub-signals record speech activity and are not irrelevant noise. Next, the mean matrix and variance matrix are determined based on the correlation matrix, and the ratio of the mean matrix to the variance matrix (mean matrix / variance matrix) is calculated. Millimeter-wave sub-signals and audio sub-signals with a ratio greater than a preset ratio are selected for retention. A higher ratio indicates higher average energy. Based on extensive experimental analysis, a preset ratio of 5 is typically set. The mean matrix is ​​obtained by calculating the average value of each column of the correlation matrix, and the variance matrix is ​​obtained by calculating the variance of each row. Then, Principal Component Analysis (PCA) is used to reconstruct the selected millimeter-wave sub-signals and audio sub-signals, generating a reconstructed signal, resulting in the denoised millimeter-wave signal and the denoised audio signal.

[0076] In this embodiment, millimeter-wave signals and audio signals are decomposed to obtain millimeter-wave sub-signals and audio sub-signals. Correlation is calculated based on the millimeter-wave sub-signals and audio sub-signals. The millimeter-wave sub-signals and audio sub-signals are then filtered based on the correlation. The filtered millimeter-wave sub-signals and audio sub-signals are then recombined to obtain denoised millimeter-wave signals and denoised audio signals. This process can remove irrelevant interference, such as environmental noise and body movement, and ensure high quality of the received signal.

[0077] In one embodiment of this application, the step of performing liveness detection based on the millimeter-wave signal and the audio signal to obtain the liveness millimeter-wave signal and the liveness audio signal includes:

[0078] S601: Extract millimeter-wave liveness features and audio liveness features based on the millimeter-wave signal and audio signal, respectively.

[0079] S603: Calculate similarity coefficients based on the millimeter-wave liveness features and audio liveness features, and aggregate the similarity coefficients to generate a dual-mode reference signal.

[0080] S605: Input the dual-modal reference signal into the classification model to obtain the live millimeter-wave signal and the live audio signal, wherein the classification model is trained using a standard dataset.

[0081] In one embodiment of this application, firstly, a short-time Fourier transform is performed on the audio signal to obtain the audio spectrum. The transformed audio spectrum is then segmented into low-frequency and high-frequency components within the audible range. Constant cepstral coefficients derived from acoustic organ characteristics are extracted from the low-frequency and high-frequency components. These coefficients represent the bioactive characteristics of speech. For millimeter-wave signals, a squarer is used to process the reflected mid-frequency millimeter waves to obtain the DC component, i.e., the low-frequency signal. This low-frequency signal represents the bioactive characteristics of millimeter waves. When the millimeter-wave signal reaches the human body, due to the limited submillimeter penetration depth, its electromagnetic radiation is mainly confined to the human skin. The amplitude of the millimeter-wave signal undergoes heterogeneous attenuation due to the difference in the heterogeneous dielectric constant determined by the biological tissue of the human skin. This attenuation is determined by the human body's reflection coefficient, preserving the bioactive characteristics and distinguishing it from artificially created malicious signals.

[0082] Then, similarity coefficients were calculated based on millimeter-wave liveness features and audio liveness features, respectively, using the following formula.

[0083]

[0084] Among them, S M S is the similarity coefficient for millimeter-wave living features. V Let M be the similarity coefficient of the audio liveness feature, and M be the millimeter-wave liveness feature. T V represents the standard millimeter-wave liveness characteristic, and V represents the audio liveness characteristic. T This is a standard audio liveness feature.

[0085] Subsequently, a dual-mode reference signal is generated based on similarity coefficient aggregation, which is mainly based on the following formula.

[0086]

[0087] Finally, the obtained bimodal reference signal is input into the classification model to obtain the live millimeter wave signal and live audio signal from a real living organism. Optionally, the classification model is a class of support vector machines trained in advance using a standard dataset.

[0088] In this embodiment, millimeter-wave liveness features and audio liveness features are extracted based on millimeter-wave signals and audio signals, respectively. Similarity coefficients are calculated based on the millimeter-wave liveness features and audio liveness features. Based on the similarity coefficients, a dual-modal reference signal is generated. The dual-modal reference signal is input into the classification model to obtain live millimeter-wave signals and live audio signals. This can identify and distinguish signals from genuine users and malicious deception signals, resist various malicious attacks, and increase the reliability and security of voiceprint recognition.

[0089] In one embodiment of this application, the step of fusing features based on the live millimeter-wave signal and the live audio signal to obtain a live speech signal fusion response map includes:

[0090] S701: Generate millimeter-wave response map and audio response map based on the live millimeter-wave signal and the live audio signal, respectively.

[0091] S703: The millimeter-wave response map and the audio response map are fused to obtain the live speech signal fusion response map.

[0092] In one embodiment of this application, such as Figure 4 As shown, firstly, based on the live millimeter-wave signal and the live audio signal, a discriminative correlation filter is used to generate millimeter-wave response maps and audio response maps, respectively. Specifically, the correlation filter tracks and learns the effective features of the target through frame stream tracking. The target can be the spectral energy trajectory of speech or wireless signals. It models the region of interest and generates response maps to track temporal and spatial changes in the target. During the tracking of spectral trajectory changes, the correlation filter gradually generates response maps. The generated response maps adaptively adjust weights to enhance speaker-specific characteristics in low and high frequencies related to pronunciation habits and vocal organs. Redundancy and environmental noise related to speech content are assigned low weights to suppress irrelevant information. Then, the millimeter-wave response map and the audio response map are fused to obtain a fused response map of the live speech signal. Specifically, the millimeter-wave response map and the audio response map are convolved in segments to generate fused features. The convolutional fusion operation enhances the relevant information between the millimeter-wave and audio signals. The final fused response map of the live speech signal still retains features derived separately from the millimeter-wave and audio signals.

[0093] In this embodiment, millimeter-wave response maps and audio response maps are generated based on live millimeter-wave signals and live audio signals, respectively. The millimeter-wave response maps and audio response maps are then fused to obtain a live speech signal fusion response map. This can utilize the correlation between the two signals to enhance the relevant information between millimeter waves and audio, and mitigate the negative impact of cross-modal unrelated interference.

[0094] In one embodiment of this application, the identity recognition based on the liveness speech signal fusion response map includes:

[0095] The fused response map of the live speech signal is input into the identity recognition network to obtain the user identity label, wherein the identity recognition network includes a channel attention module and a spatial attention module.

[0096] In one embodiment of this application, a live speech signal fusion response map is input into an identity recognition network, which is an attention residual network. The residual neural network serves as the backbone model, and two attention-based modules are introduced: a channel attention module and a spatial attention module. In each residual module of the residual neural network, an efficient attention layer (ECA) is introduced. ECA is an attention-based block composed of convolutional layers, designed to simulate the interdependencies between convolutional feature channels. ECA applies global average pooling (GAP) to learn contextual information across all receptive domains of the network, rather than learning a limited local domain as in traditional convolutional layers. Based on information from all channels, ECA enables the network to focus on capturing more important regions through adaptive channel attention. Assume the output of a convolutional layer is X = [x1, x2, ..., x...]. c ],X∈R H×W×C Where H, W, and C are the width, height, and channel dimensions, and x is the channel size. c This refers to the channel features generated by the c-th filter in the convolutional layer. Then, GAP is applied to establish the channel feature modulus Z = [z1, z2, ..., z]. c ], Z∈R 1×1×C The c-th element in Z is calculated using the following formula.

[0097]

[0098] The channel feature Z contains statistical information for all channels, including channel-specific statistics. Then, we calculate the attention feature.

[0099] A=σ(C1D k (Z))

[0100] Where A = [a1, a2, ..., a c ], A∈R 1×1×C σ is an activation function, C1D k This represents a one-dimensional convolution with a kernel size of k.

[0101] The final output of the ECA layer It is obtained by the channel dot product between X and A.

[0102]

[0103] Here, ⊙ represents scalar multiplication. The attention feature A contains dynamic channel information and is continuously optimized during the iteration process.

[0104] By concatenating a typical residual block with an ECA layer as a basic module of the backbone network, it can be described as follows:

[0105]

[0106] Among them, the function This represents multiple convolutional layers used to capture features, Y represents the final output of the residual block, and ECA(-) represents the ECA layer. Operations This represents a shortcut connection. The outputs of multiple consecutive convolutional layers flow to the ECA layer. After the result is calculated through attention, a shortcut connection adds the input of the residual block and the result of the ECA layer to obtain the final output.

[0107] Besides ECA, the feature extraction capability of attention residual networks is also determined by the channel attention module and the spatial attention module. The properties of attention modules have been widely validated, helping neural networks understand the "content" and "location" of feature maps in both channels and space. Specifically, the channel attention module and the spatial attention module guide the residual network to select and reinforce meaningful knowledge in the mixed feature maps. In the channel module of the attention residual network, given an input feature map F∈R... H×W×C Its height is H, its width is W, and it has C channels. A channel module aggregates multiple channel modules using average pooling. Avg and maximum pooling Max The operation involves collecting the channel spatial information of the feature map, as shown in the following function.

[0108] F c =δ(Conv(Pool) Avg (F))+Conv(Pool Max (F)))

[0109] Where δ(-) represents the ReLU function, Conv is a one-dimensional convolution with a 1×1 kernel, and F c This is a diagram of the channel.

[0110] Unlike the channel module, the spatial module generates a spatial attention graph F. s The location of encoding emphasis or suppression is shown in the following formula.

[0111] F s =δ(Conv 3×3 ([Pool Avg (F); Pool Max (F)]))

[0112] The average and maximum ensemble features of each channel are concatenated and convolved through a convolutional layer to generate a two-dimensional spatial attention map. The channel attention module guides the residual neural network to select and enhance meaningful knowledge in the mixed feature map. The spatial module generates a spatial attention mapping map, which encodes the locations of beneficial or harmful features, helping the network memorize and learn effective feature locations and avoid harmful features. In this system, the channel module and the spatial module are placed sequentially after each residual block to improve the network's identity recognition performance. In this embodiment, the channel module and the spatial module are placed sequentially after each residual block, and the identity recognition network consists of four residual blocks embedded with one channel module and one spatial module. The last layer of the network is an average pooling layer with a kernel size of 7 and 512 channels and a linear layer with two output units. The fused response map of the previously processed live speech signal is input into this network, and the output is a label representing the user's identity.

[0113] In this embodiment, by inputting the live voice signal fusion response map into the identity recognition network, a user identity label is obtained, which can accurately output the user identity and ensure efficient and accurate identification of the user identity in any scenario.

[0114] The following specific implementation steps of the identity recognition method are illustrated with a concrete example, such as... Figure 5 As shown, firstly, in step S801, a signal to be identified is acquired, which includes a millimeter-wave signal and an audio signal. Then, in step S803, voice activity detection is performed on the millimeter-wave signal and the audio signal to obtain millimeter-wave signals and audio signals with voice activity. Specifically, in steps S805-S809, the millimeter-wave signal and the audio signal are sampled to obtain millimeter-wave sampled signals and audio sampled signals. The phase of the millimeter-wave sampled signals is obtained, the phase difference between millimeter-wave sampled signals of the same frequency is determined, and low-pass filtering is performed based on the phase difference and the audio sampled signals to obtain millimeter-wave voice signals and audio signals with voice activity. Next, in step S811, the millimeter-wave signal and audio signal containing speech activity are subjected to noise reduction processing to obtain a noise-reduced millimeter-wave signal and a noise-reduced frequency signal. Specifically, in steps S813-S817, the millimeter-wave signal and audio signal are decomposed to obtain millimeter-wave sub-signals and audio sub-signals. Correlation is calculated based on the millimeter-wave sub-signals and audio sub-signals. The millimeter-wave sub-signals and audio sub-signals are filtered based on the correlation. The filtered millimeter-wave sub-signals and audio sub-signals are then recombined to obtain the noise-reduced millimeter-wave signal and the noise-reduced frequency signal.

[0115] Then, in step S819, liveness detection is performed based on the millimeter-wave signal and the audio signal to obtain a live millimeter-wave signal and a live audio signal. Specifically, in steps S821-S825, millimeter-wave liveness features and audio liveness features are extracted based on the millimeter-wave signal and the audio signal, respectively. A similarity coefficient is calculated based on the millimeter-wave liveness features and the audio liveness features. A bimodal reference signal is generated based on the similarity coefficient. The bimodal reference signal is input into a classification model to obtain a live millimeter-wave signal and a live audio signal. The classification model is trained using a standard dataset.

[0116] Then, in step S827, feature fusion is performed based on the live millimeter wave signal and the live audio signal to obtain a live speech signal fusion response map. Specifically, in steps S829-S831, a millimeter wave response map and an audio response map are generated based on the live millimeter wave signal and the live audio signal, respectively. The millimeter wave response map and the audio response map are then fused to obtain a live speech signal fusion response map.

[0117] Finally, in S833, identity recognition is performed based on the live speech signal fusion response map. Specifically, in S835, the live speech signal fusion response map is input into the identity recognition network to obtain the user identity label. The identity recognition network includes a channel attention module and a spatial attention module.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides an identity recognition device for implementing the aforementioned identity recognition method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more identity recognition device embodiments provided below can be found in the limitations of the identity recognition method described above, and will not be repeated here.

[0120] In one embodiment, such as Figure 6As shown, an identity recognition device 600 is provided, including: a signal acquisition module 601, a liveness detection module 603, a feature fusion module 605, and an identity recognition module 607, wherein:

[0121] The signal acquisition module 601 is used to acquire the signal to be identified, which includes millimeter wave signals and audio signals.

[0122] The liveness detection module 603 is used to perform liveness detection based on the millimeter wave signal and the audio signal to obtain the live millimeter wave signal and the live audio signal.

[0123] The feature fusion module 605 is used to perform feature fusion based on the live millimeter wave signal and the live audio signal to obtain a live speech signal fusion response map.

[0124] The identity recognition module 607 is used to perform identity recognition based on the live voice signal fusion response map.

[0125] The identity recognition device also includes a voice activity detection and noise reduction module.

[0126] In one embodiment of this application, the voice activity detection and noise reduction module is used for:

[0127] Speech activity detection is performed on the millimeter-wave signal and audio signal to obtain millimeter-wave signal and audio signal with speech activity;

[0128] The millimeter-wave signal and audio signal containing voice activity are subjected to noise reduction processing to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal.

[0129] In one embodiment of this application, the voice activity detection and noise reduction module is further configured to:

[0130] The millimeter-wave signal and the audio signal are sampled to obtain a millimeter-wave sampled signal and an audio sampled signal;

[0131] Obtain the phase of the millimeter-wave sampling signal and determine the phase difference between millimeter-wave sampling signals of the same frequency;

[0132] Low-pass filtering is performed based on the phase difference and audio sampling signal to obtain millimeter-wave speech and audio signals with speech activity.

[0133] In one embodiment of this application, the voice activity detection and noise reduction module is further configured to:

[0134] The millimeter-wave signal and the audio signal are decomposed to obtain millimeter-wave sub-signals and audio sub-signals;

[0135] The correlation between the millimeter-wave sub-signal and the audio sub-signal is calculated, and the millimeter-wave sub-signal and the audio sub-signal are then filtered based on the correlation.

[0136] The filtered millimeter-wave sub-signal and audio sub-signal are recombined to obtain a noise-reduced millimeter-wave signal and a noise-reduced audio signal.

[0137] In one embodiment of this application, the liveness detection module is further configured to:

[0138] Millimeter-wave liveness features and audio liveness features are extracted based on the millimeter-wave signal and audio signal, respectively;

[0139] A similarity coefficient is calculated based on the millimeter-wave liveness feature and the audio liveness feature, and a dual-mode reference signal is generated based on the similarity coefficient.

[0140] The dual-modal reference signal is input into the classification model to obtain the live millimeter-wave signal and the live audio signal, wherein the classification model is trained using a standard dataset.

[0141] In one embodiment of this application, the feature fusion module is further configured to:

[0142] Millimeter-wave response map and audio response map are generated based on the live millimeter-wave signal and the live audio signal, respectively;

[0143] By fusing the millimeter-wave response map and the audio response map, a live speech signal fusion response map is obtained.

[0144] In one embodiment of this application, the identity recognition module is further configured to:

[0145] The fused response map of the live speech signal is input into the identity recognition network to obtain the user identity label, wherein the identity recognition network includes a channel attention module and a spatial attention module.

[0146] Each module in the aforementioned identity recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0147] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an identification method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0148] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0150] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0151] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0153] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An identity recognition method, characterized in that, The method includes: Acquire the signal to be identified, which includes millimeter-wave signals and audio signals; Speech activity detection is performed on the millimeter-wave signal and audio signal to obtain millimeter-wave signal and audio signal with speech activity; noise reduction processing is performed on the millimeter-wave signal and audio signal with speech activity to obtain noise-reduced millimeter-wave signal and noise-reduced audio signal. Liveness detection is performed based on the denoised millimeter-wave signal and the denoised audio signal to obtain the live millimeter-wave signal and the live audio signal. Based on the feature fusion of the live millimeter wave signal and the live audio signal, a live speech signal fusion response map is obtained; The fused response map of the live voice signal is input into the identity recognition network for identity recognition; The step of denoising the millimeter-wave signal and audio signal containing speech activity to obtain denoised millimeter-wave signal and denoised audio signal includes: decomposing the millimeter-wave signal and audio signal to obtain millimeter-wave sub-signals and audio sub-signals; calculating the correlation based on the millimeter-wave sub-signals and audio sub-signals; filtering the millimeter-wave sub-signals and audio sub-signals based on the correlation; and recombining the filtered millimeter-wave sub-signals and audio sub-signals to obtain denoised millimeter-wave signal and denoised audio signal.

2. The method according to claim 1, characterized in that, The step of performing speech activity detection on the millimeter-wave signal and audio signal to obtain millimeter-wave signal and audio signal with speech activity includes: The millimeter-wave signal and the audio signal are sampled to obtain a millimeter-wave sampled signal and an audio sampled signal; Obtain the phase of the millimeter-wave sampling signal and determine the phase difference between millimeter-wave sampling signals of the same frequency; Low-pass filtering is performed based on the phase difference and audio sampling signal to obtain millimeter-wave signal and audio signal with speech activity.

3. The method according to claim 1, characterized in that, The liveness detection based on the denoised millimeter-wave signal and the denoised audio signal, to obtain the liveness millimeter-wave signal and the liveness audio signal, includes: Based on the denoised millimeter-wave signal and the denoised audio signal, millimeter-wave liveness features and audio liveness features are extracted respectively; A similarity coefficient is calculated based on the millimeter-wave liveness feature and the audio liveness feature, and a dual-mode reference signal is generated based on the similarity coefficient. The dual-modal reference signal is input into the classification model to obtain the live millimeter-wave signal and the live audio signal, wherein the classification model is trained using a standard dataset; The extraction of millimeter-wave liveness features and audio liveness features based on the denoised millimeter-wave signal and the denoised audio signal respectively includes: Short-time Fourier transform is performed on the denoised audio signal to obtain the audio spectrum. The converted audio spectrum is segmented into low-frequency and high-frequency components within the audible range. Constant cepstral coefficients derived from acoustic organ characteristics are extracted from the low-frequency and high-frequency components to determine the liveness characteristics of the audio signal. For the noise-reduced millimeter-wave signal, a squarer is used to process the reflected intermediate-frequency millimeter-wave to extract the DC component and determine the millimeter-wave liveness characteristics. The similarity coefficient is calculated based on the millimeter-wave liveness features and audio liveness features, including: ； Among them, S M S is the similarity coefficient for millimeter-wave living features. V M is the similarity coefficient of the audio liveness feature, and M is the millimeter-wave liveness feature. T V represents the standard millimeter-wave liveness characteristic, and V represents the audio liveness characteristic. T Standard audio liveness characteristics; The generation of a dual-modal reference signal based on the similarity coefficient aggregation includes: ； Where C is the dual-mode reference signal, S M S is the similarity coefficient for millimeter-wave living features. V The similarity coefficient is the audio liveness feature.

4. The method according to claim 1, characterized in that, The step of fusing features based on the live millimeter-wave signal and the live audio signal to obtain the live speech signal fusion response map includes: Millimeter-wave response map and audio response map are generated based on the live millimeter-wave signal and the live audio signal, respectively; By fusing the millimeter-wave response map and the audio response map, a live speech signal fusion response map is obtained; The fusion of the millimeter-wave response map and the audio response map yields a live speech signal fusion response map, which includes: segmented convolutional millimeter-wave response map and audio response map, to determine the live speech signal fusion response map.

5. The method according to claim 1, characterized in that, The step of inputting the fused response map of the live speech signal into the identity recognition network for identity recognition includes: The fused response map of the live speech signal is input into the identity recognition network to obtain the user identity label, wherein the identity recognition network includes a channel attention module and a spatial attention module.

6. An identity recognition device, characterized in that, The device includes: A signal acquisition module is used to acquire a signal to be identified, which includes millimeter wave signals and audio signals. The feature fusion module is used to perform feature fusion based on live millimeter-wave signals and live audio signals to obtain a live speech signal fusion response map; The identity recognition module is used to input the fused response map of the live voice signal into the identity recognition network for identity recognition; The liveness detection module is also used to detect speech activity in the millimeter wave signal and audio signal to obtain millimeter wave signal and audio signal with speech activity; and to perform noise reduction processing on the millimeter wave signal and audio signal with speech activity to obtain noise-reduced millimeter wave signal and noise-reduced audio signal. The liveness detection module is also used to decompose the millimeter-wave signal and the audio signal to obtain millimeter-wave sub-signals and audio sub-signals; calculate the correlation based on the millimeter-wave sub-signals and the audio sub-signals; filter the millimeter-wave sub-signals and the audio sub-signals based on the correlation; and reconstruct the filtered millimeter-wave sub-signals and the audio sub-signals to obtain the denoised millimeter-wave signal and the denoised audio signal. The liveness detection module is used to perform liveness detection based on the denoised millimeter-wave signal and the denoised audio signal to obtain the live millimeter-wave signal and the live audio signal.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images