VR user identity authentication method based on multi-modal perception of vocalization vibration signals
By synchronously acquiring and processing accelerometer and microphone signals, and utilizing modality conversion networks and recurrent generative adversarial networks for feature extraction, the problem of insufficient security in VR user authentication is solved, achieving efficient and reliable authentication results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2025-09-29
- Publication Date
- 2026-06-26
AI Technical Summary
Existing VR user authentication methods are difficult to effectively utilize biological or behavioral characteristics collected by multiple sensors for real-time identity verification in headset scenarios, and are vulnerable to forgery attacks, resulting in insufficient security.
By acquiring the output of the accelerometer sensor and microphone, craniofacial vibration and speech signals are collected simultaneously. Modality conversion network and recurrent generative adversarial network are used for feature extraction and identity authentication. By combining resampling, time axis alignment and adversarial training, an implicit combination of acceleration and speech modality is achieved, which improves the authentication effect and security.
It enables multimodal perception based on sound vibration signals in VR devices, improving the accuracy and security of identity authentication, effectively defending against forgery attacks, and enhancing the robustness and reliability of user identity authentication.
Smart Images
Figure CN121260163B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of user identification, and in particular to a VR user identification method based on multimodal perception of acoustic vibration signals. Background Technology
[0002] Accelerometer Sensor: An accelerometer is an inertial sensor that detects changes in the acceleration of an object. It is widely integrated into mobile devices and virtual reality headsets as a core motion capture component. Its main function is to sense the linear acceleration changes of the device in three-dimensional space in real time, assisting in functions such as head posture estimation, dynamic interaction recognition, and spatial positioning. In addition to tracking the macroscopic motion of the device (such as rotation and movement), accelerometers in head-mounted VR devices can also collect minute craniofacial vibration signals generated when the user speaks, chews, or engages in facial muscle movements.
[0003] VR / AR User Authentication: VR / AR user authentication refers to a technical system that uses various sensors (cameras, IMU, microphones, infrared optics, controllers, etc.) built into the head-mounted display and its peripherals to collect biological or behavioral characteristics and verify the user's identity in real time when the head-mounted display is worn.
[0004] CycleGAN: An extended architecture of Generative Adversarial Networks (GANs), its core consists of two sets of "adversarial" models: two generators (G{A→B} and G{B→A}) learn the data distribution mapping from domain A to domain B and from domain B to domain A, respectively; two discriminators (D_A and D_B) estimate the probability that a sample belongs to the true data of the corresponding domain; at the same time, by introducing a cycle consistency loss, adversarial training learns the mapping relationship between data in different domains. Summary of the Invention
[0005] The purpose of this invention is to provide a VR user authentication method based on multimodal perception of acoustic vibration signals.
[0006] The objective of this invention can be achieved through the following technical solutions:
[0007] A VR user authentication method based on multimodal perception of acoustic vibration signals includes:
[0008] Step S1: Obtain the output of the accelerometer sensor and the microphone, and obtain the original acceleration signal and the original voice signal after alignment based on the time axis. The microphone and the accelerometer sensor are both located on the head-mounted VR device, and the accelerometer sensor collects craniofacial vibrations.
[0009] Step S2: Perform peak detection based on the original speech signal to identify the vocal segments, and extract the vocal segments as the first speech signal;
[0010] Step S3: Based on the start and end times of the speech segment, extract the first acceleration signal corresponding to each first speech signal from the original acceleration signal;
[0011] Step S4: Generate the acceleration amplitude spectrum of the first acceleration signal as the first acceleration spectrum;
[0012] Step S5: The first acceleration spectrum is converted into a speech Mel spectrum as the second Mel spectrum by the first generator of the trained modality conversion network. Then, the second Mel spectrum is converted into an acceleration amplitude spectrum as the second acceleration spectrum by the second generator of the trained modality conversion network. The modality conversion network learns the unsupervised mapping relationship between the acceleration signal and the speech spectrum to realize the feature conversion from acceleration to speech mode and from speech to acceleration mode.
[0013] Step S6: Extract the first feature by performing cycle consistency feature extraction on the first acceleration spectrum and the second acceleration spectrum;
[0014] Step S7: Input the first feature into the trained first detection model to obtain the detection result.
[0015] Step S1 includes:
[0016] Step S1-1: Acquire the outputs of the accelerometer sensor and microphone, and perform frequency filtering on each;
[0017] Step S1-2: Based on the filtered data, resampling and time axis alignment are performed to obtain the original acceleration signal and the original speech signal, wherein the original acceleration signal and the original speech signal have the same sampling rate.
[0018] Step S6 includes:
[0019] Step S6-1: Calculate the logarithmic spectral distance, mean square error, and STFT spectral difference between the first acceleration spectrum and the second acceleration spectrum, respectively;
[0020] Step S6-2: The first feature is obtained by fusing the calculated logarithmic spectral distance, mean square error and STFT spectral difference.
[0021] The first detection model is a binary classification model.
[0022] The modality transfer network further includes a first discriminator and a second discriminator, and the modality transfer network is trained based on the first sample.
[0023] The construction process of the first sample includes:
[0024] The outputs of the accelerometer sensor and microphone under interference-free conditions are obtained, and the raw acceleration signal and raw speech signal are obtained after time axis alignment.
[0025] The vocal segment is identified and extracted as the first speech signal;
[0026] Based on the start and end times of the speech segment, the first acceleration signal corresponding to each first speech signal is extracted from the original acceleration signal;
[0027] The acceleration amplitude spectrum of the first acceleration signal is used as the first acceleration spectrum, and the speech Mel spectrum of the first speech signal is used as the first Mel spectrum.
[0028] The corresponding first acceleration spectrum and first Mel spectrum are used as the first sample for acceleration and speech modality conversion.
[0029] The first discriminator is based on a convolutional neural network architecture and is trained adversarially against the first generator based on the first Mel spectrogram and the second Mel spectrogram.
[0030] The second discriminator employs a convolutional neural network architecture and performs adversarial training against the second generator based on the first and second acceleration spectra.
[0031] The loss function of the mode conversion network is:
[0032]
[0033] in: Let X be the loss function, and Y be the speech mode. For adversarial loss from acceleration mode to speech mode, For adversarial loss from speech modality to acceleration modality, For the cycle consistency loss of acceleration modes, For the cycle consistency loss of the speech modality, λ cyc For the weights of the cycle consistency loss, λ rec To reconstruct the loss weights, The reconstruction loss is for the acceleration mode to speech mode. λ is the reconstruction loss from speech mode to acceleration mode. idt To maintain weight in one's identity, To preserve the identity of acceleration modes, Loss due to loss in voice modality identity preservation.
[0034] A VR user authentication device based on multimodal perception of acoustic vibration signals includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the method described above.
[0035] A storage medium having a program stored thereon, which, when executed, implements the method described above.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] 1. By using two generators and two discriminators obtained through adversarial training, information from acceleration and speech modality is implicitly combined. During subsequent authentication, only the acceleration signal is needed to achieve identity authentication, thereby improving both authentication effectiveness and security.
[0038] 2. Resampling and time axis alignment yield the original acceleration signal and the original speech signal, thereby achieving synchronization of the two signals.
[0039] 3. By using a specially designed model architecture and loss function, the effect of conversion between acceleration and speech modalities is improved. The unique cyclic consistency features of individuals are extracted, thereby improving the accuracy of identity authentication in real-world scenarios and enhancing the security of identity authentication. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the technical route and a system flowchart of the present invention;
[0041] Figure 2 The resulting acceleration spectrum and speech Mel spectrum;
[0042] Figure 3 This is a schematic diagram of the structure of the first generator;
[0043] Figure 4 This is a schematic diagram of the second generator.
[0044] Figure 5 A schematic diagram of the result of a cyclically generated adversarial network (acceleration);
[0045] Figure 6 A schematic diagram of the results of a recurrent generative adversarial network (audio).
[0046] Figure 7 A schematic diagram illustrating the process of converting acceleration into speech and then into simulated acceleration.
[0047] Figure 8 This represents a multi-user clustering representation.
[0048] Figure 9 This is a schematic diagram of the main steps of the method of the present invention. Detailed Implementation
[0049] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0050] This application utilizes bone conduction vibration signals from the face and skull generated by a user's natural vocalization while wearing a virtual reality (VR) device. The theoretical basis is that during human vocalization, vocal cord vibration not only transmits sound waves via airflow, which are captured by a microphone, but also induces weak but detectable mechanical vibrations in structures such as the skull, pharynx, and other parts of the head. This vibration is transmitted through facial soft tissue and bones to the contact area of the VR headset, where it can be sensed by an inertial measurement unit (IMU) such as an accelerometer embedded in the device, allowing information related to human vocalization to be extracted.
[0051] A VR user authentication method based on multimodal perception of acoustic vibration signals, such as Figure 1 and Figure 9 As shown, it includes:
[0052] Step S1: Acquire the outputs of the accelerometer sensor and microphone, and obtain the raw acceleration signal and raw speech signal after time-axis alignment. Both the microphone and accelerometer sensor are mounted on the head-mounted VR device, and the accelerometer sensor collects craniofacial vibrations.
[0053] In this invention, audio signals and facial acceleration signals are acquired simultaneously: audio signals are implicitly acquired by a built-in microphone, while acceleration signals are passively received by an accelerometer embedded in the VR device, in order to capture the minute vibrations of muscles and bones during vocalization.
[0054] This step includes:
[0055] Step S1-1: Acquire the outputs of the accelerometer sensor and microphone, and perform frequency filtering on each;
[0056] Considering the differences between the two types of signals in terms of sampling rate, signal-to-noise ratio, and physical source, the system performed bandpass filtering on the acceleration signal. Studies have shown that the influence of human head movement is mainly below 20 Hz, and the influence of human vocalization is mainly below 500 Hz. Therefore, acceleration data within the range of 20 Hz to 500 Hz were retained.
[0057] Step S1-2: Based on the filtered data, resampling and time axis alignment are performed to obtain the original acceleration signal and the original speech signal, wherein the original acceleration signal and the original speech signal have the same sampling rate.
[0058] The sampling rate of the acceleration signal was 1000, while the sampling rate of the speech signal was 48000. Therefore, in order to ensure the consistency of subsequent experimental conversion, the two were interpolated and resampled to 10448Hz. At the same time, the speech signal was converted to mono, so that the two remained aligned in the time and frequency domains.
[0059] Step S2: Perform peak detection based on the original speech signal to identify the vocal segments, and extract the vocal segments as the first speech signal;
[0060] In order to remove as much non-speech-related information as possible, the peak of the speech signal is found to confirm the vocal segment, and then the ACC segment is extracted.
[0061] Step S3: Based on the start and end times of the speech segment, extract the first acceleration signal corresponding to each first speech signal from the original acceleration signal;
[0062] Step S4: Generate the acceleration amplitude spectrum of the first acceleration signal as the first acceleration spectrum;
[0063] Step S5: The first acceleration spectrum is converted into a speech Mel spectrum as the second Mel spectrum by the first generator of the trained modality conversion network. Then, the second Mel spectrum is converted into an acceleration amplitude spectrum as the second acceleration spectrum by the second generator of the trained modality conversion network. The modality conversion network learns the unsupervised mapping relationship between the acceleration signal and the speech spectrum to realize the feature conversion from acceleration to speech mode and from speech to acceleration mode.
[0064] Step S6: Extract the first feature by performing cycle consistency feature extraction on the first acceleration spectrum and the second acceleration spectrum, including:
[0065] Step S6-1: Calculate the logarithmic spectral distance, mean square error, and STFT spectral difference between the first acceleration spectrum and the second acceleration spectrum, respectively;
[0066] Step S6-2: The first feature is obtained by fusing the calculated logarithmic spectral distance, mean square error and STFT spectral difference.
[0067] Step S7: Input the first feature into the trained first detection model to obtain the detection result, wherein the first detection model is a binary classification model.
[0068] In this embodiment, the modality switching network further includes a first discriminator and a second discriminator, and the modality switching network is trained based on the first sample.
[0069] The process of constructing the first sample includes:
[0070] The outputs of the accelerometer sensor and microphone under interference-free conditions are obtained, and the raw acceleration signal and raw speech signal are obtained after time axis alignment.
[0071] The vocal segment is identified and extracted as the first speech signal;
[0072] Based on the start and end times of the speech segment, the first acceleration signal corresponding to each first speech signal is extracted from the original acceleration signal;
[0073] The acceleration amplitude spectrum of the first acceleration signal is used as the first acceleration spectrum, and the speech Mel spectrum of the first speech signal is used as the first Mel spectrum.
[0074] The corresponding first acceleration spectrum and first Mel spectrum are used as the first sample for acceleration and speech modality conversion.
[0075] During the training phase, the speech and acceleration signals were then subjected to spectral conversion to obtain the acceleration amplitude spectrum, squared amplitude spectrum, speech spectrum, and Mel spectrum corresponding to a specific data point. Figure 2 .
[0076] Among them, such as Figure 3 As shown, the first discriminator is based on a convolutional neural network architecture and is trained adversarially against the first generator based on the first and second Mel spectrograms; similarly, as... Figure 4 As shown, the second discriminator adopts a convolutional neural network architecture and performs adversarial training with the second generator based on the first and second acceleration spectra.
[0077] Two generators (GeneratorAccToAudio and GeneratorAudioToAcc) are defined to implement feature conversion from acceleration to speech modality and from speech to acceleration modality, respectively. Both generator networks employ an Encoder-Decoder convolutional architecture based on U-Net, and introduce attention mechanisms, including multi-scale convolutional modules (MultiScaleConv), residual connections (ResidualBlock), and attention modules, to enhance feature extraction capabilities, enabling the model to simultaneously capture local and global feature differences. Furthermore, discriminators (DiscriminatorAudio and DiscriminatorAcc) judge the authenticity of the generated features. Both discriminators employ a convolutional neural network (CNN) architecture, enabling adversarial learning between the generator and discriminator, and continuously optimizing the generation performance.
[0078] The loss functions of the modality transfer network mainly include adversarial loss (adc_loss), reconstruction loss (rec_loss), cycle consistency loss (cyc_lss), and identity mapping loss (idt_loss). Since our natural advantage in the identity authentication process is the availability of paired data, which is an inherent advantage enabling supervised training, we added a reconstruction loss to ensure reconstruction effectiveness. The weighted combination of the four loss functions is then iteratively updated using the Adam optimizer.
[0079] Specifically, the loss function is:
[0080]
[0081] in: Let X be the loss function, and Y be the speech mode. For adversarial loss from acceleration mode to speech mode, For adversarial loss from speech modality to acceleration modality, For the cycle consistency loss of acceleration modes, For the cycle consistency loss of the speech modality, λ cyc For the weights of the cycle consistency loss, λ rec To reconstruct the loss weights, The reconstruction loss is for the acceleration mode to speech mode. λ is the reconstruction loss from speech mode to acceleration mode. idt To maintain weight in one's identity, To preserve the identity of acceleration modes, Loss due to loss in voice modality identity preservation.
[0082] Adversarial loss is used to improve the realism of the generated spectrum.
[0083]
[0084] Cycle Consistency Loss ensures that the mappings from acceleration to speech to acceleration and speech to acceleration to speech remain consistent.
[0085]
[0086] Reconstruction Loss is used to improve the detail fidelity of the generated spectrum;
[0087]
[0088] Identity loss constrains the generator to maintain an identity mapping when the input and output belong to the same modality, in order to avoid unnecessary offsets.
[0089]
[0090] MultiScaleConv captures speech and acceleration pattern information at different scales in the input data through multiple convolutional branches (with kernel sizes of 1×1, 3×3, and 5×5, respectively), comprehensively extracting complex features.
[0091] TimeFreqAttention calculates attention weights in both the frequency and time dimensions, helping the generator to more effectively focus on key time-frequency features during mode transitions and suppress noise and redundant information. The GeneratorAccToAudio generator uses this module to effectively compensate for the loss of frequency information caused during convolution through global weighting in the frequency direction.
[0092] In GeneratorAccToAudio, ResidualBlock effectively alleviates network degradation by introducing residual blocks during feature extraction, enhances the network's expressive power, improves training stability, and maintains continuity between modal transitions.
[0093] The Axial Attention mechanism, used in GeneratorAudioToAcc, performs self-attention learning along both the temporal and frequency domains to capture the spatiotemporal structure information of the feature map more deeply. It effectively learns which structural information in the speech "responds" in the acceleration mode, thereby further improving the quality of the generated features.
[0094] Taking the implementation of the accelerometer-to-speech generator (GeneratorAccToAudio) as an example, the Encoder uses MultiScaleConv to extract local and global features at different scales. Each convolutional layer is followed by BatchNormalization and ReLU activation functions to help stabilize features and provide non-linear representation. Residual connections and TimeFreqAttention are added to enhance feature extraction. The Decoder uses deconvolution to upsample the feature maps and fuses low-level and high-detail features from the encoder through U-Net skip connections. The output of each layer in the Decoder stage refines the features through convolutional layers and gradually restores the size of the input data. The final output is activated by the Tanh activation function to ensure that the feature values are stable within a specific range. Specifically, the input feature—accelerometer spectrum—[batch, 1, 64, 80], is processed by MultiScaleConv to extract high-dimensional features and downsampled to a smaller spatial dimension. TimeFreqAttention significantly enhances the weights of speech-related features. Residual feature processors then perform residual connections to further strengthen feature representation. Deconvolution and skip connections reconstruct the feature map, gradually restoring it to the input size. A fully connected layer (fc) further refines the spectral dimension mapping, ultimately outputting a pseudo-speech Mel-spectrum map with the same size as the input, still [Batch, 1, 64, 80]. The generated result is as follows: Figure 5 , Figure 6 .
[0095] The process involves generating an acceleration signal to convert it into fake speech features, then reversing this process to convert it into a fake acceleration spectrum, and finally performing cyclic consistency feature extraction with the original acceleration signal. Figure 6 This enables the system to capture the stability differences exhibited by individuals across modalities and uncover discriminative features.
[0096] At the feature extraction level, the system extracts differential indicators for the cyclic consistency process, including LSD (Log Spectral Distance), MSE (Mean Squared Error), and STFT spectral difference between the acceleration signal and the pseudo-acceleration signal. Specifically, these include acc_LSD_real_fake, acc_LSD_real_cycle, acc_MSE_real_fake, acc_MSE_real_cycle, acc_STFTdiff_real_fake, and acc_STFTdiff_real_cycle. The system also jointly extracts the statistical features of the original bone conduction acceleration signal in the time-frequency domain, including acc_time_mean, acc_time_std, acc_time_skewness, acc_time_kurtosis, acc_time_rms, and acc_spectral_entropy, to comprehensively reflect the physiological structural differences and individual stability of the wearer's behavioral patterns during the vocalization process.
[0097] Based on the aforementioned multimodal fusion features, the system transforms the identity authentication problem into a feature-based binary classification task. Specifically, a Support Vector Machine (SVM) combined with a Radial Basis Function (RBF) is used as the classifier to distinguish between data from target users and non-target users. SVM is widely used in biometric recognition tasks due to its generalization ability under small sample conditions and robustness to high-dimensional features. The formula is as follows:
[0098]
[0099] Where f(x) is the output of the classification function, α i For support vector weights, y i Here, x represents the training sample labels, γ represents the parameters of the RBF kernel function, and x represents the training sample labels. i Let x be the feature vector of the training sample, x be the feature vector of the input test sample, and b be the bias term.
[0100] By training and judging the above features, the system can effectively improve the robustness and security of identity authentication, and demonstrate higher reliability and applicability in scenarios such as single-modality vulnerability to forgery attacks.
[0101] This implementation method achieves accurate and implicit authentication of the identity of users wearing VR devices, providing an innovative and practical technical approach for multimodal biometric recognition systems.
[0102] Experiments were conducted in a real-world environment to evaluate the practicality and reliability of this application. Data was collected using an HTC Vive headset, and a Lenovo Y900p laptop served as the system's host computer for signal processing and model training. Five volunteers were recruited: one as a registered user and four as intruders. The registered user first registered their identity in the system. During the authentication phase, the registered user and intruders performed user authentication in single-person and two-person scenarios, respectively.
[0103] The system's performance in terms of registered user data is shown in Table 1.
[0104] Table 1
[0105] Metric precision recall f1-score support other Users 0.95 0.97 0.96 240 User1 0.89 0.85 0.87 80 accuracy 0.94 320 macro avg 0.92 0.91 0.92 320 weighted avg 0.94 0.94 0.94 320
[0106] Overall, it performed well with an accuracy rate of 94%. It excelled in identifying attackers (other Users) (F1-score = 0.96), achieving its goal of maximizing attacker defense; its multi-user clustering performance was as follows... Figure 8 The system demonstrates clear differentiation for target users, but there is still room for improvement in differentiating other registered users. The overall F1 weighted average is 0.94, indicating high classification capability and validating the reliability and practicality of this invention in multimodal scenarios.
[0107] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A VR user authentication method based on multimodal perception of acoustic vibration signals, characterized in that, include: Step S1: Obtain the output of the accelerometer sensor and the microphone, and obtain the original acceleration signal and the original voice signal after alignment based on the time axis. The microphone and the accelerometer sensor are both located on the head-mounted VR device, and the accelerometer sensor collects craniofacial vibrations. Step S2: Perform peak detection based on the original speech signal to identify the vocal segments, and extract the vocal segments as the first speech signal; Step S3: Based on the start and end times of the speech segment, extract the first acceleration signal corresponding to each first speech signal from the original acceleration signal; Step S4: Generate the acceleration amplitude spectrum of the first acceleration signal as the first acceleration spectrum; Step S5: The first acceleration spectrum is converted into a speech Mel spectrum as the second Mel spectrum by the first generator of the trained modality conversion network. Then, the second Mel spectrum is converted into an acceleration amplitude spectrum as the second acceleration spectrum by the second generator of the trained modality conversion network. The modality conversion network learns the unsupervised mapping relationship between the acceleration signal and the speech spectrum to realize the feature conversion from acceleration to speech mode and from speech to acceleration mode. Step S6: Extract the first feature by performing cycle consistency feature extraction on the first acceleration spectrum and the second acceleration spectrum; Step S7: Input the first feature into the trained first detection model to obtain the detection result; Step S6 includes: Step S6-1: Calculate the logarithmic spectral distance, mean square error, and STFT spectral difference between the first acceleration spectrum and the second acceleration spectrum, respectively; Step S6-2: The first feature is obtained by fusing the calculated logarithmic spectral distance, mean square error and STFT spectral difference.
2. The VR user authentication method based on multimodal perception of acoustic vibration signals according to claim 1, characterized in that, Step S1 includes: Step S1-1: Acquire the outputs of the accelerometer sensor and microphone, and perform frequency filtering on each; Step S1-2: Based on the filtered data, resampling and time axis alignment are performed to obtain the original acceleration signal and the original speech signal, wherein the original acceleration signal and the original speech signal have the same sampling rate.
3. The VR user authentication method based on multimodal perception of acoustic vibration signals according to claim 1, characterized in that, The first detection model is a binary classification model.
4. The VR user authentication method based on multimodal perception of acoustic vibration signals according to claim 1, characterized in that, The modality transfer network further includes a first discriminator and a second discriminator, and the modality transfer network is trained based on the first sample.
5. The VR user authentication method based on multimodal perception of acoustic vibration signals according to claim 4, characterized in that, The construction process of the first sample includes: The outputs of the accelerometer sensor and microphone under interference-free conditions are obtained, and the raw acceleration signal and raw speech signal are obtained after time axis alignment. The vocal segment is identified and extracted as the first speech signal; Based on the start and end times of the speech segment, the first acceleration signal corresponding to each first speech signal is extracted from the original acceleration signal; The acceleration amplitude spectrum of the first acceleration signal is used as the first acceleration spectrum, and the speech Mel spectrum of the first speech signal is used as the first Mel spectrum. The corresponding first acceleration spectrum and first Mel spectrum are used as the first sample for acceleration and speech modality conversion.
6. The VR user authentication method based on multimodal perception of acoustic vibration signals according to claim 5, characterized in that, The first discriminator is based on a convolutional neural network architecture and is trained adversarially against the first generator based on the first Mel spectrogram and the second Mel spectrogram. The second discriminator employs a convolutional neural network architecture and performs adversarial training against the second generator based on the first and second acceleration spectra.
7. A VR user authentication method based on multimodal perception of acoustic vibration signals according to claim 6, characterized in that, The loss function of the mode conversion network is: in: Let X be the loss function, and Y be the speech mode. For adversarial loss from acceleration mode to speech mode, For adversarial loss from speech modality to acceleration modality, For the cycle consistency loss of acceleration modes, For the cycle consistency loss of speech modalities, For cycle consistency loss weights, To reconstruct the loss weights, The reconstruction loss is for the acceleration mode to speech mode. The reconstruction loss is for the speech mode to the acceleration mode. To maintain weight in one's identity, To preserve the identity of acceleration modes, Loss due to loss in voice modality identity preservation.
8. A VR user authentication device based on multimodal perception of acoustic vibration signals, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.
9. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-7.