VR device user authentication method based on empirical wavelet and multi-scale attention

Abstract

The present invention belongs to the fields of deep learning and network security technology, and specifically relates to a VR device user authentication method based on empirical wavelets and multi-scale attention. The method comprises collecting data from motion sensors on a VR device while the VR device is in use; performing rationality verification and normalization on the motion sensor data, and then using it as user feature data. The user feature data is then labeled to form a training sample set; a user authentication model is constructed, and the user authentication model is optimized and trained using the training sample set. The user feature data of the user to be authenticated is input into the optimized and trained user authentication model, and the output of the user authentication model is used as the identity authentication result of the user to be authenticated. The present invention achieves seamless, real-time, and accurate identification of VR device users.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and network security technology, and specifically relates to a VR device user authentication method based on empirical wavelet and multi-scale attention. Background Art

[0002] Given the privacy concerns inherent in the hardware, software, and usage scenarios of virtual reality (VR) devices, developing a suitable and reliable authentication model is crucial to protect user information security and prevent intrusion and theft of software and private information stored on VR devices. User-insensitive authentication uses motion sensors to collect and calculate data to assess whether the current user is an intruder. This process uses the user's motion behavior data for identification.

[0003] Current research on user authentication for devices is steadily advancing toward intelligent and comfortable applications. This research area encompasses numerous technologies, including multi-sensor fusion, deep learning, mobile computing, and human-computer interaction. Current research has achieved some success in areas such as behavioral perception and sample collection. However, overall, this research is still in its infancy, primarily focused on academic exploration and prototype verification. A systematic research framework has yet to be established for key technical areas, leaving a significant gap between practical application and future applications. Furthermore, different devices may have different motion sensor models and categories. Current research can only identify data samples from devices with a specific sensor combination, lacking general applicability.

[0004] In view of the above reasons, a more ideal and practical VR device user authentication technology must be proposed to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a VR device user authentication method based on empirical wavelet and multi-scale attention, so as to realize the seamless, real-time and accurate identification of VR device users.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A VR device user authentication method based on empirical wavelet and multi-scale attention, comprising:

[0008] Collecting data from motion sensors on VR devices when the VR devices are in use;

[0009] After the motion sensor data is rationally checked and normalized, it is used as user feature data, and the user feature data is labeled to form a training sample set;

[0010] Construct a user authentication model and optimize and train it using a training sample set. The user authentication model uses a twin network as a framework, with a spatiotemporal feature fusion module that extracts parallel features using deep separable convolution and echo state network as the backbone network, and an empirical wavelet transform-multiscale attention mechanism module connected to the backbone network.

[0011] The user feature data of the user to be authenticated is input into the optimized and trained user authentication model, and the output of the user authentication model is taken as the identity authentication result of the user to be authenticated, and the identity authentication result is a legitimate user or an intruder.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0013] Preferably, collecting data from motion sensors on the VR device includes:

[0014] For VR devices with built-in Android systems, the data collection process is as follows: obtain the Android system's sensor management service, then obtain the target type of motion sensor, register a listener for the motion sensor, and obtain the newly collected data from the motion sensor through the listener;

[0015] For VR devices connected to the host, the data collection process is as follows: use smart pointers to configure the VR device to enter research mode; obtain the number of motion sensors in the VR device and the descriptors of the motion sensors; find the target motion sensor by traversing the descriptors of the motion sensors; open the sensor stream of the target motion sensor; after opening the sensor stream, obtain the current data frame and block until a new data frame is obtained.

[0016] Preferably, performing rationality verification and normalization processing on the motion sensor data as user feature data includes:

[0017] Over-limit abnormality check: If the data collected by the motion sensor exceeds the threshold range, it is considered that an over-limit abnormality has occurred and the current motion sensor data is discarded; otherwise, no over-limit abnormality has occurred;

[0018] Equivalent anomaly check: Calculate the equivalent anomaly rate of the current motion sensor data. If the equivalent anomaly rate of the current motion sensor data exceeds the equivalent anomaly threshold, it is considered that an equivalent anomaly occurs and the current motion sensor data is discarded; otherwise, no equivalent anomaly occurs.

[0019] Jump point abnormality check: If the current motion sensor data has jump point abnormality, the current motion sensor data will be discarded;

[0020] The data of the motion sensor that has passed the over-limit anomaly check, the equal value anomaly check, and the jump point anomaly check is normalized, and the normalized value is used as the user feature data.

[0021] Preferably, the user authentication model includes two weight-sharing sub-networks, the outputs of the two sub-networks are respectively passed through a flattening layer, and then the distance between the outputs of the two flattening layers is measured by a distance calculation layer, and then the distance is sequentially passed through two densely connected layers to obtain the output of the user authentication model;

[0022] The sub-network includes a dense connection layer, a spatiotemporal feature fusion module, an empirical wavelet transform-multi-scale attention mechanism module and a dense connection layer connected in sequence.

[0023] Preferably, the spatiotemporal feature fusion module includes a depthwise separable convolution, an echo state network, and a multi-head dynamic gating unit;

[0024] The depthwise separable convolution and echo state network are set in parallel, the outputs of the depthwise separable convolution and echo state network serve as the input of the multi-head dynamic gating unit, and the output of the multi-head dynamic gating unit serves as the output of the spatiotemporal feature fusion module.

[0025] Preferably, the multi-head dynamic gating unit has the following processing process:

[0026] Multi-head projection: Input features and input features Projected into the multi-head space, we get The first projection characteristics of the head Hedi Second projection characteristics of the head ;

[0027] Dynamic gate generation: Input features and input features After splicing, it is used as the input of the mapping operation, and the output of the mapping operation is obtained. Gating weight of each head , where the mapping operation consists of two densely connected layers;

[0028] Multi-head fusion: Gating weight of each head As the fusion weight, The first projection characteristics of the head Hedi Second projection characteristics of the head Perform fusion operation and get the Fusion characteristics of the head ,Then the fusion features of all heads are aggregated to obtain the aggregated features as the output of the multi-head dynamic gating unit.

[0029] Preferably, the processing process of the empirical wavelet transform-multi-scale attention mechanism module is as follows:

[0030] The output of the spatiotemporal feature fusion module is taken as the input of the empirical wavelet transform-multi-scale attention mechanism module, and the input feature is recorded as ;

[0031] For input features Perform empirical wavelet transform to obtain multi-scale features ;

[0032] Input features and multi-scale features Splice by channel dimension to get fusion features ;

[0033] The fusion features After the multi-scale attention mechanism, the output of the multi-scale attention mechanism is obtained as the output of the empirical wavelet transform-multi-scale attention mechanism module.

[0034] As an advantage, the fusion feature After a multi-scale attention mechanism, including:

[0035] Take fusion features The shape is [B, T, C+S], where B is the batch size, T is the sequence length, and C is the input feature The number of channels, S is the multi-scale feature The number of scales;

[0036] According to the fusion features Generate query vector, key vector and value vector, each of which has the shape of [B, T, C];

[0037] Split the query vector, key vector, and value vector into the scale number W and rearrange them into [B, W, T, D], where D = C / / W is the dimension of each scale.

[0038] Calculate the matrix multiplication of the query vector and the key vector to obtain the attention score, which has the shape of [B, W, T, T]. Apply the Softmax function to the attention score to obtain the attention weight, which has the shape of [B, W, T, T]. Then perform matrix multiplication of the attention weight with the value vector to obtain the weighted multi-scale feature, which has the shape of [B, W, T, D]. Finally, merge the scales of the multi-scale feature to obtain the output of the multi-scale attention mechanism, which has the shape of [B, T, C].

[0039] The VR device user authentication method based on empirical wavelet and multi-scale attention provided by the present invention has the following advantages compared with the existing technology:

[0040] (1) A spatiotemporal feature fusion module containing parallel extraction of deep separable convolution-echo state network (ESN) is introduced as the backbone network, and a Siamese twin neural network is used as the framework. The calculation process includes weight sharing, which can better deal with the problem of sample label imbalance while extracting deep temporal semantic information. (2) A gating mechanism of multi-head projection-dynamic gating generation-multi-head fusion is used to adapt the multimodal feature interaction and nonlinear modeling between deep separable convolution-echo state network. (3) Through the empirical wavelet transform (EWT)-multi-scale attention mechanism module, changes and patterns in different frequency bands can be captured, features of different scales can be learned and optimized, and the model's feature representation ability can be enhanced, thereby improving the accuracy and robustness of authentication. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the VR device user authentication method based on empirical wavelet and multi-scale attention of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of the user authentication model of the present invention;

[0043] Figure 3 Schematic diagram of the structure of the spatiotemporal feature fusion module of the present invention;

[0044] Figure 4 Schematic diagram of data processing of a single head in the multi-head dynamic gating unit of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0047] To achieve user-unaware VR device identity authentication and practical applications, this paper provides a VR device user authentication method based on empirical wavelet transform and multi-scale attention. This method integrates an empirical wavelet transform-multi-scale attention mechanism module into a twin architecture of the backbone network, using a spatiotemporal feature fusion module extracted in parallel using a deep separable convolution-echo state network. This enables seamless, real-time, and accurate identification of VR device users.

[0048] like Figure 1 As shown, the VR device user authentication method based on empirical wavelet and multi-scale attention in this embodiment includes the following steps:

[0049] (1) Collect data from motion sensors on VR devices when the VR devices are in use.

[0050] (1-1) Pre-preparation and parameter construction: There are four types of commonly used motion sensors and they form two combinations. In actual use, for each VR device user authentication, one combination of data can be obtained and used. Each type of sensor has , , Three axes. This embodiment uses represents the acceleration sensor, Represents the gravity sensor, represents the gyroscope, Indicates the geomagnetic sensor. Based on this, the two combinations are as follows: 、 and For the first combination, 、 and For the second combination, the parameter Indicates the time dimension The sensor reading at a moment.

[0051] VR devices can be divided into two categories based on whether they operate independently of a host computer. One type can independently run its own operating system (using Android as an example in this example), while the other type requires a connection to a host computer, leveraging the host's computing power to render VR applications. For Android-based VR devices, developers register sensors as register monitors and then refresh sensor data at a 0.02s (50Hz) sampling rate. For consistency, devices that require a host computer also use a 50Hz sampling rate.

[0052] This embodiment continuously monitors the device based on the sampling frequency while the user is actively using it. Motion sensor readings during this period effectively reflect the user's status. The loading phase duration is set to 2 to 4 seconds, based on sensitivity analysis. When the duration is less than 3 seconds, the final accuracy improves over time; when the duration exceeds 3 seconds, the final accuracy decreases. Therefore, this embodiment chooses to set the data collection duration to 3 seconds. To determine whether the user is actively using the device, this embodiment defines the following two rules. Sensor data is recorded only when both rules are met: 1. The VR device is manually turned on; 2. A new application is started. Rule 2 requires the cooperation of Rule 1 for the application to be properly activated. This is because many applications with auto-start permissions exist on the device. Only by complying with Rule 1 can it be proven that the application was opened manually.

[0053] (1-2) Data Collection Process: For VR devices with native Android systems, call the getSystemService(Context.SENSOR_SERVICE) method in the Context abstract class to obtain the system's sensor management service. Use the getDefaultSensor(int type) method in the SensorManager abstract class to obtain a sensor of a specified type. Then, call the registerListener() method in the SensorManager abstract class to register a listener for the specified sensor and implement the onSensorChanged() method in the SensorEventListener interface. The onSensorChanged() method is called back when the sensor value changes to return the newly collected data from the motion sensor.

[0054] For devices that need to connect to a host (e.g., with Windows), you need to enter the Windows Mixed Reality portal (Windows Mixed Reality) research mode and configure it correctly. This example uses the following method:

[0055] (1-2-1) Use the smart pointer ComPtr to configure the VR device to enter research mode. ComPtr is a smart pointer used to manage the lifecycle of COM (Component Object Model) objects. It automatically handles reference counting to ensure that objects are properly released when no longer needed. Specifically, the smart pointer ComPtr is used to define the COM interface IResearchModeSensorDevice representing the research mode. It provides methods to access device sensors and defines a pointer variable pSensorDevice to store a pointer to the IResearchModeSensorDevice interface. The research mode configuration is then completed by calling the CreateResearchModeSensorDevice(&pSensorDevice) method, where &pSensorDevice is the address of the pSensorDevice variable.

[0056] (1-2-2) Get the number of sensors and descriptors: Initialize the variable sensorCount to 0 to store the number of sensors, then call the GetSensorCount() method of the IResearchModeSensorDevice interface to get the number of available sensors in the device, and store the number of motion sensors in the VR device in the variable sensorCount; then create a dynamic array container sensorDescriptors with a size of sensorCount to allocate space for storing the descriptors of each motion sensor; use the sensorDescriptors.data() method to get the pointer to the underlying array of the dynamic array container, and call the GetSensorDescriptors() method of the IResearchModeSensorDevice interface again to fill the descriptor data of the motion sensor into the dynamic array container sensorDescriptors. After the GetSensorDescriptors() method is executed, sensorCount will be updated to the number of sensor descriptors actually obtained.

[0057] (1-2-3) Find the target motion sensor by traversing the descriptors of the motion sensors.

[0058] (1-2-4) Open the sensor stream: Open the sensor stream by calling the OpenStream() method.

[0059] (1-2-5) Get sensor frame: When getting the target motion sensor data, manage the lifecycle of the pointer pSensorFrame through the smart pointer ComPtr. After calling the OpenStream() method to open the sensor stream, get the current data frame by calling the pSensor->GetNextBuffer(&pSensorFrame) method and block until a new data frame is available. After getting the data frame, access the data in the data frame through the pointer pSensorFrame.

[0060] (2) After the motion sensor data is rationally checked and normalized, it is used as user feature data, and the user feature data is labeled to form a training sample set. The training sample set supports and covers a variety of motion sensor signal samples, and has feature data and labels used by VR devices in the user authentication process.

[0061] (2-1) Out-of-limit anomaly: Each sensor's reading must be within a reasonable range. If the user places the device flat on a table, this will produce samples that are not conducive to training and certification. Therefore, if the collected motion sensor data exceeds the threshold range, it is considered an out-of-limit anomaly and the current motion sensor data is discarded. Otherwise, no out-of-limit anomaly has occurred.

[0062] This embodiment sets the following boundaries for possible gravity sensors. If a reading meets all of the following conditions, the sample will be deleted:

[0063]

[0064]

[0065]

[0066] Likewise, a smaller range is set for the magnetometer, and samples will be dropped if the reading exceeds the following boundaries:

[0067]

[0068]

[0069]

[0070] (2-2) Equal value anomaly: At a sampling rate of 50 Hz, even if the device is stationary, the values ​​between two adjacent samples collected by the motion sensor will vary slightly. However, low-performance motion sensors can cause equal value anomalies. If two or more consecutive values ​​in the time series data are equal, it is called an equal value anomaly. Equal value anomalies are machine anomalies and should be eliminated. Continuous and constant values ​​will interfere with the construction of the user model. This embodiment uses the following formula for judgment:

[0071]

[0072] in, represents the equivalent anomaly rate, The maximum number of identical values ​​in the current motion sensor data. Indicates the length of the current motion sensor data. If the equal value anomaly rate is greater than 0.75, the corresponding time series data is considered abnormal and is removed. For the identification of the same number of values, if the error is taken into account, the threshold can be set to 2 or other values, where is the idealized standard deviation.

[0073] (2-3) Jumping point anomaly: Jumping point anomaly refers to the sudden appearance of one or more data points in the sequence that are significantly different from the previous data points. The appearance of these data points will cause the continuity of the sequence to be interrupted and appear as a jump mutation. If the following conditions exist, the sample is considered to have a jumping point anomaly:

[0074]

[0075] in, 、 and are three consecutive data points, represents the acceleration due to gravity, If it does not meet the requirements, the sample will be discarded.

[0076] (2-4) Normalization: Normalization can improve the performance of the model and enhance its generalization and versatility. This embodiment normalizes the data of the motion sensor after passing the over-limit anomaly check, equal value anomaly check, and jump point anomaly check, and uses the normalized value as the user feature data. Specifically, the data is normalized to a specific interval [-1, 1]. The normalization method uses maximum-minimum normalization to scale the data proportionally to the interval 0, 1. The calculation formula is:

[0077]

[0078] in, is the original data value, and are the minimum and maximum values ​​in the data set, respectively. is the normalized data.

[0079] (3) Construct a user authentication model and use the training sample set to optimize the user authentication model. Figure 2 As shown, the user authentication model of this embodiment includes two sub-networks with shared weights. The outputs of the two sub-networks pass through a flattening layer (Flatten layer) respectively, and then the distance between the outputs of the two flattening layers is measured by a distance calculation layer. The distance is then passed through two densely connected layers (Dense layers) in sequence to obtain the output of the user authentication model; the sub-network includes a densely connected layer, a spatiotemporal feature fusion module, an empirical wavelet transform-multi-scale attention mechanism module and a densely connected layer connected in sequence.

[0080] The twin neural network framework, through its symmetrical structure and weight sharing, can cluster samples of the same category or concept together to capture and learn semantic similarities between them, and exhibits excellent generalization capabilities. The spatiotemporal feature fusion module, which uses parallel extraction of deep separable convolutional and echo state networks, extracts temporal dynamic features and local features, providing a more comprehensive feature representation and improving computational efficiency and model robustness. The empirical wavelet transform-multiscale attention layer can more comprehensively capture and utilize multi-scale features in time series data, enhancing the model's expressive power.

[0081] (3-1) Densely connected layer: This layer uses a weight matrix and activation function to linearly transform and nonlinearly activate the input data, thereby achieving nonlinear mapping of the input features. This enables the network to learn more complex patterns and features, improving the model's expressiveness and fitting capabilities. It also allows the network to combine and interact with various input features to generate new feature representations. The number of units in this densely connected layer is 64.

[0082] (3-2) Spatiotemporal feature fusion module: Figure 3 As shown in Figure 1, the spatiotemporal feature fusion module consists of a parallel architecture consisting of a depthwise separable convolutional branch and an echo state network branch. The outputs of these two branches serve as the input to a multi-head dynamic gating system. The echo state network can capture dynamic features and long-term dependencies in the temporal dimension, while the depthwise separable convolutional branch can capture local patterns and structural information in space.

[0083] (3-2-1) Depthwise Separable Convolution: Depthwise separable convolution consists of channel-wise convolution and point-wise convolution. Channel-wise convolution performs convolution operations on each input channel separately, and the number of output channels is the same as the number of input channels. Point-wise convolution uses a 1×1 convolution kernel to perform a weighted combination of the output feature maps of the channel-wise convolution in the depth direction, thereby expanding the number of output channels.

[0084] (3-2-2) Echo State Network: The echo state network uses a fixed-weight "reservoir" to capture the dynamic characteristics of the input sequence. It mainly includes two main steps: initialization and recursive calculation. In initialization, the reservoir weight, input weight and bias need to be initialized. The reservoir weight initialization requires generating a random matrix and setting some elements in the matrix to zero. The spectral radius of the matrix is ​​calculated through singular value decomposition (SVD) and adjusted to the target value. In the recursive state, for each time step , reservoir status Updated by the following formula:

[0085]

[0086] in, is the first time steps, is the reservoir state at the previous time step, is the input weight matrix, is the reservoir weight matrix, is the bias vector.

[0087] Depthwise separable convolutions excel at extracting local spatial features and high-frequency information, while echo state networks excel at capturing long-term temporal dependencies and low-frequency dynamics. Using these two in parallel allows for more comprehensive feature extraction. Furthermore, convolutional networks are sensitive to local variations, while echo state networks are sensitive to global dynamics. Using these two in parallel improves the model's robustness to various types of noise and variation.

[0088] (3-3) Multi-head dynamic gating mechanism: The multi-head dynamic gating mechanism of this embodiment receives the output of the depthwise separable convolution and the echo state network as input. The detailed steps are as follows:

[0089] (3-3-1) If Figure 4 As shown, multi-head projection: the input features (Input 1) and input features (Input 2) is projected into the multi-head space, and the formula is as follows:

[0090]

[0091]

[0092] Where, For the The first projection feature of the head, Indicates the The first projection operation of the head, input feature is the output of a depthwise separable convolution or echo state network, For the The second projection feature of the head, Indicates the The second projection operation of the head, input feature The output of the echo state network or depthwise separable convolution. The projection operation used in this embodiment is to pass the input through a densely connected layer (which requires multiplying the number of heads by the original number of units) and a reshape operation.

[0093] (3-3-2) Multi-head gating: Input features and input features After splicing, it is used as a mapping operation The final output is the gating weight of each head, which is expressed as follows:

[0094]

[0095] Where, For the The gating weight of the head, is the Sigmoid activation function, Represents input features and input features The features after splicing, The operation consists of two densely connected layers. The number of units in the first densely connected layer is a preset fixed value, and the activation function is the Swish activation function. The number of units in the second densely connected layer is consistent with the number of heads, and the activation function is the Sigmoid activation function.

[0096] (3-3-3) Multi-head fusion: This step includes the calculation of fusion features for each head and the aggregation of multiple heads. The formula is as follows:

[0097]

[0098]

[0099] Where, Indicates the The fusion characteristics of the head, is the total number of multi-head dynamic gating units, It is the aggregated feature of the fusion features of all heads and is the output of the multi-head dynamic gating unit.

[0100] (3-4) Empirical Wavelet Transform-Multi-Scale Attention Mechanism Module: The processing process of this module is: take the output of the spatiotemporal feature fusion module as the input of the Empirical Wavelet Transform-Multi-Scale Attention Mechanism module, and record the input feature as ; For input features Perform empirical wavelet transform to obtain multi-scale features ; Input features and multi-scale features Splice by channel dimension to get fusion features , the original features and the multi-scale features extracted by the empirical wavelet transform are spliced ​​together to form a richer feature representation; the fusion features After the multi-scale attention mechanism, the output of the multi-scale attention mechanism is obtained as the output of the empirical wavelet transform-multi-scale attention mechanism module.

[0101] (3-4-1) Among them, for the input features Perform empirical wavelet transform to obtain multi-scale features , the processing process is as follows:

[0102] Fast Fourier Transform (FFT): Convert the input signal to the frequency domain to obtain the spectrum. First, the input feature Convert to complex type, then perform fast Fourier transform on the complex tensor to convert the time domain signal into frequency domain representation. The shape of the complex representation in the frequency domain is the same as the input feature. The same. Then the amplitude of the frequency domain signal is obtained. Since the fast Fourier transform result is symmetrical, usually only the first half of the amplitude of the frequency domain signal is taken as the spectrum.

[0103] Boundary detection: Detect local maxima and their locations in the spectrum through the maximum pooling operation, then sort the peaks in descending order and select the first nb_scales (for example, 16 in this example) - 1 as the band boundaries:

[0104] Filter bank construction: Generate a time axis from 0 to 1 and generate a Meyer filter for each frequency band. Each filter is constructed by multiplying a Hamming window and a cosine function. Finally, all filters are stacked into a filter bank.

[0105] Multi-scale feature extraction: Apply a filter bank to the input signal via tensor product to extract features at different scales.

[0106] (3-4-2) Among them, the fusion feature After the multi-scale attention mechanism, the processing process is as follows:

[0107] Take fusion features The shape is [B, T, C+S], where B is the batch size, T is the sequence length, and C is the input feature The number of channels, S is the multi-scale feature The number of scales;

[0108] QKV generation: Map the fused features to query (Q), key (K) and value (V) space. Through the dense connection layer, the fused features are mapped to the query (Q), key (K) and value (V) space. Map to a space with 3 times the number of channels to generate QKV features with a shape of [B, T, 3*C]. Then split the QKV features into query vector, key vector, and value vector according to the channel dimension. The shape of each vector is [B, T, C].

[0109] Multi-head splitting: The query vector, key vector, and value vector are split according to the number of scales W through the split operation, and rearranged into the shape [B, W, T, D], where D = C / / W is the dimension of each scale.

[0110] Calculate attention weights: Calculate the matrix multiplication of the query vector and the key vector to obtain an attention score with a shape of [B, W, T, T]. Then, use the trainable temperature parameter to adjust the attention score. Apply the Softmax function to the attention score to obtain the attention weight with a shape of [B, W, T, T]. Then, perform matrix multiplication of the attention weight with the value vector to obtain the weighted multi-scale feature with a shape of [B, W, T, D]. Finally, merge the various scales of the multi-scale feature to obtain the output of the multi-scale attention mechanism with a shape of [B, T, C].

[0111] (3-5) Flattening layer: Flattens the multi-dimensional tensor into a one-dimensional vector to facilitate distance calculation in the subsequent twin neural network architecture.

[0112] (3-6) Distance Calculation Layer: After the flattening layer, both ends of the twin neural network have become one-dimensional vectors. The distance calculation in this embodiment is the L1 distance, which calculates the absolute value difference between the two.

[0113] (3-7) Densely connected layers: Finally, densely connected layers are used to map the L1 distances to a low-dimensional space. The first densely connected layer uses the ReLU activation function to increase the model's nonlinearity, while the second densely connected layer uses the Sigmoid activation function to constrain the output to the range [0, 1]. If multi-user classification is required, the final activation function can be changed from the Sigmoid activation function to the Softmax function.

[0114] (4) Inputting the user feature data of the user to be authenticated into the optimized and trained user authentication model, taking the output of the user authentication model as the identity authentication result of the user to be authenticated, and the identity authentication result is a legitimate user or an intruder.

[0115] The feature data of the VR user to be authenticated is obtained as the input of the optimized and updated user authentication model, and the authentication result of whether the user to be authenticated is a legitimate user is obtained based on the output of the authentication model. After obtaining the training sample set in steps (1) and (2), the training sample set needs to be pushed to the cloud for training. The server is built and processed based on Django. In this embodiment, the user authentication model is implemented using the Python-based TensorFlow library. When the network model successfully learns the user features, the trained weight parameters are pushed back to the original device to realize knowledge transfer. The method of obtaining the feature data of the VR device user to be authenticated in the authentication stage is consistent with steps (1) and (2) and will not be repeated here. The local computer judges the obtained verification sample by loading the weight parameters pushed by the cloud and returns the judgment similarity. If the similarity is higher than the threshold, the current user identity is determined to be a legitimate user and the authentication is successful; otherwise, the identity is obtained to be illegal and the authentication fails. If the authentication fails, the program will assist in locking the system. It needs to be unlocked through other verification methods of the system brought by the VR device before it can be reactivated.

[0116] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A VR device user authentication method based on empirical wavelet and multi-scale attention, characterized in that: The VR device user authentication method based on empirical wavelet and multi-scale attention includes: Collecting data from motion sensors on VR devices when the VR devices are in use; After the motion sensor data is rationally checked and normalized, it is used as user feature data, and the user feature data is labeled to form a training sample set; Construct a user authentication model and optimize and train it using a training sample set. The user authentication model uses a twin network as a framework, with a spatiotemporal feature fusion module that extracts parallel features using deep separable convolution and echo state network as the backbone network, and an empirical wavelet transform-multiscale attention mechanism module connected to the backbone network. Inputting the user feature data of the user to be authenticated into the optimized and trained user authentication model, taking the output of the user authentication model as the identity authentication result of the user to be authenticated, and the identity authentication result is a legitimate user or an intruder; The user authentication model includes two weight-sharing sub-networks. The outputs of the two sub-networks are respectively passed through a flattening layer. The distance between the outputs of the two flattening layers is then measured through a distance calculation layer. The distance is then passed through two densely connected layers in sequence to obtain the output of the user authentication model. The sub-network includes a dense connection layer, a spatiotemporal feature fusion module, an empirical wavelet transform-multi-scale attention mechanism module and a dense connection layer connected in sequence, and the empirical wavelet transform-multi-scale attention mechanism module has the following processing process: The output of the spatiotemporal feature fusion module is taken as the input of the empirical wavelet transform-multi-scale attention mechanism module, and the input feature is recorded as ; For input features Perform empirical wavelet transform to obtain multi-scale features ; Input features and multi-scale features Splice by channel dimension to get fusion features ; The fusion features After the multi-scale attention mechanism, the output of the multi-scale attention mechanism is obtained as the output of the empirical wavelet transform-multi-scale attention mechanism module.

2. The VR device user authentication method based on empirical wavelet and multi-scale attention according to claim 1 is characterized in that: The data collected from the motion sensor on the VR device includes: For VR devices with built-in Android systems, the data collection process is as follows: obtain the Android system's sensor management service, then obtain the target type of motion sensor, register a listener for the motion sensor, and obtain the newly collected data from the motion sensor through the listener; For VR devices connected to the host, the data collection process is as follows: use smart pointers to configure the VR device to enter research mode; obtain the number of motion sensors in the VR device and the descriptors of the motion sensors; find the target motion sensor by traversing the descriptors of the motion sensors; open the sensor stream of the target motion sensor; after opening the sensor stream, obtain the current data frame and block until a new data frame is obtained.

3. The VR device user authentication method based on empirical wavelet and multi-scale attention according to claim 1 is characterized in that: The rationality check and normalization processing of the motion sensor data as user feature data includes: Over-limit abnormality check: If the data collected by the motion sensor exceeds the threshold range, it is considered that an over-limit abnormality has occurred and the current motion sensor data is discarded; otherwise, no over-limit abnormality has occurred; Equivalent anomaly check: Calculate the equivalent anomaly rate of the current motion sensor data. If the equivalent anomaly rate of the current motion sensor data exceeds the equivalent anomaly threshold, it is considered that an equivalent anomaly occurs and the current motion sensor data is discarded; otherwise, no equivalent anomaly occurs. Jump point abnormality check: If the current motion sensor data has jump point abnormality, the current motion sensor data will be discarded; The data of the motion sensor that has passed the over-limit anomaly check, the equal value anomaly check, and the jump point anomaly check is normalized, and the normalized value is used as the user feature data.

4. The VR device user authentication method based on empirical wavelet and multi-scale attention according to claim 1 is characterized in that: The spatiotemporal feature fusion module includes a depthwise separable convolution, an echo state network, and a multi-head dynamic gating unit; The depthwise separable convolution and echo state network are set in parallel, the outputs of the depthwise separable convolution and echo state network serve as the input of the multi-head dynamic gating unit, and the output of the multi-head dynamic gating unit serves as the output of the spatiotemporal feature fusion module.

5. The VR device user authentication method based on empirical wavelet and multi-scale attention according to claim 4 is characterized in that: The multi-head dynamic gating unit processes as follows: Multi-head projection: Input features and input features Projected into the multi-head space, we get The first projection characteristics of the head Hedi Second projection characteristics of the head ; Dynamic gate generation: Input features and input features After splicing, it is used as the input of the mapping operation, and the output of the mapping operation is obtained. Gating weight of each head , where the mapping operation consists of two densely connected layers; Multi-head fusion: Gating weight of each head As the fusion weight, The first projection characteristics of the head Hedi Second projection characteristics of the head Perform fusion operation and get the Fusion characteristics of the head ,Then the fusion features of all heads are aggregated to obtain the aggregated features as the output of the multi-head dynamic gating unit.

6. The VR device user authentication method based on empirical wavelet and multi-scale attention according to claim 1 is characterized in that: The fusion feature After a multi-scale attention mechanism, including: Take fusion features The shape is [B, T, C+S], where B is the batch size, T is the sequence length, and C is the input feature The number of channels, S is the multi-scale feature The number of scales; According to the fusion features Generate query vector, key vector and value vector, each of which has the shape of [B, T, C]; Split the query vector, key vector, and value vector into the scale number W and rearrange them into [B, W, T, D], where D = C / / W is the dimension of each scale. Calculate the matrix multiplication of the query vector and the key vector to obtain the attention score, which has the shape of [B, W, T, T]. Apply the Softmax function to the attention score to obtain the attention weight, which has the shape of [B, W, T, T]. Then perform matrix multiplication of the attention weight with the value vector to obtain the weighted multi-scale feature, which has the shape of [B, W, T, D]. Finally, merge the scales of the multi-scale feature to obtain the output of the multi-scale attention mechanism, which has the shape of [B, T, C].

Images