Game controller sensing method, device, equipment and storage medium

Abstract

The present application relates to the field of game controller sensing technology, and discloses a game controller sensing method, device, equipment and storage medium, the method comprising: performing digital filtering and time-series synchronization on the angular velocity data of a gyroscope sensor, the motion data of an acceleration sensor and the pressure intensity data of a pressure sensor to obtain original action data; performing feature segmentation and modal cross-mapping to obtain an action feature sequence; inputting a multi-level deep convolutional neural network to perform spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; calculating a dynamic compensation value; performing gradient iterative operation and error correction to obtain real-time control parameters; performing position control signal and force feedback signal processing based on the real-time control parameters, and outputting a target execution instruction, thereby ensuring the accurate execution of the control instruction and greatly improving the control experience of the game controller.

Description

Technical Field

[0001] The present application relates to the field of game controller sensing technology, and in particular to a game controller sensing method, device, equipment and storage medium. Background Art

[0002] Traditional game controllers rely primarily on single button and joystick input, making them incapable of meeting the precision and immersion demands of modern games. While newer generation game controllers incorporate multiple sensing elements, such as gyroscopes, accelerometers, and pressure sensors, effectively integrating and precisely controlling these sensor data remains challenging.

[0003] Existing game controller sensing methods often rely on simple data fusion strategies, failing to fully exploit the complex relationships between multimodal sensor data. Especially when handling high-speed movements and delicate manipulations, sensor data noise, latency, and differences in player operating habits can affect control accuracy. Furthermore, random delays in wireless transmission can lead to unstable position control and force feedback responses, impacting the consistency of the gaming experience. Summary of the Invention

[0004] The present application provides a sensing method, apparatus, device and storage medium for a game controller, thereby ensuring the accurate execution of control instructions and greatly improving the control experience of the game controller.

[0005] A first aspect of the present application provides a sensing method for a game controller, the sensing method for the game controller comprising:

[0006] Perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data;

[0007] Performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence;

[0008] Inputting the action feature sequence into a multi-layer deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set;

[0009] Based on the operation behavior feature set, calculating the position state quantity and the force feedback state quantity to obtain a dynamic compensation value;

[0010] Performing gradient iterative calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters;

[0011] The position control signal and the force feedback signal are processed based on the real-time control parameters, and a target execution instruction is output.

[0012] A second aspect of the present application provides a sensing device for a game controller, the sensing device comprising:

[0013] The synchronization module is used to digitally filter and synchronize the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data;

[0014] A mapping module, configured to perform feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence;

[0015] A recognition module is used to input the action feature sequence into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set;

[0016] A calculation module, configured to calculate a position state quantity and a force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value;

[0017] a correction module, configured to perform gradient iterative calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters;

[0018] The output module is used to process the position control signal and the force feedback signal based on the real-time control parameters and output the target execution instruction.

[0019] A third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory to enable the electronic device to execute the above-mentioned game controller sensing method.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned game controller sensing method.

[0021] Compared with the existing technology, the present application has the following beneficial effects: by introducing a bidirectional cross-sensing mechanism and a multi-level deep convolutional neural network, efficient fusion and feature extraction of multimodal sensor data are achieved, thereby improving control accuracy; an adaptive learning algorithm is used to model user operating habits, so that the control system can dynamically adjust parameters according to the characteristics of different players; a delay prediction and compensation mechanism based on Kalman filtering is designed to effectively solve the problem of wireless transmission delay, and the position error and force feedback error are controlled within the preset range respectively; through gradient iterative optimization and error correction strategy, real-time updating and optimization of control parameters are ensured; the hierarchical drive and priority scheduling mechanism is adopted to ensure the precise execution of control instructions, greatly improving the control experience of the game controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] The structures, proportions, sizes, etc. depicted in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not intended to limit the conditions under which the present invention can be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportional relationships, or adjustments in size should still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and objectives that can be achieved by the present invention.

[0024] Figure 1 1 is a flow chart of a sensing method for a game controller provided by an embodiment of the present invention;

[0025] Figure 2 is a schematic block diagram of the structure of a sensing device of a game controller provided by an embodiment of the present invention;

[0026] Figure 3 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0029] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should be further understood that the term "and / or" used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 In one embodiment of the present application, a method for sensing a game controller includes:

[0031] Step 100: digitally filter and time-series synchronize the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain raw motion data;

[0032] It is understandable that the execution subject of the present application can be a sensing device of a game controller, or a terminal or a server, and the specific implementation is not limited here. The embodiment of the present application is described by taking the server as the execution subject as an example.

[0033] Specifically, the X-, Y-, and Z-axis angular velocity data collected by the gyroscope sensor undergoes 16-bit digital quantization, converting the analog continuous signals into discrete digital signals to produce digitized angular velocity data. Simultaneously, the X-, Y-, and Z-axis motion data collected by the acceleration sensor undergoes 16-bit digital quantization to produce digitized motion data. The pressure data collected by the pressure sensor undergoes 12-bit digital quantization to produce digitized pressure data. 12-bit quantization is chosen based on the characteristics of pressure data, taking into account the relative range of pressure variation and its impact on operation. This 12-bit quantization ensures measurement accuracy while meeting hardware processing efficiency requirements. High-frequency noise and low-frequency drift are removed from the digitized angular velocity data and motion data to produce attitude filtered data. Eliminating high-frequency noise results in more stable and reliable attitude information. Furthermore, since long-term measurements can cause low-frequency drift in the sensor output, which can cause errors in position and attitude measurements, low-frequency drift correction is performed to ensure data accuracy. Simultaneously, the digitized pressure data is smoothed to reduce the impact of transient spikes and jitter caused by user pressure variations on the controller operation. This effectively filters out minor fluctuations during operation, resulting in a smoother pressure signal and improved system response consistency, generating force-filtered data. Time-stamp alignment and difference compensation are performed on the posture and force-filtered data. This time-stamp alignment synchronizes data from different sources on the same timeline, ensuring that they accurately reflect the controller's motion state at the same moment. Furthermore, difference compensation is performed to account for time discrepancies caused by different sampling rates or processing steps, reducing errors caused by these time differences and generating more accurate aligned data. The aligned data is then divided into continuous data segments according to a preset time window, and each segment is smoothed along its boundaries. By dividing the data into continuous time windows, the controller's motion state is finely sliced ​​in time, enabling independent analysis of data within each time window and achieving precise motion tracking. Boundary smoothing reduces discontinuities between time windows, ensuring a more natural and smooth transition between data and preventing the impact of transient errors introduced by window boundary fluctuations on subsequent calculations. The segmented data sequences are normalized and dimensionally unified to obtain the raw motion data. Data from different sources are represented on the same scale to avoid deviations caused by varying numerical ranges. Dimensional unification also integrates multi-source data into a single data structure, allowing subsequent pattern recognition and feature extraction steps to effectively utilize this multi-dimensional information and obtain the raw motion data used for control and feedback.

[0034] Step 200: performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence;

[0035] Specifically, the raw motion data is divided into angular velocity data sets, motion data sets, and pressure data sets based on sensor type, facilitating independent extraction and analysis of each data type's characteristics. Different sensor data have different dynamic characteristics and physical meanings. The angular velocity data set contains three-axis angular velocity information measured by the gyroscope sensor, the motion data set consists of three-axis motion information measured by the accelerometer sensor, and the pressure data set contains information on the player's key pressure. An intra-modal association unit is constructed to extract and fuse features from these different data sets. This intra-modal association unit comprises a feature extraction layer and a feature fusion layer. The feature extraction layer employs a sliding time window structure. By setting a sliding time window, it dynamically calculates local features for the angular velocity, motion, and pressure data, capturing subtle changes in movement during operation. For the angular velocity and motion data sets, the feature extraction layer extracts mean, variance, and peak features, respectively. The mean describes the overall controller movement trend over a specific period of time, the variance reflects movement volatility, and the peak captures extreme changes in movement. For the pressure data set, the extracted features include mean, duration, and rate of change. The mean reflects the average pressure applied by the player, the duration characterizes the duration of the pressure, and the rate of change characterizes the speed with which the pressure is applied. The extracted data is fused, and a feature fusion layer is set up within the intra-modal association unit. This feature fusion layer uses a multi-layer perceptron architecture and consists of two fully connected layers. The first fully connected layer uses the ReLU activation function, which introduces nonlinear characteristics, allowing the fused features to better represent complex relationships. The second fully connected layer uses the Sigmoid activation function, normalizing the output to keep the eigenvalues ​​within a limited range. This control prevents extreme values ​​from affecting model training and improves the stability of subsequent calculations. After feature extraction and fusion, the angular velocity data set, motion data set, and pressure data set are respectively input into the intra-modal association unit to generate single-modal feature vectors, which describe the characteristics of the handle motion in each modality. A bidirectional cross-perception mechanism is constructed to fuse and map features from different modalities. This mechanism consists of a mapping layer, an association layer, and a self-preservation layer. The mapping layer employs an attention mechanism, using self-attention to calculate the weight matrix between different modalities. The self-attention mechanism dynamically assigns weights based on the importance of different modal features, thereby highlighting more meaningful features and suppressing irrelevant or redundant information. The association layer utilizes a graph convolutional network structure. By constructing data from different modalities into a graph, the association layer effectively captures the topological relationships between modalities through graph convolution. These structured relationships help identify potential connections and synergies between modalities. The self-preservation layer uses a long short-term memory (LSTM) network structure to store and maintain feature combinations with high recognition accuracy.LSTM can retain important information in time series data while forgetting unimportant parts, ensuring that effective feature combinations can be accurately captured even during long-term operations. Unimodal feature vectors are input into a bidirectional cross-sensing mechanism. Through attention calculations in the mapping layer, graph convolution processing in the association layer, and a memory network in the self-preservation layer, a cross-feature matrix is ​​generated. This cross-feature matrix is ​​then subjected to temporal reconstruction and structuring, reorganizing the feature matrix so that it not only represents the temporal information of the features themselves but also effectively reflects the relationships and interactions between different features, ultimately resulting in a sequence of action features.

[0036] Step 300: Input the action feature sequence into a multi-layer deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set;

[0037] It should be noted that the action feature sequence is divided into temporal and spatial feature data blocks, and the network is used to independently extract and learn data of different dimensions. A multi-level deep convolutional neural network is constructed, consisting of a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure and captures detailed characteristics and overall trends in the feature data through multi-level convolution operations, achieving efficient extraction of spatiotemporal features. The feature extraction network contains three convolutional modules, each of which processes different types of input data. The first convolutional module contains two convolutional layers, each using a 3×3 convolution kernel with a stride of 2, effectively downsampling the input features to capture features within a larger receptive field. Furthermore, the first convolutional module includes a BatchNorm layer for normalizing the convolved data. The introduction of BatchNorm can accelerate network training, effectively prevent gradient vanishing and gradient exploding, and improve network stability and generalization. When processing spatial features, the second convolutional module of the feature extraction network contains three convolutional layers. Each convolutional layer also uses a 3×3 convolution kernel with a stride of 2 and introduces the ReLU activation function. The nonlinear characteristics of the ReLU activation function enable the network to learn more complex and diverse features. The third convolutional module contains three convolutional layers. The convolution kernel is still 3×3, but the stride is adjusted to 1, and the LeakyReLU activation function is used. Compared with the standard ReLU, the LeakyReLU has better characteristics. It retains a portion of the gradient when the input is less than zero, alleviating the "neuron death" problem that occurs during the training process of ReLU, improving the model's responsiveness to negative features, and enabling the convolutional network to learn the information in the features more comprehensively. During the input stage, the time domain feature data block is input into the first convolutional module of the feature extraction network. After multi-scale convolution operations, the time domain feature map is obtained. It describes the changing pattern of the action in the time dimension and can reflect the dynamic behavior characteristics of the handle throughout the operation process. At the same time, the spatial feature data block is input into the second and third convolution modules of the feature extraction network. Through multi-level convolution operations, a spatial feature map is obtained, which describes the distribution characteristics and local changes of the action in the spatial dimension and can effectively capture the operation characteristics in different directions and positions. A feature recognition network is constructed, which consists of two fully connected layers and one classification layer. The number of neurons in the fully connected layers is 512 and 256 respectively, which gradually reduces the dimension of the feature vector and achieves effective dimensionality reduction and abstraction of high-dimensional features. Through the two fully connected layers, the network can fuse information from different feature maps, allowing the temporal and spatial domain features to be integrated and understood at a global level.The classification layer uses the Softmax activation function to convert the network output into a probability distribution. This normalizes each category so that the sum of all category probabilities is 1. This output facilitates the determination of the action category and provides a probability value for each category, indicating the network's confidence in each action category. The temporal and spatial feature maps are input into the feature recognition network. Feature fusion and classification operations are used to generate an action category vector, representing the type of the current action. The probability output of the Softmax layer is then used to calculate the probability distribution of each category. The action category vector is then subjected to probability distribution calculation and label mapping to obtain the action feature set.

[0038] Step 400: Calculate the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value;

[0039] Specifically, the operational behavior feature set is divided into position state features and force feedback state features based on state attributes, effectively distinguishing features related to handle position from features related to force feedback, and optimizing these two types of features separately. Position state features describe information such as the handle's position, velocity, and acceleration in space, while force feedback state features describe force-related factors such as feedback intensity and direction. A second-order dynamic model of the handle's position is established based on the position state features. This second-order dynamic model effectively describes the changing position of the handle during different operations. By correlating the three state variables, position, velocity, and acceleration, a position state equation is established. The position state equation describes how the handle's position evolves over time and how velocity and acceleration affect its current state. This equation is used to describe and predict the handle's motion state at different time points. Simultaneously, a spring-damper force feedback model is established based on the force feedback state features. This model describes the force feedback applied to the handle during operation. The spring-damper model simulates the force feedback experienced in actual operation. The force feedback state equation is derived by calculating the force response to position deviation and velocity. In the force feedback state equation, position deviation reflects the degree of deviation between the current handle position and the target position, while velocity indicates the speed of the handle movement. The feedback force associated with these states is calculated using a spring-damper model. The position state equation and force feedback state equation are input into a Kalman state predictor to obtain a state prediction matrix. The Kalman predictor is an effective tool for state estimation, particularly in the presence of noise. It can accurately predict the system state by continuously updating and revising the state estimate. By inputting the position state equation and force feedback state equation, the Kalman predictor combines the current state variables with historical observations to predict the future state of the handle, forming a state prediction matrix containing state information at multiple future time points. The state prediction matrix is ​​subjected to state variable separation and time series expansion to obtain a delayed prediction sequence. Because actual control processes often involve processing delays, the predicted states are gradually expanded using time series expansion to accurately describe the changes in the system state during the delay period, resulting in a delayed prediction sequence. Based on the delayed prediction sequence, a compensation model is constructed to accumulate the state changes during the predicted delay period to obtain the state compensation. The compensation model pre-estimates and accumulates the state deviations of the controller during the delay period for correction during subsequent control. The state compensation compensates for the deviations incurred by the system during the processing delay period. By substituting the state compensation into the position state equation and the force feedback state equation, the system state is updated to obtain the compensated state variables. The compensated state variables effectively correct for control deviations caused by delays, improving the response accuracy and stability of the controller. Dynamic correction and boundary constraint processing are performed on the compensated state variables to obtain the dynamic compensation value.Dynamic correction eliminates the cumulative errors that occur in the system's state estimation and control process by introducing additional correction factors, while boundary constraint processing is used to ensure that the compensated state variables are within a reasonable physical range. For example, the intensity of force feedback should not exceed the maximum tolerance of the handle, and the position state should not exceed the operable boundary of the handle.

[0040] Step 500: performing gradient iterative calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters;

[0041] Specifically, the dynamic compensation values ​​and the operational behavior feature set are combined to generate a parameter optimization dataset. Based on the parameter optimization dataset, a user operation habit model is constructed. By analyzing and calculating operation speed, force, and duration, characteristic parameters of the user's controller usage are extracted. Operation speed reflects the speed of the user's controller movement, force describes the strength of the user's button press, and duration represents how long the user maintains a particular action. Through a comprehensive analysis of these characteristics, operational characteristic parameters are derived, reflecting the user's individualized usage patterns and preferences. Based on the operational characteristic parameters, the model parameters are initialized to obtain the model's initial parameters. These initial parameters serve as the basis for subsequent gradient iterations. Proper initialization accelerates model convergence and improves optimization effectiveness. The model's initial parameters are input into a gradient descent optimizer, and the parameter gradients are calculated using the backpropagation algorithm to determine the parameter update direction. During the backpropagation process, the partial derivatives of the loss function with respect to each parameter are calculated to determine the update direction for each parameter in the current state, enabling the model to continuously adjust towards reducing error. Adaptive step size adjustment is performed on the parameter update direction, dynamically adjusting the step size based on the current gradient changes, thereby achieving more stable and efficient optimization during the iterative process. This adaptive step size adjustment prevents overshooting the optimal solution due to excessively large step sizes when the error is small, and increases the step size to accelerate convergence when the error is large. Furthermore, momentum accumulation is performed during the parameter update process to determine the parameter iteration amount. By retaining momentum information from the previous update, the parameter update direction is smoother, reducing jitter caused by drastic gradient changes and ensuring more stable model convergence to the optimal solution during optimization. The parameter iteration amount is compared with the current parameter error calculation, and parameters with errors below a set threshold are stored to obtain the optimized parameter set. The optimized parameter set is then verified using position error and force feedback error calculations. Position error verification tests the accuracy of the controller's position response during actual operation, while force feedback error verification verifies whether the force feedback provided by the controller meets the expected user experience. Verification of position error and force feedback error ensures that the optimized parameters not only meet the numerical requirements but also provide a good user experience in actual operation. Through verification, a control parameter set is obtained, including parameters related to handle position control and force feedback control, ensuring that users experience more accurate and realistic feedback when operating the handle. The control parameter set is assigned parameters according to the control execution cycle to obtain real-time control parameters. At each fixed time interval, the handle operation is updated and adjusted to ensure real-time control. In this way, real-time control parameters are dynamically adjusted based on user operating habits and the physical characteristics of the handle to provide optimal response and feedback.

[0042] Step 600: Process the position control signal and the force feedback signal based on the real-time control parameters, and output a target execution instruction.

[0043] Specifically, the real-time control parameter undergoes first and second signal separation to obtain a first position control parameter and a first force feedback control parameter. The first position control parameter describes the required position change of the handle, while the first force feedback control parameter describes the force feedback the handle needs to provide to the player during operation. The first position control parameter is input into a first frequency converter for signal modulation, adjusting the frequency of the control signal to better meet the input requirements of the driver module, thereby obtaining a second position control parameter. Similarly, the first force feedback control parameter is input into a second frequency converter for signal modulation, thereby obtaining a second force feedback control parameter. The first driver module performs pulse width modulation and amplitude conversion on the second position control parameter to obtain a first execution signal. Pulse width modulation is a control method that achieves precise control of the handle position by adjusting the pulse width to control the average energy of the signal. Amplitude conversion adjusts the signal amplitude to meet the driving requirements of the actuator unit, ensuring that the handle position change accurately reflects the user's operational intent. Simultaneously, the second driver module performs current intensity modulation and direction control on the second force feedback control parameter to obtain a second execution signal. Current intensity modulation controls the strength of the force feedback signal, allowing the player to experience varying degrees of feedback force, while directional control determines the direction of the feedback force to provide appropriate feedback in different player scenarios. The first and second execution signals are input into the timing processing unit for bidirectional signal synchronization and priority determination, resulting in a third execution signal. Bidirectional signal synchronization ensures that the position control signal and force feedback signal are time-coordinated, avoiding operational delays or feedback distortion caused by signal asynchrony. Priority determination coordinates multiple control signals to prioritize control signals with greater impact on the gaming experience during critical operations, thereby enhancing the user experience. Through processing, the timing processing unit generates a combined third execution signal containing synchronized position and force feedback information. The third driver module then performs wireless signal encoding and transmission intensity modulation on the third execution signal to produce a fourth execution signal. Wireless signal encoding converts the execution signal into a format suitable for wireless transmission, enabling effective communication between the controller and the control terminal. Transmission intensity modulation is used to adjust the transmission intensity of the signal to ensure that the signal can be transmitted stably in different transmission environments. For example, the transmission intensity is increased in an environment with interference, and the intensity is reduced in an environment with good signals to save energy. The fourth execution signal is parsed and scheduled according to the execution priority and timing requirements to obtain the target execution instruction. Instruction parsing is to separate the various parts of the fourth execution signal and extract specific control information, such as the target position of the handle, the amount of feedback force to be applied, etc. Scheduling and allocation is to send the control information to the corresponding execution device according to the execution priority and timing requirements, so that the corresponding action can be performed at the right time.

[0044] In the embodiments of the present application, by introducing a bidirectional cross-sensing mechanism and a multi-level deep convolutional neural network, efficient fusion and feature extraction of multimodal sensor data are achieved, thereby improving control accuracy; an adaptive learning algorithm is used to model user operating habits, so that the control system can dynamically adjust parameters according to the characteristics of different players; a delay prediction and compensation mechanism based on Kalman filtering is designed to effectively solve the problem of wireless transmission delay, and control the position error and force feedback error within the preset range respectively; through gradient iterative optimization and error correction strategy, real-time updating and optimization of control parameters are ensured; a hierarchical drive and priority scheduling mechanism is adopted to ensure the precise execution of control instructions, greatly improving the control experience of the game controller.

[0045] In a specific embodiment, the process of executing step 100 may specifically include the following steps:

[0046] The X-axis, Y-axis, and Z-axis angular velocity data collected by the gyroscope sensor are subjected to 16-bit digital quantization processing to obtain digital angular velocity data; the X-axis, Y-axis, and Z-axis motion data collected by the acceleration sensor are subjected to 16-bit digital quantization processing to obtain digital motion data; the pressure intensity data collected by the pressure sensor are subjected to 12-bit digital quantization processing to obtain digital pressure data;

[0047] Eliminate high-frequency noise and low-frequency drift on the digitized angular velocity data and digitized motion data to obtain attitude filter data, and perform data smoothing on the digitized pressure data to obtain force filter data;

[0048] The posture filter data and the force filter data are time-stamped and difference-compensated to obtain aligned data. The aligned data are divided into continuous data segments according to the time window and the boundaries are smoothed to obtain a segmented sequence. The segmented sequence is then normalized and dimensionally unified to obtain the original motion data.

[0049] Specifically, the X-axis, Y-axis, and Z-axis angular velocity data collected by the gyroscope sensor are subjected to 16-bit digital quantization processing to obtain digital angular velocity data, the X-axis, Y-axis, and Z-axis motion data collected by the acceleration sensor are subjected to 16-bit digital quantization processing to obtain digital motion data, and the pressure intensity data collected by the pressure sensor are subjected to 12-bit digital quantization processing to obtain digital pressure data. The data collected by the gyroscope, acceleration sensor, and pressure sensor are all analog signals, expressed as voltage values, and the range is a specified interval, such as ±5V or ±3.3V. In order to convert these analog signals into digital signals, analog-to-digital conversion is performed, that is, a certain number of quantization bits are used to convert these voltage values ​​into discrete digital values. In this embodiment, the data quantization of the gyroscope and acceleration sensor adopts 16 bits, that is, the range of the quantized values ​​is between 0 and 65535. The quantized angular velocity data is expressed as:

[0050] ;

[0051] in, Represents the angular velocity data collected by the gyroscope, and Respectively represent the lower and upper limits of the gyroscope's range, and is the quantized digital angular velocity data. Similarly, the quantization formula of acceleration data is expressed as:

[0052] ;

[0053] in, Represents the motion data collected by the acceleration sensor, and are the lower and upper limits of the acceleration range, It is the quantized digital motion data. For the pressure sensor, 12-bit quantization is used, and the formula is as follows:

[0054] ;

[0055] in, Indicates the collected pressure intensity data. and Represent the lower and upper limits of the pressure sensor, respectively, and Represents quantized pressure data. After data quantization is completed, the digitized angular velocity data and acceleration data are subjected to high-frequency noise and low-frequency drift elimination to obtain attitude filtered data. High-frequency noise is caused by external environmental interference and the instability of the sensor itself. For example, when the user shakes the sensor quickly, large instantaneous jitter will be generated. To reduce this interference, a low-pass filter is used. The low-pass filter is expressed as follows:

[0056] ;

[0057] in, Represents the original input data of the nth sampling point, Represents the output data after filtering, is the filter coefficient, which controls the intensity of the filter and ranges from 0 to 1. When the value is smaller, the low-pass filter has a stronger effect on suppressing high-frequency noise. Low-frequency drift refers to the tendency of the output signal to gradually change when the sensor is used for a long time. This change will affect the accurate measurement of angular velocity and acceleration. To eliminate low-frequency drift, a high-pass filter is used. The formula is as follows:

[0058] ;

[0059] in, is the coefficient of the high-pass filter, and adjusting it controls the filtering effect. After processing high-frequency noise and low-frequency drift, stable posture filter data is obtained. For the digitized pressure data, data smoothing is performed to eliminate instantaneous spikes and jitter caused by changes in user operation force. Smoothing is achieved through the sliding average method, and its formula is:

[0060] ;

[0061] in, represents the size of the sliding window, represents the pressure data of the i-th sampling point, Represents the smoothed pressure data. By using a sliding average, random jitter in the pressure data is effectively reduced, making the signal more stable. Time stamp alignment and difference compensation are performed on the posture filter data and the force filter data. Time stamp alignment is achieved through interpolation, such as linear interpolation:

[0062] ;

[0063] in, and Known time points and The data on For the time point that needs to be interpolated, the time point is obtained by linear interpolation The data values ​​on the sensor are aligned to achieve alignment. At the same time, difference compensation is performed to eliminate the differences caused by the different response characteristics of the sensors, so that the data can be better matched. After the time marker alignment is completed, the aligned data is divided into continuous data segments according to the time window, and the boundary smoothing is performed to obtain a segmented sequence. The time window is divided by setting a fixed length time period, for example, the aligned data is divided into a window of 100 milliseconds, and the original continuous data is divided into multiple small segments. The boundary smoothing process is to reduce the discontinuity between time windows. The data at the boundary is processed in a manner similar to weighted averaging to ensure a more natural transition between windows. The formula is as follows:

[0064] ;

[0065] in, Represents the original data at the boundary, is the corresponding weight. The larger the weight, the greater the impact of the data point on the smoothing result. By using the weighted average method, the data mutation at the window boundary is effectively reduced. The segmented sequence is normalized and dimensionally unified to obtain the original action data. Data from different sources are represented on the same scale to avoid deviations caused by different numerical ranges. Normalization is achieved using the following formula:

[0066] ;

[0067] in, Represents the original data, is the mean of the data, is the standard deviation of the data. After normalization, the data has a distribution with a mean of 0 and a standard deviation of 1. Dimensional unification integrates different types of data into the same feature space for subsequent pattern recognition and feature extraction. For example, angular velocity, acceleration, and pressure data can be spliced ​​into a multidimensional vector in a certain order to form the original motion data.

[0068] In a specific embodiment, the process of executing step 200 may specifically include the following steps:

[0069] The original motion data is divided into angular velocity data group, motion data group and pressure data group according to sensor type;

[0070] Construct an intra-modal correlation unit, which includes a feature extraction layer and a feature fusion layer. The feature extraction layer adopts a sliding time window structure, and the feature fusion layer adopts a multi-layer perceptron structure. The feature extraction layer extracts mean, variance, and peak features from the angular velocity data set, mean, variance, and peak features from the motion data set, and mean, duration, and rate of change features from the pressure data set. The feature fusion layer contains two fully connected layers. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function.

[0071] Inputting the angular velocity data group, the motion data group and the pressure data group into the intramodal correlation unit to obtain a single modal feature vector;

[0072] Construct a bidirectional cross-perception mechanism, which includes a mapping layer, an association layer, and a self-preservation layer. The mapping layer uses an attention mechanism structure, the association layer uses a graph convolutional network structure, and the self-preservation layer uses a memory network structure. The mapping layer calculates the weight matrix between different modalities through self-attention, the association layer uses a graph convolutional network to construct the topological relationship between modalities, and the self-preservation layer contains a long short-term memory network to store feature combinations with high recognition accuracy.

[0073] The single-modal feature vector is input into the bidirectional cross-perception mechanism, and the cross-feature matrix is ​​obtained through feature mapping and association learning. The cross-feature matrix is ​​then subjected to temporal reconstruction and structural processing to obtain the action feature sequence.

[0074] Specifically, the original motion data is classified according to the type of sensor from which it comes, that is, it is divided into angular velocity data group, motion data group and pressure data group. The angular velocity data group comes from the gyroscope, which mainly indicates the rotation speed of the handle on the X, Y and Z axes; the motion data group comes from the acceleration sensor, which describes the linear acceleration of the handle in different directions; and the pressure data group comes from the pressure sensor, which indicates the force applied by the user to the handle buttons during operation. Classification helps to extract independent features for different data types in order to better reflect the status of the handle. Construct an intra-modal association unit, which consists of a feature extraction layer and a feature fusion layer. The feature extraction layer adopts a sliding time window structure to extract local features from the data, and the length of the sliding time window is set to , the data in each time window is used to calculate the corresponding features. The feature extraction layer extracts the mean, variance, and peak features of the angular velocity data set. These features describe the average rotation speed of the handle within the time window, the degree of rotation fluctuation, and the maximum rotation speed. For example, for the first time window of the angular velocity data set, the mean feature is represented as:

[0075] ;

[0076] in, Indicates the time window The angular velocity value at a moment, The variance feature is used to measure the degree of discreteness of the data, which is expressed as:

[0077] ;

[0078] The peak feature is expressed as:

[0079] ;

[0080] Similarly, for the motion data set, mean, variance, and peak features were extracted using the same sliding time window. For the pressure data set, in addition to the mean and peak values, duration and rate of change features were also extracted. Duration represents the length of time the user maintains a certain pressure, while rate of change describes the rate at which pressure changes over time, expressed as:

[0081] ;

[0082] in, and Respectively represent the pressure values ​​at the start and end of the time window, represents the rate of change of pressure. By extracting features from angular velocity, motion, and pressure data, we obtain a feature set for each modality. The output of the feature extraction layer is passed to the feature fusion layer, which uses a multi-layer perceptron structure to fuse different features. The feature fusion layer consists of two fully connected layers. The first fully connected layer uses the ReLU activation function, and its output is:

[0083] ;

[0084] in, is the weight matrix of the first fully connected layer, is the input feature vector, is the bias term, and the ReLU activation function is defined as:

[0085] ;

[0086] The second fully connected layer uses the Sigmoid activation function, and its output is:

[0087] ;

[0088] Among them, the Sigmoid function is defined as:

[0089] ;

[0090] Through the feature fusion layer, the extracted features are abstracted into single-modal feature vectors to describe the operation behavior of the handle under each mode. The angular velocity, motion and pressure data groups are input into the intra-modal association unit to obtain the single-modal feature vector of each mode. The single-modal feature vector is input into the bidirectional cross-sensing mechanism to obtain the relationship and combination characteristics between the modes. The bidirectional cross-sensing mechanism includes a mapping layer, an association layer and a self-preservation layer, which captures the complex correlation characteristics between modalities in a variety of different ways. The mapping layer adopts an attention mechanism to calculate the weight matrix between different modalities through the self-attention mechanism to measure the contribution of each modality to the operation behavior in a specific time period. For example, given the angular velocity feature vector and motion feature vector , the attention weight matrix is ​​expressed as:

[0091] ;

[0092] in, represents the first angular velocity feature vector The element of the motion feature vector The attention weights of each element are calculated. This step allows for dynamic weight distribution between modalities, highlighting the modal features that are more important to the current operation. The association layer uses a graph convolutional network structure to construct the topological relationship between modalities. The feature vectors of different modalities are considered nodes in the graph, and the edges between nodes represent the relationship between modalities. The graph convolution operation is expressed as:

[0093]

[0094] in, is the adjacency matrix of the graph, which represents the connection relationship between modes. For the The node feature matrix of the layer, For the The weight matrix of the layer, and is an activation function (such as ReLU). Through graph convolution, the association layer can effectively capture the relationship between different modalities and build the topological structure between them. The self-preservation layer uses a long short-term memory network (LSTM) structure to store feature combinations with high recognition accuracy. LSTM is a special recurrent neural network (RNN) that can maintain long-term dependency information in time series data. Its state update formula is:

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] in, 、 and They are forget gate, input gate and output gate respectively. and The LSTM is a weight and bias matrix. This structure enables it to store important feature combinations at different time steps, preserving features that demonstrate high recognition accuracy across multiple operations. After processing in the mapping, association, and self-preservation layers, the unimodal feature vectors are combined into a cross-feature matrix. This cross-feature matrix undergoes time-series reconstruction and structuring, ultimately yielding an action feature sequence. Time-series reconstruction aims to reorganize the feature matrix into a time series format that reflects the state changes of the controller at different time points. Structuring combines features from different modalities according to predetermined rules to form a complete action feature description.

[0101] In a specific embodiment, the process of executing step 300 may specifically include the following steps:

[0102] The action feature sequence is divided into a time domain feature data block and a space domain feature data block;

[0103] Construct a multi-level deep convolutional neural network, which includes a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure, and the feature recognition network adopts a multi-scale fusion structure. The feature extraction network contains three convolution modules. The first convolution module contains two convolution layers, each of which uses a 3×3 convolution kernel with a stride of 2 and a BatchNorm layer. The second convolution module contains three convolution layers, each of which uses a 3×3 convolution kernel with a stride of 2 and a ReLU activation function. The third convolution module contains three convolution layers, each of which uses a 3×3 convolution kernel with a stride of 1 and a LeakyReLU activation function.

[0104] The time domain feature data block is input into the first convolution module of the feature extraction network, and the time domain feature map is obtained through multi-scale convolution operation. The spatial domain feature data block is input into the second and third convolution modules of the feature extraction network, and the spatial domain feature map is obtained through multi-level convolution operation.

[0105] Construct a feature recognition network. The feature recognition network consists of two fully connected layers and one classification layer. The number of neurons in the fully connected layers is 512 and 256 respectively. The classification layer uses the Softmax function.

[0106] The temporal feature map and the spatial feature map are input into the feature recognition network, and the action category vector is obtained through feature fusion and classification operations. The action category vector is then subjected to probability distribution calculation and label mapping processing to obtain the operation behavior feature set.

[0107] Specifically, the motion feature sequence is deconstructed to extract temporal and spatial information. The motion feature sequence is a high-dimensional feature matrix derived from the fusion of multiple sensor data, containing various characteristic information about the user's controller operation. The temporal feature data block represents the temporal evolution of the motion. These features reflect how the controller's motion changes over time, such as speed and acceleration. The spatial feature data block describes the spatial distribution of the motion, reflecting the posture and orientation of the controller at different spatial locations. A multi-level deep convolutional neural network is constructed for feature extraction and recognition. The multi-level deep convolutional neural network consists of a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure, capturing the different scale characteristics of the motion feature data through multi-level convolution operations. The spatial pyramid structure effectively extracts both local and global information, making the network robust to features at different scales. The feature recognition network, on the other hand, adopts a multi-scale fusion structure. By fusing features at different scales, the network has better recognition capabilities for complex motion patterns. The feature extraction network consists of three convolutional modules, each processing a different feature data block. The first convolution module contains two convolutional layers, each of which uses a 3×3 convolution kernel with a stride of 2 and a BatchNorm layer. The role of BatchNorm is to normalize the output of each layer to accelerate network training and improve its stability. , the convolution operation is expressed as:

[0108] ;

[0109] in, is the convolution kernel weight, * represents the convolution operation, is the bias term, is an activation function, such as ReLU or LeakyReLU. The BatchNorm layer normalizes the convolution output as follows:

[0110] ;

[0111] in, and are the mean and standard deviation of the batch, The second convolution module contains three convolution layers, each of which also uses a 3×3 convolution kernel with a step size of 2 and a ReLU activation function. The formula for the ReLU activation function is:

[0112] ;

[0113] The nonlinear characteristics of ReLU effectively prevent the vanishing gradient problem, allowing the network to better learn complex feature relationships. Through convolution operations, the second convolution module can extract input features layer by layer, allowing the network to better perceive the local details of the input features. The third convolution module also contains three convolution layers, each of which uses a 3×3 convolution kernel with a stride of 1 and uses LeakyReLU as the activation function. The activation formula of LeakyReLU is:

[0114] ;

[0115] in, A small constant (e.g., 0.01) is used to ensure gradient propagation even when the input is negative, thus resolving the "neuron death" issue caused by ReLU. The third convolutional module allows the network to further extract spatial features and capture detailed variations in features through a refined convolution step size. During feature extraction, the temporal feature data block is input into the first convolutional module of the feature extraction network. Multi-scale convolution operations generate a temporal feature map representing the temporal variations of the handle motion, such as the temporal variation of rotation speed. Simultaneously, the spatial feature data block is input into the second and third convolutional modules of the feature extraction network. Multi-level convolution operations are used to extract a spatial feature map, which describes the spatial variations of the motion, such as the changes in the handle's posture in different directions. After obtaining the temporal and spatial feature maps, a feature recognition network is constructed to fuse and classify these features. The feature recognition network consists of two fully connected layers and one classification layer, with 512 and 256 neurons in each fully connected layer, respectively. The fully connected layer fuses and reduces the extracted temporal and spatial features to extract high-level features with strong discriminative power. The operation of the fully connected layer is expressed as:

[0116] ;

[0117] ;

[0118] in, and is the weight matrix of the fully connected layer, and is the bias term, is the input feature vector, and are the outputs of the fully connected layer. The activation function uses ReLU to increase the nonlinearity of the feature. The classification layer uses the Softmax activation function to convert the network output into a probability distribution for action category judgment. The definition of the Softmax function is:

[0119] ;

[0120] in, is the output vector of the fully connected layer, Indicates the The scores of the categories, Indicates that the input belongs to The probability of each category. Through the Softmax function, the network can normalize each category so that the sum of the probabilities of each category output is 1, thereby determining which action category the input feature belongs to. The time domain feature map and the spatial domain feature map are input into the feature recognition network. Through the feature fusion of the fully connected layer and the operation of the classification layer, the action category vector is obtained, which represents the different action categories to which the current handle operation belongs and its corresponding probability distribution. The action category vector is subjected to probability distribution calculation and label mapping processing to obtain an operation behavior feature set. The label mapping process is to convert the action category probability output by the network into a specific operation behavior label, such as waving, pressing, rotating, etc. These operation behavior labels can be used to generate control instructions and process feedback signals.

[0121] In a specific embodiment, the process of executing step 400 may specifically include the following steps:

[0122] The operation behavior feature set is divided into position state feature and force feedback state feature according to state attributes;

[0123] According to the position state characteristics, a second-order dynamic model of the handle position is established, and the three state variables of position, velocity and acceleration are correlated and calculated to obtain the position state equation;

[0124] A spring damping force feedback model is established based on the force feedback state characteristics, and the force response of the position deviation and velocity is calculated to obtain the force feedback state equation;

[0125] Input the position state equation and the force feedback state equation into the Kalman state predictor to obtain the state prediction matrix;

[0126] Perform state variable separation and time series expansion operations on the state prediction matrix to obtain the delay prediction sequence;

[0127] A compensation model is constructed based on the delayed prediction sequence, and the state change amount within the predicted delay time is accumulated to obtain the state compensation amount;

[0128] The state compensation amount is substituted into the position state equation and the force feedback state equation for state update to obtain the compensation state variable, and the compensation state variable is subjected to dynamic correction and boundary constraint processing to obtain the dynamic compensation value.

[0129] Specifically, the operation behavior feature set is divided into position state features and force feedback state features according to state attributes. The operation behavior feature set includes the user's operation actions and their physical attributes. According to the state attributes, the features are divided into position state features and force feedback state features. The position state features include the spatial position, movement speed, acceleration and other information of the handle, which are used to describe the movement trajectory and state changes of the handle in the physical space; while the force feedback state features include the interaction force between the handle and the user, such as the magnitude and direction of the feedback force given to the user by the force feedback motor. These features are used to describe the response of the handle to the user's actions. After completing the division of state features, a second-order dynamic model of the handle is established based on the position state features. This model effectively describes the movement state of the handle in space. The movement of the handle conforms to the classical physical dynamics principles. The position state features mainly include position ,speed and acceleration Three state variables. The basic equation of the second-order kinetic model is:

[0130] ;

[0131] in, Indicates the quality of the handle, is the acceleration, is the damping coefficient, For speed, is the elastic coefficient, For location, Represents the external force. By calculating the relationship between the three state variables of position, velocity and acceleration, the motion equation of the handle is obtained, which is recorded as the position state equation:

[0132] ;

[0133] in, is the state vector, representing position and velocity, is the system matrix, which is used to describe the relationship between state variables. is the input matrix, describing the influence of external forces on the system, Represents the rate of change of state. A spring-damper force feedback model is established based on the force feedback state characteristics. The principle of force feedback is described by a spring-damper system, which can simulate the reaction force generated by the handle during user operation. The basic equation of the spring-damper model is:

[0134] ;

[0135] in, is the feedback force, is the spring stiffness coefficient, which indicates the effect of position deviation on the feedback force. Target location is the current handle position, and the position deviation is is the damping coefficient, is the speed of the handle, describing the effect of speed on the feedback force. By calculating the force response of position deviation and speed, the force feedback state equation is obtained, which is used to generate the feedback force of the handle, thereby providing tactile feedback to the user. The position state equation and the force feedback state equation are input into the Kalman state predictor to obtain the state prediction matrix. The Kalman predictor is a recursive algorithm that can estimate the system state in the presence of noise interference. It continuously updates the predicted value of the system state and combines it with the observed data to gradually reduce the prediction error. The state update equation of the Kalman prediction is:

[0136] ;

[0137] ;

[0138] in, Indicates that at time step The predicted value of the state, is the forecast error covariance matrix, is the process noise covariance matrix, which describes the influence of random noise in the system. Through the prediction process of the Kalman filter, a state prediction matrix containing state information such as position and velocity is obtained. The state prediction matrix is ​​subjected to state variable separation and time series expansion operations to obtain a delayed prediction sequence. Since there is a certain delay in the signal processing and response of the system in actual operation, the predicted state is time-expanded so that the state changes of the system can still be accurately described in future time steps. The purpose of state variable separation is to separate different state variables such as position, velocity and feedback force so that they can be analyzed and processed independently, while time series expansion is to extend these state variables in time to form a prediction sequence, providing the system with state estimates within a longer time range. A compensation model is constructed based on the delayed prediction sequence, and the state changes within the prediction delay time are accumulated to obtain the state compensation. The compensation model accumulates the state deviation generated during the delay time to compensate in advance, thereby reducing the error caused by the delay. The calculation of the state compensation is expressed as:

[0139] ;

[0140] in, is the state compensation, is the predicted state, For the actual state, by the delay time The state change is integrated to obtain the required state compensation. The state compensation is substituted into the position state equation and the force feedback state equation for state update to obtain the compensation state variable. The state update corrects the position and feedback force by introducing the compensation, allowing the system to maintain high response accuracy even in the presence of delay. The equation after the state update is:

[0141] ;

[0142] in, The updated state variables represent the position, velocity, and other states after compensation. Dynamic correction and boundary constraint processing are performed on the compensated state variables to obtain the dynamic compensation value. Dynamic correction eliminates the effects of nonlinear errors or external disturbances in the system by introducing additional correction factors. Boundary constraint processing is used to ensure that the compensated state variables are within a reasonable physical range. For example, the position should not exceed the maximum operating range of the handle, and the feedback force should not exceed the handle's tolerance limit. Through dynamic correction and boundary constraints, the final dynamic compensation value is ensured to be not only highly accurate but also meets the physical constraint requirements of the system.

[0143] In a specific embodiment, the process of executing step 500 may specifically include the following steps:

[0144] The dynamic compensation value and the operation behavior feature set are combined to obtain a parameter optimization data set;

[0145] Build a user operation habit model for the parameter optimization data set, and obtain operation characteristic parameters through analysis and calculation of operation speed, force and duration;

[0146] Initialize the model parameters based on the operational characteristic parameters to obtain the initial model parameters, and input the initial model parameters into the gradient descent optimizer. Calculate the parameter gradient through backpropagation to obtain the parameter update direction.

[0147] Adaptively adjust the step size of the parameter update direction and perform momentum accumulation calculation to obtain the parameter iteration amount;

[0148] Compare and calculate the error between the parameter iteration amount and the current parameter, store the parameters whose error is less than the threshold, and obtain the optimized parameter group;

[0149] The position error and force feedback error of the optimized parameter group are verified and calculated to obtain the control parameter group, and the control parameter group is parameter-assigned according to the control execution cycle to obtain the real-time control parameters.

[0150] Specifically, the dynamic compensation value is combined with the operation behavior feature set to jointly describe the user's action characteristics and system response during the operation of the game controller. The dynamic compensation value represents the dynamic response correction of the controller under force and feedback, while the operation behavior feature set covers the user's action characteristics, including position, speed, acceleration, etc. By combining these two types of information, a parameter optimization data set is obtained. Let the dynamic compensation value be , the operational behavior feature set is , then the parameter optimization data set Expressed as:

[0151] ;

[0152] in, and F are a collection of multiple variables. The combined D contains the user's operating behavior and the dynamic feedback information of the handle. A user operating habit model is constructed based on the parameter optimization dataset. The user operating habit model aims to capture the characteristics of the user when operating the handle. These characteristics include the speed, force, and duration of the operation, and use this information to describe the user's operating style and habits. The operation speed is calculated by taking the derivative of the position change in the time series. The formula is as follows:

[0153] ;

[0154] in, Indicates that at time step The operating speed at and are the handle positions of adjacent time steps, is the time interval. The operation force is directly obtained from the pressure sensor and is set to , duration indicates the length of time the user maintains an action, By analyzing and calculating the speed, force and duration, the operating characteristic parameters are obtained. , describes the personalized characteristics of the user when operating the handle. Based on the operation characteristic parameters, the model parameters are initialized to obtain the initial model parameters. Reasonable initialization enables the model to converge quickly in the subsequent optimization process and avoid falling into the local optimum. Let the model parameters be , then the parameters after initialization are expressed as:

[0155] ;

[0156] in, are the initialized model parameters, Indicates operating characteristic parameters Initialization function. After the initialization is completed, the model initial parameters Input into the gradient descent optimizer to calculate the gradient of the parameters through the back-propagation algorithm, thereby determining the direction of parameter update. Gradient descent gradually reduces the loss function by continuously adjusting the parameters. The gradient of the loss function is expressed as:

[0157] ;

[0158] in, is the loss function, which represents the error between the model prediction and the actual value. The loss function parameter The gradient of is used to determine the adjustment direction of the model under the current parameters. According to the calculated gradient, the direction of updating the parameters is the negative gradient direction, that is:

[0159] ;

[0160] in, is the learning rate, which controls the step size of parameter updates. To improve the efficiency and stability of gradient descent, the step size in the parameter update direction is adaptively adjusted, and momentum accumulation is performed to obtain the parameter iteration amount. Adaptive step size adjustment is achieved through the Adam optimization algorithm, and momentum accumulation smoothes parameter updates by introducing a momentum term. The formula for momentum accumulation is:

[0161] ;

[0162] ;

[0163] in, is momentum, is the momentum decay coefficient, which indicates the degree of retention of historical gradients. is the parameter update amount after iteration. By introducing momentum, oscillation is reduced, making the parameter update smoother and accelerating convergence. The error between the parameter iteration amount and the current parameter is compared to determine whether the parameter is close to the optimal value. By comparing the error between the current parameter and the target value, if the error is less than the preset threshold, the parameter is considered to have reached the optimal value and is stored. The error comparison formula is:

[0164] ;

[0165] like ,in If the value of the parameter exceeds the set threshold, the parameter is considered to have reached the optimal value and is stored in the optimized parameter group. The position error and force feedback error are verified and calculated for the optimized parameter group to obtain the final control parameter group. The purpose of position error verification is to check whether the position signal of the handle under the current parameter setting is consistent with the expected target, while the force feedback error verification is to check whether the feedback force given by the handle to the user meets the design requirements. The position error is expressed as:

[0166] ;

[0167] in, is the actual location, is the target position, is the number of samples. The force feedback error is calculated in a similar way:

[0168] ;

[0169] By calculating the position error and force feedback error, the effectiveness of the current optimized parameter group is verified, and finally the control parameter group is obtained. The control parameter group ensures that the handle can accurately respond to the user's actions during the user operation and provide appropriate force feedback, thereby improving the user experience. The control parameter group is assigned parameters according to the control execution cycle to obtain real-time control parameters. The control execution cycle is set to update the control parameters of the handle within each fixed time period to ensure that the handle always maintains the best control state during the operation. Let the control execution cycle be , the real-time control parameters are , then the control parameter update of each cycle is expressed as:

[0170] ;

[0171] Through periodic updates, the real-time control parameters of the handle are dynamically adjusted according to the user's operating habits and the system's response characteristics to ensure the accuracy of operation and real-time response.

[0172] In a specific embodiment, the process of executing step 600 may specifically include the following steps:

[0173] Performing first signal separation and second signal separation on the real-time control parameter to obtain a first position control parameter and a first force feedback control parameter;

[0174] Inputting the first position control parameter into a first frequency converter for signal modulation to obtain a second position control parameter, and inputting the first force feedback control parameter into a second frequency converter for signal modulation to obtain a second force feedback control parameter;

[0175] Performing pulse width modulation and amplitude conversion on the second position control parameter through the first driving module to obtain a first execution signal, and performing current intensity modulation and direction control on the second force feedback control parameter through the second driving module to obtain a second execution signal;

[0176] Inputting the first execution signal and the second execution signal into the timing processing unit for bidirectional signal synchronization and priority determination to obtain a third execution signal;

[0177] The third execution signal is wirelessly encoded and transmitted with intensity modulated by the third driving module to obtain a fourth execution signal, and the fourth execution signal is instruction parsed and dispatched according to the execution priority and timing requirements to obtain a target execution instruction.

[0178] Specifically, the real-time control parameter is subjected to first signal separation and second signal separation to obtain the first position control parameter and the first force feedback control parameter. Assume that the real-time control parameter is , including position control signals and force feedback signals , the first position control parameter and the first force feedback control parameter are obtained through signal separation operation:

[0179] ;

[0180] Among them, Split Represents the signal separation operation, is the first position control parameter, is the first force feedback control parameter. The signal is input to the first frequency converter for signal modulation to obtain the second position control parameter. The function of the frequency converter is to adjust the frequency characteristics of the signal to make it more suitable for the subsequent drive module operation. Signal modulation is achieved by frequency up-conversion, and its basic expression is:

[0181] ;

[0182] in, is the second position control parameter after modulation, is the modulation frequency, is a time variable. By frequency modulating the first position control parameter, it is adjusted to a frequency band suitable for the drive module, thereby ensuring that the signal can be effectively transmitted and processed. Similarly, the first force feedback control parameter Input to the second frequency converter for modulation to obtain the second force feedback control parameter , its modulation method is similar to the position control signal:

[0183] ;

[0184] By modulating the force feedback signal and adjusting it to a specific frequency, it can be transmitted and processed in a different frequency band from the position control signal, thus avoiding signal interference. After completing the signal modulation, the first drive module adjusts the second position control parameter Pulse width modulation (PWM) and amplitude conversion are performed to obtain the first execution signal. The purpose of pulse width modulation is to convert the signal into a series of pulses with different duty cycles, thereby controlling the average energy of the signal. The expression is:

[0185] ;

[0186] in, is the first execution signal, Amplitude conversion factor, used to adjust the amplitude of the signal, PWM Indicates the pulse signal obtained by pulse width modulation of the second position control parameter. In this way, the continuous control signal is converted into a discrete pulse signal, which is convenient for transmission and control through the drive module. At the same time, the second force feedback control parameter is modulated by the second drive module. Perform current intensity modulation and direction control to obtain the second execution signal The purpose of current intensity modulation is to control the current intensity according to the magnitude of the force feedback signal, thereby adjusting the magnitude of the feedback force. Its expression is:

[0187] ;

[0188] in, is the maximum current intensity, is the maximum value of the force feedback signal, The direction control function of the signal determines the direction of the current, thereby controlling the direction of the feedback force. Through current intensity modulation and direction control, it can ensure that the intensity and direction of the force feedback signal can accurately reflect the user's operation requirements, thereby providing appropriate force feedback. and the second execution signal Input to the timing processing unit for bidirectional signal synchronization and priority determination to obtain the third execution signal The purpose of bidirectional signal synchronization is to ensure that the position control signal and the force feedback signal are coordinated in time to avoid operation delays or feedback distortion caused by timing asynchrony. Synchronization is achieved by calculating the delay between the two signals. The basic formula is:

[0189] ;

[0190] in, To achieve the best delay time, synchronization is achieved by minimizing the difference between signals. The purpose of priority determination is to select between multiple control signals to ensure that the control signal with greater impact on user experience is executed first during critical operations. For example, during high-speed steering, the position control signal needs to be responded to first, while the force feedback signal needs to be responded to first during fine adjustments. The third execution signal obtained through synchronization and priority determination of the timing processing unit is: It is a comprehensive control signal that contains coordinated control information of position and feedback. The third execution signal is wirelessly encoded and transmitted with intensity modulation by the third driving module to obtain the fourth execution signal. The purpose of wireless signal coding is to convert the control signal into a format suitable for wireless transmission, which is achieved through modulation and coding, such as frequency shift keying (FSK) or amplitude shift keying (ASK). Its basic expression is:

[0191] ;

[0192] in, represents the modulation of the third execution signal, The transmission intensity modulation factor (TIMF) controls the transmission strength of the wireless signal. Through coding and modulation, it ensures stable and efficient signal transmission in various transmission environments. The fourth execution signal is parsed and scheduled according to execution priority and timing requirements to obtain the target execution instruction. The purpose of instruction parsing is to separate the various components of the fourth execution signal to extract specific control information, such as the target position of the handle and the amount of feedback force required. Scheduling sends this control information to the corresponding actuator according to priority and timing requirements, ensuring that the corresponding action is executed at the correct time. The basic principle of scheduling is to schedule based on the importance and timing requirements of the control signal to ensure that important control actions are executed promptly, while lower-priority control actions are executed later. The resulting target execution instruction drives the handle's actuator to adjust position and perform feedback control, thereby realizing the user's intended operation.

[0193] The above describes the sensing method of the game controller in the embodiment of the present application. The following describes the sensing device 10 of the game controller in the embodiment of the present application. Figure 2 In one embodiment of the present application, the sensing device 10 of the game controller includes:

[0194] The synchronization module 11 is used to perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain raw motion data;

[0195] A mapping module 12 is used to perform feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence;

[0196] The recognition module 13 is used to input the action feature sequence into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set;

[0197] A calculation module 14 is used to calculate the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value;

[0198] The correction module 15 is used to perform gradient iterative calculation and error correction on the dynamic compensation value and the operation behavior feature set to obtain real-time control parameters;

[0199] The output module 16 is used to process the position control signal and the force feedback signal based on the real-time control parameters and output the target execution instruction.

[0200] Through the collaborative cooperation of the above components, by introducing a bidirectional cross-sensing mechanism and a multi-level deep convolutional neural network, efficient fusion and feature extraction of multimodal sensor data are achieved, thereby improving control accuracy; an adaptive learning algorithm is used to model user operating habits, enabling the control system to dynamically adjust parameters according to the characteristics of different players; a delay prediction and compensation mechanism based on Kalman filtering is designed to effectively solve the problem of wireless transmission delay, and control the position error and force feedback error within the preset range respectively; through gradient iterative optimization and error correction strategies, real-time updating and optimization of control parameters are ensured; the hierarchical drive and priority scheduling mechanism is adopted to ensure the precise execution of control instructions, greatly improving the control experience of the game controller.

[0201] See also Figure 3 , Figure 3 This is a schematic block diagram of the structure of an electronic device 300 provided in an embodiment of the present application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected via a device bus 303, wherein the memory 302 may include a non-volatile storage medium and an internal memory.

[0202] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 301, the processor 301 can execute any of the above-mentioned game controller sensing methods.

[0203] The processor 301 is used to provide computing and control capabilities to support the operation of the entire electronic device 300 .

[0204] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 301, the processor 301 can execute any of the above-mentioned game controller sensing methods.

[0205] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device 300 involved in the solution of the present application. The specific electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0206] It should be understood that the processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0207] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device 300 described above can refer to the corresponding process of the sensing method of the aforementioned game controller, and will not be repeated here.

[0208] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the sensing method of the game controller provided in the embodiment of the present application.

[0209] The computer-readable storage medium may be an internal storage unit of the electronic device 300 in the aforementioned embodiment, such as a hard disk or memory of the electronic device 300. The computer-readable storage medium may also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc., equipped with the electronic device 300.

[0210] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0212] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for sensing a game controller, characterized in that: The method comprises: The angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor and the pressure intensity data of the pressure sensor are digitally filtered and time-series synchronized to obtain original motion data; specifically comprising: performing 16-bit digital quantization processing on the X-axis, Y-axis and Z-axis angular velocity data collected by the gyroscope sensor to obtain digitized angular velocity data; performing 16-bit digital quantization processing on the X-axis, Y-axis and Z-axis motion data collected by the acceleration sensor to obtain digitized motion data; performing 12-bit digital quantization processing on the pressure intensity data collected by the pressure sensor to obtain digitized pressure data; performing high-frequency noise and low-frequency drift elimination on the digitized angular velocity data and the digitized motion data to obtain posture filter data, and performing data smoothing processing on the digitized pressure data to obtain force filter data; performing time mark alignment and difference compensation on the posture filter data and the force filter data to obtain aligned data, dividing the aligned data into continuous data segments according to the time window and performing boundary smoothing processing to obtain a segmented sequence, and performing data normalization and dimension unification processing on the segmented sequence to obtain original motion data; Performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; Inputting the action feature sequence into a multi-layer deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; Based on the operation behavior feature set, calculating the position state quantity and the force feedback state quantity to obtain a dynamic compensation value; Performing gradient iterative calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters; The position control signal and the force feedback signal are processed based on the real-time control parameters, and a target execution instruction is output.

2. The sensing method of the game controller according to claim 1, characterized in that: The performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence includes: dividing the raw motion data into an angular velocity data group, a motion data group, and a pressure data group according to sensor types; Constructing an intra-modal association unit, the intra-modal association unit including a feature extraction layer and a feature fusion layer, the feature extraction layer adopting a sliding time window structure, and the feature fusion layer adopting a multi-layer perceptron structure; the feature extraction layer extracts mean, variance, and peak features from the angular velocity data group, extracts mean, variance, and peak features from the motion data group, and extracts mean, duration, and rate of change features from the pressure data group; the feature fusion layer includes two fully connected layers, the first fully connected layer uses a ReLU activation function, and the second fully connected layer uses a Sigmoid activation function; Inputting the angular velocity data group, the motion data group, and the pressure data group into the intra-modal association unit to obtain a single-modal feature vector; Construct a bidirectional cross-perception mechanism, which includes a mapping layer, an association layer, and a self-preservation layer. The mapping layer adopts an attention mechanism structure, the association layer adopts a graph convolutional network structure, and the self-preservation layer adopts a memory network structure. The mapping layer calculates the weight matrix between different modalities through self-attention, the association layer uses a graph convolutional network to construct the topological relationship between modalities, and the self-preservation layer contains a long short-term memory network to store feature combinations with high recognition accuracy. The single-modal feature vector is input into the bidirectional cross-sensing mechanism, a cross-feature matrix is ​​obtained through feature mapping and association learning, and the cross-feature matrix is ​​subjected to time series reconstruction and structural processing to obtain an action feature sequence.

3. The sensing method of the game controller according to claim 2, characterized in that: The action feature sequence is input into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set, including: dividing the action feature sequence into a time domain feature data block and a space domain feature data block; Construct a multi-level deep convolutional neural network, which includes a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure, and the feature recognition network adopts a multi-scale fusion structure. The feature extraction network includes three convolution modules. The first convolution module includes two convolution layers, each convolution layer uses a 3×3 convolution kernel with a stride of 2 and a BatchNorm layer. The second convolution module includes three convolution layers, each convolution layer uses a 3×3 convolution kernel with a stride of 2 and a ReLU activation function. The third convolution module includes three convolution layers, each convolution layer uses a 3×3 convolution kernel with a stride of 1 and a LeakyReLU activation function. Inputting the time domain feature data block into the first convolution module of the feature extraction network, and obtaining a time domain feature map through multi-scale convolution operation, and inputting the spatial domain feature data block into the second and third convolution modules of the feature extraction network, and obtaining a spatial domain feature map through multi-level convolution operation; Construct a feature recognition network, wherein the feature recognition network includes two fully connected layers and one classification layer, the number of neurons in the fully connected layers is 512 and 256 respectively, and the classification layer uses a Softmax function; The time domain feature map and the spatial domain feature map are input into the feature recognition network, an action category vector is obtained through feature fusion and classification operations, and the action category vector is subjected to probability distribution calculation and label mapping processing to obtain an operation behavior feature set.

4. The sensing method of the game controller according to claim 3, characterized in that: The calculating of the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain the dynamic compensation value includes: dividing the operation behavior feature set into position state features and force feedback state features according to state attributes; A second-order dynamic model of the handle position is established according to the position state characteristics, and the three state variables of position, velocity and acceleration are correlated and calculated to obtain the position state equation; Establishing a spring damping force feedback model according to the force feedback state characteristics, performing force response calculation on the position deviation and velocity, and obtaining a force feedback state equation; Inputting the position state equation and the force feedback state equation into a Kalman state predictor to obtain a state prediction matrix; performing state variable separation and time series expansion operations on the state prediction matrix to obtain a delay prediction sequence; Building a compensation model based on the delay prediction sequence, and performing cumulative calculation on the state change amount within the prediction delay time to obtain the state compensation amount; The state compensation amount is substituted into the position state equation and the force feedback state equation for state update to obtain a compensation state variable, and the compensation state variable is subjected to dynamic correction and boundary constraint processing to obtain a dynamic compensation value.

5. The sensing method of the game controller according to claim 4, characterized in that: The step of performing gradient iterative calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters includes: Combining the dynamic compensation value and the operational behavior feature set to obtain a parameter optimization data set; Constructing a user operation habit model for the parameter optimization data set, and obtaining operation characteristic parameters through analysis and calculation of operation speed, force and duration; Initializing the model parameters based on the operational characteristic parameters to obtain the model initial parameters, and inputting the model initial parameters into the gradient descent optimizer, calculating the parameter gradients through back propagation, and obtaining the parameter update direction; Adaptively adjusting the step size of the parameter update direction and performing momentum accumulation calculation to obtain the parameter iteration amount; Compare and calculate the error between the parameter iteration amount and the current parameter, store the parameters whose error is less than the threshold, and obtain the optimized parameter group; Verification calculations of position error and force feedback error are performed on the optimized parameter group to obtain a control parameter group, and parameter allocation is performed on the control parameter group according to a control execution cycle to obtain real-time control parameters.

6. The sensing method of the game controller according to claim 5, characterized in that: The processing of the position control signal and the force feedback signal based on the real-time control parameter and outputting the target execution instruction includes: Performing first signal separation and second signal separation on the real-time control parameter to obtain a first position control parameter and a first force feedback control parameter; Inputting the first position control parameter into a first frequency converter for signal modulation to obtain a second position control parameter, and inputting the first force feedback control parameter into a second frequency converter for signal modulation to obtain a second force feedback control parameter; performing pulse width modulation and amplitude conversion on the second position control parameter by a first driving module to obtain a first execution signal, and performing current intensity modulation and direction control on the second force feedback control parameter by a second driving module to obtain a second execution signal; Inputting the first execution signal and the second execution signal into a timing processing unit for bidirectional signal synchronization and priority determination to obtain a third execution signal; The third execution signal is wirelessly encoded and transmitted with intensity modulated by the third driving module to obtain a fourth execution signal, and the fourth execution signal is instruction parsed and scheduled according to the execution priority and timing requirements to obtain a target execution instruction.

7. A sensing device for a game controller, characterized in that: A sensing method for executing a game controller according to any one of claims 1 to 6, wherein the sensing device of the game controller comprises: The synchronization module is used to digitally filter and synchronize the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data; A mapping module, configured to perform feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; A recognition module is used to input the action feature sequence into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; A calculation module, configured to calculate a position state quantity and a force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value; a correction module, configured to perform gradient iterative calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters; The output module is used to process the position control signal and the force feedback signal based on the real-time control parameters and output the target execution instruction.

8. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the electronic device to execute the sensing method of the game controller according to any one of claims 1 to 6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the sensing method of the game controller according to any one of claims 1 to 6 is implemented.

Images