Sensing method, device and equipment of gamepad and storage medium

Abstract

The invention relates to the technical field of gamepad induction, and discloses a gamepad induction method, device and equipment and a storage medium, the method comprises the following steps: carrying out digital filtering and time sequence synchronization on angular velocity data of a gyroscope sensor, motion data of an acceleration sensor and pressing force data of a pressure sensor to obtain original motion data; performing feature segmentation and modal cross mapping to obtain an action feature sequence; inputting a multi-level deep convolutional neural network to perform spatio-temporal feature extraction and pattern recognition to obtain an operation behavior feature set; calculating to obtain a dynamic compensation value; carrying out gradient iterative operation and error correction to obtain real-time control parameters; and the position control signal and the force feedback signal are processed based on the real-time control parameters, and the target execution instruction is output, so that accurate execution of the control instruction is ensured, and the control experience of the gamepad is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of game controller sensing technology, and in particular to a game controller sensing method, device, equipment and storage medium. Background Art

[0002] Traditional game controllers mainly rely on single button and joystick input, which is difficult to meet the requirements of modern games for accuracy and immersion. Although the new generation of game controllers have introduced multiple sensing elements such as gyroscopes, accelerometers and pressure sensors, the effective fusion and precise control of these sensor data still face challenges.

[0003] Existing game controller sensing methods often use simple data fusion strategies and fail to fully utilize the complex correlation between multimodal sensor data. Especially when dealing with high-speed actions and fine operations, the noise, delay and operating habits of different players in sensor data will affect the accuracy of control. In addition, random delay problems in wireless transmission will also lead to instability in position control and force feedback response, affecting the consistency of the game experience. Summary of the invention

[0004] The present application provides a sensing method, device, equipment and storage medium for a game controller, thereby ensuring the accurate execution of control instructions and greatly improving the control experience of the game controller.

[0005] A first aspect of the present application provides a sensing method for a game controller, the sensing method for the game controller comprising: Perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data; Performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; Inputting the action feature sequence into a multi-layer deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; Based on the operation behavior feature set, calculating the position state quantity and the force feedback state quantity to obtain a dynamic compensation value; Performing gradient iteration calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters; The position control signal and the force feedback signal are processed based on the real-time control parameters, and a target execution instruction is output.

[0006] A second aspect of the present application provides a sensing device for a game controller, the sensing device for the game controller comprising: A synchronization module is used to perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data; A mapping module, used for performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; A recognition module, used for inputting the action feature sequence into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; A calculation module, used to calculate the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value; A correction module, used for performing gradient iteration calculation and error correction on the dynamic compensation value and the operation behavior feature set to obtain a real-time control parameter; The output module is used to process the position control signal and the force feedback signal based on the real-time control parameters and output the target execution instruction.

[0007] A third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned game controller sensing method.

[0008] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned game controller sensing method.

[0009] Compared with the prior art, the present application has the following beneficial effects: by introducing a bidirectional cross-sensing mechanism and a multi-level deep convolutional neural network, efficient fusion and feature extraction of multimodal sensor data are achieved, thereby improving control accuracy; an adaptive learning algorithm is used to model user operating habits, so that the control system can dynamically adjust parameters according to the characteristics of different players; a delay prediction and compensation mechanism based on Kalman filtering is designed, which effectively solves the problem of wireless transmission delay and controls the position error and force feedback error within a preset range respectively; through gradient iterative optimization and error correction strategies, real-time updating and optimization of control parameters are ensured; a hierarchical drive and priority scheduling mechanism is adopted to ensure the precise execution of control instructions, greatly improving the control experience of the game controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0012] Figure 1 is a flow chart of a sensing method for a game controller provided by an embodiment of the present invention; Figure 2 is a schematic block diagram of the structure of a sensing device of a game controller provided by an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0015] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0016] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 , an embodiment of the sensing method of the game controller in the embodiment of the present application includes: Step 100: digitally filter and time-series synchronize the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain original motion data; It is understandable that the execution subject of the present application may be a sensing device of a game controller, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0017] Specifically, the X-axis, Y-axis, and Z-axis angular velocity data collected by the gyroscope sensor are subjected to 16-bit digital quantization processing, and the analog continuous signal is converted into a discrete digital signal to obtain digital angular velocity data. At the same time, the X-axis, Y-axis, and Z-axis motion data collected by the acceleration sensor are subjected to 16-bit digital quantization processing to obtain digital motion data. The pressing force data collected by the pressure sensor is subjected to 12-bit digital quantization processing to obtain digital pressure data. The selection of 12-bit digitization is based on the characteristics of the pressing force data. Considering the relative range of the pressing force change and its impact on the operation, the 12-bit quantization not only ensures the accuracy of the measurement, but also meets the requirements of hardware processing efficiency. The digitized angular velocity data and the digitized motion data are subjected to high-frequency noise and low-frequency drift elimination to obtain attitude filtering data. By eliminating high-frequency noise, more stable and reliable attitude information is obtained. At the same time, since long-term measurement will cause low-frequency drift in the sensor output, this drift will cause errors in the measurement of position and attitude, and low-frequency drift correction is performed to ensure data accuracy. At the same time, the digital pressure data is smoothed to reduce the impact of instantaneous spikes and jitters caused by the user's pressure changes on the handle operation, effectively filter out small fluctuations during the operation, make the pressure signal more stable, improve the response consistency of the system, and obtain force filtering data. Time mark alignment and difference compensation are performed on the posture filter data and the force filter data. Through the time mark alignment operation, data from different sources are synchronously processed on the same time axis to ensure that these data can accurately reflect the action state of the handle at the same time. At the same time, in order to solve the time difference caused by different sampling rates or processing processes, difference compensation is performed to reduce the error caused by time difference and obtain more accurate alignment data. The aligned data is divided into continuous data segments according to the preset time window, and the boundary of each data segment is smoothed. By dividing the data into continuous time window segments, the action state of the handle is finely sliced ​​in time, so that the data in each time window can be analyzed independently to achieve accurate tracking of the action. The boundary smoothing process can reduce the discontinuity between time windows, ensure that the data transition is more natural and smooth, and avoid the impact of the instantaneous error introduced by the change of the window boundary on subsequent calculations. The segmented data sequence is normalized and dimensionally unified to obtain the original motion data. Data from different sources are represented on the same scale to avoid deviations caused by different numerical ranges. At the same time, dimension unification is to fuse multi-source data into the same data structure so that subsequent pattern recognition and feature extraction steps can effectively utilize these multi-dimensional information to obtain the original motion data for control and feedback.

[0018] Step 200: performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; Specifically, the original motion data is divided into angular velocity data group, motion data group and pressure data group according to the sensor type, so as to extract and analyze the characteristics of each type of data independently. Different sensor data have different dynamic characteristics and physical meanings. The angular velocity data group contains the three-axis angular velocity information measured by the gyroscope sensor, the motion data group consists of the three-axis motion information measured by the acceleration sensor, and the pressure data group contains the player's operation force information on the button. Construct an intra-modal association unit for feature extraction and fusion of these different data groups. The intra-modal association unit includes a feature extraction layer and a feature fusion layer. The feature extraction layer adopts a sliding time window structure. By setting a sliding time window, it dynamically calculates local features of the angular velocity, motion and pressure data to capture the subtle changes in the action during the operation. For the angular velocity data group and the motion data group, the feature extraction layer extracts the mean, variance and peak features respectively. The mean is used to describe the overall action trend of the handle in a specific period of time, the variance reflects the volatility of the action, and the peak is used to capture the extreme changes of the action. For the pressure data set, the extracted features include mean, duration and rate of change. The mean reflects the average force of the player's pressing, the duration represents the length of time of pressing, and the rate of change is used to characterize the speed of the player's pressure application. The data after feature extraction is fused, and a feature fusion layer is set in the intra-modal association unit. The feature fusion layer adopts a multi-layer perceptron structure and contains two fully connected layers. The first fully connected layer uses the ReLU activation function, which can introduce nonlinear characteristics, so that the fused features can better express complex relationships. The second fully connected layer uses the Sigmoid activation function. By normalizing the output, the eigenvalue is within a limited range, controlling the range of the eigenvalue, preventing extreme values ​​from affecting the training of the model, and facilitating the stability of subsequent calculations. After feature extraction and feature fusion, the angular velocity data set, the motion data set and the pressure data set are respectively input into the intra-modal association unit to obtain single-modal feature vectors, which respectively describe the characteristics of the handle action under each mode. A bidirectional cross-perception mechanism is constructed to fuse and map features of different modalities. The bidirectional cross-perception mechanism includes a mapping layer, an association layer, and a self-preservation layer. The mapping layer adopts an attention mechanism structure, and the weight matrix between different modalities is calculated through the self-attention mechanism. The self-attention mechanism dynamically assigns weights according to the importance of different modal features, thereby highlighting more meaningful features and suppressing irrelevant or redundant information; the association layer adopts a graph convolutional network structure. By constructing different modal data in the form of a graph, the association layer can effectively capture the topological relationship between modalities through graph convolution. The structured relationship helps identify the potential connection and synergy between the modalities; the self-preservation layer adopts a long short-term memory network (LSTM) structure to store and maintain feature combinations with high recognition accuracy.LSTM can retain important information in time series data and forget unimportant parts, ensuring that effective feature combinations can still be accurately captured during long-term operations. The unimodal feature vector is input into the bidirectional cross-sensing mechanism, and the cross-feature matrix is ​​obtained through the attention calculation of the mapping layer, the graph convolution processing of the association layer, and the memory network of the self-preservation layer. The cross-feature matrix is ​​reconstructed and structured in a temporal manner, and the feature matrix is ​​reorganized so that it can not only represent the temporal information of the feature itself, but also effectively reflect the relationship and interaction characteristics between different features, and finally obtain the action feature sequence.

[0019] Step 300: Input the action feature sequence into a multi-layer deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; It should be noted that the action feature sequence is divided into time domain feature data blocks and spatial domain feature data blocks, and the network is used to independently extract and learn data of different dimensions. A multi-level deep convolutional neural network is constructed, which consists of a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure, and captures the detailed characteristics and overall trends in the feature data through multi-level convolution operations to achieve efficient extraction of spatiotemporal features. The feature extraction network contains three convolution modules, each of which processes different types of input data. The first convolution module contains two convolution layers, each of which uses a 3×3 convolution kernel with a step size of 2 to effectively downsample the input features, thereby capturing features in a larger receptive field. At the same time, the first convolution module contains a BatchNorm layer for normalizing the convolutional data. The introduction of BatchNorm can accelerate the training speed of the network, effectively prevent gradient disappearance and gradient explosion, and improve the stability and generalization ability of the network. When processing spatial features, the second convolution module of the feature extraction network contains three convolution layers, each of which also uses a 3×3 convolution kernel with a step size of 2, and introduces the ReLU activation function. The nonlinear characteristics of the ReLU activation function enable the network to learn more complex and diverse features. The third convolution module contains three convolution layers, the convolution kernel is still 3×3, but the step size is adjusted to 1, and the LeakyReLU activation function is used. Compared with the standard ReLU, LeakyReLU has better characteristics. It retains a part of the gradient when the input is less than zero, alleviates the "neuron death" problem of ReLU during training, improves the model's responsiveness to negative features, and enables the convolution network to learn the information in the features more comprehensively. In the input stage, the time domain feature data block is input into the first convolution module of the feature extraction network. After multi-scale convolution operations, the time domain feature map is obtained, which describes the change law of the action in the time dimension and can reflect the dynamic behavior characteristics of the handle during the entire operation process. At the same time, the spatial feature data block is input into the second and third convolution modules of the feature extraction network. The spatial feature map is obtained through multi-level convolution operations, which describes the distribution characteristics and local changes of the action in the spatial dimension, and can effectively capture the operation characteristics in different directions and positions. Construct a feature recognition network, which contains two fully connected layers and one classification layer. The number of neurons in the fully connected layer is 512 and 256 respectively, which gradually reduces the dimension of the feature vector and realizes the effective dimensionality reduction and abstraction of high-dimensional features. Through two fully connected layers, the network can fuse information from different feature maps, so that the time domain and spatial domain features can be integrated and understood at the global level.The classification layer uses the Softmax activation function to convert the network output into a probability distribution, thereby normalizing each category so that the sum of the probabilities of all categories is 1. This output form is convenient for judging the category of the action and gives the probability value of each category, indicating the network's confidence in each action category. The time domain feature map and the spatial domain feature map are input into the feature recognition network, and the action category vector is obtained through feature fusion and classification operations. This vector represents the type of the current operation behavior, and the probability distribution of each category is calculated through the probability output of the Softmax layer. The action category vector is subjected to probability distribution calculation and label mapping processing to obtain the operation behavior feature set.

[0020] Step 400: Calculate the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value; Specifically, the operation behavior feature set is divided into position state features and force feedback state features according to the state attributes, effectively distinguishing the features related to the handle position and the features related to the force feedback, and optimizing the two types of features respectively. The position state features describe the position, velocity and acceleration of the handle in space, while the force feedback state features describe the factors related to the force, such as the intensity and direction of the feedback. A second-order dynamic model of the handle position is established based on the position state features. The second-order dynamic model can effectively describe the change law of the handle position during different operations. The position state equation is established by correlating the three state variables of position, velocity and acceleration. The position state equation describes how the position of the handle evolves over time and how velocity and acceleration affect the current state of the handle. Through this equation, the motion state of the handle at different time points is described and predicted. At the same time, a spring damping force feedback model is established based on the force feedback state features. The model is used to describe the force feedback received by the handle during the operation. The spring damping model can simulate the force feedback in actual operation. The force feedback state equation is obtained by calculating the force response of the position deviation and velocity. In the force feedback state equation, the position deviation reflects the degree of deviation between the current position of the handle and the target position, while the speed indicates how fast the handle moves. The feedback force related to these states is calculated through the spring damping model. The position state equation and the force feedback state equation are input into the Kalman state predictor to obtain the state prediction matrix. The Kalman predictor is an effective tool for state estimation. Especially in the case of noise interference, the Kalman predictor can obtain an accurate prediction of the system state by continuously updating and correcting the state estimate. By inputting the position state equation and the force feedback state equation, the Kalman predictor combines the current state variables and historical observation data to predict the future state of the handle to form a state prediction matrix, which contains the state information at multiple time points in the future. The state prediction matrix is ​​separated and time-series expanded to obtain a delayed prediction sequence. Since there is a certain processing delay in the actual control process, the predicted state is gradually expanded by time-series expansion to accurately describe the change of the system state during the delay time and obtain the delayed prediction sequence. Based on the delayed prediction sequence, a compensation model is constructed to accumulate the state change amount during the predicted delay time to obtain the state compensation amount. Through the compensation model, the state deviation of the handle during the delay time is pre-estimated and accumulated so as to be corrected in the subsequent control. The state compensation is a compensation for the deviation of the system during the processing delay time. By substituting the state compensation into the position state equation and the force feedback state equation, the state of the system is updated to obtain the compensated state variable. The compensated state variable can effectively correct the control deviation caused by the delay and improve the response accuracy and stability of the handle. The compensated state variable is subjected to dynamic correction and boundary constraint processing to obtain the dynamic compensation value.Dynamic correction eliminates the cumulative errors that occur in the system's state estimation and control process by introducing additional correction factors, while boundary constraint processing is used to ensure that the compensated state variables are within a reasonable physical range. For example, the strength of force feedback should not exceed the maximum tolerance of the handle, and the position state should not exceed the operable boundary of the handle.

[0021] Step 500, performing gradient iteration calculation and error correction on the dynamic compensation value and the operation behavior feature set to obtain real-time control parameters; Specifically, the dynamic compensation value and the operation behavior feature set are combined to obtain a parameter optimization data set. Based on the parameter optimization data set, a user operation habit model is constructed. By analyzing and calculating the operation speed, force and duration, the characteristic parameters of the user when using the handle are extracted. The operation speed reflects the speed of the user's handle movement, while the force describes the intensity of the user's pressing the button, and the duration represents the length of time the user maintains a certain action. By comprehensively analyzing these characteristics, the operation characteristic parameters are obtained to reflect the user's individualized usage and preferences. The model parameters are initialized based on the operation characteristic parameters to obtain the initial parameters of the model. These initial parameters are the basis for subsequent gradient iterative operations. Through reasonable initialization, the convergence speed of the model is accelerated and the optimization effect is improved. The initial parameters of the model are input into the gradient descent optimizer, and the parameter gradient is calculated by the back propagation algorithm to obtain the parameter update direction. In the process of back propagation, the update direction of each parameter in the current state is determined by calculating the partial derivatives of the loss function with respect to each parameter, so that the model can be continuously adjusted in the direction of reducing the error. The step size is adaptively adjusted in the parameter update direction so that the step size can be dynamically adjusted according to the change of the current gradient, thereby achieving more stable and efficient optimization in the iterative process. The adaptive adjustment of the step size prevents the optimal solution from being exceeded due to excessive step size when the error is small, and increases the step size to accelerate convergence when the error is large. At the same time, momentum accumulation calculation is performed on the parameter update process to obtain the parameter iteration amount. By retaining the momentum information of the previous round of updates, the direction of parameter update can be smoother, thereby reducing the jitter problem caused by the drastic change of the gradient, so that the model can converge to the optimal solution more stably during the optimization process. The parameter iteration amount is compared with the current parameter for error calculation, and the parameters with errors less than the set threshold are stored to obtain the optimized parameter group. The position error and force feedback error are verified and calculated for the optimized parameter group. The position error verification is to detect whether the position response of the handle in actual operation is accurate, while the force feedback error verification is to check whether the force feedback given to the user by the handle meets the expectations. By verifying the position error and force feedback error, it is ensured that the optimized parameters not only meet the requirements in terms of numerical value, but also provide a good user experience in actual operation. Through verification, a control parameter group is obtained, which includes parameters related to the handle position control and force feedback control parameters to ensure that the user can feel more accurate and real feedback when operating the handle. The control parameter group is assigned parameters according to the control execution cycle to obtain real-time control parameters. At each fixed time interval, the operation of the handle is updated and adjusted to ensure the real-time nature of the control. In this way, the real-time control parameters are dynamically adjusted according to the user's operating habits and the physical characteristics of the handle to provide the best response and feedback effect.

[0022] Step 600: Process the position control signal and the force feedback signal based on the real-time control parameters, and output a target execution instruction.

[0023] Specifically, the real-time control parameter is subjected to first signal separation and second signal separation to obtain a first position control parameter and a first force feedback control parameter. The first position control parameter is used to describe the position change requirements of the handle, while the first force feedback control parameter describes the force feedback that the handle needs to give to the player during operation. The first position control parameter is input into the first frequency converter for signal modulation, and the frequency of the control signal is adjusted to make it more suitable for the input requirements of the driving module to obtain the second position control parameter. Similarly, the first force feedback control parameter is input into the second frequency converter for signal modulation to obtain the second force feedback control parameter. The second position control parameter is pulse-width modulated and amplitude-converted by the first driving module to obtain the first execution signal. Pulse width modulation is a control method that achieves precise control of the handle position by adjusting the average energy of the pulse width control signal. The purpose of amplitude conversion is to adjust the amplitude of the signal to meet the driving requirements of the execution unit and ensure that the position change of the handle can accurately reflect the user's operation intention. At the same time, the second force feedback control parameter is current-intensity modulated and direction-controlled by the second driving module to obtain the second execution signal. Current intensity modulation is used to control the intensity of the force feedback signal so that the player can feel different degrees of feedback force, while direction control is used to determine the direction of the feedback force so as to provide appropriate feedback in different operation scenarios of the player. The first execution signal and the second execution signal are input into the timing processing unit for two-way signal synchronization and priority determination to obtain a third execution signal. The two-way signal synchronization is to ensure that the position control signal and the force feedback signal can be coordinated in time to avoid operation delays or feedback distortion problems caused by signal asynchrony. Priority determination is to coordinate between multiple control signals to ensure that the control signal with a greater impact on the game experience is executed first during key operations to enhance the user's operation experience. Through processing, the timing processing unit generates a comprehensive third execution signal, which includes synchronized position information and force feedback information. The third execution signal is wirelessly encoded and transmitted with intensity modulation by the third driving module to obtain a fourth execution signal. Wireless signal encoding is to convert the execution signal into a format suitable for wireless transmission so that effective communication can be carried out between the handle and the control terminal. Transmission intensity modulation is used to adjust the transmission intensity of the signal to ensure that the signal can be transmitted stably in different transmission environments, such as increasing the transmission intensity in an environment with interference, and reducing the intensity in an environment with good signals to save energy. The fourth execution signal is parsed and scheduled according to the execution priority and timing requirements to obtain the target execution instruction. Instruction parsing is to separate the various parts of the fourth execution signal and extract specific control information, such as the target position of the handle, the amount of feedback force to be applied, etc. Scheduling and allocation is to send the control information to the corresponding actuator according to the execution priority and timing requirements, so as to perform the corresponding action at the right time.

[0024] In the embodiments of the present application, by introducing a bidirectional cross-sensing mechanism and a multi-level deep convolutional neural network, efficient fusion and feature extraction of multimodal sensor data are achieved, thereby improving control accuracy; an adaptive learning algorithm is used to model user operating habits, so that the control system can dynamically adjust parameters according to the characteristics of different players; a delay prediction and compensation mechanism based on Kalman filtering is designed to effectively solve the problem of wireless transmission delay, and control the position error and force feedback error within a preset range respectively; through gradient iterative optimization and error correction strategies, real-time updating and optimization of control parameters are ensured; a hierarchical drive and priority scheduling mechanism is adopted to ensure the precise execution of control instructions, greatly improving the control experience of the game controller.

[0025] In a specific embodiment, the process of executing step 100 may specifically include the following steps: The X-axis, Y-axis, and Z-axis angular velocity data collected by the gyroscope sensor are subjected to 16-bit digital quantization processing to obtain digital angular velocity data; the X-axis, Y-axis, and Z-axis motion data collected by the acceleration sensor are subjected to 16-bit digital quantization processing to obtain digital motion data; the pressing force data collected by the pressure sensor are subjected to 12-bit digital quantization processing to obtain digital pressure data; Eliminate high-frequency noise and low-frequency drift on the digitized angular velocity data and the digitized motion data to obtain attitude filter data, and perform data smoothing on the digitized pressure data to obtain force filter data; The posture filter data and the force filter data are time-stamped and difference-compensated to obtain aligned data. The aligned data are divided into continuous data segments according to the time window and the boundaries are smoothed to obtain a segmented sequence. The segmented sequence is then normalized and dimensionally unified to obtain the original motion data.

[0026] Specifically, the X-axis, Y-axis, and Z-axis angular velocity data collected by the gyroscope sensor are subjected to 16-bit digital quantization processing to obtain digital angular velocity data, the X-axis, Y-axis, and Z-axis motion data collected by the acceleration sensor are subjected to 16-bit digital quantization processing to obtain digital motion data, and the pressure intensity data collected by the pressure sensor are subjected to 12-bit digital quantization processing to obtain digital pressure data. The data collected by the gyroscope, acceleration sensor, and pressure sensor are all analog signals, expressed as voltage values, and the range is a specified interval, such as ±5V or ±3.3V. In order to convert these analog signals into digital signals, analog-to-digital conversion is performed, that is, a certain number of quantization bits are used to convert these voltage values ​​into discrete digital values. In this embodiment, the data quantization of the gyroscope and acceleration sensor adopts 16 bits, that is, the quantized value range is between 0 and 65535. The quantized angular velocity data is expressed as: ; in, Represents the angular velocity data collected by the gyroscope, and Respectively represent the lower and upper limits of the gyroscope's range, and is the quantized digital angular velocity data. Similarly, the quantization formula of acceleration data is expressed as: ; in, Represents the motion data collected by the acceleration sensor, and are the lower and upper limits of the acceleration range, It is the quantized digital motion data. For the pressure sensor, 12-bit quantization is used, and the formula is as follows: ; in, Indicates the collected pressure data. and Respectively represent the lower and upper limits of the pressure sensor, and Represents the quantized pressure data. After the data is quantized, the high-frequency noise and low-frequency drift of the digitized angular velocity data and acceleration data are eliminated to obtain the attitude filtering data. High-frequency noise is caused by external environmental interference and the instability of the sensor itself. For example, when the user shakes quickly, a large instantaneous jitter will be generated. In order to reduce these interferences, a low-pass filter is used. The low-pass filter is expressed by the following expression: ; in, Represents the original input data of the nth sampling point, Represents the output data after filtering, is the filter coefficient, which controls the strength of the filter and ranges from 0 to 1. When it is smaller, the low-pass filter has a stronger suppression effect on high-frequency noise. Low-frequency drift refers to the tendency of the output signal to gradually change when the sensor is used for a long time. This change will affect the accurate measurement of angular velocity and acceleration. In order to eliminate low-frequency drift, a high-pass filter is used. The formula is as follows: ; in, is the coefficient of the high-pass filter, and adjusting it controls the filtering effect. After processing high-frequency noise and low-frequency drift, stable posture filtering data is obtained. For the digitized pressure data, data smoothing is performed to eliminate instantaneous spikes and jitters caused by changes in user operation force. Smoothing is achieved through the sliding average method, and its formula is: ; in, represents the size of the sliding window, represents the pressure data of the i-th sampling point, Represents the smoothed pressure data. By using sliding average, the random jitter in the pressure data is effectively reduced, making the signal more stable. The posture filter data and the force filter data are time-stamped and the difference is compensated. The time-stamp alignment is achieved by interpolation, such as linear interpolation: ; in, and The known time points and The data on For the time point that needs to be interpolated, the time point is obtained by linear interpolation The data values ​​on the sensors are aligned to achieve alignment. At the same time, difference compensation is performed to eliminate the differences caused by the different response characteristics of the sensors so that the data can be better matched. After the time marker alignment is completed, the aligned data is divided into continuous data segments according to the time window, and the boundaries are smoothed to obtain a segmented sequence. The time window is divided by setting a time period of a fixed length, for example, the aligned data is divided into a window of 100 milliseconds, and the original continuous data is divided into multiple small segments. The boundary smoothing process is to reduce the discontinuity between time windows. The data at the boundary is processed in a manner similar to weighted averaging to ensure a more natural transition between windows. The formula is as follows: ; in, represents the original data at the boundary, is the corresponding weight. The larger the weight, the greater the impact of the data point on the smoothing result. The weighted average method can effectively reduce the data mutation at the window boundary. The segmented sequence is normalized and dimensionally unified to obtain the original action data. Data from different sources are represented on the same scale to avoid deviations caused by different value ranges. Normalization is achieved through the following formula: ; in, Represents the original data, is the mean of the data, is the standard deviation of the data. The normalized data has a distribution with a mean of 0 and a standard deviation of 1. Dimensional unification is to integrate different types of data into the same feature space for subsequent pattern recognition and feature extraction. For example, angular velocity, acceleration, and pressure data are spliced ​​into a multidimensional vector in a certain order to form the original motion data.

[0027] In a specific embodiment, the process of executing step 200 may specifically include the following steps: The original motion data is divided into an angular velocity data group, a motion data group and a pressure data group according to the sensor type; Construct an intra-modal correlation unit, which includes a feature extraction layer and a feature fusion layer. The feature extraction layer adopts a sliding time window structure, and the feature fusion layer adopts a multi-layer perceptron structure. The feature extraction layer extracts mean, variance, and peak features for the angular velocity data group, extracts mean, variance, and peak features for the motion data group, and extracts mean, duration, and rate of change features for the pressure data group. The feature fusion layer contains two fully connected layers. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function. Inputting the angular velocity data group, the motion data group and the pressure data group into the intra-modal association unit to obtain a single-modal feature vector; Construct a bidirectional cross-perception mechanism, which includes a mapping layer, an association layer, and a self-preservation layer. The mapping layer adopts an attention mechanism structure, the association layer adopts a graph convolutional network structure, and the self-preservation layer adopts a memory network structure. The mapping layer calculates the weight matrix between different modalities through self-attention, the association layer uses a graph convolutional network to construct the topological relationship between modalities, and the self-preservation layer contains a long short-term memory network to store feature combinations with high recognition accuracy. The unimodal feature vector is input into the bidirectional cross-perception mechanism, and the cross-feature matrix is ​​obtained through feature mapping and association learning. The cross-feature matrix is ​​then subjected to temporal reconstruction and structural processing to obtain the action feature sequence.

[0028] Specifically, the raw motion data is classified according to the type of sensor from which it comes, that is, it is divided into angular velocity data group, motion data group and pressure data group. The angular velocity data group comes from the gyroscope, which mainly indicates the rotation speed of the handle on the X, Y and Z axes; the motion data group comes from the acceleration sensor, which describes the linear acceleration of the handle in different directions; and the pressure data group comes from the pressure sensor, which indicates the force applied by the user to the handle buttons during operation. Classification helps to extract independent features for different data types in order to better reflect the state of the handle. Construct an intra-modal association unit, which consists of a feature extraction layer and a feature fusion layer. The feature extraction layer adopts a sliding time window structure to extract local features from the data. The length of the sliding time window is set to , the data in each time window is used to calculate the corresponding features. The feature extraction layer extracts the mean, variance and peak features of the angular velocity data set, which respectively describe the average rotation speed of the handle in the time window, the degree of rotation fluctuation and the maximum value of the rotation speed. For example, for the first time window of the angular velocity data set, the mean feature is expressed as: ; in, Indicates the time window The angular velocity value at a moment, Then is the mean of the angular velocity in the window. The variance feature is used to measure the discreteness of the data, which is expressed as: ; The peak feature is expressed as: ; Similarly, the motion data set is extracted with the same sliding time window to extract mean, variance and peak features. For the pressure data set, in addition to extracting mean and peak, duration and rate of change features are extracted. Duration indicates the length of time the user maintains a certain pressure, while the rate of change describes the rate of change of pressure over time, which is expressed as: ; in, and Respectively represent the pressure values ​​at the start and end of the time window, Represents the pressure change rate. By extracting the features of angular velocity, motion and pressure data, the feature set of each modal data is obtained. The output of the feature extraction layer is passed to the feature fusion layer. The feature fusion layer uses a multi-layer perceptron structure to fuse different features. The feature fusion layer contains two fully connected layers. The first fully connected layer uses the ReLU activation function, and its output is: ; in, is the weight matrix of the first fully connected layer, is the input feature vector, is the bias term, and the ReLU activation function is defined as: ; The second fully connected layer uses the Sigmoid activation function, and its output is: ; Among them, the Sigmoid function is defined as: ; Through the feature fusion layer, the extracted features are abstracted into single-modal feature vectors to describe the operation behavior of the handle under each mode. The angular velocity, motion and pressure data groups are input into the intra-modal association unit to obtain the single-modal feature vector of each mode. The single-modal feature vector is input into the bidirectional cross-sensing mechanism to obtain the relationship and combination characteristics between the modes. The bidirectional cross-sensing mechanism includes a mapping layer, an association layer and a self-preservation layer, which captures the complex correlation characteristics between modalities in a variety of different ways. The mapping layer adopts an attention mechanism to calculate the weight matrix between different modalities through the self-attention mechanism to measure the contribution of each modality to the operation behavior in a specific time period. For example, given the angular velocity feature vector and motion feature vector , the attention weight matrix is ​​expressed as: ; in, represents the first angular velocity feature vector The element in the motion feature vector The attention weight of each element is calculated. Through this step, dynamic weight distribution between modalities is achieved to highlight the modal features that are more important to the current operation behavior. The association layer uses a graph convolutional network structure to construct the topological relationship between modalities. The feature vectors of different modalities are regarded as nodes in the graph, and the edges between nodes represent the relationship between modalities. The operation of graph convolution is expressed as: in, is the adjacency matrix of the graph, which represents the connection relationship between modes. For the The node feature matrix of the layer, For the The weight matrix of the layer, and is an activation function (such as ReLU). Through graph convolution, the association layer can effectively capture the relationship between different modalities and build the topological structure between modalities. The self-preservation layer uses a long short-term memory network (LSTM) structure to store feature combinations with high recognition accuracy. LSTM is a special recurrent neural network (RNN) that can maintain long-term dependency information in time series data. Its state update formula is: ; ; ; ; ; in, , and They are forget gate, input gate and output gate respectively. and The weight and bias matrices are used to store important feature combinations at different time steps, and retain features that show high recognition accuracy in multiple operations. After being processed by the mapping layer, the association layer, and the self-preservation layer, the single-modal feature vectors are combined into a cross-feature matrix. The cross-feature matrix is ​​reconstructed and structured to finally obtain the action feature sequence. The purpose of time series reconstruction is to reorganize the feature matrix into a time series form to reflect the state changes of the handle at different time points. The structured processing is to combine the features of different modes according to predetermined rules to form a complete action feature description.

[0029] In a specific embodiment, the process of executing step 300 may specifically include the following steps: The action feature sequence is divided into a time domain feature data block and a space domain feature data block; Construct a multi-level deep convolutional neural network, which includes a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure, and the feature recognition network adopts a multi-scale fusion structure. The feature extraction network contains three convolutional modules. The first convolutional module contains two convolutional layers, each of which uses a 3×3 convolutional kernel with a step size of 2 and a BatchNorm layer. The second convolutional module contains three convolutional layers, each of which uses a 3×3 convolutional kernel with a step size of 2 and a ReLU activation function. The third convolutional module contains three convolutional layers, each of which uses a 3×3 convolutional kernel with a step size of 1 and a LeakyReLU activation function. Input the time domain feature data block into the first convolution module of the feature extraction network, obtain the time domain feature map through multi-scale convolution operation, and input the spatial domain feature data block into the second convolution module and the third convolution module of the feature extraction network, obtain the spatial domain feature map through multi-level convolution operation; Construct a feature recognition network. The feature recognition network contains two fully connected layers and one classification layer. The number of neurons in the fully connected layers is 512 and 256 respectively. The classification layer uses the Softmax function. The time domain feature map and the spatial domain feature map are input into the feature recognition network, and the action category vector is obtained through feature fusion and classification operations. The action category vector is then subjected to probability distribution calculation and label mapping processing to obtain the operation behavior feature set.

[0030] Specifically, the action feature sequence is deconstructed to extract the time-related information and space-related information. The action feature sequence is a high-dimensional feature matrix obtained by fusing multiple sensor data, which contains various feature information when the user operates the handle. The time domain feature data block represents the evolution characteristics of the action in the time dimension. These features reflect how the action of the handle changes over time, such as the speed and acceleration of the action; the spatial domain feature data block describes the distribution characteristics of the action in space, reflecting the posture and direction of the handle at different spatial positions. A multi-level deep convolutional neural network is constructed to extract and identify features. The multi-level deep convolutional neural network consists of a feature extraction network and a feature recognition network. The feature extraction network adopts a spatial pyramid structure to capture different scale characteristics in the action feature data through multi-level convolution operations. The spatial pyramid structure effectively extracts local and global information, making the network robust to features of different scales. The feature recognition network adopts a multi-scale fusion structure. By fusing features of different scales, the network has better recognition capabilities for complex action patterns. The feature extraction network contains three convolution modules, each of which processes different feature data blocks. The first convolution module contains two convolutional layers, each of which uses a 3×3 convolution kernel with a stride of 2 and a BatchNorm layer. The role of BatchNorm is to normalize the output of each layer to speed up the training of the network and improve its stability. Suppose the input is , the convolution operation is expressed as: ; in, is the convolution kernel weight, * represents the convolution operation, is the bias term, is an activation function, such as ReLU or LeakyReLU. The BatchNorm layer normalizes the convolution output as follows: ; in, and are the mean and standard deviation of the batch, respectively. is the normalized output. The second convolution module contains three convolution layers, each of which also uses a 3×3 convolution kernel with a step size of 2 and an activation function of ReLU. The formula for the ReLU activation function is: ; The nonlinear characteristics of ReLU effectively prevent the gradient vanishing problem, allowing the network to better learn complex feature relationships. Through convolution operations, the second convolution module can extract input features layer by layer, allowing the network to better perceive the local details of the input features. The third convolution module also contains three convolution layers, each of which uses a 3×3 convolution kernel with a step size of 1, and the activation function uses LeakyReLU. The activation formula of LeakyReLU is: ; in, is a small constant (e.g. 0.01) to ensure that there is still gradient transmission when the input is negative, solving the "neuron death" problem caused by ReLU. Through the third convolution module, the network can further extract spatial features and capture the detailed changes of features through refined convolution steps. In the process of feature extraction, the time domain feature data block is input into the first convolution module of the feature extraction network, and the time domain feature map is obtained through multi-scale convolution operations, which represents the changes of the handle action in the time dimension, such as the change pattern of rotation speed over time. At the same time, the spatial domain feature data block is input into the second and third convolution modules of the feature extraction network, and the spatial domain feature map is extracted through multi-level convolution operations. The spatial domain feature map describes the changes of the action in the spatial dimension, such as the posture changes of the handle in different directions. After obtaining the time domain feature map and the spatial domain feature map, a feature recognition network is constructed to fuse and classify these features. The feature recognition network contains two fully connected layers and one classification layer, and the number of neurons in the two fully connected layers is 512 and 256 respectively. The fully connected layer fuses and reduces the extracted temporal and spatial features to extract high-level features with strong discriminative power. The operation of the fully connected layer is expressed as: ; ; in, and is the weight matrix of the fully connected layer, and is the bias term, is the input feature vector, and are the outputs of the fully connected layer respectively. The activation function uses ReLU to increase the nonlinearity of the feature. The classification layer uses the Softmax activation function to convert the network output into a probability distribution for action category judgment. The definition of the Softmax function is: ; in, is the output vector of the fully connected layer, Indicates The scores of the categories, Indicates that the input belongs to The probability of each category. Through the Softmax function, the network can normalize each category so that the sum of the probabilities of each category output is 1, thereby determining which action category the input feature belongs to. The time domain feature map and the spatial domain feature map are input into the feature recognition network. Through the feature fusion of the fully connected layer and the operation of the classification layer, the action category vector is obtained, which represents the different action categories to which the current handle operation belongs and their corresponding probability distribution. The action category vector is subjected to probability distribution calculation and label mapping processing to obtain the operation behavior feature set. The label mapping process is to convert the action category probability output by the network into a specific operation behavior label, such as waving, pressing, rotating, etc. These operation behavior labels can be used to generate control instructions and process feedback signals.

[0031] In a specific embodiment, the process of executing step 400 may specifically include the following steps: The operation behavior feature set is divided into position state feature and force feedback state feature according to state attributes; According to the position state characteristics, the second-order dynamic model of the handle position is established, and the three state variables of position, velocity and acceleration are correlated and calculated to obtain the position state equation; A spring damping force feedback model is established according to the force feedback state characteristics, and the force response calculation is performed on the position deviation and velocity to obtain the force feedback state equation; The position state equation and the force feedback state equation are input into the Kalman state predictor to obtain a state prediction matrix; Perform state variable separation and time series expansion operations on the state prediction matrix to obtain a delay prediction sequence; A compensation model is constructed based on the delayed prediction sequence, and the state change amount within the predicted delay time is accumulated to obtain the state compensation amount; The state compensation amount is substituted into the position state equation and the force feedback state equation for state update to obtain the compensation state variable, and the compensation state variable is subjected to dynamic correction and boundary constraint processing to obtain the dynamic compensation value.

[0032] Specifically, the operation behavior feature set is divided into position state features and force feedback state features according to state attributes. The operation behavior feature set includes the user's operation actions and their physical attributes. According to the state attributes, the features are divided into position state features and force feedback state features. The position state features include information such as the spatial position, movement speed, acceleration, etc. of the handle, which are used to describe the motion trajectory and state changes of the handle in the physical space; while the force feedback state features include the interaction force between the handle and the user, such as the magnitude and direction of the feedback force given to the user by the force feedback motor. These features are used to describe the response of the handle to the user's actions. After completing the division of state features, a second-order dynamic model of the handle is established based on the position state features. This model effectively describes the motion state of the handle in space. The movement of the handle conforms to the classical principles of physical dynamics. The position state features mainly include position ,speed and acceleration Three state variables. The basic equation of the second-order kinetic model is: ; in, Indicates the quality of the handle, is the acceleration, is the damping coefficient, For speed, is the elastic coefficient, For location, Represents the external force. By calculating the relationship between the three state variables of position, velocity and acceleration, the motion equation of the handle is obtained, which is recorded as the position state equation: ; in, is the state vector, representing position and velocity, is the system matrix, which is used to describe the relationship between state variables. is the input matrix, describing the influence of external forces on the system, Represents the rate of change of state. A spring damping force feedback model is established based on the force feedback state characteristics. The principle of force feedback is described by a spring damping system, which can simulate the reaction force generated by the handle during user operation. The basic equation of the spring damping model is: ; in, is the feedback force, is the spring stiffness coefficient, which indicates the influence of position deviation on the feedback force. The target location is the current handle position, and the position deviation is is the damping coefficient, is the speed of the handle, describing the effect of speed on the feedback force. The force feedback state equation is obtained by calculating the force response of the position deviation and speed, which is used to generate the feedback force of the handle, thereby providing tactile feedback to the user. The position state equation and the force feedback state equation are input into the Kalman state predictor to obtain the state prediction matrix. The Kalman predictor is a recursive algorithm that can estimate the system state in the presence of noise interference. It gradually reduces the prediction error by continuously updating the predicted value of the system state and combining the observed data. The state update equation of the Kalman prediction is: ; ; in, Indicates that at time step The predicted value of the state, is the forecast error covariance matrix, is the process noise covariance matrix, which describes the influence of random noise in the system. Through the prediction process of the Kalman filter, a state prediction matrix containing state information such as position and speed is obtained. The state prediction matrix is ​​subjected to state variable separation and time series expansion operations to obtain a delayed prediction sequence. Since there is a certain delay in the signal processing and response of the system in actual operation, the predicted state is time-expanded so that the state change of the system can still be accurately described in future time steps. The purpose of state variable separation is to separate different state variables such as position, speed and feedback force so that they can be analyzed and processed independently, while time series expansion is to extend these state variables in time to form a prediction sequence to provide the system with a state estimate over a longer time range. A compensation model is constructed based on the delayed prediction sequence, and the state change amount within the prediction delay time is accumulated to obtain the state compensation amount. The compensation model accumulates the state deviation generated during the delay time to compensate in advance, thereby reducing the error caused by the delay. The calculation of the state compensation amount is expressed as: ; in, is the state compensation amount, For the predicted state, For the actual state, by the delay time Integrate the state change in the state to obtain the required state compensation. Substitute the state compensation into the position state equation and the force feedback state equation for state update to obtain the compensation state variable. The state update introduces compensation to correct the position and feedback force, so that the system can still maintain a high response accuracy in the presence of delay. The equation after the state update is: ; in, is the updated state variable, which indicates the position and speed after compensation. The compensated state variables are subjected to dynamic correction and boundary constraint processing to obtain the dynamic compensation value. Dynamic correction eliminates the influence of nonlinear errors or external disturbances in the system by introducing additional correction factors. Boundary constraint processing is used to ensure that the compensated state variables are within a reasonable physical range. For example, the position should not exceed the maximum operating range of the handle, and the feedback force should not exceed the limit value that the handle can withstand. Through dynamic correction and boundary constraints, it is ensured that the final dynamic compensation value not only has high accuracy, but also meets the physical constraint requirements of the system.

[0033] In a specific embodiment, the process of executing step 500 may specifically include the following steps: The dynamic compensation value and the operation behavior feature set are combined to obtain a parameter optimization data set; A user operation habit model is constructed for the parameter optimization data set, and the operation characteristic parameters are obtained through analysis and calculation of operation speed, strength and duration; Initialize the model parameters based on the operating characteristic parameters to obtain the model initial parameters, and input the model initial parameters into the gradient descent optimizer, calculate the parameter gradient through back propagation, and obtain the parameter update direction; Adaptively adjust the step size of the parameter update direction and perform momentum accumulation calculation to obtain the parameter iteration amount; Compare and calculate the error between the parameter iteration amount and the current parameter, store the parameters whose error is less than the threshold, and obtain the optimized parameter group; The position error and force feedback error of the optimized parameter group are verified and calculated to obtain the control parameter group, and the control parameter group is assigned parameters according to the control execution cycle to obtain the real-time control parameters.

[0034] Specifically, the dynamic compensation value is combined with the operation behavior feature set to jointly describe the user's action characteristics and system response during the operation of the game controller. The dynamic compensation value represents the dynamic response correction of the controller under force and feedback, while the operation behavior feature set covers the user's action characteristics, including position, speed, acceleration, etc. By combining these two types of information, a parameter optimization data set is obtained. Let the dynamic compensation value be , the operational behavior feature set is , then the parameter optimization data set It is expressed as: ; in, and F are a collection of multiple variables. The combined D contains the user's operation behavior and the dynamic feedback information of the handle. The user operation habit model is constructed based on the parameter optimization data set. The user operation habit model aims to capture the characteristics of the user when operating the handle. These characteristics include the speed, strength, and duration of the operation, and describe the user's operation style and habits through this information. For the operation speed, it is calculated by taking the derivative of the position change in the time series. The formula is as follows: ; in, Indicates that at time step The operating speed at and are the handle positions of adjacent time steps, is the time interval. The operating force is directly obtained from the pressure sensor and is set to , duration indicates the length of time a user maintains an action, By analyzing and calculating the speed, force and duration, the operating characteristic parameters are obtained. , describes the personalized characteristics of the user when operating the handle. The model parameters are initialized based on the operation characteristic parameters to obtain the initial parameters of the model. Reasonable initialization enables the model to converge quickly in the subsequent optimization process and avoid falling into the local optimum. Let the model parameters be , then the parameters after initialization are expressed as: ; in, are the initialized model parameters, Indicates the parameters based on the operating characteristics After the initialization is completed, the model initial parameters Input into the gradient descent optimizer to calculate the gradient of the parameters through the back propagation algorithm to determine the direction of parameter update. Gradient descent gradually reduces the loss function by continuously adjusting the parameters. The gradient of the loss function is expressed as: ; in, is the loss function, which represents the error between the model prediction and the actual value. The loss function parameter The gradient of , through the gradient calculation, determines the adjustment direction of the model under the current parameters. According to the calculated gradient, the direction of updating the parameters is the negative gradient direction, that is: ; in, is the learning rate, which controls the step size of parameter update. In order to improve the efficiency and stability of gradient descent, the step size of parameter update direction is adaptively adjusted, and momentum accumulation calculation is performed to obtain the parameter iteration amount. The step size adaptive adjustment is achieved through the Adam optimization algorithm, and momentum accumulation is achieved by introducing momentum terms to smooth parameter updates. The formula for momentum accumulation is: ; ; in, is momentum, is the momentum decay coefficient, which indicates the degree of retention of historical gradients. is the parameter update amount after iteration. By introducing momentum and reducing oscillation, the parameter update is smoother, thereby accelerating convergence. The error comparison calculation between the parameter iteration amount and the current parameter is performed to determine whether the parameter is close to the optimal value. By comparing the error between the current parameter and the target value, if the error is less than the preset threshold, it is considered that the parameter has reached the optimal value and is stored. The formula for error comparison is: ; like ,in If the value is greater than the set threshold, the parameter is considered to be optimal and is stored in the optimized parameter group. The position error and force feedback error are verified and calculated for the optimized parameter group to obtain the final control parameter group. The purpose of position error verification is to check whether the position signal of the handle under the current parameter setting is consistent with the expected target, while the force feedback error verification is to check whether the feedback force given by the handle to the user meets the design requirements. The position error is expressed as: ; in, is the actual location, is the target location, is the number of samples. The force feedback error is calculated in a similar way: ; By calculating the position error and force feedback error, the effectiveness of the current optimization parameter group is verified, and finally the control parameter group is obtained. The control parameter group ensures that the handle can accurately respond to the user's actions during the user operation and provide appropriate force feedback, thereby improving the user experience. The control parameter group is assigned parameters according to the control execution cycle to obtain real-time control parameters. The control execution cycle is set to update the control parameters of the handle in each fixed time period to ensure that the handle always maintains the best control state during the operation. Suppose the control execution cycle is , the real-time control parameters are , then the control parameter update of each cycle is expressed as: ; Through periodic updates, the real-time control parameters of the handle are dynamically adjusted according to the user's operating habits and the system's response characteristics to ensure the accuracy of the operation and the real-time response.

[0035] In a specific embodiment, the process of executing step 600 may specifically include the following steps: Performing first signal separation and second signal separation on the real-time control parameter to obtain a first position control parameter and a first force feedback control parameter; Inputting the first position control parameter into the first frequency converter for signal modulation to obtain the second position control parameter, and inputting the first force feedback control parameter into the second frequency converter for signal modulation to obtain the second force feedback control parameter; Performing pulse width modulation and amplitude conversion on the second position control parameter through the first driving module to obtain a first execution signal, and performing current intensity modulation and direction control on the second force feedback control parameter through the second driving module to obtain a second execution signal; Inputting the first execution signal and the second execution signal into the timing processing unit for bidirectional signal synchronization and priority determination to obtain a third execution signal; The third execution signal is wirelessly encoded and transmitted with intensity modulated by the third driving module to obtain a fourth execution signal, and the fourth execution signal is instruction parsed and dispatched according to the execution priority and timing requirements to obtain a target execution instruction.

[0036] Specifically, the real-time control parameter is subjected to first signal separation and second signal separation to obtain a first position control parameter and a first force feedback control parameter. Assume that the real-time control parameter is , including position control signals and force feedback signal , the first position control parameter and the first force feedback control parameter are obtained through signal separation operation: ; Among them, Split Represents the signal separation operation, is the first position control parameter, is the first force feedback control parameter. The first position control parameter The signal is input into the first frequency converter for signal modulation to obtain the second position control parameter. The function of the frequency converter is to adjust the frequency characteristics of the signal to make it more suitable for the subsequent drive module operation. Signal modulation is achieved by frequency up-conversion, and its basic expression is: ; in, is the second position control parameter after modulation, is the modulation frequency, is a time variable. By frequency modulating the first position control parameter, it is adjusted to a frequency band suitable for the drive module, thereby ensuring that the signal can be effectively transmitted and processed. Similarly, the first force feedback control parameter The input is modulated into the second frequency converter to obtain the second force feedback control parameter , and its modulation method is similar to the position control signal: ; By modulating the force feedback signal and adjusting it to a specific frequency, it can be transmitted and processed in a different frequency band from the position control signal, thereby avoiding signal interference. Pulse width modulation (PWM) and amplitude conversion are performed to obtain the first execution signal. The purpose of pulse width modulation is to convert the signal into a series of pulses with different duty cycles, thereby controlling the average energy of the signal, and its expression is: ; in, is the first execution signal, is the amplitude conversion factor, which is used to adjust the amplitude of the signal, PWM represents a pulse signal obtained by pulse width modulation of the second position control parameter. In this way, the continuous control signal is converted into a discrete pulse signal, which is convenient for transmission and control through the drive module. At the same time, the second force feedback control parameter is modulated by the second drive module. Perform current intensity modulation and direction control to obtain a second execution signal The purpose of current intensity modulation is to control the intensity of the current according to the size of the force feedback signal, thereby adjusting the size of the feedback force. Its expression is: ; in, is the maximum current intensity, is the maximum value of the force feedback signal, The direction control function of the signal determines the direction of the current, thereby controlling the direction of the feedback force. Through current intensity modulation and direction control, it can ensure that the intensity and direction of the force feedback signal can accurately reflect the user's operating requirements, thereby providing appropriate force feedback. and the second execution signal Input to the timing processing unit for bidirectional signal synchronization and priority determination to obtain the third execution signal The purpose of bidirectional signal synchronization is to ensure that the position control signal and the force feedback signal are coordinated in time to avoid operation delays or feedback distortion problems caused by timing asynchrony. Synchronization is achieved by calculating the delay between the two signals, and the basic formula is: ; in, To achieve the best delay time, synchronization is achieved by minimizing the difference between signals. The purpose of priority determination is to select between multiple control signals to ensure that control signals that have a greater impact on user experience are executed first during critical operations. For example, during high-speed steering, position control signals need to be responded to first, while force feedback signals need to be responded to first during fine adjustments. The third execution signal obtained through synchronization and priority determination of the timing processing unit is It is a comprehensive control signal, including the coordinated control information of position and feedback. The third execution signal is wirelessly encoded and transmitted with intensity modulation by the third driving module to obtain the fourth execution signal. The purpose of wireless signal coding is to convert the control signal into a format suitable for wireless transmission, which is achieved through modulation and coding, such as frequency shift keying (FSK) or amplitude shift keying (ASK). Its basic expression is: ; in, represents the modulation of the third execution signal, is the transmission intensity modulation factor, which is used to control the transmission intensity of the wireless signal. Through coding and modulation, the signal can be transmitted stably and efficiently in different transmission environments. According to the execution priority and timing requirements, the fourth execution signal is parsed and scheduled to obtain the target execution instruction. The purpose of instruction parsing is to separate the various parts of the fourth execution signal to extract specific control information, such as the target position of the handle, the feedback force to be applied, etc. The scheduling allocation is to send these control information to the corresponding actuator according to the priority and timing requirements, so that the corresponding action is performed at the right time. The basic principle of scheduling is to schedule according to the importance and timing requirements of the control signal to ensure that important control actions can be executed in time, while low-priority control actions are executed later. The target execution instruction obtained drives the actuator of the handle to adjust the position and feedback control, thereby realizing the user's operation intention.

[0037] The above describes the sensing method of the game controller in the embodiment of the present application. The following describes the sensing device 10 of the game controller in the embodiment of the present application. Figure 2 In one embodiment of the present application, the sensing device 10 of the game controller includes: The synchronization module 11 is used to perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain original motion data; A mapping module 12 is used to perform feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; The recognition module 13 is used to input the action feature sequence into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; A calculation module 14 is used to calculate the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value; The correction module 15 is used to perform gradient iteration calculation and error correction on the dynamic compensation value and the operation behavior feature set to obtain real-time control parameters; The output module 16 is used to process the position control signal and the force feedback signal based on the real-time control parameters and output the target execution instruction.

[0038] Through the coordinated cooperation of the above components, by introducing a two-way cross-sensing mechanism and a multi-level deep convolutional neural network, efficient fusion and feature extraction of multimodal sensor data are achieved, thereby improving control accuracy. An adaptive learning algorithm is used to model user operating habits, so that the control system can dynamically adjust parameters according to the characteristics of different players. A delay prediction and compensation mechanism based on Kalman filtering is designed to effectively solve the problem of wireless transmission delay, and control the position error and force feedback error within the preset range respectively. Through gradient iterative optimization and error correction strategies, real-time updating and optimization of control parameters are ensured. The hierarchical drive and priority scheduling mechanism are adopted to ensure the precise execution of control instructions, greatly improving the control experience of the game controller.

[0039] See also Figure 3 , Figure 3 This is a schematic block diagram of the structure of an electronic device 300 provided in an embodiment of the present application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected via a device bus 303, wherein the memory 302 may include a non-volatile storage medium and an internal memory.

[0040] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 301, the processor 301 can execute any of the above-mentioned game controller sensing methods.

[0041] The processor 301 is used to provide computing and control capabilities to support the operation of the entire electronic device 300 .

[0042] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 301, the processor 301 can execute any of the above-mentioned game controller sensing methods.

[0043] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the electronic device 300 involved in the present application scheme. The specific electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0044] It should be understood that the processor 301 may be a central processing unit (CPU), and the processor 301 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0045] It should be noted that those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the electronic device 300 described above can refer to the corresponding process of the sensing method of the aforementioned game controller, and will not be repeated here.

[0046] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the sensing method of the game controller provided in the embodiment of the present application.

[0047] The computer-readable storage medium may be an internal storage unit of the electronic device 300 in the aforementioned embodiment, such as a hard disk or memory of the electronic device 300. The computer-readable storage medium may also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the electronic device 300.

[0048] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0049] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0050] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sensing method for a game controller, characterized in that: The method comprises: Perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data; Performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; Inputting the action feature sequence into a multi-layer deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; Based on the operation behavior feature set, calculating the position state quantity and the force feedback state quantity to obtain a dynamic compensation value; Performing gradient iteration calculation and error correction on the dynamic compensation value and the operating behavior feature set to obtain real-time control parameters; The position control signal and the force feedback signal are processed based on the real-time control parameters, and a target execution instruction is output.

2. The sensing method of the game controller according to claim 1, characterized in that: The angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor are digitally filtered and time-series synchronized to obtain the original motion data, including: The X-axis, Y-axis, and Z-axis angular velocity data collected by the gyroscope sensor are subjected to 16-bit digital quantization processing to obtain digital angular velocity data; the X-axis, Y-axis, and Z-axis motion data collected by the acceleration sensor are subjected to 16-bit digital quantization processing to obtain digital motion data; the pressing force data collected by the pressure sensor are subjected to 12-bit digital quantization processing to obtain digital pressure data; Eliminating high-frequency noise and low-frequency drift on the digitized angular velocity data and the digitized motion data to obtain attitude filtering data, and performing data smoothing processing on the digitized pressure data to obtain force filtering data; The posture filter data and the force filter data are subjected to time mark alignment and difference compensation to obtain aligned data, the aligned data are divided into continuous data segments according to time windows and boundary smoothing is performed to obtain a segmented sequence, and the segmented sequence is subjected to data normalization and dimension unification to obtain original motion data.

3. The sensing method of the game controller according to claim 2, characterized in that: The performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence includes: The raw motion data is divided into an angular velocity data group, a motion data group and a pressure data group according to sensor types; Constructing an intra-modal correlation unit, the intra-modal correlation unit includes a feature extraction layer and a feature fusion layer, the feature extraction layer adopts a sliding time window structure, and the feature fusion layer adopts a multi-layer perceptron structure; the feature extraction layer extracts mean, variance, and peak features from the angular velocity data group, extracts mean, variance, and peak features from the motion data group, and extracts mean, duration, and rate of change features from the pressure data group, and the feature fusion layer includes two fully connected layers, the first fully connected layer uses a ReLU activation function, and the second fully connected layer uses a Sigmoid activation function; Inputting the angular velocity data group, the motion data group and the pressure data group into the intra-modal association unit to obtain a single-modal feature vector; Construct a bidirectional cross-perception mechanism, which includes a mapping layer, an association layer, and a self-preservation layer. The mapping layer adopts an attention mechanism structure, the association layer adopts a graph convolutional network structure, and the self-preservation layer adopts a memory network structure. The mapping layer calculates the weight matrix between different modalities through self-attention, the association layer uses a graph convolutional network to construct the topological relationship between modalities, and the self-preservation layer contains a long short-term memory network for storing feature combinations with high recognition accuracy. The unimodal feature vector is input into the bidirectional cross-sensing mechanism, a cross-feature matrix is ​​obtained through feature mapping and association learning, and the cross-feature matrix is ​​subjected to time series reconstruction and structural processing to obtain an action feature sequence.

4. The sensing method of the game controller according to claim 3, characterized in that: The action feature sequence is input into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set, including: Dividing the action feature sequence into a time domain feature data block and a space domain feature data block; Constructing a multi-level deep convolutional neural network, the multi-level deep convolutional neural network includes a feature extraction network and a feature recognition network, the feature extraction network adopts a spatial pyramid structure, and the feature recognition network adopts a multi-scale fusion structure; the feature extraction network includes three convolution modules, the first convolution module includes two convolution layers, each convolution layer uses a 3×3 convolution kernel with a step size of 2 and a BatchNorm layer, the second convolution module includes three convolution layers, each convolution layer uses a 3×3 convolution kernel with a step size of 2 and a ReLU activation function, and the third convolution module includes three convolution layers, each convolution layer uses a 3×3 convolution kernel with a step size of 1 and a LeakyReLU activation function; Input the time domain feature data block into the first convolution module of the feature extraction network, obtain the time domain feature map through multi-scale convolution operation, and input the spatial domain feature data block into the second convolution module and the third convolution module of the feature extraction network, obtain the spatial domain feature map through multi-level convolution operation; Constructing a feature recognition network, wherein the feature recognition network includes two fully connected layers and one classification layer, the number of neurons in the fully connected layers is 512 and 256 respectively, and the classification layer uses a Softmax function; The time domain feature map and the spatial domain feature map are input into the feature recognition network, an action category vector is obtained through feature fusion and classification operations, and probability distribution calculation and label mapping processing are performed on the action category vector to obtain an operation behavior feature set.

5. The sensing method of the game controller according to claim 4, characterized in that: The calculating the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain the dynamic compensation value includes: The operation behavior feature set is divided into position state features and force feedback state features according to state attributes; A second-order dynamic model of the handle position is established according to the position state characteristics, and the three state variables of position, velocity and acceleration are correlated and calculated to obtain a position state equation; A spring damping force feedback model is established according to the force feedback state characteristics, and force response calculation is performed on the position deviation and the speed to obtain a force feedback state equation; Inputting the position state equation and the force feedback state equation into a Kalman state predictor to obtain a state prediction matrix; Performing state variable separation and time series expansion operations on the state prediction matrix to obtain a delay prediction sequence; Building a compensation model based on the delay prediction sequence, accumulating and calculating the state change amount within the prediction delay time to obtain the state compensation amount; The state compensation amount is substituted into the position state equation and the force feedback state equation for state updating to obtain a compensation state variable, and the compensation state variable is subjected to dynamic correction and boundary constraint processing to obtain a dynamic compensation value.

6. The sensing method of the game controller according to claim 5, characterized in that: The step of performing gradient iteration calculation and error correction on the dynamic compensation value and the operation behavior feature set to obtain real-time control parameters includes: Combining the dynamic compensation value and the operation behavior feature set to obtain a parameter optimization data set; Constructing a user operation habit model for the parameter optimization data set, and obtaining operation characteristic parameters through analysis and calculation of operation speed, force and duration; Initializing the model parameters based on the operating characteristic parameters to obtain the model initial parameters, and inputting the model initial parameters into the gradient descent optimizer, calculating the parameter gradient through back propagation, and obtaining the parameter update direction; Adaptively adjusting the step size of the parameter update direction and performing momentum accumulation calculation to obtain the parameter iteration amount; Compare and calculate the error between the parameter iteration amount and the current parameter, store the parameters whose error is less than the threshold, and obtain the optimized parameter group; The position error and force feedback error of the optimization parameter group are verified and calculated to obtain a control parameter group, and the control parameter group is assigned parameters according to the control execution cycle to obtain real-time control parameters.

7. The sensing method of the game controller according to claim 6, characterized in that: The processing of the position control signal and the force feedback signal based on the real-time control parameter and outputting the target execution instruction comprises: Performing first signal separation and second signal separation on the real-time control parameter to obtain a first position control parameter and a first force feedback control parameter; Inputting the first position control parameter into a first frequency converter for signal modulation to obtain a second position control parameter, and inputting the first force feedback control parameter into a second frequency converter for signal modulation to obtain a second force feedback control parameter; Performing pulse width modulation and amplitude conversion on the second position control parameter through a first driving module to obtain a first execution signal, and performing current intensity modulation and direction control on the second force feedback control parameter through a second driving module to obtain a second execution signal; Inputting the first execution signal and the second execution signal into a timing processing unit for bidirectional signal synchronization and priority determination to obtain a third execution signal; The third execution signal is wirelessly encoded and transmitted with intensity modulated by the third driving module to obtain a fourth execution signal, and the fourth execution signal is instruction parsed and scheduled according to the execution priority and timing requirements to obtain a target execution instruction.

8. A sensing device for a game controller, characterized in that: The sensing method for executing the game controller according to any one of claims 1 to 7, wherein the sensing device of the game controller comprises: A synchronization module is used to perform digital filtering and time-series synchronization on the angular velocity data of the gyroscope sensor, the motion data of the acceleration sensor, and the pressure intensity data of the pressure sensor to obtain the original motion data; A mapping module, used for performing feature segmentation and modal cross-mapping on the original motion data to obtain a motion feature sequence; A recognition module, used for inputting the action feature sequence into a multi-level deep convolutional neural network for spatiotemporal feature extraction and pattern recognition to obtain an operation behavior feature set; A calculation module, used to calculate the position state quantity and the force feedback state quantity based on the operation behavior feature set to obtain a dynamic compensation value; A correction module, used for performing gradient iteration calculation and error correction on the dynamic compensation value and the operation behavior feature set to obtain a real-time control parameter; The output module is used to process the position control signal and the force feedback signal based on the real-time control parameters and output the target execution instruction.

9. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the sensing method of the game controller as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the sensing method of the game controller as described in any one of claims 1 to 7 is implemented.

Images