Virtual drum set playing control method and device based on posture perception

Abstract

This invention discloses a virtual drum kit playing control method and device based on posture perception, belonging to the field of electronic musical instruments and motion control technology. It includes: acquiring data output from a nine-axis sensor to obtain standardized sensor data; fusing the standardized sensor data using adaptive complementary filtering to obtain posture quaternions and converting them into posture angles; converting acceleration data into acceleration vectors in the world coordinate system based on the posture angles, reconstructing the drumstick motion trajectory through time integration, and locking the strike point coordinates when a sudden acceleration change is detected; extracting feature parameters and strike duration, using them along with the strike point coordinates as input, performing valid strike determination and drum position matching, generating a drum position code, and sending it to the receiving end via BLE communication. This invention achieves a high-precision, low-latency virtual drum kit playing experience without a physical drumhead, solving the technical problems of traditional drum kits being large and noisy, and existing virtual drum solutions having low recognition accuracy and high communication latency.

Description

Technical Field

[0001] This invention relates to the field of electronic musical instruments and motion control technology, and in particular to a virtual drum kit playing control method and device based on posture perception, which is applicable to scenarios such as motion-sensing music entertainment, smart wearable interaction, and portable music practice. Background Technology

[0002] As a classic percussion instrument, the drum kit is loved by music lovers. However, traditional drum kits have inherent drawbacks such as large size, large space occupation, inconvenience to carry, and high noise, making it difficult to meet the practice and entertainment needs of users in various scenarios such as home and outdoors. Especially for users living in densely populated residential areas such as apartments, the noise problem seriously limits the usage scenarios and usage time.

[0003] To address these issues, existing technologies include electronic drums and some virtual drum kit solutions. While electronic drums can reduce noise and size, they still rely on physical drum surfaces, resulting in insufficient portability. Furthermore, high-end electronic drums are expensive, and mid-to-low-end products have poor tactile feedback and dynamic response, failing to accurately reproduce the playing experience of a real drum kit. They also present issues of maintenance costs and space requirements.

[0004] Existing virtual drum kit solutions mostly employ a single sensor (such as a six-axis gyroscope) or visual recognition technology, which has significant drawbacks: Single-sensor solutions can only recognize simple striking motions, cannot accurately pinpoint the striking location, have low recognition accuracy, weak anti-interference capabilities, are prone to false triggers, and cannot stably recognize striking commands from different areas in front, making it difficult to simulate the multi-drum position playing logic of a real drum kit; visual recognition technology solutions are greatly affected by ambient light, have high recognition latency, and require additional equipment such as cameras, resulting in poor portability, limited applicability, and insufficient compatibility, with some devices unable to achieve stable connections.

[0005] Furthermore, existing wireless communication solutions for virtual drum kits mostly use 2.4G transmission or traditional Bluetooth technology. 2.4G transmission suffers from high power consumption and weak anti-interference capabilities, while traditional Bluetooth technology has the drawback of high latency, making it impossible to achieve real-time synchronization between striking actions and audio output, thus affecting the performance experience. At the same time, existing solutions lack a simple, low-power, and portable solution that can accurately identify six specific positions in front, making it difficult to balance recognition accuracy, portability, and smooth performance. This makes it impossible to meet users' needs for lightweight and high-precision air drum kits, and it is also difficult to adapt to the rapid linkage of ordinary audio equipment.

[0006] In existing technologies, traditional drum kits and electronic drum kits are limited by the physical striking surface, resulting in drawbacks such as large size, high noise, and poor portability. Virtual drum kit solutions based on a single sensor or visual recognition suffer from low accuracy in striking position recognition, weak anti-interference ability, and high communication latency, making it impossible to stably achieve accurate positioning of multiple drum positions and low-latency real-time synchronized performance. Summary of the Invention

[0007] To address this, this application provides a virtual drum kit playing control method and device based on posture perception, which helps to solve the shortcomings of traditional drum kits and electronic drums, which are limited by the physical striking surface and have large size, high noise, and poor portability; while virtual drum kit solutions based on a single sensor or visual recognition have problems such as low accuracy of striking position recognition, weak anti-interference ability, and high communication latency.

[0008] To achieve the above objectives, this application adopts the following technical solution:

[0009] In a first aspect, this application provides a virtual drum kit playing control method based on posture awareness, including: The sensor data is collected from the output of the nine-axis sensor; the sensor data includes: three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data; The sensor data output by the nine-axis sensor is filtered, calibrated, normalized, and timestamped to generate standardized sensor data. The standardized sensor data is fused using an adaptive complementary filter to obtain a fused attitude quaternion, and the fused attitude quaternion is then converted into attitude angles; the attitude angles include: pitch angle, roll angle and yaw angle. Based on the attitude angle, the triaxial acceleration data under the standardized sensor data is converted into an acceleration vector in the world coordinate system, and the displacement is obtained by time integration. The spatial motion trajectory of the drumstick is calculated based on the displacement and the real-time rotation angle. Based on the spatial motion trajectory, when a sudden acceleration change is detected, the corresponding spatial coordinates are locked as the striking point coordinates. The real-time rotation angle is obtained by time integration of the triaxial gyroscope angular velocity data under the standardized sensor data. The peak acceleration, rate of change of angular velocity, and duration of impact are extracted from the standardized sensor data and used as input along with the coordinates of the impact point. Valid impact determination and drum position matching are performed to generate the drum position code corresponding to the matching result, which is then sent to the receiving end via BLE communication.

[0010] Secondly, this application provides a posture-aware virtual drum kit playing control device, the device comprising: The attitude acquisition module is used to acquire the sensing data output by the nine-axis sensor, including three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data; The core control module is connected to the posture acquisition module and configured to execute any of the above-described posture-aware virtual drum kit playing control methods. The BLE communication module is used to transmit the drum position code to the receiving end via the BLE wireless link.

[0011] The application employs the above technical solution and has at least the following beneficial effects: 1. Improve the accuracy and anti-interference capability of tap location recognition. By collecting raw data from a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer from a nine-axis sensor, and generating standardized sensor data through filtering, calibration, normalization, and timestamp alignment, adaptive complementary filtering is used for multi-sensor fusion to obtain attitude quaternions, which are then converted into pitch, roll, and yaw angles. This fusion strategy fully leverages the complementary characteristics of the three types of sensors: the gyroscope has a fast dynamic response and is used to capture high-frequency dynamics of the impact; the accelerometer and magnetometer have good long-term stability and are used to provide a low-frequency attitude reference—and the fusion weights are adaptively adjusted by the degree to which the acceleration magnitude deviates from gravitational acceleration. When stationary, the correction weight is increased to suppress drift, and during vigorous movement, the dynamic weight is increased to avoid distortion. Based on this, the calculated attitude angles are highly accurate and drift-free, providing an accurate directional reference for subsequent drum position positioning, effectively overcoming the shortcomings of single-sensor recognition such as low accuracy, weak anti-interference ability, and susceptibility to false triggering.

[0012] 2. Achieve a portable playing experience without a physical drumhead. This method achieves virtual drum kit performance entirely based on motion data collected by a nine-axis sensor, eliminating the need for a physical drumhead or external equipment such as a camera. By converting acceleration data into acceleration vectors in a world coordinate system and reconstructing the spatial trajectory of the drumsticks through time integration, the method locks the strike point coordinates when a sudden acceleration change is detected. The entire performance process is completed solely using a handheld sensor and a core algorithm. This significantly reduces the size and weight of the device, improving portability and making it suitable for various scenarios such as home, outdoor, and office settings. Simultaneously, it eliminates the noise interference problem inherent in traditional drum kits.

[0013] 3. Reduce wireless transmission latency to achieve real-time synchronization between tapping and audio. After generating a drum position code through effective strike detection and drum position matching, BLE communication is used to send the code to the receiving end. BLE technology features low power consumption and low latency. Combined with frequency hopping communication strategy and retransmission mechanism, communication latency can be controlled to an extremely low level, ensuring that the striking action and sound output are synchronized in real time. This solves the problems of high latency and high power consumption of traditional Bluetooth technology, significantly improving the smoothness of the performance.

[0014] 4. Simplify equipment structure to improve compatibility and accessibility. This method integrates core functions such as posture calculation, trajectory reconstruction, tap recognition, and drum position matching at the algorithm level onto the sensor and general-purpose processing hardware. It outputs standard drum position codes to the BLE receiver, eliminating the need for additional dedicated receiving equipment or complex audio interfaces. This reduces dependence on audio hardware, improves compatibility and ease of integration with ordinary audio equipment, lowers overall costs, and promotes the lightweighting and popularization of motion-sensing music entertainment devices.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart illustrating a posture-aware virtual drum kit playing control method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the composition of a posture-aware virtual drum kit playing control device according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating the overall composition of a posture-aware virtual drum kit playing control device according to an exemplary embodiment; Figure 4 This is a schematic diagram of the posture acquisition module of a posture-aware virtual drum kit playing control device according to an exemplary embodiment; Figure 5 This is a schematic diagram of the core control module of a posture-aware virtual drum kit playing control device according to an exemplary embodiment; Figure 6 This is a schematic diagram of the button control unit of a posture-aware virtual drum kit playing control device according to an exemplary embodiment; Figure 7 This is a schematic diagram of the state feedback unit of a posture-aware virtual drum kit playing control device according to an exemplary embodiment. Figure 8 This is a schematic diagram of the power supply and charging unit of a posture-aware virtual drum kit playing control device according to an exemplary embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Example 1 Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a posture-aware virtual drum kit playing control method according to an exemplary embodiment, the method comprising: Step S1: Collect sensor data output from the nine-axis sensor; the sensor data includes three-axis accelerometer data, three-axis gyroscope data, and three-axis magnetometer data. It should be noted that these three types of data measure the linear acceleration, rotational angular velocity, and magnetic field strength of the drumstick in space, respectively.

[0020] Step S2: Filter, calibrate, normalize, and align the sensing data with timestamps to generate standardized sensing data. The filtering process employs Kalman filtering and extended Kalman filtering to eliminate environmental noise and system errors. Specifically, Kalman filtering removes random noise such as environmental vibrations and electromagnetic interference, while extended Kalman filtering addresses the filtering problem of nonlinear systems and corrects system errors caused by sensor drift. The calibration process sequentially performs zero-point calibration, sensitivity calibration, and magnetometer magnetic bias calibration to eliminate the sensor's own zero bias, sensitivity deviation, and environmental magnetic interference. The normalization process maps the calibrated data to the [0, 1] interval to eliminate the differences between data from different ranges. The timestamp alignment process uses a double buffer mechanism and circular queue management to ensure that the three types of data are synchronized in time.

[0021] Step S3: The standardized sensor data is fused using adaptive complementary filtering. High-frequency dynamic information is extracted from the gyroscope data via high-pass filtering, and low-frequency reference information is extracted from the accelerometer and magnetometer data via low-pass filtering. The data are then fused using adaptive weighting to obtain the optimal attitude quaternion q=[q0,q1,q2,q3]. T The attitude quaternions are then converted into pitch, roll, and yaw angles. These three attitude angles are output in real time at a frequency of 1 kHz.

[0022] Step S4: Based on the attitude angle, convert the three-axis acceleration data in the standardized sensor data into an acceleration vector in the world coordinate system; perform a time integration on the gyroscope angular velocity data to obtain the real-time rotation angle, and perform two consecutive time integrations on the acceleration vector with a sampling period of 0.01s to obtain the linear displacement. Based on the real-time rotation angle and linear displacement, determine the spatial coordinates (X, Y, Z) of the drumstick in the world coordinate system point by point, and reconstruct the complete spatial motion trajectory of the drumstick from raising the stick, accelerating, striking, to returning to its original position. During the trajectory reconstruction process, continuously monitor the change in acceleration amplitude. When a drastic change in acceleration amplitude is detected in a very short time (acceleration change point), lock the spatial coordinates of the current sampling moment as the striking point coordinates.

[0023] Step S5: Extract the peak acceleration, rate of change of angular velocity, and duration of the strike from the standardized sensor data. Use these parameters, along with the strike point coordinates, as input to perform a valid strike determination. If the peak acceleration is not less than 0.5g, the rate of change of angular velocity is not less than 100° / s², and the strike duration is between 10ms and 50ms, the strike is determined to be valid, and the process proceeds to drum position matching; otherwise, the strike is determined to be invalid.

[0024] During drum position matching, the Y-axis and Z-axis coordinates of the strike point are compared with six preset drum position coordinate ranges. The drum position where the coordinates fall is taken as the matching result, generating a corresponding three-digit drum position code. This drum position code is then encapsulated into a data frame with an 8-bit CRC checksum appended. It is then transmitted to the receiving end (audio module) via frequency hopping communication using the BLE 5.4 protocol. Communication latency is controlled within 20ms, and automatic retransmission occurs upon transmission failure, up to a maximum of three times.

[0025] Example 2 This embodiment further defines the preprocessing steps, mainly including the following steps: The first step is noise reduction. Kalman filtering and extended Kalman filtering are used to process the sensor data. Kalman filtering removes random noise such as environmental vibrations and electromagnetic interference, while extended Kalman filtering corrects systematic errors caused by sensor drift. The core formula of Kalman filtering is: State prediction: Xk - =A·Xk-1+B·uk Among them: Xk - Xk is the prior state estimate at time k; 1 represents the state estimate after time k-1; A: state transition matrix; B: control input matrix; uk: control vector.

[0026] Specifically, based on the filtered sensor state estimate Xk-1 from the previous time step, the current sensor state Xk is predicted using the state transition matrix A.- This provides a priori estimates for subsequent measurement corrections.

[0027] Covariance prediction: Pk - =A·Pk-1·A T +Q Among them: Pk - To estimate the covariance matrix a priori; Pk 1 represents the posterior covariance matrix of the previous time step; Q represents the process noise covariance matrix.

[0028] Specifically, based on the uncertainty of the state Pk at the previous moment... 1. Predict the error range Pk of the current state estimation. - Q is used to describe the uncertainties of the sensor model itself, such as gyroscope drift and modeling errors.

[0029] Kalman gain: Kk = Pk - ·H T ·(H·Pk - ·H T +R) - ¹ Where: Kk is the Kalman gain; H is the observation matrix; R is the measurement noise covariance matrix.

[0030] Specifically, the weighting coefficient Kk between the predicted state and the sensor measurement value is calculated; where R describes the sensor measurement noise and determines whether the filtering algorithm relies more on the "predicted value" or the "measured value" to adapt to the dynamic changes of the tapping action.

[0031] State correction: Xk=Xk - +Kk·(Zk-H·Xk - ) Where: Xk is the posterior state estimate at time k; Zk is the sensor measurement.

[0032] Specifically, the predicted state Xk is corrected using the current sensor measurement value Zk. - The filtered optimal state Xk is obtained, which is the noise-reduced acceleration, angular velocity, and magnetic field strength data, providing input for subsequent attitude calculation.

[0033] Covariance correction: Pk=(I-Kk·H)·Pk - Where: Pk is the posterior estimated covariance matrix at time k; I is the identity matrix.

[0034] Application: The uncertainty Pk of the updated state estimate will gradually converge as the filtering iterates, thus improving the stability of subsequent attitude calculations.

[0035] Meanwhile, by combining the sensor's built-in low-pass filter, high-frequency noise above the preset cutoff frequency of 100Hz is filtered out, further improving data stability.

[0036] The second step is sensor calibration. Three types of calibration are performed sequentially on the filtered data: Zero-point calibration: During device initialization, the device is left to stand still for 3 seconds, and the zero-point output value of each channel of the nine-axis sensor is recorded. In subsequent operation, all raw data collected are subtracted from the zero-point output value of the corresponding channel to eliminate the zero-point offset of the sensor itself.

[0037] Sensitivity calibration: The sensitivity coefficient of each channel (i.e., the digital output value corresponding to a unit physical quantity) is calibrated using standard acceleration and standard angular velocity signals; subsequent data acquisition is multiplied by the corresponding sensitivity coefficient to ensure accurate and consistent measurement values ​​under different forces and rotation speeds.

[0038] Magnetometer magnetic deflection calibration: The user rotates the sensor 360° around its own axis continuously and collects triaxial magnetic field strength data at each angle during the rotation; based on the collected data, the magnetic deflection parameters are fitted to correct the heading angle error caused by geomagnetic field interference, ensuring the accuracy of spatial positioning under different usage environments.

[0039] The third step is normalization and synchronization. The calibrated triaxial accelerometer, triaxial gyroscope, and triaxial magnetometer data are normalized to the [0, 1] interval to eliminate differences in the range and resolution of different sensors. Subsequently, a double-buffering mechanism and circular queue management are used to align the timestamps of the three types of sensor data, ensuring that acceleration, angular velocity, and magnetic field strength data at the same sampling time remain synchronized. The data after all the above processing is used as the standardized sensor data for subsequent fusion algorithms.

[0040] Example 3 This embodiment further defines the steps for fusing adaptive complementary filtering and converting it into attitude angles.

[0041] The fusion strategy fully leverages the complementary characteristics of the three types of sensors: gyroscopes have fast dynamic response and high short-time integration accuracy, making them suitable for capturing high-frequency dynamic processes of striking actions; accelerometers and magnetometers have no static drift and good long-term stability, making them suitable for providing low-frequency attitude reference correction.

[0042] The specific integration steps are as follows: The first step is high-pass filtering to extract dynamic information. High-pass filtering is applied to the three-axis gyroscope data in the standardized sensor data to retain the high-frequency dynamic components of the tapping motion and suppress low-frequency errors caused by slow gyroscope drift. The high-pass filtering formula is: hp_out(n)=in(n)-in(n-1)+(1-α)·hp_out(n-1) Wherein(n) is the gyroscope data at the current sampling time, hp_out(n) is the current high-pass filter output, hp_out(n-1) is the high-pass filter output at the previous time, and α is the filter coefficient. In this embodiment, α=0.05, which corresponds to a cutoff frequency of approximately 20Hz.

[0043] The second step is low-pass filtering to extract reference information. The triaxial accelerometer and triaxial magnetometer data in the standardized sensor data are low-pass filtered to remove high-frequency motion interference caused by the impact action, extracting a stable low-frequency attitude reference. The low-pass filtering formula is: lp_out(n)=(1-α)·out(n-1)+α·in(n) Wherein(n) is the accelerometer or magnetometer data at the current sampling time, lp_out(n) is the current low-pass filter output, out(n-1) is the output at the previous time, and α is also taken as 0.05.

[0044] The third step is adaptive weighted fusion. The high-frequency dynamic information obtained from the high-pass filtering and the low-frequency reference information obtained from the low-pass filtering are weighted and fused according to the fusion weights to obtain the optimal attitude quaternion q=[q0,q1,q2,q3]. T The core fusion formula is: q_est=α_fusion·q_gyro+(1-α_fusion)·q_acc / mag Where q_gyro is the real-time attitude quaternion obtained by integrating the gyroscope angular velocity, q_acc / mag is the observed attitude quaternion calculated by the accelerometer and magnetometer, and q_est is the optimal attitude quaternion after fusion.

[0045] The fusion weight α_fusion is adaptively adjusted based on the degree to which the acceleration modulus |a|=√(ax²+ay²+az²) deviates from the gravitational acceleration g, where g is taken as 9.81 m / s². When |a|∈[g-0.5g,g+0.5g] (i.e., approximately 4.9~14.7m / s²), it is determined to be a stationary or slow-moving state. At this time, the accelerometer and magnetometer data are reliable. Setting α_fusion=0.9 increases the correction weight of the accelerometer and magnetometer, effectively suppressing the gyroscope integral drift.

[0046] When |a| does not belong to [g-0.5g, g+0.5g], it is judged as a violent motion state such as knocking. At this time, the accelerometer is severely affected by motion interference. Setting α_fusion=0.98 increases the dynamic weight of the gyroscope to avoid attitude distortion caused by motion acceleration interference and ensure high dynamic response.

[0047] The fourth step is to convert the fused attitude quaternions into pitch, roll, and yaw using the first set of formulas.

[0048] Example 4 This embodiment specifically defines the set of formulas for converting attitude quaternions into attitude angles. Specifically, the first set of formulas includes the following three conversion formulas: Formula (1): Pitch = arcsin(2(q0q2- q1q3)) Where Pitch is the pitch angle, representing the rotation angle of the drumstick around the Y-axis; q0, q1, q2, and q3 are the four components of the quaternion, where q0 is the real part and q1, q2, and q3 are the imaginary parts, collectively describing the spatial rotation state of the sensor; this formula is the standard conversion formula for quaternion to Euler angles, calculating the pitch angle through the quaternion components for subsequent coordinate transformation of acceleration data. The arcsin function constrains the output value within the range of [-90°, 90°], conforming to the actual physical range of the pitch angle.

[0049] Formula (2): Roll = arctan2(2(q0q1+ q2q3), 1-2(q1² + q2²)) Where Roll is the roll angle, representing the angle by which the drumstick rotates around the X-axis, i.e., the degree of "rolling" of the drumstick. arctan2 is a two-parameter arctangent function, which determines the correct quadrant based on the signs of the numerator and denominator, and the output range is [-180°, 180°], avoiding angle jumps.

[0050] Formula (3): Yaw = arctan2(2(q0q3+ q1q2), 1-2(q2² + q3²)) Here, Yaw is the heading angle, representing the angle by which the drumstick rotates around the Z-axis, i.e., the left and right direction of the drumstick on the horizontal plane. This angle is directly used in subsequent drum position matching to determine whether the user struck the left or right virtual drumhead.

[0051] The three posture angles mentioned above are output in real time at a high frequency of 1kHz, providing high-precision and real-time posture input for subsequent trajectory and coordinate calculation, supporting the spatial trajectory restoration of the striking action and precise positioning of the drum position.

[0052] Example 5 This embodiment further defines the steps of trajectory restoration and tapping point coordinate locking described above.

[0053] The first step is to establish a world coordinate system. Using the spatial position of the nine-axis sensor when the device is initially stationary after power-on as the origin O(0,0,0), and with the user's front as the X-axis, the horizontal leftward direction as the Y-axis, and the vertical upward direction as the Z-axis, a right-handed three-dimensional world coordinate system is established. All subsequent spatial position calculations are based on this coordinate system, constraining the striking motion within a semi-cylindrical space in front of the user, thus adapting to the natural swinging posture of the human body during performance.

[0054] The second step is to construct the rotation matrix. Using the pitch angle (Pitch), roll angle (Roll), and yaw angle (Yaw) output from step S3, a rotation matrix R is generated from the sensor's local coordinate system to the world coordinate system. R is a 3×3 matrix, constructed as follows: calculate the rotation matrix Rz around the Z-axis using the Yaw angle, calculate the rotation matrix Ry around the Y-axis using the Pitch angle, and calculate the rotation matrix Rx around the X-axis using the Roll angle. Multiply these matrices in the order Z→Y→X to obtain R=Rz×Ry×Rx.

[0055] The three columns of the rotation matrix R have clear physical meanings: the first column is the projection component of the sensor's own X-axis onto the X, Y, and Z axes of the world coordinate system in the current posture; the second column is the projection component of the sensor's own Y-axis; and the third column is the projection component of the sensor's own Z-axis. These three column vectors describe the directions in the real world for the sensor's "front," "right," and "up" directions, thus determining the directional correspondence between the sensor's local coordinate system and the world coordinate system.

[0056] The third step is acceleration coordinate transformation. The triaxial acceleration data from the standardized sensor data is multiplied by the rotation matrix R corresponding to the current posture to obtain the acceleration vector in the world coordinate system. This transformation eliminates the influence of the drumstick holding posture on the judgment of motion direction—regardless of whether the user holds the drumstick with a right-handed grip, a left-handed grip, or at any tilt angle, the transformed acceleration components accurately reflect the drumstick's motion changes in all directions in real space, with a posture adaptation response time within 100ms.

[0057] The fourth step is displacement calculation and trajectory reconstruction. Integration is performed in two parallel channels: Angular velocity integral channel: Perform a time integral on the three-axis gyroscope angular velocity data in the standardized sensor data to obtain the real-time rotation angle, which is used to continuously update the current attitude direction.

[0058] Acceleration integration channel: The acceleration vector in the world coordinate system is integrated twice in a preset sampling period of 0.01s—the first integration yields the real-time velocity, and the second integration yields the linear displacement in three-dimensional space.

[0059] Based on the real-time rotation angle and the linear displacement, the spatial coordinates (X, Y, Z) of the drumstick in the world coordinate system are determined point by point. The rotation angle determines the direction of movement of the drumstick, and the linear displacement determines the distance of movement. By combining the two, the position of the drumstick in space at each sampling moment can be accurately calculated. By continuously recording these coordinate points, the complete spatial motion trajectory from raising the drumstick, accelerating, striking, to resetting can be reconstructed.

[0060] Example 6 This embodiment adds a drift suppression step to the trajectory restoration process.

[0061] Pure integration accumulates sensor noise and minute errors over time, causing the reconstructed trajectory to gradually deviate from the true position, resulting in drift. To suppress this drift, two absolute references are introduced during the integration process for closed-loop correction: The first approach uses the gravity vector as the absolute reference in the vertical direction. The gravity vector is the acceleration vector pointing towards the Earth's center, measured by the triaxial accelerometer in a stationary state; its direction is always vertically downwards, and its magnitude is approximately 9.81 m / s². During integration, the pitch and roll angles in the integration result are compared with the theoretical projections of the gravity vector onto each axis of the sensor. When a deviation in the vertical direction is detected, a gradient descent algorithm is used to adjust the integration result to converge towards the gravity vector direction, correcting the accumulated errors in the pitch and roll angles.

[0062] The second approach uses the geomagnetic field vector as the absolute horizontal reference. This geomagnetic field vector is the direction vector of the Earth's magnetic field measured by the triaxial magnetometer, pointing towards the geomagnetic north pole. Magnetometer calibration is performed every second, comparing the heading angle in the integration result with the theoretical direction of the geomagnetic field vector on the horizontal plane. When a deviation in the heading angle is detected, a gradient descent algorithm is used to adjust the integration result to converge towards the direction of the geomagnetic field vector, correcting the accumulated error in the heading angle.

[0063] Through continuous closed-loop correction using the two absolute references mentioned above, it is ensured that the trajectory and spatial coordinates remain stable without significant drift during long-term continuous performance (such as continuous use for tens of minutes). The angular error of the trajectory calculation is controlled within 5°, and the spatial positioning error is controlled within 2cm.

[0064] Example 7 This embodiment further defines the specific logic for determining a valid tap. After extracting the peak acceleration, rate of change of angular velocity, and tap duration from the standardized sensor data, the following logic is used to determine each item: The first criterion is to determine whether the peak acceleration is not lower than the first threshold of 0.5g. The peak acceleration refers to the maximum value of the acceleration modulus |a| = √(ax² + ay² + az²) at the moment of impact, where g is the acceleration due to gravity (9.81 m / s²). If the peak acceleration is less than 0.5g, it indicates that the force of the action is too light, belonging to a light tap, a stroking, or triggered by environmental vibration, and is directly judged as an invalid action, and the process terminates. This condition is used to distinguish the difference in force between a light tap and a heavy strike.

[0065] The second criterion is whether the rate of change of angular velocity is not lower than the second threshold of 100° / s². The rate of change of angular velocity reflects the instantaneous intensity of wrist rotation during the striking action. If the rate of change of angular velocity is <100° / s², it indicates that the wrist rotation is too slow, belonging to slow movement, posture adjustment, or other non-striking actions, and is directly judged as an invalid action, terminating the process. This condition is used to exclude invalid actions such as slight shaking or slow swinging.

[0066] The third criterion is to determine whether the tapping duration falls within the preset time range of 10ms to 50ms. If the duration is less than 10ms, it is highly likely to be transient electrical noise or signal glitches from the sensor, rather than a genuine physical tap. If the duration is greater than 50ms, it is a non-tapping action such as pressing, pushing, or prolonged pressing. Only when the duration falls within the 10ms to 50ms range does it conform to the pulse-like mechanical characteristics of a genuine tap. This condition is used to eliminate invalid actions such as noise, accidental touches, and prolonged pressing.

[0067] The current action is considered a valid tap and proceeds to the subsequent drum position matching step only if all three conditions are met simultaneously. If any condition is not met, the action is considered invalid and no further processing is performed, thus achieving a low false alarm rate.

[0068] Example 8 This embodiment further defines the specific matching method for drum position matching.

[0069] Within the semi-cylindrical space in front of the user, six virtual drum positions are pre-defined, corresponding to the six core drumheads of a traditional drum kit. Each drum position is defined by a Y-axis coordinate range (corresponding to the yaw angle, representing the horizontal left-right direction) and a Z-axis coordinate range (corresponding to the pitch angle, representing the vertical up-down direction), with the coordinate unit being degrees (°). The six drum positions are divided by angle based on the initial stationary position as follows: The snare drum is located at the bottom center: Y∈[-30, 30], Z∈[-90, 30], drum position code 002. This corresponds to the snare drum, which is located at the bottom center in a traditional drum kit, and is consistent with the range of motion of striking with the arm hanging horizontally and naturally.

[0070] The two tom-toms are located above the center: Y∈[-30, 30], Z∈[30, 90], drum position code 005. They are located directly above the snare drum, corresponding vertically to it, and are positioned at the height of a slightly raised arm when striking.

[0071] The cymbals are located at the lower left: Y∈[-90, -30], Z∈[-90, 30], drum position code 001. They are immediately adjacent to the left of the snare drum and at the same horizontal level (consistent with the Z-axis range).

[0072] The left hanging cymbal is located at the upper left: Y∈[-90, -30], Z∈[30, 90], drum position code 004. It is located directly above the foot cymbal, at the same horizontal level as the two tom-toms (consistent Z-axis range).

[0073] The three-tum drum is located at the lower right: Y∈[30, 90], Z∈[-90, 30], drum position code 003. It is immediately adjacent to the right side of the snare drum and at the same horizontal level (consistent with the Z-axis range).

[0074] The cymbals are located in the upper right: Y∈[30, 90], Z∈[30, 90], drum position code 006. It is located directly above the three-tum drum, at the same horizontal level as the two-tum drum (the Z-axis range is the same).

[0075] The six drum positions are arranged in a top-bottom layer and left-right symmetrical layout: the three upper drumheads (left cymbal, double tom-tom, and ding-ding cymbal) all have a Z-axis range of [30, 90]; the three lower drumheads (boot cymbal, snare drum, and triple tom-tom) all have a Z-axis range of [-90, 30]; the two left drumheads (boot cymbal and left cymbal) all have a Y-axis range of [-90, -30]; the two middle drumheads (snare drum and double tom-tom) all have a Y-axis range of [-30, 30]; and the two right drumheads (triple tom-tom and ding-ding cymbal) all have a Y-axis range of [30, 90].

[0076] After a valid strike is detected, the Y-axis and Z-axis coordinates of the strike point are compared with the coordinate ranges of the six drum positions mentioned above. The drum position whose coordinates fall within both the Y and Z ranges is taken as the matching result, and a corresponding three-digit code is generated and output to the BLE communication module for transmission.

[0077] In summary, this invention effectively solves the technical problems of traditional drum kits, such as large size, noise pollution, and inconvenience of carrying, as well as the low recognition accuracy, weak anti-interference ability, and high communication latency of existing virtual drum kit solutions. It realizes a high-precision, low-latency virtual drum kit playing experience without the need for a physical drumhead, and is widely applicable to home, outdoor, and office scenarios.

[0078] Example 9 Please see Figure 2 This embodiment provides a posture-aware virtual drum kit playing control device, including: The attitude acquisition module 10 is used to acquire the sensing data output by the nine-axis sensor, including three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data; The core control module 20 is connected to the posture acquisition module and is configured to execute any of the above embodiments of the posture-aware virtual drum kit playing control method; BLE communication module 30 is used to send the drum position code to the receiving end via a BLE wireless link.

[0079] See Figure 3 Specifically, the hardware circuit of this invention adopts a miniaturized, low-power, and highly integrated design. The core is divided into five major circuit modules (power circuit, attitude acquisition module, core control module, BLE communication module, and audio receiving circuit). The circuit modules are electrically connected through PCB copper foil and work together to complete the core functions of "acquisition-processing-transmission-output". The electrical connection relationship, function and working principle of each circuit module are described in detail below, which is completely consistent with the schematic diagram layout.

[0080] For details, please refer to Figure 4 The attitude acquisition module 10 is the core of motion sensing, used to acquire sensing data output by the nine-axis sensor.

[0081] This module consists of a six-axis inertial sensor chip WJ601 and a three-axis AMR magnetic sensor chip QMC6309. The WJ601 outputs three-axis accelerometer data (range ±16g) and three-axis gyroscope data (range ±2000dps) to capture the dynamic characteristics of the drumstick swing; the QMC6309 collects three-axis magnetic field strength data in space in real time, providing a stable data source for determining the direction, speed, and force of the strike.

[0082] In terms of electrical connectivity, both chips communicate with the core control module via the I2C bus. Sensor_SDA and Sensor_SCL are connected to the PA4 and PA5 pins of the main control chip, respectively. The main control chip interacts with the two chips through a specified I2C address. For power supply, the sensor's VDD and VDDIO are provided by the LDO chip ME6211. The power switch is controlled by the main control chip's PA1 pin. C10 and C14 are decoupling capacitors to ensure the stability of the sensor's analog and digital power supplies. Regarding low-power design, this module supports dynamic sampling rate adjustment and single-trigger sampling. When there is no tapping for more than 10 minutes, the main control chip shuts down the ME6211 output via the PA1 pin to cut off the sensor's power supply. Combined with the main control chip's sleep mechanism, the standby current is reduced to 2μA, maximizing battery life.

[0083] See Figure 5The core control module 20 is connected to the posture acquisition module 10 and is configured to execute any of the above-described performance control method embodiments to complete sensor data reception, posture calculation, trajectory restoration, strike recognition, drum position matching, MIDI instruction generation, and system low power consumption control.

[0084] Specifically, the core control module uses an MCU main control chip with an integrated BLE 5.4 protocol stack. The electrical connections are as follows: Clock domain: The 24MHz crystal oscillator (Y1) is connected to the BT_OSCI / BT_OSCO pins of the main controller, providing a high-precision clock reference for BLE RF and system core.

[0085] Sensor interface: PA4 / PA5 pins are configured as I2C interfaces to enable data communication with the attitude acquisition module.

[0086] Peripheral control: PA0 pin is configured as GPIO-PWM output to drive the vibration motor; PA1 pin is configured as GPIO output to control the sensor power supply switch; PA2 pin is configured as GPIO-PWM output to control the RGB indicator light; PA3 pin is configured as GPIO input to detect button events; PB6 pin is configured as an ADC channel to acquire the battery NTC temperature voltage divider signal; PB9 pin is configured as VUSB detection to identify the charging insertion status.

[0087] RF front end: The VBT / VDD pins are connected to the RF antenna matching network composed of L3 / L4 / C14 / C15 to achieve efficient transmission and reception of BLE signals.

[0088] When there is no data interaction, the core control module 20 executes a low-power strategy: enters a deep sleep mode, retaining only the button detection circuit and the USB insertion detection circuit to maximize battery life.

[0089] Specifically, the BLE communication module 30 is integrated into the core control module and is used to send the drum position code to the receiving end (audio receiver module) via the BLE wireless link.

[0090] Wireless communication employs the BLE 5.4 protocol, encoding the tap position information with 16-bit binary data. This data is transmitted at a rate of 500kbps via a custom proprietary protocol, with a transmission time of less than 10ms and communication latency controlled within 20ms. A frequency-hopping communication strategy enhances anti-interference capabilities, and a three-retransmission mechanism ensures a retransmission success rate of over 99%. When there is no data transmission, the BLE module automatically enters sleep mode, automatically waking up and transmitting when a new drum position code is generated.

[0091] The RF front end is connected to the RF antenna matching network consisting of L3 / L4 / C14 / C15 via pins such as VBT / VDD to ensure efficient transmission and reception of BLE signals.

[0092] Please see Figure 6 The button control module is the system's human-machine interface, enabling power control and device reset functions.

[0093] Electrical connection: One end of the button is grounded, and the other end is connected to VCC via a pull-up resistor (R20: 10kΩ), and simultaneously connected to the POWER_KEY pin of the main control chip. When not pressed, POWER_KEY is high; when pressed, it is low, triggering a main control interrupt.

[0094] Function definition: A short press once sets the current orientation to the initial zero point (positioning the initial position); a long press for 3 seconds performs a power-on / power-off operation, turning the device on or entering a low-power state; a long press for 8 seconds triggers the nine-axis sensor calibration process.

[0095] Please see Figure 7 The status feedback module provides visual and tactile status feedback to convey the device's working status to the user.

[0096] RGB indicator light: It adopts an integrated RGB LED chip WS2812D-F5-12mA-C1 (U1), whose control pin DIN is connected to the main control GPIO (PA2) through a current limiting resistor (R4:75Ω), and the color and brightness are adjusted by PWM duty cycle.

[0097] The indicator logic is as follows: the green light flashes once when the device is powered on; the light goes out when the device is powered off; the red light stays on while charging and goes out when charging is complete; the red light stays on when the battery is low; during drumstick calibration, the red light breathes to indicate that calibration is in progress, the blue light breathes to indicate that calibration is in progress, and the green light breathes to indicate that calibration is complete; the blue light flashes once when the drumstick is positioned. When a drum is struck effectively, it displays a different color depending on the drum position: snare drum - yellow light flashes twice, duton - magenta light flashes twice, triton - blue light flashes twice, cymbal - green light flashes twice, ding-ding cymbal - orange light flashes twice, and left cymbal - cyan light flashes twice; for power indicators, 100%-75% is green, 75%-50% is blue, 50%-25% is yellow, and 25%-0% is red.

[0098] Vibration Motor: A miniature vibration motor is used, driven by a P-channel enhancement-mode MOSFET (Q1: SI2302). The MOTOR_PWM signal is connected to the main control GPIO (PA0) to control the MOSFET base. A diode (D1: 1N5819WT) is connected in parallel across the motor to absorb the back electromotive force generated by the motor coil when the MOSFET is turned off, protecting the driver transistor. When the core control module determines a valid impact, the motor generates a vibration feedback.

[0099] Please see Figure 8 The power supply and charging module provides a stable power supply to all circuits in the device, and the main control chip has built-in charging management functions. This module consists of four cooperating circuit paths: The USB charging path draws power from the outside. A 5V voltage is introduced through the VBUS pin of the Type-C interface (USB1). After safety review by the overvoltage and overcurrent protection chip SA8206A, a stable USB 5V is output through the current-limiting resistor R1 to power the battery for charging and system debugging. The battery power supply path provides energy to the device. The positive terminal VBAT of the lithium battery is directly connected to the VBAT pin of the main control chip. The integrated LDO in the main control chip converts the VBAT into multiple independent power supplies, such as VCC, VDD, and VDDBT, to power different power consumption domains, including sensors, indicator lights, and Bluetooth RF, ensuring that RF, analog, and digital circuits do not interfere with each other. The power filtering network consists of three 1μF capacitors (C7, C8, and C9) connected in parallel next to each power pin of the main control chip. When the power consumption changes drastically due to violent drumstick swinging, the stored charge is released nearby to suppress power ripple, ensuring stable and reliable circuit operation during motion scenarios. The NTC temperature detection network is responsible for safety monitoring. It uses the battery's built-in NTC thermistor and a fixed resistor to form a voltage divider circuit. The divided voltage is then connected to the main control ADC channel PB6. The main control collects this voltage and calculates the battery's real-time temperature. If the casing temperature exceeds the safety threshold during charging, it automatically triggers a power reduction or alarm mechanism to prevent the battery from overheating.

[0100] In terms of charging management features, this module supports USB Type-C plug-and-play charging, automatically identifies power supply priority, and prioritizes external power supply for charging the battery when USB is plugged in. It integrates multiple protections against overvoltage, overcurrent, and overtemperature, effectively preventing damage to components from hot-plug surges. The aforementioned power supply and charging module works in conjunction with the attitude acquisition module, core control module, BLE communication module, button control module, and status feedback module to form a complete closed loop of attitude acquisition → core processing → BLE transmission → status feedback. This enables a high-precision, low-latency virtual drum kit playing device without a physical drumhead, widely adaptable to home, outdoor, and office scenarios.

[0101] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0102] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A virtual drum kit playing control method based on posture perception, characterized in that, The method includes: The sensor data is collected from the output of the nine-axis sensor; the sensor data includes: three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data; The sensor data output by the nine-axis sensor is filtered, calibrated, normalized, and timestamped to generate standardized sensor data. The standardized sensor data is fused using an adaptive complementary filter to obtain a fused attitude quaternion, and the fused attitude quaternion is then converted into attitude angles; the attitude angles include: pitch angle, roll angle and yaw angle. Based on the attitude angle, the triaxial acceleration data under the standardized sensor data is converted into an acceleration vector in the world coordinate system, and the displacement is obtained by time integration. The spatial motion trajectory of the drumstick is calculated based on the displacement and the real-time rotation angle. Based on the spatial motion trajectory, when a sudden acceleration change is detected, the corresponding spatial coordinates are locked as the striking point coordinates. The real-time rotation angle is obtained by time integration of the triaxial gyroscope angular velocity data under the standardized sensor data. The peak acceleration, rate of change of angular velocity, and duration of impact are extracted from the standardized sensor data and used as input along with the coordinates of the impact point. Valid impact determination and drum position matching are performed to generate the drum position code corresponding to the matching result, which is then sent to the receiving end via BLE communication.

2. The method according to claim 1, characterized in that, The process of filtering, calibrating, normalizing, and aligning the sensor data output by the nine-axis sensor with timestamps to generate standardized sensor data includes: The sensor data is processed using Kalman filtering and extended Kalman filtering to filter out noise above a preset cutoff frequency; After filtering, zero-point calibration, sensitivity calibration, and magnetometer magnetic deflection calibration are performed sequentially. The calibrated triaxial accelerometer data, triaxial gyroscope data, and triaxial magnetometer data are normalized to the [0, 1] interval and then timestamped to serve as the standardized sensor data.

3. The method according to claim 1, characterized in that, The process of fusing the standardized sensor data using adaptive complementary filtering to obtain fused attitude quaternions, and then converting the fused attitude quaternions into attitude angles, includes: The triaxial gyroscope data in the standardized sensor data is high-pass filtered to extract high-frequency dynamic information; The triaxial accelerometer and triaxial magnetometer data in the standardized sensor data are low-pass filtered to extract low-frequency reference information; The high-frequency dynamic information and the low-frequency reference information are fused based on preset fusion weights to obtain a fused attitude quaternion; The fused attitude quaternion is converted into attitude angles using a preset first set of formulas. The attitude angles include pitch angle, roll angle and yaw angle. The fusion weights are adaptively adjusted based on the degree to which the acceleration modulus deviates from the gravitational acceleration: when the acceleration modulus is within a first preset range, the correction weights of the accelerometer and magnetometer are increased; when the acceleration modulus exceeds this range, the dynamic weights of the gyroscope are increased.

4. The method according to claim 3, characterized in that, The first set of formulas includes: Pitch = arcsin(2(q_0 q_2 - q_1 q_3))(1) Where Pitch is the pitch angle, representing the rotation angle of the drumstick around the Y-axis; q0, q1, q2, and q3 are the four components of the quaternion, where q0 is the real part and q1, q2, and q3 are the imaginary parts, which together describe the spatial rotation state of the sensor; this formula is the standard conversion formula for quaternion to Euler angle, and the pitch angle is calculated through the quaternion components for subsequent coordinate transformation of acceleration data; Roll = arctan2(2(q_0 q_1 + q_2 q_3), 1-2(q_1^2 + q_2^2))(2) Where Roll is the roll angle, representing the angle by which the drumstick rotates around the X-axis; Yaw = arctan2(2(q_0 q_3 + q_1 q_2), 1-2(q_2^2 + q_3^2))(3) Where Yaw is the heading angle, representing the angle by which the drumstick rotates around the Z-axis.

5. The method according to claim 1, characterized in that, Based on the attitude angle, the triaxial acceleration data under the standardized sensor data is converted into an acceleration vector in the world coordinate system, and the displacement is obtained by time integration. The spatial motion trajectory of the drumstick is calculated based on the displacement and the real-time rotation angle, including: Using the attitude angle, a rotation matrix is ​​generated from the sensor's local coordinate system to the world coordinate system; wherein, the three columns of the rotation matrix are the projection components of the sensor's own X-axis, Y-axis, and Z-axis onto each axis of the world coordinate system under the current attitude; Multiply the triaxial acceleration data in the standardized sensor data with the rotation matrix corresponding to the current attitude to obtain the acceleration vector in the world coordinate system; The real-time rotation angle is obtained by performing a time integration on the triaxial gyroscope angular velocity data in the standardized sensor data; the linear displacement is obtained by performing two consecutive time integrations on the acceleration vector in the world coordinate system according to a preset sampling period. Based on the real-time rotation angle and the linear displacement, the spatial coordinates of the drumstick in the world coordinate system are determined point by point, and the spatial motion trajectory of the drumstick is reconstructed.

6. The method according to claim 5, characterized in that, Also includes: Using the gravity vector and geomagnetic field vector as references, a closed-loop correction is performed on the time integration results to suppress accumulated errors; Wherein, the gravity vector is the acceleration vector pointing towards the Earth's center measured by the triaxial accelerometer, and the geomagnetic field vector is the direction vector of the Earth's magnetic field measured by the triaxial magnetometer.

7. The method according to claim 1, characterized in that, The determination of a valid tap includes: Determine whether the peak acceleration is not lower than a first threshold, whether the rate of change of angular velocity is not lower than a second threshold, and whether the duration of the tapping is within a preset time interval. If all three conditions are met, the tapping is deemed valid and the process proceeds to drum position matching. Otherwise, it will be considered an invalid action and will not be processed further.

8. The method according to claim 1, characterized in that, The method for determining drum position matching includes: The Y-axis and Z-axis coordinates of the striking point are compared with the preset range of six drum position coordinates, and the drum position in which the coordinates fall is taken as the matching result. The coordinate ranges of the six drum positions are divided according to angles, based on the initial static position: Snare drum: Y∈[-30, 30], Z∈[-90, 30]; Two-ton drum: Y∈[-30, 30], Z∈[30, 90]; Cymbals: Y∈[-90, -30], Z∈[-90, 30]; Left cymbal: Y∈[-90,-30], Z∈[30,90]; Triple-tum: Y∈[30, 90], Z∈[-90, 30]; Ding ding cymbals: Y∈[30, 90], Z∈[30, 90].

9. A virtual drum kit playing control device based on posture perception, characterized in that, The device includes: The attitude acquisition module is used to acquire the sensing data output by the nine-axis sensor, including three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data; The core control module is connected to the posture acquisition module and configured to execute the posture-aware virtual drum kit playing control method according to any one of claims 1-8; The BLE communication module is used to transmit the drum position code to the receiving end via the BLE wireless link.

10. The apparatus according to claim 9, characterized in that, The attitude acquisition module includes at least: an inertial sensor for outputting the data from the three-axis accelerometer and the three-axis gyroscope, and a magnetic sensor for outputting the data from the three-axis magnetometer. The inertial sensor and the magnetic sensor are respectively communicatively connected to the core control module.

Images