Wearable virtual music interaction device and computing method thereof

Abstract

A wearable virtual music interaction device, comprising a main hand body, a motion positioning feedback module, a human-computer interaction module and a battery module; the motion positioning feedback module is arranged on the main hand body and is used for acquiring hand actions of a user; the actions are converted into data signals and transmitted to an internal processor; the internal processor sends the generated data to the human-computer interaction module through a data transmission submodule; the human-computer interaction module analyzes the data, obtains actual values of note parameters input by the user, and plays the actual values of the note parameters through a sound emitting component to visually, aurally and tactilely feedback the performance of the user; and the battery module is used for supplying power to the motion positioning feedback module. The device is a virtual musical instrument that can normally be played as long as it is worn on the hand, is comfortable to wear and convenient to carry, supports use anywhere, can realize multiple tasks in parallel while ensuring freedom of the hands of the user. Meanwhile, a calculation method of actual values of note parameters is given.

Description

Technical Field

[0001] This invention belongs to the field of wearable smart device technology, specifically relating to a wearable virtual music interaction device and its implementation method that simulates playing musical instruments by moving the arm and / or hand in the air. Background Technology

[0002] The importance of music is undeniable, and playing music through instruments is a way for performers to resonate with the music itself. However, not every instrument is easy to carry around and play anywhere.

[0003] However, the cost of using musical instruments is relatively high. For example, the price of an entry-level home piano can easily range from several thousand to tens of thousands of yuan. Common woodwind instruments like the clarinet range from thousands for student-level models to over one hundred thousand yuan for expert-level models. Even relatively popular instruments like the guitar or violin are not cheap. At the same time, instruments take up relatively much space and are often quite heavy, making them inconvenient to carry. An upright piano is about 130 cm high, 155 cm long, and 60 cm wide, weighing up to 450 kg and occupying about one square meter of space. The long bench used for playing also takes up considerable space, making them difficult to move. Other large instruments, such as the tuba, double bass, timpani, and harp, also suffer from the same problem of being bulky and difficult to move. Finally, the maintenance costs of instruments are high. For example, the tension of the strings in stringed instruments decreases, affecting pitch and inevitably impacting the sound quality. Saxophones and oboes, in addition to basic instrument maintenance, also require frequent reed replacements. The costs of instrument maintenance and consumables add to the extra expenses for players after purchasing the instrument.

[0004] Wearable devices represent a major breakthrough in the history of technological development. Devices such as smart bracelets and holographic projectors offer users a superior user experience. Therefore, it is necessary to provide a device that virtualizes physical musical instruments, offering greater convenience for music use while significantly reducing the expenses for performers to purchase and maintain their instruments.

[0005] In existing technologies, patents CN103235641A and CN101388149A both collect user motion input information through visual signals and then convert and calculate the visual signals to derive the corresponding music. This method of motion acquisition via visual images has a narrow scope of application; for example, it is difficult to work in dimly lit or frequently changing environments. Furthermore, this image recognition method has low accuracy and slow processing speed, making it difficult to keep up with the performer's rhythm. The image acquisition device also requires placement, adjustment, and fixation, occupying a large area and being inconvenient to carry. Patent CN106445168A discloses a smart glove design; however, this design is merely a general frame module and lacks specific application areas. Patents CN112318531A and CN111667737A provide specific music teaching platforms; however, the former is fixed and lacks flexibility, while the latter is limited to teaching purposes. Summary of the Invention

[0006] This invention provides a wearable virtual music interaction device and its implementation method. The device is a virtual musical instrument that can be played normally as long as it is worn on the hand. It is comfortable to wear, easy to carry, and can be used anywhere. While ensuring the user's hands are free, it can perform multiple tasks in parallel.

[0007] This invention operates without physical contact with an object; even moving fingers in the air can simulate playing a musical instrument. Once the user wears the device correctly and initializes the positioning, they only need to move their fingers to achieve the effect of playing a musical instrument, and the device can play the corresponding musical notes.

[0008] This invention employs a combination of hardware and software to collect user finger movement data. The generated data is transmitted via hardware sensors and received and processed by software, supporting high-precision finger movement recognition with short response times. Furthermore, this invention can also synchronously display the playing process through an interface as needed.

[0009] To meet the above technical requirements, the present invention includes a hardware part and a software part, wherein the hardware part has four modules: a main hand body, a motion positioning feedback module, a human-computer interaction module, and a power supply module.

[0010] The back of the main hand unit has a processor and other components that can acquire the user's hand movement and finger tapping actions.

[0011] The inertial navigation system module generates position data and data signals based on the acquired actions, and then sends the above information to the host through the data transmission submodule.

[0012] As one of the preferred embodiments of the present invention, an inertial navigation module can be added to the user's arm to improve the accuracy of motion tracking, and a power module can be installed on the user's arm.

[0013] Upon receiving the aforementioned information, the software maps the user's finger movements onto musical note images. During software initialization, each finger is initially initialized as a static note; as the user moves the device, the corresponding note changes according to the finger's movement. The software then displays a virtual musical instrument image, simultaneously showing the finger positions on the image to create an animation or AR / VR scene. This allows the user to see the hand's position in virtual space and the note being played.

[0014] Specifically, the deflection angle, acceleration, and time obtained by the inertial navigation module are used to calculate the user's hand position in the body coordinate system. Combined with the initial note parameter settings of the virtual instrument, the actual values ​​of the note parameters to be played by the user are calculated.

[0015] The wearable virtual music interaction device provided by this invention offers users convenience in playing musical instruments while significantly reducing costs. By setting one or more inertial navigation modules in the motion positioning feedback module, those skilled in the art can balance the accuracy of the positioning logic with manufacturing costs, and flexibly choose the specific implementation method by comprehensively considering the adaptability to different types of instruments and the corresponding scalable algorithms. Furthermore, this invention avoids injury to the user's hands from physical musical instruments. Because this invention is lightweight and does not come into contact with the physical keys or strings of an instrument, it avoids irreversible injuries such as tendinitis, calluses, fingerprint fading, and finger deformities. Attached Figure Description

[0016] Figure 1 This is the overall system architecture diagram of the present invention.

[0017] Figure 2 This is a schematic diagram of the wearable device of the present invention.

[0018] Figure 3 This is a system block diagram of one specific embodiment of the present invention.

[0019] Figure 4 The MPU6050 or MPU6000 sensor is selected for the inertial navigation module in one of the specific embodiments of the present invention.

[0020] Figure 5 The flowchart of the STM32 program used in one of the specific embodiments of the present invention is shown.

[0021] Figure 6This is a software state diagram of one specific embodiment of the present invention.

[0022] Figure 7 This is the interface initialization effect of one specific embodiment of the present invention.

[0023] Figure 8a , Figure 8b This is an example of the interface playing an instrument (illustrated for a keyboard instrument) as one of the specific embodiments of the present invention.

[0024] Figure 9 This is an example of the effect of playing a wind instrument during one of the specific embodiments of the present invention.

[0025] Figure 10 The black and white keys of a keyboard musical instrument and their corresponding input values ​​are one of the specific embodiments of the present invention.

[0026] Figure 11 This is an accelerator computing model, which is one of the specific embodiments of the present invention.

[0027] Figure 12 This is a gravity disturbance compensation model, which is one of the specific embodiments of the present invention.

[0028] Figure 13 This is an acceleration model during motion, which is one of the specific embodiments of the present invention.

[0029] Figure 14 This is a velocity model during motion, which is one of the specific embodiments of the present invention.

[0030] Figure 15 This is a displacement model during motion, which is one of the specific embodiments of the present invention.

[0031] Figure 16 This is a schematic diagram of a user moving their arm, which is one of the specific embodiments of the present invention.

[0032] Figure 17 This is a schematic diagram of arm yaw in one specific embodiment of the present invention.

[0033] Figure 18 To refine the actual pitch played by each finger by setting up a sub-inertial navigation module or key for each finger. Detailed Implementation

[0034] To make the technical means, creative features, achieved objectives and effects of this invention easily understood, the invention is described in detail below with reference to embodiments and accompanying drawings. The embodiments of this invention include, but are not limited to, the following embodiments. The terms "first," "second," and "third," etc., used in the specification and claims of this invention are used to distinguish different objects themselves and do not represent a complete limitation on the order of events.

[0035] Example 1

[0036] A wearable virtual music interaction device includes a main hand unit, a motion positioning feedback module, a human-computer interaction module, and a power module. Its overall system architecture diagram, wearable illustration, and system block diagram are attached. Figures 1-3 STM32 program flowchart, software state diagram, and interface initialization effect reference attached. Figures 5-7 For the effects of playing the game on the interface, the notes on the black and white keys, and the corresponding input values, please refer to Figures 8-10.

[0037] The main hand part can be in the form of a glove, or it can be in the form of a bracelet, ring, finger cot, or other forms. While ensuring the product's functionality, those skilled in the art can flexibly choose the specific implementation method while considering factors such as cost and ease of use.

[0038] The power module is used to supply power to the motion positioning feedback module.

[0039] The motion positioning feedback module is located on the main hand body, specifically on the palm, back of the hand, or wrist, and is used to acquire the user's hand movements; convert the movements into data signals and transmit them to the internal processor; the internal processor sends the generated data to the human-computer interaction module through the data transmission submodule; the data transmission submodule can be implemented using Bluetooth or other wireless transmission methods, or other wired transmission methods.

[0040] The motion positioning feedback module further includes a first inertial navigation module for collecting hand motion parameters and tracking the corresponding motion trajectory. The inertial navigation module specifically includes a gyroscope and an accelerometer, obtaining a body coordinate system with the user's initial position as the origin. In a specific configuration, the body coordinate system is defined as follows in this embodiment:

[0041] A body coordinate system is obtained with the user's initial position as the origin. The positive x-axis of this body coordinate system points to the right horizontally of the body, the positive y-axis points forward, and the positive z-axis points upward. However, those skilled in the art should understand that the specific directions of the x, y, and z axes can be flexibly chosen, and subsequent calculation steps will correspond to different basic x, y, and z axis definitions. Optional inertial navigation modules are listed in the appendix. Figure 4 .

[0042] The body coordinate system is obtained with the user's initial position as the origin. The specific method for obtaining hand movements is as follows:

[0043] The angle changes output by the first inertial navigation module are identified as quaternions q0, q1, q2, and q3. These quaternions are then converted into Euler angles through calculations, specifically including the pitch, roll, and yaw angles of the hand movements, representing the rotation angles in the body coordinate system about the x, y, and z axes, respectively. The specific calculation formulas are as follows:

[0044]

[0045] Based on the three rotation angles, combined with the acceleration values ​​and movement time obtained by the first inertial navigation module, the displacement of the hand in the three directions of x, y, and z axes is calculated, and the pitch of the actual value of the note parameters played by the user is further calculated using the displacement.

[0046] The human-computer interaction module parses the data to obtain the actual value of the note parameters input by the user, and plays the actual value of the note parameters through the sound-producing component to provide feedback on the user's performance.

[0047] The actual values ​​of note parameters include, but are not limited to: pitch, duration, and intensity.

[0048] The human-computer interaction module may also include a display component, which provides visual feedback on the position of the virtual instrument played by the user.

[0049] The visual feedback method includes mapping motion data obtained through the motion positioning feedback module to the keyboard positions of keyboard instruments, key hole positions of wind instruments, string positions of string instruments, or striking positions of percussion instruments, initialized on the display component. This mapping allows the display component to show a triggered indicator for the corresponding keyboard position, key hole position, string position, or striking position when the user's hand moves to different positions. The display component can be a display screen, projection device, or AR / VR glasses, etc., and is not specifically limited to these in this invention.

[0050] The types of virtual musical instruments include, but are not limited to, keyboard instruments, wind instruments, string instruments, and percussion instruments. The human-computer interaction module can also provide mode selection for different types of instruments. The types and modes of virtual musical instruments include, but are not limited to, keyboard instruments such as piano, wind instruments such as oboe, string instruments such as guitar, and percussion instruments such as timpani.

[0051] This application uses specific examples of keyboard instruments such as pianos, organs, and electronic keyboards for illustration. However, those skilled in the art will understand that the same method of recognizing the performer's hand gestures and providing the actual values ​​of the played note parameters can also be implemented when playing many other types of instruments. That is, the virtual music device of this application can provide users with multiple selectable instrument modes. After selecting the corresponding instrument mode, the reference pitch and relative note values ​​are initialized according to the characteristics of different instruments. Alternatively, a single set of calculation methods for the same reference pitch, relative note values, and corresponding actual note parameter values ​​can be used for different instruments.

[0052]

Example 2

[0053] The wearable virtual music interaction device according to Embodiment 1 further includes a MIDI interface, which is disposed on the main hand unit and the human-computer interaction module. Those skilled in the art can refer to existing technologies to configure the MIDI interface, thereby adding general music interaction functions to the device and improving its compatibility.

[0054]

Example 3

[0055] According to the wearable virtual music interaction device described in Embodiment 1, the specific method for calculating the actual pitch of the notes played by the user is as follows:

[0056] A rotation matrix is ​​constructed using the rotation angle obtained from the first inertial navigation module; the rotation matrix is ​​transposed to return to the body coordinate system, multiplied by the accelerations in the three directions of the xyz axes obtained from the first inertial navigation module, and the accelerations are integrated twice to obtain the displacement of the hand in three-dimensional space, that is, the displacement of the hand relative to the origin of the body coordinate system on the xyz axes.

[0057] After obtaining the displacement of the user's hand on the xyz axis relative to the origin of the body coordinate system, the pitch of the note played by the user can be calculated based on the virtual instrument range set in the initialization, as well as the displacement width and direction corresponding to each pitch.

[0058] The rotation matrix R is constructed using the displacement angle obtained from the first inertial navigation module, specifically as follows:

[0059]

[0060] Where P = pitch, R = roll, and Y = yaw.

[0061] In practical use, it's impossible to guarantee that the hand will always be horizontal to the ground; there will inevitably be some tilting and shaking. Since the main body of the motion positioning feedback module is located in the hand, the module also rotates when the hand rotates, and the measured acceleration is also after rotation. This means that the direction axis of the motion positioning feedback module's sensor and the direction axis of the body coordinate system form an angle. We need to calculate the acceleration in the three directions before rotation based on the acceleration in the three directions after rotation to determine the hand's position relative to the body.

[0062] Furthermore, the following calculation is performed using the formula:

[0063]

[0064] We obtain sx, sy, and sz, which represent the displacements of the hand along the x, y, and z axes, respectively, where R T Let be the transpose of the rotation matrix, where ax, ay, and az are the accelerations in the three directions obtained by the sensor, and t is the movement time.

[0065] The specific method for calculating the actual pitch played by the user based on the hand's displacement along the xyz axes is as follows:

[0066] The system initializes the width of the keyboard keys, wind instrument keyholes, string spacing, or percussion surface width of the virtual instrument. It also initializes the user's body coordinate system origin to a fixed pitch. The system calculates the hand's displacement along the x, y, and z axes of the body coordinate system, divides this displacement by the width of the keyboard, keyholes, strings, or percussion surface of the initialized virtual instrument, and rounds down to obtain the actual pitch played by the user relative to the fixed pitch after hand movement.

[0067] For example, when playing a keyboard instrument, the initial key width is set to wide = 2cm. Then, the displacement length sx divided by the key width wide is rounded down. This rounded result represents the number of keys the hand has moved relative to the initial key position, thus determining the actual key position triggered by the hand. In a specific embodiment, if the initial hand position (right thumb) is middle C, and the calculated sx is 2cm, the number of keys moved is 1. Therefore, the right thumb is playing the note D to the right of middle C, and the index finger is playing the note E. A negative value indicates a movement to the left (opposite to the right) under the same initialization conditions. In this embodiment, only the case where the finger-triggered pitches are sequentially arranged is considered.

[0068] If the value of sy is greater than or equal to a specific threshold, which in one embodiment can be selected as 2cm, then it can be determined that the semitone black key is triggered.

[0069] In cases where the keyboard has multiple rows stacked vertically, such as a modern electronic keyboard or a classical pipe organ, if the value of sz is greater than or equal to a specific threshold (in one embodiment, the threshold can be selected as 8cm), it can be determined that the keyboard at different heights is triggered.

[0070] Furthermore, the instrument's range can be set as needed during initialization, depending on the instrument type. Those skilled in the art will also recognize that the pitch values ​​for playing other types of virtual instruments can be calculated by initializing the origin and the displacement values ​​of the x, y, and z axes. This calculation is based on the characteristics of different instrument types, specifically the relative positions of the key holes or strings between different pitches set during initialization. Simultaneously, the instrument's range or the striking position of individual key holes can be customized for user-friendliness. For example, users with shorter arms can set smaller, denser key layouts to suit their needs during initialization. For other types of instruments, hand posture can be used to determine the instrument's position and make corresponding adjustments.

[0071]

Example 4

[0072] Based on Embodiment 1, the movement time of the hand is first obtained through the motion positioning feedback module. Combined with the angle change obtained by the first or second inertial navigation module in the motion positioning feedback module, the pitch angle, roll angle, and yaw angle of the hand movement are calculated. The intensity and duration of the actual parameter values ​​of the notes played by the user are then calculated.

[0073] Furthermore, the duration of the actual pitch value of the note played by the user is obtained by holding the position for the duration after the hand triggers the actual pitch at a specific location, i.e., the actual trigger duration; furthermore, the intensity of the note is obtained by the magnitude of the angular velocity of the pitch at the moment when a certain pitch is actually triggered, which is collected by the first or second inertial navigation module; specifically, the magnitude of the angular velocity of the pitch is positively correlated with the magnitude of the intensity.

[0074] In a setup that conforms to MIDI specifications, the formula for calculating intensity v is as follows:

[0075]

[0076] Where ω is angular velocity, the sensor can directly return the angular velocity, ω Max 2 is the maximum angular velocity that the sensor can acquire. 7 This is the maximum intensity value allowed by the MIDI specification.

[0077] Example 5

[0078] According to the wearable virtual music interaction device described in Embodiment 1, the device can also set the actual values ​​of note parameters through the human-computer interaction module, further including: changes in intensity, continuity of notes, and discontinuity of notes.

[0079] The setting of these note parameter values ​​takes into account the expressiveness of musical performance. For example, note continuity is represented by slurs on the score, while dynamic changes correspond to the markings f and p on the score. For wind instruments, staccato is particularly distinctive, specifically known as tonguing. This playing style can express the dynamism of music and create a lively atmosphere. The aforementioned note parameter values ​​that enhance musical expressiveness can be preset in the human-computer interaction module or set as needed by real-time detection of the motion parameters of the main hand unit via a motion positioning feedback module. For example, when a large unidirectional movement of the main hand unit exceeding a set threshold is detected, settings such as increasing intensity or increasing note staccato can be activated. Another possible implementation is to set one or more foot pedals, similar to piano pedals, where pressing the corresponding pedal triggers the setting of the aforementioned note parameter values. Without any inventive effort, those skilled in the art can also obtain the note parameter values ​​described in this embodiment through other means.

[0080] In a further preferred embodiment, the key of the music being played can also be set through the human-computer interaction module, such as C major or E minor.

[0081] Example 6

[0082] According to the wearable virtual music interaction device described in Embodiment 1, the motion positioning feedback module also has a second inertial navigation module mounted on the user's forearm. This second inertial navigation module is detachable or fixed and is used to collect the forearm yaw angle (arm yaw) and calculate the arm displacement. Combined with the calculation results of the first inertial navigation module, it serves as the distance the user moves along the xyz axes in three-dimensional space. Furthermore, based on the virtual instrument range set during initialization, as well as the displacement width and orientation corresponding to each pitch, the actual pitch value played by the user can be calculated.

[0083] The arm yaw angle acquired by the second inertial navigation module is calculated using the following formula:

[0084]

[0085] Please refer to the appendix for further information. Figures 16-17Where D is the arm yaw distance relative to the x-axis, L is the length from the second inertial navigation module of the arm to the center of the motion positioning feedback module on the main hand body, and arm yaw is the forearm movement angle on the x-axis collected by the second inertial navigation module; then the obtained arm displacement D can be used to replace the sx value as the horizontal displacement on the x-axis in the body coordinate system, and thus obtain the pitch.

[0086] Furthermore, in a simplified alternative embodiment for keyboard instruments, the corresponding pitch value f can be obtained by directly calculating the following formula, thereby skipping the calculation of the D value and improving the system's calculation speed and response time:

[0087]

[0088] Where 'a' is the arm yaw angle and 'f' is the pitch, rounding the result of the calculation of 'f' allows us to directly obtain the relationship between the angle input by the gyroscope of the inertial navigation module and the pitch.

[0089] Example 7

[0090] According to the wearable virtual music interaction device described in Embodiment 3, the motion positioning feedback module further includes a second inertial navigation module mounted on the user's forearm and a third inertial navigation module mounted on the user's upper arm. The second and third inertial navigation modules are detachable or fixed. The forearm yaw angle (arm yaw) collected by the second inertial navigation module and the upper arm yaw angle (arm yaw2) and pitch angle (arm pitch2) collected by the third inertial navigation module are obtained, and the arm displacement is calculated as the distance the user moves along the xyz axes in three-dimensional space. Furthermore, the actual pitch value played by the user can be calculated based on the virtual instrument range set in the initialization, as well as the displacement width and orientation corresponding to each pitch.

[0091] The arm yaw angle (arm yaw2) and pitch angle (arm pitch2) acquired by the third inertial navigation module are calculated using the following formulas:

[0092]

[0093] D2 represents the obtained x-axis displacement of the hand, and D2Y represents the obtained y-axis displacement of the hand. The obtained arm displacement D2 can be used to replace the sx value, and D2Y can be used to replace the sy value, as the horizontal displacement on the x-axis in the body coordinate system, thereby obtaining the pitch.

[0094] In a simplified alternative embodiment for keyboard instruments, when the device is used with one hand, the longest keyboard is 2L (L is the forearm length). 2L corresponds to a forearm angle from 0 to 180 degrees, which is the length corresponding to the arm's full lateral extension and retraction. 'a' is the yaw angle of the forearm sensor. When 'a' is in the range (0°, 180°), the keyboard length is 2L. Assuming the user's forearm length is 30, the keyboard length is 60. However, if the user dislikes large, sweeping movements or is limited by a shorter arm, the range of 'a' can be appropriately reduced, for example, to (45°, 135°). In this case, the keyboard length is the square root of 2L. The actual key width is the keyboard length divided equally, but in practice, the calculation only divides the angle range equally (e.g., 88 for a standard piano). If (0°, 180°) is used as the setting, 0 corresponds to the leftmost key A2, and 180 corresponds to C5. B2 (the key to the right of A2) corresponds to the angle (0 + 180 / 88). If it's not an 88-key keyboard, the pitch range will change, and the corresponding keys for 0 and 180 degrees will also change. However, generally speaking, the smallest angle corresponds to the leftmost key, and the largest angle corresponds to the rightmost key. Both the key and angle ranges are adjustable.

[0095] Corresponding to the foregoing embodiments, in a simplified alternative embodiment for keyboard instruments, the corresponding pitch value f can be obtained by directly calculating the following formula, thereby skipping the calculation of the D2 value and improving the system's calculation speed and response time:

[0096]

[0097] Where a1 is the arm yaw, a2 is the arm yaw2, and L and R are the arm and big arm lengths, with a default value of 30, which can be modified according to the actual situation.

[0098] Example 8

[0099] After obtaining the hand displacement and calculating the corresponding pitch value, the pitch value can be determined as the actual pitch value of the player's hand at the moment of playing. For example, in the percussion instrument playing mode, it is not necessary to distinguish the precise relative displacement of each finger.

[0100] However, many other types of musical instruments require precise judgment of the relative displacement of each finger to obtain the pitch value of each finger.

[0101] In one implementation, hand displacement can be taken as the displacement of any finger (e.g., the right thumb). The pitch value of the right index finger at this time is equivalent to the pitch value of the thumb plus one, the middle finger plus two, and so on.

[0102] In other, more refined implementations, the following approach is used:

[0103] According to the wearable virtual music interaction device described in Embodiment 3, a sub-inertial navigation module or button is set on each finger to improve upon the previous embodiment and refine the actual pitch played by each finger.

[0104] By using a sub-finger inertial navigation module, the precise angle and value of the xyz axis displacement of each finger can be obtained. Furthermore, the actual pitch value can be calculated from the angle and displacement values. For example, when the initial reference pitch is the right thumb, the displacement of the other fingers relative to the right thumb can be calculated to further obtain the actual pitch value relative to the reference pitch of the right thumb. Specific examples of using the sub-finger inertial navigation module can be found in the previous embodiments.

[0105] However, considering factors such as cost, user convenience and flexibility, and signal processing speed (response time), precise finger-playing pitch can be obtained using buttons. Specifically, an elastic connection is provided near the fingertip of each finger on the main hand body. This elastic connection is connected to a button switch, with one end connected to a high level and the other end connected to a low level. Based on the triggering of the button switch, the actual pitch of the note played by each finger is calculated. (See attached diagram.) Figure 18 .

[0106] On the main hand unit, each finger has a flexible wire near its fingertip connected to a push-button switch. One end of the button is connected to a high-level voltage, and the other end is connected to an I / O port of the embedded system board of the motion positioning feedback module and grounded through a pull-down resistor. When a finger moves, if the button is not triggered, the I / O port receives a low-level voltage; if the button is triggered, it receives a high-level voltage. Each of the five buttons corresponds to one of the five fingers and one I / O port. Thus, when a finger presses its corresponding button, the processor of the motion positioning feedback module can determine which finger triggered the button by detecting the I / O port, thereby accurately determining the actual pitch value played by each finger.

[0107] Simultaneously, the yaw angle collected by the first inertial navigation module on the main hand body can be used to determine the position of the finger's pitch range, that is, to determine whether the key value of the finger is offset to the left or right. By default, the finger and the key should be parallel, but considering cross-range playing under this cost constraint, the following judgment mechanism is set: if the yaw angle collected by the first inertial navigation module turns more than 45 degrees to the right, it is determined that the pitch range played by the hand at this moment is one pitch higher than the previous moment, so the key value of the little finger is increased by 1 or the key value of the thumb is increased by 1; if the yaw angle turns more than 45 degrees to the left, it is determined that the pitch range played by the hand at this moment is one octave lower than the previous moment, so the key value of the little finger is decreased by 1 or the key value of the thumb is decreased by 1.

[0108] Example 9

[0109] According to the wearable virtual music interaction device described in Embodiment 3, gravity compensation can be performed by calculating the gravitational acceleration component or by setting a digital low-pass filter (LPF).

[0110] This embodiment aims to address the accuracy issue of accelerometers. Theoretically, the sum of the velocities of the hand and arm from start to stop should eventually equal zero. However, this is not the case in practice. Due to limitations in the accelerometer's operation and the influence of gravity on the accelerometer and related modules, irregular hand movements can affect accuracy. (See appendix.) Figure 11 If the hand is at a relative horizontal angle, the gravity that is originally on the z-axis will produce components on the x and y axes. If these components are not reduced, then when the hand tilts and shakes, this tilting and shaking will always be considered as having acceleration in the x-axis direction, and will then be continuously integrated to generate velocity and displacement, thus affecting the accuracy of the calculation results.

[0111] Gravity compensation by calculating the components of gravitational acceleration specifically includes:

[0112] The components of gravitational acceleration g in the three directions, gx, gy, and gz, are calculated using the roll angle, pitch angle, and yaw angle, respectively. The specific formulas are as follows:

[0113]

[0114] Furthermore, by subtracting the components of gravitational acceleration in the three directions from the sensor's measured values, the true accelerations of the sensor in the three directions ax, ay, and az are obtained:

[0115]

[0116] Where a'x, a'y, and a'z are the measured values ​​of sensor acceleration. (See attached reference for gravity disturbance compensation model.) Figure 12 .

[0117] In addition, a digital low-pass filter can be used to filter out interference. Gravity compensation using a digital low-pass filter (LPF) specifically includes:

[0118] The specific formulas for calculating the values ​​of gx, gy, and gz are as follows:

[0119]

[0120] Where s is the inertial parameter, gx', gy', and gz' represent the gx, gy, and gz values ​​obtained in the previous sampling period, respectively, and a'x, a'y, and a'z are the measured values ​​of the sensor acceleration.

[0121] Furthermore, by subtracting the calculated value of the LPF from the sensor's measured value, the true accelerations of the sensor in the three directions ax, ay, and az are obtained:

[0122]

[0123] Example 10

[0124] To further address the aforementioned accuracy issue, the wearable virtual music interaction device described in Embodiment 9 includes a movement flag to indicate the hand's state. Here, the movement flag (moveflag) is calculated as velocity × acceleration.

[0125] If `moveflag` < 0, it means the acceleration and velocity are in opposite directions, and the hand's velocity is decreasing; if `moveflag` = 0 and the acceleration is 0, the velocity value is reset to zero, meaning the hand has stopped. See the attached reference for acceleration, velocity, and displacement models. Figures 13-15 .

[0126] Example 11

[0127] It also features an X-axis or Z-axis linear motor mounted on the main hand or fingers to provide force feedback from the virtual instrument to the user during performance, simulating the feel of playing a real instrument. Alternatively, in a lower-cost implementation, as in the aforementioned embodiments, the buttons can also provide force feedback when triggered.

[0128] Example 12

[0129] Meanwhile, the present invention provides a computer-readable storage medium storing one or more programs, which, when executed by a computer processor, implement the calculation method of the wearable virtual music interaction device in the foregoing embodiments, to calculate the actual values ​​of the aforementioned note parameters and to provide feedback on the user's performance.

[0130] Note: The above content is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited to the specific embodiments. The terms "comprising," "including," or variations thereof indicate a non-exclusive inclusion relationship, meaning that the existence of other elements is not limited to those listed. Any changes, substitutions, or improvements that can be easily conceived by those skilled in the art within the spirit and principles of the technical solutions presented in this invention, without creative effort, are included within the scope of protection of this invention.

Claims

1. A wearable virtual music interaction device, comprising a main hand body, a motion positioning feedback module, a human-computer interaction module, and a power module; The motion positioning feedback module is installed on the main hand body and is used to acquire the user's hand movements; convert the movements into data signals and transmit them to the internal processor; the internal processor sends the generated data to the human-computer interaction module through the data transmission submodule. The human-computer interaction module parses the data to obtain the actual value of the note parameters input by the user, and plays the actual value of the note parameters through the sound-producing component to provide feedback on the user's performance. The power module is used to supply power to the motion positioning feedback module; The actual values ​​of the note parameters include at least two of the following characteristics: pitch, duration, and intensity; The motion positioning feedback module also includes a first inertial navigation module, which is used to collect the motion parameters of the hand and perform corresponding motion trajectory tracking; The motion positioning feedback module also has a second inertial navigation module installed on the user's forearm. The second inertial navigation module is detachable or fixed and is used to collect the forearm yaw angle and calculate the arm displacement as the distance the user moves along the xyz axis in three-dimensional space. Furthermore, the actual pitch value played by the user can be calculated based on the virtual instrument range set during initialization, as well as the displacement width and orientation corresponding to each pitch.

2. The wearable virtual music interaction device according to claim 1, characterized in that, The human-computer interaction module also includes a display component, which provides visual feedback on the position of the virtual instrument played by the user. The visual feedback method includes mapping the motion data obtained through the motion positioning feedback module to the keyboard position of a keyboard instrument, the key hole position of a wind instrument, the string position of a string instrument, or the striking position of a percussion instrument after initialization on the display component, so that when the user's hand moves to different positions, the corresponding keyboard position, key hole position, string position, or striking position is displayed on the display component as a trigger indicator. Virtual musical instruments include keyboard instruments, wind instruments, string instruments, and percussion instruments.

3. The wearable virtual music interaction device according to claim 1, characterized in that, The first inertial navigation module and the second inertial navigation module specifically include a gyroscope and an accelerometer.

4. The wearable virtual music interaction device according to claim 3, characterized in that, The motion positioning feedback module also has a third inertial navigation module installed on the user's upper arm. The third inertial navigation module is detachable or fixed and is used to collect the yaw angle and pitch angle of the upper arm and calculate the arm displacement as the distance the user moves along the xyz axis in three-dimensional space. Furthermore, the actual pitch value played by the user can be calculated based on the virtual instrument range set during initialization, as well as the displacement width and orientation corresponding to each pitch.

5. The wearable virtual music interaction device according to claim 1, characterized in that, The human-computer interaction module can also provide mode selection for different types of musical instruments. The types of virtual musical instruments include keyboard instruments, wind instruments, string instruments, and percussion instruments.

6. The wearable virtual music interaction device according to claim 1, characterized in that, The actual values ​​of note parameters can also be set through the human-computer interaction module, including: changes in intensity, continuity of notes, and discontinuity of notes.

7. The wearable virtual music interaction device according to claim 1, characterized in that, The motion positioning feedback module also includes a sub-inertial navigation module set on each finger, which is used to accurately collect the displacement data of each finger on the xyz axis, and then calculate the relative pitch of the actual value of the note parameters played by each finger of the user.

8. The wearable virtual music interaction device according to claim 1, characterized in that, An elastic connection is provided on the main hand body near the fingertip of each finger. The elastic connection is connected to a button switch. One end of the button switch is connected to a high level and the other end is connected to a low level. Based on the triggering of the button switch, the pitch of the actual value of the note parameter played by each finger of the user is calculated.

9. A calculation method applied to wearable virtual music interactive devices, characterized in that, The specific method for calculating the actual values ​​of the note parameters played by the user is as follows: The body coordinate system is obtained with the user's initial position as the origin. The specific method for obtaining hand movements is as follows: the Euler angles obtained by the first inertial navigation module are obtained, specifically including the pitch angle, roll angle, and yaw angle of the hand movements, which respectively represent the rotation angles in the body coordinate system with the xyz axis as the rotation axis. Based on the pitch, roll, and yaw angles of the hand movements, combined with the acceleration values ​​and movement time obtained by the first inertial navigation module, the displacement of the hand in the three directions of the x, y, and z axes is calculated. The pitch of the actual value of the note parameters played by the user is then obtained through further calculation based on the displacement. The specific method for calculating the actual pitch of the notes played by the user is as follows: A rotation matrix is ​​constructed using the rotation angle obtained from the first inertial navigation module; the rotation matrix is ​​transposed to return to the body coordinate system, multiplied by the accelerations in the three directions of the xyz axis obtained from the first inertial navigation module, and the accelerations are integrated twice to obtain the displacement of the hand in three-dimensional space, that is, the displacement of the hand relative to the origin of the body coordinate system on the xyz axis. After obtaining the displacement of the user's hand on the xyz axis relative to the origin of the body coordinate system, the pitch of the note played by the user can be calculated based on the virtual instrument range set in the initialization, as well as the displacement width and direction corresponding to each pitch.

10. The calculation method for wearable virtual music interaction devices according to claim 9, characterized in that, The specific method for calculating the actual pitch played by the user based on the hand's displacement along the xyz axes is as follows: The system initializes the width of the keyboard keys, wind instrument keyholes, string spacing, or percussion surface width. It also initializes the user's body coordinate system origin to a fixed pitch. The system calculates the hand's displacement along the x, y, and z axes within the body coordinate system, divides this displacement by the initialized keyboard key width, wind instrument keyhole width, string spacing, or percussion surface width, and rounds down to obtain the actual pitch played by the user relative to the fixed pitch after hand movement.

11. The calculation method for wearable virtual music interactive devices according to claim 9, characterized in that, Gravity compensation can be achieved by calculating the components of gravitational acceleration or by setting a digital low-pass filter (LPF).

12. The calculation method for wearable virtual music interactive devices according to claim 9, characterized in that, The motion positioning feedback module obtains the hand movement time, combines it with the angle change obtained by the first or second inertial navigation module in the motion positioning feedback module, and calculates the pitch angle, roll angle, and yaw angle of the hand movement, and further calculates the intensity and duration of the actual parameter values ​​of the notes played by the user. Furthermore, by measuring the duration of the hand holding the position after triggering the actual pitch at a specific location, i.e., the actual trigger duration, the duration of the actual parameter value of the note played by the user can be obtained. Furthermore, the intensity of the note is obtained by using the angular velocity of the pitch angle at the moment when a certain pitch is actually triggered, which is collected by the first or second inertial navigation module; specifically, the magnitude of the angular velocity of the pitch angle is positively correlated with the magnitude of the intensity.

13. The calculation method for wearable virtual music interactive devices according to claim 9, characterized in that, A movement flag is set to indicate the state of the hand, where the movement flag (moveflag) = velocity × acceleration; If motionflag < 0, it means that acceleration and velocity are in opposite directions, and the hand speed is decreasing; If moveeflag=0 and acceleration is 0, then the velocity value is cleared to zero, which means the hand stops.

14. The calculation method for wearable virtual music interactive devices according to any one of claims 9 to 10, characterized in that, The forearm yaw angle (arm yaw) of the second inertial navigation module installed on the user's forearm is obtained. The displacement D of the forearm on the x-axis is calculated using the forearm yaw angle. Using the forearm displacement D, and the y-axis displacement sy and z-axis displacement sz calculated by the first inertial navigation module, the displacement of the user's hand on the xyz axes in three-dimensional space relative to the origin of the body coordinate system is obtained. The pitch of the actual value of the note parameter played by the user is further calculated using the displacement.

15. The calculation method for wearable virtual music interactive devices according to any one of claims 9 to 10, characterized in that, The forearm yaw angle (arm yaw) of the second inertial navigation module set on the user's forearm, and the upper arm yaw angle (arm yaw2) and upper arm pitch angle (arm pitch2) of the third inertial navigation module set on the user's upper arm are obtained. The horizontal displacement D2 of the arm on the x-axis and the displacement D2Y on the y-axis are calculated. Using the forearm displacement D and the z-axis displacement sz calculated by the first inertial navigation module, the displacement of the user's hand relative to the origin of the body coordinate system on the xyz axes in three-dimensional space is obtained. The pitch of the note played by the user is further calculated using the displacement.

Images