Musical instrument keyboard key feedback and finger track cross validation method based on MR

Abstract

The invention relates to the technical field of musical instruments, and particularly discloses an MR-based musical instrument keyboard key feedback and finger trajectory cross validation method, which comprises the following steps of: capturing hand actions through MR head display equipment or an external camera, tracking a finger trajectory by utilizing a computer vision algorithm, and constructing a three-dimensional coordinate system; keyboard key pressing information is acquired by means of a pressure sensor, an optical sensor or MIDI data; fusing a finger track and key feedback data in an MR space, and matching and verifying whether a finger presses a correct key at a correct time and a correct position based on a specific algorithm by defining parameters such as a time error threshold value and a position matching radius; according to the verification result, visual enhancement feedback such as color identification, vibration prompt and audio feedback is generated, and personalized exercise suggestions such as hand shape correction and speed adjustment are provided through augmented reality information projected by the MR head-mounted display. Accurate cross validation is achieved, real-time interactive feedback is achieved, the method is suitable for various keyboard instruments, and keyboard playing accuracy and learning efficiency can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of musical instruments, and particularly to a method for cross-verification of keyboard key feedback and finger trajectory of a musical instrument based on MR. BACKGROUND

[0002] In the current music education, performance training and intelligent musical instrument development, using computer vision and MR technology for performance feedback has become a research hotspot. However, the existing technology mainly focuses on:

[0003] Gesture recognition based on camera (such as optical tracking, Leap Motion, etc.), but cannot combine keyboard feedback data for accurate cross-verification.

[0004] Augmented reality (AR) assisted teaching, but it is limited to visual guidance and lacks recognition of key feedback, and cannot verify whether the actual performance is correct.

[0005] Traditional MIDI input feedback, but without real-time interactive correction combined with finger trajectory.

[0006] Therefore, how to combine MR visual recognition while accurately capturing finger trajectory and keyboard key feedback to realize cross-verification mechanism is the key technical breakthrough of the present application.

[0007] Therefore, the method for cross-verification of keyboard key feedback and finger trajectory of a musical instrument based on MR has become a problem to be solved. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a method for cross-verification of keyboard key feedback and finger trajectory of a musical instrument based on MR, to realize cross-verification between keyboard key feedback and finger trajectory information of a musical instrument, to improve the accuracy of keyboard performance, and to provide real-time feedback and performance correction functions.

[0009] To solve the above technical problems, the technical solution provided by the present application is as follows: a method for cross-verification of keyboard key feedback and finger trajectory of a musical instrument based on MR, comprising the following steps:

[0010] S1, using MR head-mounted device or external camera system to capture user hand movements in real time; using computer vision algorithm to track finger movement trajectory and establish three-dimensional space coordinate system;

[0011] S2, using pressure sensor, optical sensor or MIDI data acquisition musical instrument keyboard to record key pressing information, record key force, timestamp and note information, and synchronize to MR processing unit;

[0012] S3, in the MR space, the finger trajectory data and the keyboard key feedback data are fused, the algorithm matching is performed, it is verified whether the finger is pressed on the correct key at the correct time and the correct position, and visual enhanced feedback is generated, the visual enhanced feedback includes color identification, vibration prompt and audio feedback;

[0013] S4, the augmented reality information is projected through the MR head-mounted device, the user is guided to correctly adjust the hand position, the personalized practice suggestion is provided according to the error detection, and the personalized practice suggestion includes hand shape correction and performance speed adjustment.

[0014] Further, in step S1, an image or a video stream containing a user's hand is collected using an MR head-mounted device or an external camera system, the collected image is preprocessed, the preprocessed image is input into an Open Pose model, and the two-dimensional coordinate positions of hand key points in the image are extracted through the Open Pose model, and the two-dimensional coordinates are converted into three-dimensional coordinates (X, Y, Z) according to the intrinsic matrix K, the extrinsic matrix [R|t] and the depth information d of the camera, and the specific conversion formula is:

[0015]

[0016] Where (u, v) is the two-dimensional coordinate of the detected finger joint node on the image plane, c x and c y are the optical center coordinates, f x and f y are the focal lengths;

[0017] The position of the camera is taken as the origin, the optical axis direction of the camera is taken as the positive direction of the Z axis, the horizontal right direction is taken as the positive direction of the X axis, and the vertical upward direction is taken as the positive direction of the Y axis, a three-dimensional coordinate system is established, and all three-dimensional coordinates of the finger joint nodes are unified to the coordinate system.

[0018] Further, the specific method for fusing the finger trajectory data and the keyboard key feedback data in the MR space is as follows:

[0019] The finger trajectory data and the keyboard key feedback data are sorted in time sequence, and the finger trajectory data set is obtained Wherein is the timestamp of the i-th finger trajectory data, is the three-dimensional coordinate;

[0020] The keyboard key feedback data set is Wherein is the timestamp of the j-th key event, is the three-dimensional position of the key, is the note information, To the key force;

[0021] With the timestamp of the keyboard key feedback data as the basis, find the data points in the finger trajectory data that are close in time, associate these data, and form a fusion data set; that is, define a time error threshold Δt, and the fusion data set The construction method is:

[0022]

[0023] Further, through algorithm matching, verify whether the finger is pressed on the correct key at the correct time and the correct position. The specific method is as follows:

[0024] (1) Position matching

[0025] Let the position range of each key in the three-dimensional space be the center coordinate And the radius ∈ p Definition; for each key event in the fusion data set If there is a finger coordinate Satisfies:

[0026]

[0027] Then record that the finger and the key have a position matching relationship, marked as PosMatch i,j =1, otherwise PosMatch i,j =0;

[0028] (2) Time matching

[0029] Let the correct performance sequence be The time error threshold is Δt; for the key event in the fusion data set If there is Make:

[0030] And And n kj =n sl ;

[0031] Then record that the key event and the correct sequence have a time matching relationship, marked as TimeMatch j =1, otherwise TimeMatch j =0;

[0032] (3) Key correctness verification

[0033] Define the key correctness verification result Verify j :

[0034] ​

[0035] Further, the specific method of generating visual enhanced feedback is as follows:

[0036] According to the verification result Verify j , the feedback signal is generated:

[0037] Color identification: if Verify j =1, color code C j =Green is assigned to the key j in the MR space, showing green; if Verify j =0, C j =Red, showing red;

[0038] Vibration prompt: define vibration intensity function V(Verify j ), when Verify j =0, V(Verify j )=V high , indicating high intensity vibration; when Verify j =1, V(Verify j )=0, indicating no vibration;

[0039] Audio feedback: define audio signal A(Verify j ), when Verify j =1, A(Verify j )=A confirm , that is, a confirmation sound is emitted; when Verify j =0, A(Verify j )=A alert , an alarm sound is emitted.

[0040] Further, the MR head-mounted device adopts Apple Vision Pro or PICO4.

[0041] The advantages of the present application compared with the prior art are:

[0042] The present application combines MR visual technology and keyboard feedback, and realizes cross verification between finger trajectory and keyboard feedback through precise algorithm matching, significantly improving the accuracy of keyboard playing precision judgment.

[0043] The present application supports multi-mode input such as visual tracking, MIDI feedback, sensor data, etc., can adapt to different hardware devices and application scenarios, and improves the applicability of the method.

[0044] The present application has real-time interactive feedback mechanism, through various ways such as color identification, vibration prompt, audio feedback, etc., to help users quickly find and correct playing errors, and improve learning and training efficiency.

[0045] The application is suitable for different types of keyboard instruments, can meet the needs of music education, professional performance training, intelligent instrument development, virtual reality music experience and other scenes, and enhances the compatibility and universality of the system. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the MR-based keyboard feedback and finger trajectory cross verification method of the application. DETAILED DESCRIPTION

[0047] Various exemplary embodiments of the application will be described in detail below with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the application unless otherwise specifically stated.

[0048] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the application or its application or uses.

[0049] Techniques, methods, and equipment known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification where appropriate.

[0050] In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative, not as limiting. Thus, other examples of the exemplary embodiments can have different values.

[0051] The MR-based keyboard feedback and finger trajectory cross verification method of the application will be described in further detail below with reference to the accompanying drawings.

[0052] In conjunction with the accompanying Figure 1 The application will be described in detail.

[0053] The MR-based keyboard feedback and finger trajectory cross verification method specifically includes the following steps:

[0054] Step S1: Finger trajectory capture and three-dimensional coordinate system establishment

[0055] An MR headset device (such as Apple Vision Pro, PICO 4) or an external camera system is used to collect images or video streams containing the user's hand, and the collected images are preprocessed, such as noise reduction and normalization. The preprocessed images are input into the OpenPose model to extract the two-dimensional coordinate positions of the hand key points in the images. According to the camera's intrinsic matrix K, extrinsic matrix [R|t] and depth information d, the two-dimensional coordinates are converted into three-dimensional coordinates (X, Y, Z), and the specific conversion formula is:

[0056]

[0057] where (u, v) is the two-dimensional coordinate of the detected finger joint on the image plane, c x and c y are the principal point coordinates, f x and f y are the focal length;

[0058] Taking the position of the camera as the origin and the direction of the optical axis of the camera as the positive direction of the Z axis, the horizontal right direction as the positive direction of the X axis, and the vertical upward direction as the positive direction of the Y axis, a three-dimensional coordinate system is established, and the three-dimensional coordinates of all finger joints are unified to this coordinate system, thereby realizing the tracking of the finger movement trajectory.

[0059] Step S2: Keyboard key feedback collection

[0060] Through the pressure sensor, optical sensor or MIDI data acquisition musical instrument keyboard, the key pressing information of the keyboard is collected, the key strength, timestamp, note information of each key are recorded, and these information are synchronized to the MR processing unit.

[0061] Step S3: Data fusion, cross-validation and feedback generation

[0062] Data fusion: sort the finger trajectory data and keyboard key feedback data in time sequence, get the finger trajectory data set as where is the timestamp of the i-th finger trajectory data, is the three-dimensional coordinate;

[0063] The keyboard key feedback data set is where is the timestamp of the j-th key event, is the three-dimensional position of the key, is the note information, is the key strength;

[0064] Take the timestamp of the keyboard key feedback data as the basis, find the data points with similar time in the finger trajectory data, associate these data to form a fusion data set; that is, define the time error threshold Δt, and the fusion data set is The construction method is:

[0065]

[0066] Cross-validation:

[0067] (1) Position matching

[0068] Let the position range of each key in the 3D space be defined by a center coordinate and a radius ∈ p ; for each key event in the fused dataset if there exists a finger coordinate satisfying:

[0069]

[0070] then record that the finger and the key have a position matching relationship, marked as PosMatch i,j = 1, otherwise PosMatch i,j = 0.

[0071] (2) Time matching

[0072] Let the correct performance sequence be and the time error threshold be Δt; for each key event in the fused dataset if there exists a time point such that:

[0073] and and

[0074] then record that the key event and the correct sequence have a time matching relationship, marked as TimeMatch j = 1, otherwise TimeMatch j = 0.

[0075] (3) Key correctness verification

[0076] Define the key correctness verification result Verify j :

[0077]

[0078] Feedback generation:

[0079] According to the verification result Verify j , generate a feedback signal:

[0080] Color identification: if Verify j = 1, then assign a color code C j = Green to the key j in the MR space, which displays green; if Verify j = 0, then C j = Red, which displays red.

[0081] Vibration prompt: define a vibration intensity function V(Verify j ), when Verify jVerify j ) = V high , indicating high intensity vibration; when Verify j = 1, V(Verify j ) = 0, indicating no vibration.

[0082] Audio feedback: define audio signal A(Verify j ), when Verify j = 1, A(Verify j ) = A confirm , that is, issue a confirmation sound; when Verify j = 0, A(Verify j ) = A alert , issue a warning sound.

[0083] Step S4: user interaction and performance optimization

[0084] Project augmented reality information through the MR head-mounted device, and provide visual guidance information for the user according to the error detection result, to help the user correctly adjust the hand position. At the same time, according to the error situation in the performance process, provide personalized practice suggestions, such as hand shape correction, performance speed adjustment, etc., to achieve the purpose of performance optimization.

[0085] The specific implementation process of the MR-based keyboard feedback and finger trajectory cross verification method of the musical instrument is as follows:

[0086] The user wears an Apple Vision Pro or PICO 4 MR head-mounted device, and enters the system preset performance interface.

[0087] The finger tracking module collects user hand images or video streams in real time through the MR head-mounted device or external camera system, and records the three-dimensional coordinate data of the hand trajectory after computer vision algorithm processing. At the same time, the keyboard feedback module collects the key pressing information of the keyboard through the pressure sensor, optical sensor or MIDI data acquisition, including key force, timestamp, note information, and synchronizes these data to the MR processing unit.

[0088] The cross verification system obtains finger trajectory data and keyboard key feedback data, and compares the finger position and key feedback according to the above data fusion and cross verification algorithm to determine whether the performance is accurate.

[0089] The MR feedback system provides real-time correction suggestions to the user through vision (such as color-coded virtual keyboard), sound (confirmation sound or warning sound), and tactile (vibration prompt) methods according to the cross verification result, guides the user to adjust the hand position and performance method, and realizes performance optimization.

[0090] The above describes the present application and its embodiments, which are not limited, and the drawings only show one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired thereby, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution are not creative, and should belong to the protection scope of the present application.

Claims

1. A method for MR-based cross-validation of keyboard key feedback and finger trajectory for a musical instrument, characterized in that, Comprise the following steps: S1, using MR head-mounted device or external camera system, real-time capture user hand action; Using computer vision algorithm to track finger movement trajectory, and establish three-dimensional space coordinate system; S2, through the pressure sensor, optical sensor or MIDI data acquisition instrument keyboard key press information, record the key strength, timestamp, note information, and synchronize to MR processing unit; S3, in the MR space, fusion finger trajectory data and keyboard key feedback data, through algorithm matching, verify whether the finger is in the correct time, correct position to press the correct key, and generate visual enhancement feedback, the visual enhancement feedback includes color identification, vibration prompt, audio feedback; S4, through the MR head-mounted device projection augmented reality information, guide the user to correctly adjust the hand position, according to the error detection, provide personalized practice suggestion, the personalized practice suggestion includes hand shape correction, performance speed adjustment.

2. The MR-based method for verifying the intersection of a keyboard key press feedback and a finger trajectory of claim 1, wherein: In step S1, the image or video stream containing the user's hand is collected using the MR head-mounted device or external camera system, and the collected image is preprocessed; The preprocessed image is input into the Open Pose model, and the two-dimensional coordinate position of the hand key point in the image is extracted through the OpenPose model; According to the camera's intrinsic matrix K, extrinsic matrix, R|t- and depth information d, the two-dimensional coordinates are converted into three-dimensional coordinates (X, Y, Z), and the specific conversion formula is: X = d x x, Y = d x y, Z = d; where (u, v) is the two-dimensional coordinate of the detected finger knuckle on the image plane, c x and c y are the principal point coordinates, f x and f y are the focal length; Taking the position of the camera as the origin, the optical axis direction of the camera as the positive direction of the Z axis, the horizontal right as the positive direction of the X axis, and the vertical upward as the positive direction of the Y axis, a three-dimensional space coordinate system is established, and all finger joint nodes are unified to the coordinate system.

3. The MR-based method for verifying the intersection of a keyboard key press feedback and a finger trajectory of claim 2, wherein: The specific method of fusing finger trajectory data and keyboard key feedback data in MR space is as follows: The finger trajectory data and the keyboard key feedback data are sorted in chronological order to obtain a finger trajectory data set wherein is a timestamp of the i-th finger trajectory data, is a three-dimensional coordinate; The keyboard key feedback data set is wherein is a timestamp of the jth key event, is a three-dimensional position of the key, is note information, is a key strength; Take the timestamp of the keyboard key feedback data as the basis, find the data points with similar time in the finger trajectory data, associate these data, and form a fusion data set; That is: define a time error threshold Δt, the fused data set The construction is:

4. The MR-based method for verifying the intersection of a keyboard key press feedback and a finger trajectory of claim 3, wherein: The specific method of verifying whether the finger is in the correct time, correct position to press the correct key through algorithm matching is as follows: (1) Position matching Let the position range of each key in three-dimensional space be defined by a center coordinate and a radius ∈ p ; for each key event in the fused data set if there exists a finger coordinate satisfying: If the position of the finger and the position of the key match, mark it as PosMatch i,j = 1, otherwise PosMatch i,j = 0; (2) Time matching Let the correct sequence of play be The time error threshold is Δt; for key events in the fused data set If there is such that: and and If the time match condition is met, the button event is marked as TimeMatch j = 1, otherwise TimeMatch j = 0; (3) Key correctness verification Defining the key correctness verification result Verify j :

5. The MR-based method for cross-verification of keyboard key feedback and finger trajectory of a musical instrument as claimed in claim 4, wherein: The specific method of generating visual enhancement feedback is as follows: According to the verification result Verify j , a feedback signal is generated: Color identification: if Verify j = 1, then assign color code C j = Green, display green; if Verify j = 0, then C j = Red, display red; Vibration prompt: Define the vibration intensity function V(Verify) j When Verify j When = 0, V(Verify) j ) = V high This indicates high-intensity vibration; when Verify j When = 1, V(Verify) j ) = 0 indicates no vibration; Audio feedback: define an audio signal A(Verify j ), when Verify j =1, A(Verify j )=A confirm , i.e. issue a confirmation sound; when Verify j =0, A(Verify j )=A alert , issue a warning sound.

6. The MR-based method of verifying the intersection of a keyboard key press feedback and a finger trajectory for a musical instrument keyboard as defined in claim 1, wherein: The MR head-mounted device uses Apple Vision Pro or PICO4.

Images