Piano playing system based on gesture recognition and region detection and construction method thereof

Abstract

The invention relates to the technical field of electronics and computers, in particular to a piano playing system based on gesture recognition and area detection and a construction method thereof, the piano playing system comprises an interactive terminal and a camera module, and a pygame library and a mediappe library are arranged in the interactive terminal; the camera shooting module is in signal connection with the interactive terminal, the interactive terminal generates an interactive window, the camera shooting module is used for displaying a shot hand picture on a screen of the interactive terminal, the pyrame library is used for drawing piano keys in the interactive window, and the media library is matched with the camera shooting module to detect joint points of a hand. The device is ingenious in design, light in weight and convenient to deploy on all platforms, only a single camera module is used for achieving three-dimensional interaction in a space area, the range of the keys can be freely adjusted in a gesture interaction mode, other specific wearable or projection devices are not needed, and the device requirement for playing is simplified.

Description

Technical Field

[0001] The present invention relates to the field of electronics and computer technology, and in particular to a piano playing system based on gesture recognition and region detection and a construction method thereof. Background Art

[0002] Existing virtual piano technologies generally have three major pain points: flat interaction logic, poor hardware adaptability, and lack of physical feedback. Existing technical solutions include:

[0003] 1. Publication No. (CN109243248A), "A Virtual Piano Based on a 3D Depth Camera Module and Its Implementation Method," utilizes a 3D depth camera module to capture a user's hand depth image in real time and transmits it to a processing module via a communication module. The module then analyzes the hand depth information based on the hand depth image and identifies the hand tapping motion. Once the gesture is correctly tracked, the system determines the keystrokes based on the key position parameters and controls the player to play the corresponding musical notes. Finally, the system provides operational feedback and performance evaluation. The projection device projects a piano key pattern and transmits the projected key position parameters to the processing module via the communication module.

[0004] 2. Publication No. (CN109448131B) "A method for constructing a virtual piano playing system based on Kinect" uses the Kinect device to complete the three-dimensional reconstruction of the scene, selects the area for creating a virtual keyboard on a suitable plane in the scene, generates virtual keys, performs key pressing detection, and sets the corresponding note playback to realize the function of playing the virtual piano. As a way of interaction between people and machines, the present invention provides a simple and convenient virtual keyboard that can be expanded to fields such as smart homes, games, and robots; it uses the OpenGL library for display, and combines the value of the fingertip position to judge the state of the keys, thereby improving the accuracy of key playing and bringing a good user experience. When realizing the virtual piano, a three-dimensional model is established to make the picture effect more three-dimensional, satisfying people's immersive experience.

[0005] 3. Publication No. (CN117850595A), "A Tactile Feedback Glove and Control Method for Virtual Piano Training," utilizes an infrared gesture recognition device to locate the left and right glove components and analyze finger gestures, thereby generating virtual hands within a virtual piano playing environment. The infrared gesture recognition device is connected to a computer via a wired connection, while the headphones are connected to the computer via a wired or Bluetooth connection. The tactile feedback glove simulates the feel of a real piano keyboard and transmits the learner's finger movements to the virtual piano scene in real time, creating a realistic and interactive experience.

[0006] Traditional solutions mostly rely on dedicated hardware (such as depth cameras, projectors, or wearable devices) to achieve gesture tracking, which not only increases the cost of use, but also limits the adaptability to different scenarios - for example, the need to flatten the projection surface or endure the constraints of gloves on flexibility. In terms of core interaction design, most systems only determine the triggering of piano keys through two-dimensional plane coordinates, simplifying the black and white keys into visual pattern differences, resulting in a lack of the key-stroke hierarchy of a real piano when playing: the triggering mechanisms of black and white keys are homogenized, which can neither simulate the physical characteristics of black keys being higher and requiring deeper pressing, nor the spatial orientation feedback during playing. What's more serious is that some solutions use a serial processing flow of "gesture confirmation-coordinate matching-sound triggering". The superposition of multiple judgment links causes delays, which seriously affects the rhythm accuracy during legato playing. Summary of the Invention

[0007] In response to the problems of the prior art, the present invention provides a method for constructing a piano playing system based on gesture recognition and area detection. The system has an ingenious design, lightweight equipment, and is easy to deploy on various platforms. It uses only a single camera module to achieve three-dimensional interaction in spatial areas, and the range of the piano keys can be freely adjusted in the form of gesture interaction. No other specific wearable or projection equipment is required, thus simplifying the equipment requirements for performance.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] The present invention provides a virtual piano playing system based on gesture recognition and area detection, which includes an interactive terminal and a camera module. The interactive terminal is provided with a pygame library and a mediapipe library; the camera module is connected to the interactive terminal by signal, the interactive terminal generates an interactive window, the camera module is used to display the captured human hand image on the screen of the interactive terminal, the pygame library is used to draw piano keys in the interactive window, the mediapipe library cooperates with the camera module to detect the joint points of the human hand, and the camera module projects the recognized part of the human hand image into the screen of the interactive terminal and cooperates with the mediapipe library to draw a gesture skeleton diagram in real time, showing the joint activity points of the human hand and the connecting lines therebetween; the five fingertip joint points are set as key-touch joint points; when the human hand moves on the screen, the mediapipe library will also move the entire gesture skeleton detected and drawn, and the drawn gesture skeleton will be drawn Figure 1 Directly fit the position of the real human hand shown in the camera module;

[0010] When there is no key-touch joint in the key detection area, it is in the initial state and the state is "waiting to be pressed". When the state is "waiting to be pressed" and a key-touch joint is detected in a specific key detection area, the key is determined to be pressed, the corresponding key sound is emitted, and then the state is changed to "pressed", and the corresponding "pressed" effect drawing is displayed on the screen of the interactive terminal;

[0011] When the state of any key is "pressed", when other key-touch joints enter the key detection area, no sound will be made and the state will not change. When no key-touch joints are detected in the key detection area, the state is replaced with "waiting to be pressed" and the "pressed" effect disappears.

[0012] Among them, the virtual piano playing system also includes a mouse connected to the interactive terminal signal. When the left button of the mouse is pressed, it is considered that the piano's damper pedal is pressed; when the left button of the mouse is released, it is considered that the piano's damper pedal is released; when the right button of the mouse is pressed, it is considered that the piano's soft pedal is pressed; when the right button of the mouse is released, it is considered that the piano's soft pedal is released.

[0013] When the sustain pedal is pressed, the sound duration will be slightly prolonged, and when the key joint leaves the detection area, the sound will still exist and the volume will gradually decrease.

[0014] If you release the sustain pedal while the note is being sustained, the sound will immediately disappear and return to the normal key touch sound detection state.

[0015] Among them, when the soft pedal is pressed, the system volume is adjusted to 50% of the original sound, so the sound volume of all piano keys will be reduced to 50% of the original sound; when the soft pedal is released, the system volume is adjusted to 100%, and the sound detection returns to normal volume.

[0016] The present invention also provides a method for constructing a piano playing system based on gesture recognition and region detection, which comprises the following steps:

[0017] Step S1: Connect the camera module to the interactive terminal and open the interactive terminal to generate an interactive window; Step S2: Use the pygame library to draw piano keys in the interactive terminal and display them on the screen;

[0018] Step S3: Lift the human hand and place it within the detection area of ​​the camera module. Use the MediaPipe library in conjunction with the camera module to detect the joints of the human hand. The camera module projects the image of the recognized hand onto the screen of the interactive terminal. The MediaPipe library then coordinates the real-time drawing of a gesture skeleton diagram, displaying the joint motion points and the connecting lines between them.

[0019] Step S4, the five fingertip joints are set as touch key joints; when the hand moves on the screen, the mediapipe library will also move the entire gesture skeleton drawn by the detection. Figure 1 Directly fit the position of the real human hand shown in the camera module; step S5, the piano keys drawn in step S2 correspond to the key position detection area of ​​the specific piano keys of the piano;

[0020] When there is no key-touch joint in the key detection area, it is in the initial state and the state is "waiting to be pressed". When the state is "waiting to be pressed" and a key-touch joint is detected in a specific key detection area, the key is determined to be pressed, the corresponding key sound is emitted, and then the state is changed to "pressed", and the corresponding "pressed" effect drawing is displayed on the screen of the interactive terminal;

[0021] Step S6: When the state of any key is "pressed", when other key-touch joints enter the key position detection area, no sound is produced and the state remains unchanged. When no key-touch joints are detected in the key position detection area, the state is changed to "waiting to be pressed" and the "pressed" effect disappears.

[0022] Step S7: cyclically and repeatedly monitor the key position information in the key position detection area and the receipt of the refresh gesture, and wait for the next playing.

[0023] The method for drawing piano keys in the interactive terminal using the pygame library in step S2 is:

[0024] The interactive window size generated by the interactive terminal is 640*480; when using the pygame library to draw piano keys, starting from the upper left corner of the interactive window, draw two white horizontal lines 200 units and 300 units downward as the upper and lower edges of the white key recognition area; starting from the left side of the interactive window, draw two white vertical lines with a thickness of 2 40 units and 590 units to the right as the leftmost and rightmost ends of the white key recognition area; starting from the leftmost white vertical line, draw a white vertical line with a length of 100 units and a thickness of 2 50 units to the right each time until it reaches the rightmost side of the white horizontal line; the above white lines constitute 11 adjacent white rectangles as the recognition areas of the 11 white piano keys;

[0025] Starting from the top of the interactive window, draw 7 pairs of black horizontal lines with a length of 30 units and a thickness of 3 at 180 units and 250 units downwards. The left and right positions on the screen are as follows: taking the leftmost end of the interactive window as a reference, from left to right at a distance of 65, 95, 130, 160, 215, 245, 275, 305, 335, 365, 415, 445, 450, and 480 units, draw black vertical lines with a thickness of 3 and a height of 70 units; the above black lines form 7 black rectangular frames, which serve as the recognition areas of the 7 black piano keys.

[0026] Among them, the recognition area of ​​a single piano black key is a black rectangle, the recognition area of ​​all piano black keys is above the display position of the recognition area of ​​all piano white keys, and adjacent piano black keys are arranged at intervals; the recognition area of ​​piano white keys is composed of multiple white rectangles of different notes connected to each other, and two adjacent white keys are arranged in a row and there is no spacing between them.

[0027] Among them, when the state of the white piano key is "pressed", the drawing effect is a white rectangle with a length of 20, a width of 20, and a downward protrusion thickness of 2, with the bottom edge of the key detection area as the length; when the state of the black piano key is "pressed", the drawing effect is a black rectangle with a length of 30, a width of 2, and a downward protrusion thickness of 2, with the bottom edge of the key detection area as the length.

[0028] Among them, when the state of the white piano key is "pressed" or "to be pressed", the monitoring conditions are: only two-dimensional detection of the xy area, that is, the plane position detection of the key touch joint point. The xy area is the picture captured by the camera module and displayed on the screen. When the position of the key touch joint point is within the detection range of a certain white rectangle, the "pressed" state of the key can be triggered.

[0029] Among them, when the state of the black piano key is "pressed" or "to be pressed", the monitoring conditions are: three-dimensional detection of the xyz area, that is, depth detection conditions are added on the basis of two-dimensional plane position detection. The xy area is the picture captured by the camera module and displayed on the screen. The z direction is the direction perpendicular to the camera module. When the touch key joint point is in a plane position within a certain black rectangular detection range and the depth condition is also met, the "pressed" state of the key can be triggered.

[0030] Beneficial effects of the present invention:

[0031] The present invention is cleverly designed, lightweight, and easy to deploy on various platforms. It uses only a single camera module to achieve three-dimensional interaction in a spatial area, and the range of the keys can be freely adjusted in the form of gesture interaction. No other specific wearable or projection equipment is required, simplifying the equipment requirements for performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a usage state diagram of a virtual piano playing system based on gesture recognition and area detection according to the present invention.

[0033] Figure 2 This is a usage status diagram within the interactive window of the present invention.

[0034] exist Figures 1 to 2 Reference numerals in the figures include:

[0035] 1. Interactive terminal; 2. Camera module; 3. Mouse. DETAILED DESCRIPTION

[0036] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the embodiments and the accompanying drawings. The contents mentioned in the embodiments are not intended to limit the present invention. The present invention will be described in detail below with reference to the accompanying drawings.

[0037] Example 1

[0038] In the first embodiment of the present application, a virtual piano playing system based on gesture recognition and area detection is provided, which includes an interactive terminal 1 and a camera module 2, the camera module 2 is a depth camera, and the interactive terminal 1 is provided with a pygame library and a mediapipe library; the camera module 2 is connected to the interactive terminal 1 by signal, the interactive terminal 1 generates an interactive window, the camera module 2 is used to display the captured human hand image on the screen of the interactive terminal 1, the pygame library is used to draw piano keys in the interactive window, the mediapipe library cooperates with the camera module 2 to detect the joint points of the human hand, and the camera module 2 puts the recognized part of the human hand image into the screen of the interactive terminal 1 and cooperates with the mediapipe library to draw a gesture skeleton diagram in real time, showing the joint activity points of the human hand and the connecting lines between them; the five fingertip joints are set as key touch joints; when the human hand moves on the screen, the mediapipe library will also move the entire detected gesture skeleton, and the drawn gesture skeleton Figure 1 Directly fit the position of the real human hand shown in the camera module 2;

[0039] Furthermore, the pygame library's scheme for drawing piano keys is as follows: the interactive window size generated by the interactive terminal 1 is 640*480; when using the pygame library to draw piano keys, starting from the upper left corner of the interactive window, two white horizontal lines are drawn 200 units and 300 units downward as the upper and lower edges of the white key recognition area; starting from the left side of the interactive window, two white vertical lines with a thickness of 2 are drawn 40 units and 590 units to the right as the leftmost and rightmost ends of the white key recognition area; starting from the leftmost white vertical line, a white vertical line with a length of 100 units and a thickness of 2 is drawn 50 units to the right each time, until it reaches the rightmost side of the white horizontal line; the above white lines constitute 11 adjacent white rectangles as the recognition areas of the 11 white piano keys;

[0040] Starting from the top of the interactive window, draw 7 pairs of black horizontal lines with a length of 30 units and a thickness of 3 at 180 units and 250 units downwards. The left and right positions on the screen are as follows: taking the leftmost end of the interactive window as a reference, from left to right at a distance of 65, 95, 130, 160, 215, 245, 275, 305, 335, 365, 415, 445, 450, and 480 units, draw black vertical lines with a thickness of 3 and a height of 70 units; the above black lines form 7 black rectangular frames, which serve as the recognition areas of the 7 black piano keys.

[0041] Among them, the recognition area of ​​a single piano black key is a black rectangle, the recognition area of ​​all piano black keys is above the display position of the recognition area of ​​all piano white keys, and adjacent piano black keys are arranged at intervals; the recognition area of ​​piano white keys is composed of multiple white rectangles of different notes connected to each other, and two adjacent white keys are arranged in a row and there is no spacing between them.

[0042] Among them, when the state of the white piano key is "pressed", the drawing effect is a white rectangle with a length of 20, a width of 20, and a downward protrusion thickness of 2, with the bottom edge of the key detection area as the length; when the state of the black piano key is "pressed", the drawing effect is a black rectangle with a length of 30, a width of 2, and a downward protrusion thickness of 2, with the bottom edge of the key detection area as the length.

[0043] When there is no key-touch joint in the key detection area, it is the initial state, and the state is "waiting to be pressed". When the state is "waiting to be pressed" and a key-touch joint is detected in a specific key detection area, it is determined that the key is pressed, and the corresponding key sound is emitted. Then the state is changed to "pressed", and the corresponding "pressed" effect drawing is displayed on the screen of the interactive terminal 1;

[0044] When the state of any key is "pressed", when other key-touch joints enter the key detection area, no sound will be made and the state will not change. When no key-touch joints are detected in the key detection area, the state is replaced with "waiting to be pressed" and the "pressed" effect disappears.

[0045] Specifically, when using the embodiment of the present application, the camera module 2 is connected to the interactive terminal 1 (which can be a computer or other intelligent terminal). The person is about two arms' distance away from the camera, and the interactive terminal 1 program is opened. The program will generate a 640*480 size window, and the depth camera will start working at the same time; the depth camera will monitor the screen in real time and obtain the depth information of the gesture joints. The user can see the real-time action feedback captured by the camera from this window. The pygame library is used to complete the drawing of piano keys: the recognition area of ​​a single black key is a black rectangle, and all black keys are displayed a little higher than the white key area, and there is a certain fixed regular spacing between the black keys; the recognition area of ​​multiple white keys is composed of white rectangles of different tones connected to each other, and there is no spacing between two adjacent white keys; it conforms to the arrangement design of a real piano. The MediaPipe library is used to detect hand joints. Lift your hand and place it within the depth camera's detectable area. The library's functions automatically identify the hand on the screen. Once detected, a real-time gesture skeleton is drawn on the screen, showing the hand's joint motion points and their connections. Five fingertip joints are set as touch joints. Each rectangle drawn on the screen corresponds to a specific piano key detection area; black rectangles represent black key detection areas, and white rectangles represent white key detection areas. The gestures drawn by the MediaPipe library use the five fingertip joints as touch joints (consistent with the logic of normal piano fingertip touch). As the hand moves on the screen, the MediaPipe library moves the entire detected gesture skeleton, ensuring the drawn gesture matches the actual hand position shown on the camera. The performer can adjust the touch joint positions by moving their hand while watching the screen.

[0046] The embodiment of the present application is cleverly designed, the device is lightweight, and is easy to deploy on various platforms. It only uses a single camera module 2 to achieve three-dimensional interaction in the spatial area, and the range of the keys can be freely adjusted in the form of gesture interaction. No other specific wearable or projection equipment is required, simplifying the equipment requirements for performance.

[0047] In an embodiment of the present application, the virtual piano playing system further includes a mouse 3 connected to the interactive terminal 1. When the left button of the mouse 3 is pressed, the piano's damper pedal is considered pressed. When the left button of the mouse 3 is released, the piano's damper pedal is considered released. When the right button of the mouse 3 is pressed, the piano's soft pedal is considered pressed. When the right button of the mouse 3 is released, the piano's soft pedal is considered released. When the damper pedal is pressed, the duration of the key-touch sound is slightly prolonged. When the key-touch joint leaves the detection area, the sound continues, and the volume gradually decreases. If the damper pedal is released while the sound is being sustained, the sound immediately disappears, returning to the normal key-touch sound detection state. When the soft pedal is pressed (i.e., the right button of the mouse 3 is pressed), the system volume is adjusted to 50% of the original volume, so the sound volume of all piano keys is reduced to 50% of the original volume. When the soft pedal is released, the system volume is adjusted to 100%, returning to the normal volume sound detection state.

[0048] Specifically, when using it, when you press the damper pedal (i.e., press the left mouse button 3), the key touch sound time will be slightly extended, and when the key touch joint leaves the detection area, the sound will also exist, and the volume will slowly decrease, which is consistent with the real sustain effect. If you release the damper pedal during sustain, the sound will immediately disappear; it will return to the normal key touch sound detection state; the specific functions are as follows: each piano key sound effect has two playback versions, one is a short sound effect played when the damper pedal is released, and the other is a long sound effect played when the damper pedal is pressed; the two sound effects with different durations simulate the playing effect of the real piano damper pedal; when the damper pedal is pressed (i.e., press the left mouse button 3), the sound effect played is the longer version. As long as the damper pedal is not released, the sound effect of triggering a key will play completely, for 5 to 7 seconds; but if the damper pedal is released while the long sound effect is still playing, the sound effect will stop; simulating the real piano sustain effect. When the soft pedal is depressed, the system volume is adjusted to 50% of its original level, reducing the volume of all keys to 50% of their original level. Releasing the soft pedal adjusts the system volume to 100%, restoring the normal volume level for sound detection. The interactive terminal 1 automatically adjusts the system volume to 100% when the program is opened. This embodiment of the application eliminates the need for external hardware such as projectors and wearable devices, and can achieve the functions of sustain and bass pedals.

[0049] Example 2

[0050] A second embodiment of the present application provides a method for constructing a piano playing system based on gesture recognition and region detection, which includes the following steps:

[0051] Step S1: Connect the camera module 2 to the interactive terminal 1, and open the interactive terminal 1 to generate an interactive window;

[0052] Step S2: Use the pygame library to draw piano keys in the interactive terminal 1 and display them on the screen;

[0053] Step S3: Lift the human hand and place it within the detection area of ​​the camera module 2. Use the mediapipe library in conjunction with the camera module 2 to detect the joint points of the human hand. The camera module 2 projects the image of the recognized human hand onto the screen of the interactive terminal 1. The mediapipe library is used to draw a real-time gesture skeleton diagram, showing the joint motion points of the human hand and the connecting lines between them.

[0054] Step S4, the five fingertip joints are set as touch key joints; when the hand moves on the screen, the mediapipe library will also move the entire gesture skeleton drawn by the detection. Figure 1 Directly fit the position of the real human hand shown in the camera module 2;

[0055] Step S5: The piano keys drawn in step S2 correspond to the key position detection area of ​​the specific piano keys of the piano;

[0056] When there is no key-touch joint in the key detection area, it is the initial state, and the state is "waiting to be pressed". When the state is "waiting to be pressed" and a key-touch joint is detected in a specific key detection area, it is determined that the key is pressed, and the corresponding key sound is emitted. Then the state is changed to "pressed", and the corresponding "pressed" effect drawing is displayed on the screen of the interactive terminal 1;

[0057] Step S6: When the state of any key is "pressed", when other key-touch joints enter the key position detection area, no sound is produced and the state remains unchanged. When no key-touch joints are detected in the key position detection area, the state is changed to "waiting to be pressed" and the "pressed" effect disappears.

[0058] Step S7: cyclically and repeatedly monitor the key position information in the key position detection area and the receipt of the refresh gesture, and wait for the next playing.

[0059] Wherein, the method for drawing piano keys in the interactive terminal 1 using the pygame library in step S2 is as follows: the size of the interactive window generated by the interactive terminal 1 is 640*480; when using the pygame library to draw piano keys, starting from the upper left corner of the interactive window, two white horizontal lines are drawn 200 units and 300 units downward as the upper and lower edges of the white key recognition area of ​​the piano key; starting from the left side of the interactive window, two white vertical lines with a thickness of 2 are drawn 40 units and 590 units to the right as the leftmost and rightmost ends of the white key recognition area; starting from the leftmost white vertical line, a white vertical line with a length of 100 units and a thickness of 2 is drawn 50 units to the right each time, until the rightmost side of the white horizontal line; the above white lines constitute 11 adjacent white rectangles as the recognition areas of 11 piano white keys;

[0060] Starting from the top of the interactive window, draw 7 pairs of black horizontal lines with a length of 30 units and a thickness of 3 at 180 units and 250 units downwards. The left and right positions on the screen are as follows: taking the leftmost end of the interactive window as a reference, from left to right at a distance of 65, 95, 130, 160, 215, 245, 275, 305, 335, 365, 415, 445, 450, and 480 units, draw black vertical lines with a thickness of 3 and a height of 70 units; the above black lines form 7 black rectangular frames, which serve as the recognition areas of the 7 black piano keys.

[0061] In the embodiment of the present application, the recognition area for a single black piano key is a black rectangle. The recognition areas for all black piano keys are located above the display locations for the recognition areas for all white piano keys, with adjacent black piano keys spaced apart. The recognition area for white piano keys is composed of a plurality of interconnected white rectangles of different tones, with adjacent white keys arranged side by side in a row with no spacing. When a white piano key is in the "pressed" state, the resulting image is a white rectangle with a width of 20 and a downward projection of 2, using the bottom edge of the key detection area as its length. When a black piano key is in the "pressed" state, the resulting image is a black rectangle with a width of 30 and a downward projection of 2, using the bottom edge of the key detection area as its length.

[0062] In the embodiment of the present application, when the state of a white piano key is "pressed" or "to be pressed", the monitoring condition is: only two-dimensional detection of the xy area, that is, the plane position detection of the key joint point, the xy area is the picture captured by the camera module 2 and displayed on the screen, when the position of the key joint point is within the detection range of a certain white rectangle, the "pressed" state of the key can be triggered. Among them, when the state of a black piano key is "pressed" or "to be pressed", the monitoring condition is: three-dimensional detection of the xyz area, that is, adding a depth detection condition on the basis of the two-dimensional plane position detection, the xy area is the picture captured by the camera module 2 and displayed on the screen, the z direction is the direction perpendicular to the camera module 2, when the plane position of the key joint point is within the detection range of a certain black rectangle and the depth condition is also met, the "pressed" state of the key can be triggered.

[0063] Specifically, the embodiment of the present application synchronously calculates the plane coordinates and spatial depth information of the hand joints in the environment of a single camera module 2, and creates a trigger logic of "white key area priority, black key depth lock": the white keys can instantly make sounds as long as the fingertips fall into the preset area of ​​the XY plane, simulating the wide and easy-to-trigger characteristics of real white keys; the black keys are superimposed with millimeter-level depth detection conditions, and the sound effects will only be activated when the fingertips meet both the plane positioning and the specific depth threshold, accurately reproducing the tactile memory of the real black keys that are higher and need to be deliberately pressed down; this hierarchical judgment mechanism not only circumvents the limitations of wearable devices on flexibility, but also encodes the physical structural characteristics of the piano into spatial interaction rules, so that the virtual performance can obtain a playing experience close to that of a physical instrument.

[0064] In the embodiment of the present application, in step S7, the program of the interactive terminal 1 will repeatedly monitor and detect key position information and refresh the drawing of gestures in a loop. The specific method is as follows: three variables ctime, fps, and ptime are set; initially, these three variables are all 0. In the loop, the time library is called, time.time() is assigned to the ctime variable, and fps is calculated as 1 / (ctime-ptime). The value of ctime is then assigned to ptime; the next round of program monitoring loop continues; therefore, the fps here is refreshed by the number of times the program runs in the loop, which is considered to be the running frame rate. In the upper left corner of the window, mediapipe will display the frame rate.

[0065] The program records the xy position of each key joint in each frame. The key joint is considered "pressed" one frame after the touch instant, while the key joint is considered "not pressed" one frame before the touch instant, indicating the key is still in the "pending" state. Dividing the y coordinates of the key joints in two consecutive frames by 2 gives the velocity at the time of key touch. Touch velocity touch_velocity = abs(y coordinate of the key joint in the frame before the touch instant – y coordinate of the key joint in the frame after the touch instant) / 2. The faster the key touch velocity, the stronger the touch force and the louder the sound. Use pygame.mixer.Sound() to set the sound effects for all keys. Variables with sound effects have a play() method to play the sound effects. Before executing play(), three conditions must be checked: the key touch velocity, whether the sustain pedal is pressed, and whether the soft pedal is pressed. The program tries to simulate the dynamic, soft, and continuous touch of a real piano as closely as possible. The present embodiment also sets a variable for the sound effect, a set_volume() method, which sets the volume of the sound effect playback through set_volume(touch_velocity / 10). The valid range of the input parameter is 0 to 1. If the range exceeds this, the sound effect is played at the maximum volume by default.

[0066] Compared with the traditional solution, the embodiment of the present application realizes dual innovation in user experience and technical architecture; at the user experience level, the coupling of depth conditions and plane detection constructs a three-dimensional interactive space - users can naturally distinguish the triggering of black and white keys through gesture depth without external devices. At the technical architecture level, a regional-depth joint monitoring mode of parallel computing is adopted to compress the serial multi-level judgment in the traditional solution into a single-frame synchronous analysis, so that the response delay is reduced to less than 20 milliseconds, achieving a better playing experience. In addition, the pure visual solution gets rid of the dependence on dedicated hardware such as Kinect, and can be implemented through a depth camera, making the system compatible with multiple platforms such as mobile phones, tablets, and computers, and can be used in scenes such as classrooms, stages, and homes. It provides a lightweight technical path for the popularization of virtual musical instruments.

[0067] The purpose of this invention is to simulate the physical key travel differences of a real piano through pure visual interaction technology, using an ordinary depth camera to synchronously detect the planar position and spatial depth of the fingertips: white keys can be triggered only by the fingertips falling into the XY area, while black keys must additionally meet preset depth conditions, thereby restoring the high and low tactile differences between the black and white keys. At the same time, external hardware such as projectors and wearable devices are discarded, and the functions of sustain and bass pedals can be realized; while reducing deployment costs and enhancing the immersiveness of performance, this provides a more novel, lightweight, and highly realistic interactive solution for virtual piano instruments.

[0068] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention is disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technology of the present invention are all within the scope of the technical solution of the present invention without departing from the content of the technical solution of the present invention.

Claims

1. A virtual piano playing system based on gesture recognition and region detection, characterized by: It includes an interactive terminal and a camera module, wherein the interactive terminal is provided with a pygame library and a mediapipe library; the camera module is connected to the interactive terminal signal, the interactive terminal generates an interactive window, the camera module is used to display the captured human hand image on the screen of the interactive terminal, the pygame library is used to draw piano keys in the interactive window, the mediapipe library cooperates with the camera module to detect the joint points of the human hand, and the camera module projects the recognized part of the human hand image into the screen of the interactive terminal and cooperates with the mediapipe library to draw a gesture skeleton diagram in real time, showing the joint activity points of the human hand and the connecting lines between them; the five fingertip joints are set as key-touch joints; when the human hand moves on the screen, the mediapipe library will also move the entire detected gesture skeleton along with it, and keep the drawn gesture skeleton diagram aligned with the position of the real human hand shown in the camera module; When there is no key-touch joint in the key detection area, it is in the initial state, and the state is "waiting to be pressed". When the state is "waiting to be pressed" and a key-touch joint is detected in a specific key detection area, the key is determined to be pressed, and the corresponding key sound is emitted. Then the state is changed to "pressed", and the corresponding "pressed" effect is displayed on the screen of the interactive terminal; When the state of any key is "pressed", when other key-touch joints enter the key detection area, no sound will be made and the state will not change. When no key-touch joints are detected in the key detection area, the state is replaced with "waiting to be pressed" and the "pressed" effect disappears.

2. The virtual piano playing system based on gesture recognition and area detection according to claim 1, characterized in that: The virtual piano playing system also includes a mouse connected to the interactive terminal signal. When the left button of the mouse is pressed, it is considered that the piano's damper pedal is pressed; when the left button of the mouse is released, it is considered that the piano's damper pedal is released; when the right button of the mouse is pressed, it is considered that the piano's soft pedal is pressed; when the right button of the mouse is released, it is considered that the piano's soft pedal is released.

3. The virtual piano playing system based on gesture recognition and area detection according to claim 2, characterized in that: When the sustain pedal is pressed, the key touch sound time will be slightly prolonged, and when the key touch joint leaves the detection area, the sound will still exist and the volume will gradually decrease; if the sustain pedal is released during the sustain sound, the sound will immediately disappear and return to the normal key touch sound detection state.

4. The virtual piano playing system based on gesture recognition and area detection according to claim 2, characterized in that: When the soft pedal is pressed, the system volume is adjusted to 50% of the original volume, so the sound volume of all piano keys will be reduced to 50% of the original sound; when the soft pedal is released, the system volume is adjusted to 100% and the sound detection returns to normal volume.

5. A method for constructing a piano playing system based on gesture recognition and region detection, characterized in that: The following steps are involved: Step S1: Connect the camera module to the interactive terminal, open the interactive terminal to generate an interactive window; Step S2: Use the pygame library to draw piano keys in the interactive terminal and display them on the screen; Step S3: Lift the human hand and place it within the detection area of ​​the camera module. Use the MediaPipe library in conjunction with the camera module to detect the joints of the human hand. The camera module projects the image of the recognized hand onto the screen of the interactive terminal. The MediaPipe library then coordinates the real-time drawing of a gesture skeleton diagram, displaying the joint motion points and the connecting lines between them. Step S4: The five fingertip joints are set as touch joints. When the hand moves on the screen, the mediapipe library will also move the entire detected gesture skeleton, and the drawn gesture skeleton image will always fit the position of the real hand shown in the camera module. Step S5: The piano keys drawn in step S2 correspond to the key position detection area of ​​the specific piano keys of the piano; When there is no key-touch joint in the key detection area, it is in the initial state, and the state is "waiting to be pressed". When the state is "waiting to be pressed" and a key-touch joint is detected in a specific key detection area, the key is determined to be pressed, and the corresponding key sound is emitted. Then the state is changed to "pressed", and the corresponding "pressed" effect is displayed on the screen of the interactive terminal; Step S6: When the state of any key is "pressed", when other key-touch joints enter the key position detection area, no sound is produced and the state remains unchanged. When no key-touch joints are detected in the key position detection area, the state is changed to "waiting to be pressed" and the "pressed" effect disappears. Step S7: cyclically and repeatedly monitor the key position information in the key position detection area and the receipt of the refresh gesture, and wait for the next playing.

6. The method for constructing a piano playing system based on gesture recognition and area detection according to claim 5, characterized in that: The method for drawing piano keys in the interactive terminal using the pygame library in step S2 is: The interactive window size generated by the interactive terminal is 640*480. When using the pygame library to draw piano keys, two white horizontal lines are drawn 200 units and 300 units downward from the upper left corner of the interactive window as the upper and lower edges of the white key recognition area. Starting from the left side of the interactive window, draw two white vertical lines with a thickness of 2 at 40 units and 590 units to the right, which serve as the leftmost and rightmost ends of the white key recognition area. Starting from the leftmost white vertical line, draw a white vertical line with a length of 100 units and a thickness of 2 every 50 units to the right, until it reaches the rightmost side of the white horizontal line. The above white lines form 11 adjacent white rectangles, which serve as the recognition area for the 11 white piano keys. Starting from the top of the interactive window, draw 7 pairs of black horizontal lines with a length of 30 units and a thickness of 3 at 180 units and 250 units downwards. The left and right positions on the screen are as follows: taking the leftmost end of the interactive window as a reference, from left to right at a distance of 65, 95, 130, 160, 215, 245, 275, 305, 335, 365, 415, 445, 450, and 480 units, draw black vertical lines with a thickness of 3 and a height of 70 units; the above black lines form 7 black rectangular frames, which serve as the recognition areas of the 7 black piano keys.

7. The method for constructing a piano playing system based on gesture recognition and region detection according to claim 6, characterized in that: The recognition area of ​​a single piano black key is a black rectangle. The recognition areas of all piano black keys are above the display position of the recognition areas of all piano white keys, and adjacent piano black keys are arranged at intervals; the recognition area of ​​piano white keys is composed of multiple white rectangles of different notes connected to each other, and two adjacent white keys are arranged in a row and side by side with no spacing.

8. The method for constructing a piano playing system based on gesture recognition and region detection according to claim 6, characterized in that: When the state of a white piano key is "pressed", the effect of its drawing is a white rectangle with a length of 20, a width of 20, and a downward protrusion thickness of 2, using the bottom edge of the key detection area as the length; when the state of a black piano key is "pressed", the effect of its drawing is a black rectangle with a length of 30, a width of 2, and a downward protrusion thickness of 2, using the bottom edge of the key detection area as the length.

9. The method for constructing a piano playing system based on gesture recognition and area detection according to claim 6, characterized in that: When the status of a white piano key is "pressed" or "to be pressed", the monitoring conditions are: only two-dimensional detection of the xy area, that is, the plane position detection of the key touch joint point. The xy area is the picture captured by the camera module and displayed on the screen. When the position of the key touch joint point is within the detection range of a certain white rectangle, the "pressed" state of the key can be triggered.

10. The method for constructing a piano playing system based on gesture recognition and area detection according to claim 6, characterized in that: When the status of a black piano key is "pressed" or "to be pressed", the monitoring conditions are: three-dimensional detection of the xyz area, that is, depth detection conditions are added on the basis of two-dimensional plane position detection. The xy area is the picture captured by the camera module and displayed on the screen. The z direction is the direction perpendicular to the camera module. When the touch joint point is within a black rectangular detection range in the plane position and the depth condition is also met, the "pressed" state of the key can be triggered.

Images