Piano gesture recognition method for complex scene and electronic device

Abstract

The present application relates to a piano gesture recognition method for complex scenes and electronic equipment, the method comprises the following steps: using a YOLO model to identify the piano and gesture positions in a piano playing image, and generating a piano bounding box and a gesture bounding box respectively; in each bounding box, using a SAM model to extract the edges of the gesture and the piano, and segmenting the gesture area and the piano area, thereby screening effective gestures; using a key point detection algorithm to obtain a key point set on each effective gesture and a feature matrix of the key points, and constructing a graph structure; based on the graph and the gesture area data of the effective gestures respectively, obtaining local features and global features of the gestures; splicing the local features and the global features of the gestures to obtain final gesture features, and classifying the final gesture features to obtain a predicted label. Compared with the prior art, the present application has the advantages of screening effective gestures, improving the accuracy and reliability of gesture recognition, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence assisted piano teaching, and particularly relates to a piano gesture recognition method for complex scenes and electronic equipment. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, AI assisted piano teaching has gradually attracted attention. Its significance lies in providing timely and personalized feedback for learners, improving learning efficiency, and reducing the formation of bad habits. In the process of piano teaching, judging the correctness of the user's playing gesture is one of the key links, which can effectively help learners to adjust and correct their posture. To achieve this purpose, accurate gesture detection and segmentation are needed for each frame in the user's piano video. However, due to the complexity of the user's playing scene, the image not only contains hands, but also includes pianos and other objects. The hands may be on the piano (i.e. valid gesture) or on other positions such as legs (i.e. invalid gesture). Judging and correcting invalid gestures not only has no meaning, but also affects the accuracy of the overall recognition. Therefore, the gesture recognition method not only needs to detect the hands, but also needs to ensure that the gesture is a valid playing gesture. In addition, existing methods often extract gesture features from a single granularity, resulting in incomplete extraction of gesture information, especially in complex scenes, the neglect of global information will reduce the accuracy of classification.

[0003] After searching, Chinese invention patent application publication No. CN115798033A discloses a piano training method, system and device based on gesture recognition and storage medium. By collecting the hand movement video of the user playing the piano, the key frame image of the hand movement in the video is extracted, the key points of the hand are extracted according to the key frame image, the key points in the image are compared with the key points in the pre-stored standard key point sequence, the standard degree of the hand posture is generated, and then the key frame image with a standard degree lower than the pre-set threshold is extracted to generate an error gesture image. Training guidance information is generated according to the error gesture image and the corresponding standard image, that is, the user's gesture action evaluation can be obtained in real time, which is convenient for the user's targeted training.

[0004] After retrieval, Chinese invention patent application publication No. CN113255470B discloses a multi-modal piano accompaniment system and method based on hand gesture estimation. The system comprises: a data acquisition module that acquires gesture information and piano sound information during performance; a data recognition module that identifies key nodes in the gesture information according to the piano sound information and a preset algorithm, and sends them to a data comparison module; the data comparison module compares the key node recognition results with a standard database to obtain error gesture information and error piano sound segment information during performance; and a result display module that labels the error gesture information and error piano sound segment information on the music score and feeds back to the user in a multi-modal form. The present application extracts and identifies gesture features in multiple ways through multiple algorithms, thereby effectively extracting the temporal and spatial feature information of the video frames, while relying on the microphone to greatly reduce the algorithmic consumption, and improving the piano accompaniment effect through the positioning of the small joints.

[0005] Both of the two existing invention applications directly recognize gestures without screening effective gestures, resulting in low accuracy of piano gesture recognition.

[0006] How to effectively screen out invalid gestures in complex scenes and improve the accuracy of piano gesture recognition has become a technical problem to be solved. SUMMARY

[0007] The purpose of the present application is to overcome the defects of the prior art and provide a piano gesture recognition method and electronic device for complex scenes.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] According to one aspect of the present application, a piano gesture recognition method for complex scenes is provided, which comprises the following steps:

[0010] Step S1, using a YOLO model to identify the positions of the piano and the gesture in the piano performance image, and generating a piano bounding box and a gesture bounding box respectively;

[0011] Step S2, in each bounding box, using a SAM model to extract the edges of the gesture and the piano, and segmenting out the gesture region and the piano region;

[0012] Step S3, screening effective gestures according to the segmented gesture region and piano region;

[0013] Step S4, using a key point detection algorithm to obtain a set of key points on each effective gesture and a feature matrix of the key points, and constructing a graph structure;

[0014] Step S5, based on the graph and the gesture region data of the effective gesture respectively, obtaining local features and global features of the gesture;

[0015] Step S6: Concatenate the local features and global features to obtain the final gesture features, and classify the final gesture features to obtain the predicted labels.

[0016] Preferably, step S2 specifically involves: in each bounding box, using the SAM model to extract the gesture edge and the piano edge respectively, the area enclosed by the gesture edge is denoted as the gesture region, and the area enclosed by the piano edge is denoted as the piano region.

[0017] Preferably, the process of filtering out valid gestures includes: calculating the overlap ratio between the gesture area and the piano area; if the overlap ratio is less than a set threshold, the gesture is considered invalid and is removed; otherwise, the gesture is retained as a valid gesture.

[0018] Preferably, the process of constructing the graph structure includes: for valid gestures, using the key point detection algorithm f(·) to obtain the key point set and the feature matrix X of each gesture; using the key points as nodes, constructing a graph G = (A, X), where A is the weighted adjacency matrix of the graph.

[0019] More preferably, the weights of the edges formed by nodes i and j in the weighted adjacency matrix A are initialized to the cosine similarity A of the node features. ij =cos(x) i ,x j ), where x i and x j The features of nodes i and j are respectively.

[0020] Preferably, obtaining the local and global features of the gesture specifically involves: inputting the graph into a graph convolutional neural network to obtain gesture features with local information; and inputting the gesture region data of the effective gesture into a ResNet network to obtain gesture features with global information.

[0021] Preferably, the process of acquiring gesture features with local information includes:

[0022] A two-layer graph convolutional neural network is used to process the features of each node, generating a high-order feature representation H for the node:

[0023]

[0024] in, Let W0 be the normalized adjacency matrix, σ(·) be the activation function, and W0 and W1 be trainable parameters.

[0025] Then, average pooling is performed on the higher-order feature representation H in the graph through a pooling layer to obtain the gesture feature h with local information. local :

[0026] h local =MeanPooling(H)

[0027] MeanPooling represents average pooling.

[0028] Preferably, a linear classifier g(·) is used to classify the final gesture features to obtain the final predicted label.

[0029] Preferably, the final predicted label includes correct gestures and incorrect gestures.

[0030] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1) This invention accurately distinguishes between gestures and the piano area, and cleverly uses the overlapping area between the two as a criterion for judgment. It can accurately filter out effective piano playing gestures in complex environments containing a variety of objects. Then, it extracts features of local and global information from the effective gestures, splices them together and classifies them. Because it eliminates the interference of irrelevant gestures, it significantly improves the accuracy of gesture recognition in complex and ever-changing scenarios.

[0033] 2) This invention adopts an advanced joint modeling strategy, which cleverly integrates the key point information of gestures with the visual features of the global image. It not only captures the overall outline of the gesture from a macro (global) perspective, but also analyzes the fine features of the gesture in detail from a micro (local) level, ensuring the comprehensiveness and accuracy of gesture features, thereby greatly enhancing the accuracy and reliability of gesture classification. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the piano gesture recognition method of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the predicted labels of incorrect gesture samples and their confidence levels in this invention;

[0036] Figure 3 This is a schematic diagram illustrating the predicted correct gesture samples and their confidence levels in this invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] This embodiment relates to a piano gesture recognition method for complex scenarios. To address the problem of piano gesture recognition in complex scenarios, it accurately filters out valid piano playing gestures by precisely segmenting the gestures and the piano and judging the overlapping area. Then, it uses graph convolutional networks and residual networks to extract gesture features from different granularities and finally classifies the correctness of the gestures.

[0039] like Figure 1 The method includes the following steps:

[0040] Step S1: Use the YOLO model to identify the positions of the piano and gestures in the piano performance image, and generate bounding boxes (rectangular boxes that exactly contain the objects) respectively. Output the piano bounding box and the gesture bounding box, which are B... piano and B hand For piano performance videos, the video is captured according to the sampling frequency and converted into multiple images.

[0041] Step S2: In each gesture bounding box and piano bounding box, use the SAM (Segment Anything Model) model to accurately segment the gesture and piano, and accurately extract the edges of the gesture and piano.

[0042] The area enclosed by the edges of the gesture is denoted as the gesture area M. hand In each piano bounding box B piano Inside, the SAM model is used to perform more detailed segmentation of the piano, accurately extracting the piano's edges. The area enclosed by the piano's edges is denoted as the piano region M. piano .

[0043] Step S3: Calculate the segmented gesture region M hand With piano area M piano The percentage of overlap between them is defined as the overlap area ratio O:

[0044]

[0045] If the overlap ratio O is less than the set threshold θ, the gesture is considered invalid and is removed; otherwise, the gesture is retained as a valid gesture.

[0046] Step S4: For the remaining valid gestures, use the key point detection algorithm to obtain the key nodes on each gesture and abstract them into a graph structure. The feature of the node is the context information of its location in the image.

[0047] For the remaining valid gestures, the keypoint detection algorithm f(·) is used to obtain the set of keypoints for each gesture and the feature matrix X = [x1, x2, ..., xn] of the keypoints. N ] T The node features contain contextual information about its location in the image. Then, using keypoints as nodes, a graph G = (A, X) is constructed, where A is the weighted adjacency matrix of the graph. The element A in the i-th row and j-th column of A... ij This represents the weight of the edge connecting nodes i and j, initialized as the cosine similarity A of the node features. ij =cos(x) i ,x j ).

[0048] Step S5: On one hand, the graph is input into a graph convolutional neural network (GCN) and pooling layers to obtain gesture features with local information. On the other hand, the complete gesture image is input into a residual network (ResNet) to obtain gesture features with global information.

[0049] On the one hand, a two-layer graph convolutional neural network (GCN) is used to process the features of each node, generating a high-order feature representation H of the node:

[0050]

[0051] in, Let W0 be the normalized adjacency matrix, σ(·) be the activation function, and W0 and W1 be trainable parameters.

[0052] Then, average pooling is performed on the node features in the graph through a pooling layer to obtain gesture features h with local information. local :

[0053] h local =MeanPooling(H)

[0054] On the other hand, the complete gesture bounding box B hand The input is fed into a residual network (ResNet) to obtain gesture features h with global information. global .

[0055] Step S6: Concatenate the gesture features containing local and global information to obtain the final gesture feature h. hand Then, a linear classifier g(·) is used to classify them to obtain the final predicted label. That is, whether the gesture is correct or incorrect.

[0056] To verify that the model can effectively distinguish valid gestures and classify their correctness, we used manually labeled datasets for validation. We evaluated the model's performance using common classification metrics: precision and recall. The final precision was 0.9678 and the recall was 0.8608. To more intuitively demonstrate the model's performance, we selected a portion of the segmented gesture images and their corresponding classification results and confidence scores.

[0057] Error gesture samples and their model confidence are as follows Figure 2 As shown, the correct gesture samples and their model confidence are as follows: Figure 3 As shown.

[0058] As can be seen, the model can accurately identify valid gestures (gestures on playing the piano, i.e. whether the hand is playing the piano) and correctly classify them with high confidence (correct and incorrect gestures for playing the piano). Therefore, the model provided by this invention has excellent performance.

[0059] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0060] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0061] The processing unit performs the various methods and processes described above. For example, in some embodiments, the methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the methods by any other suitable means (e.g., by means of firmware).

[0062] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0063] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0064] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A piano gesture recognition method for complex scenarios, characterized in that, The method includes the following steps: Step S1: Use the YOLO model to identify the positions of the piano and gestures in the piano performance image, and generate the piano bounding box and the gesture bounding box respectively; Step S2: In each bounding box, use the SAM model to extract the edges of the gesture and the piano, and segment the gesture region and the piano region. Step S3: Filter valid gestures based on the segmented gesture area and piano area; specifically: calculate the overlap ratio between the gesture area and the piano area. If the overlap ratio is less than a set threshold, the gesture is considered invalid and is removed; otherwise, the gesture is retained as a valid gesture. Step S4: Use a keypoint detection algorithm to obtain the set of keypoints and the feature matrix of the keypoints for each valid gesture, and construct a graph. Specifically, for a valid gesture, use a keypoint detection algorithm... Obtain the keypoint set and feature matrix X of each gesture; construct a graph using the keypoints as nodes. ,in Let be the weighted adjacency matrix of the graph; Step S5: Based on the graph and the gesture region data of the effective gesture, obtain the local and global features of the gesture respectively. Specifically, input the graph into the graph convolutional neural network to obtain gesture features with local information; input the gesture region data of the effective gesture into the ResNet network to obtain gesture features with global information. Step S6: Concatenate the local features and global features to obtain the final gesture features, and classify the final gesture features to obtain the predicted labels.

2. The piano gesture recognition method for complex scenarios according to claim 1, characterized in that, Step S2 specifically involves: in each bounding box, using the SAM model to extract the gesture edge and the piano edge respectively. The area enclosed by the gesture edge is denoted as the gesture region, and the area enclosed by the piano edge is denoted as the piano region.

3. The piano gesture recognition method for complex scenarios according to claim 1, characterized in that, The nodes in the weighted adjacency matrix A and The weights of the edges are initialized to the cosine similarity of the node features. ,in and They are nodes and Its characteristics.

4. The piano gesture recognition method for complex scenarios according to claim 1, characterized in that, The process of acquiring local features of a gesture includes: A two-layer graph convolutional neural network is used to process the features of each node, generating a high-order feature representation of the node. : in, The normalized adjacency matrix, For activation function, and All of these are trainable parameters; Then, the higher-order features in the graph are represented by pooling layers. Perform average pooling to obtain gesture features with local information. : in This indicates average pooling.

5. The piano gesture recognition method for complex scenarios according to claim 1, characterized in that, Using a linear classifier The final gesture features are classified to obtain the final predicted label.

6. The piano gesture recognition method for complex scenarios according to claim 1, characterized in that, The final predicted label includes correct and incorrect gestures.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

Images