Piano gesture recognition method for complex scene and electronic equipment

Abstract

The invention relates to a complex scene-oriented piano gesture recognition method and electronic equipment, and the method comprises the following steps: recognizing piano and gesture positions in a piano playing image by using a YOLO model, and respectively generating a piano bounding box and a gesture bounding box; in each bounding box, using an SAM model to extract the edges of a gesture and a piano, and segmenting a gesture area and a piano area so as to screen effective gestures; obtaining a key point set on each effective gesture and a feature matrix of key points by using a key point detection algorithm, and constructing a graph structure; acquiring a local feature and a global feature of the gesture based on the graph and the gesture region data of the effective gesture; and the local features and the global features of the potential are spliced to obtain final gesture features, and the final gesture features are classified to obtain prediction labels. Compared with the prior art, the method has the advantages that effective gestures are screened, and the accuracy and the leaning performance of gesture recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-assisted piano teaching, and in particular to a piano gesture recognition method and electronic equipment for complex scenarios. Background Art

[0002] With the continuous development of artificial intelligence technology, AI-assisted piano teaching has gradually attracted attention. Its significance lies in providing learners with timely and personalized feedback, improving learning efficiency, and reducing the formation of bad habits. In the process of piano teaching, judging the correctness of the user's piano playing gesture is one of the key links, which can effectively help learners adjust and correct their postures. To achieve this goal, it is necessary to accurately detect and segment gestures in each frame of the user's piano playing video. However, due to the complex piano playing scene of the user, the image contains not only the hands, but also the piano and other objects. The hands may be on the piano (i.e., valid gestures) or in other positions such as the legs (i.e., invalid gestures). Judging and correcting invalid gestures is not only meaningless, but also affects the accuracy of overall recognition. Therefore, gesture recognition methods not only need to detect the hands, but also ensure that the gesture is a valid piano playing gesture. In addition, existing methods often extract gesture features from a single granularity, resulting in incomplete gesture information extraction. Especially in complex scenes, the neglect of global information will reduce the accuracy of classification.

[0003] After searching, Chinese invention patent application publication number CN115798033A discloses a piano training method, system, device and storage medium based on gesture recognition. By collecting a video of the user's hand movements when playing the piano, key frame images of the hand movements are extracted from the video, and the key points of the hand are extracted according to the key frame images. The key points in the image are compared with the key points in the pre-stored standard key point sequence to generate the standard degree of the hand posture, and then the key frame images with a standard degree lower than a preset threshold are extracted to generate an erroneous gesture image. Training guidance information is generated according to the erroneous gesture image and the corresponding standard image, that is, the user's gesture action evaluation situation can be obtained in real time, which is convenient for the user to conduct targeted training.

[0004] After searching, China Invention Patent Application Publication No. CN113255470B discloses a multimodal piano accompaniment system and method based on hand gesture estimation. The system includes: a data acquisition module, which collects gesture information and piano sound information during the performance; a data recognition module, which identifies the small joint points in the gesture information according to the piano sound information and the preset algorithm, and sends it to the data comparison module; the data comparison module compares the small joint point recognition results with the standard database to obtain the wrong gesture information and wrong piano sound segment information during the performance; the result display module marks the wrong gesture information and the wrong piano sound segment information on the piano score, and feeds back to the user in a multimodal form. The present invention uses multiple algorithms to extract and recognize gesture features in multiple aspects, thereby effectively extracting the temporal feature information and spatial feature information of the video frame, while relying on microphone sound reception to greatly reduce the computing power consumption, and improve the piano accompaniment effect through small joint positioning.

[0005] Both of these existing invention applications directly recognize gestures without screening effective gestures, which leads to the problem of low accuracy in piano gesture recognition.

[0006] How to effectively screen out invalid gestures in complex scenarios and improve the accuracy of piano gesture recognition has become a technical problem that needs to be solved. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a piano gesture recognition method and electronic device for complex scenes.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] According to one aspect of the present invention, a piano gesture recognition method for complex scenes is provided, the method comprising the following steps:

[0010] Step S1, using the YOLO model to identify the piano and gesture positions in the piano performance image, and generate a piano bounding box and a gesture bounding box respectively;

[0011] Step S2: In each bounding box, use the SAM model to extract the edges of the gesture and the piano, and segment the gesture area and the piano area;

[0012] Step S3, screening valid gestures according to the segmented gesture area and piano area;

[0013] Step S4: Use a key point detection algorithm to obtain a key point set and a feature matrix of key points on each valid gesture, and construct a graph structure;

[0014] Step S5, obtaining local features and global features of the gesture based on the graph and the gesture area data of the valid gesture respectively;

[0015] Step S6: concatenate the local features and the global features to obtain the final gesture features, and classify the final gesture features to obtain the predicted labels.

[0016] Preferably, step S2 specifically includes: in each bounding box, using the SAM model to extract gesture edges and piano edges respectively, the area enclosed by the gesture edge is recorded as the gesture area, and the area enclosed by the piano edge is recorded as the piano area.

[0017] Preferably, the process of screening out valid gestures includes: calculating the overlap ratio between the gesture area and the piano area; if the overlap ratio is less than a set threshold, the gesture is considered to be an invalid gesture and is discarded; otherwise, the gesture is retained as a valid gesture.

[0018] Preferably, the process of constructing the graph structure includes: for valid gestures, using the key point detection algorithm f(·) to obtain the key point set of each gesture and the feature matrix X of the key points; using the key points as nodes, constructing a graph G = (A, X), where A is the weighted adjacency matrix of the graph.

[0019] More preferably, the weight of the edge composed of nodes i and j in the weighted adjacency matrix A is initialized to the cosine similarity A of the node features. ij =cos(x i ,x j ), where x i and x j are the features of nodes i and j respectively.

[0020] Preferably, obtaining local features and global features of a gesture is specifically as follows: inputting a graph into a graph convolutional neural network to obtain gesture features with local information; and inputting gesture area data of a valid gesture into a ResNet network to obtain gesture features with global information.

[0021] Preferably, the process of acquiring gesture features with local information includes:

[0022] Use a two-layer graph convolutional neural network to process the features of each node and generate a high-order feature representation H of the node:

[0023]

[0024] in, is the normalized adjacency matrix, σ(·) is the activation function, and W0 and W1 are both trainable parameters;

[0025] Then, the high-order feature representation H in the image is averaged through the pooling layer to obtain the gesture feature h with local information local :

[0026] h local =MeanPooling(H)

[0027] Among them, MeanPooling means average pooling.

[0028] Preferably, a linear classifier g(·) is used to classify the final gesture features to obtain a final predicted label.

[0029] Preferably, the final predicted labels include correct gestures and incorrect gestures.

[0030] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1) The present invention can accurately screen out effective piano-playing gestures in a complex environment containing various objects by accurately distinguishing between gestures and piano areas and cleverly using the overlapping area between the two as a basis for judgment. It then extracts the features of local and global information from the effective gestures, and classifies them after splicing. Because the interference of irrelevant gestures is eliminated, the accuracy of gesture recognition is significantly improved in complex and changeable scenarios.

[0033] 2) The present invention adopts an advanced joint modeling strategy, which cleverly integrates the key point information of the gesture with the visual features of the global image. It not only captures the overall outline of the gesture from a macro (global) perspective, but also analyzes the fine features of the gesture from a micro (local) level, ensuring the comprehensiveness and accuracy of the gesture features, thereby greatly enhancing the accuracy and reliability of gesture classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the piano gesture recognition method in the present invention;

[0035] Figure 2 A schematic diagram of a gesture sample with a predicted label of being an error and its confidence level in the present invention;

[0036] Figure 3 A schematic diagram of a gesture sample predicted to be labeled as correct and its confidence in the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] This embodiment relates to a piano gesture recognition method for complex scenes. For the piano gesture recognition problem in complex scenes, effective piano gestures are accurately screened out through precise segmentation of gestures and piano and judgment of overlapping areas, and gesture features are extracted from different granularities using graph convolutional networks and residual networks, and finally the correctness of the gestures is classified.

[0039] like Figure 1 , the method comprises the following steps:

[0040] Step S1: Use the YOLO model to identify the piano and gesture positions in the piano performance image, and generate bounding boxes (rectangular boxes that just contain the objects) respectively, and output the piano bounding box and the gesture bounding box, which are B and B respectively. piano and B hand For the piano performance video, the video is collected according to the sampling frequency and converted into multiple images.

[0041] Step S2: In each gesture bounding box and piano bounding box, the SAM (Segment Anything Model) model is used to accurately segment the gesture and the piano, and the edges of the gesture and the piano are accurately extracted.

[0042] The area enclosed by the edge of the gesture is recorded as the gesture area M hand , in each piano bounding box B piano The SAM model is used to segment the piano more carefully and accurately extract the piano edge. The area enclosed by the piano edge is recorded as the piano area M. piano .

[0043] Step S3: Calculate the segmented gesture area M hand With Piano Area M piano The overlap ratio between them is defined as the overlap area ratio O:

[0044]

[0045] When the overlap ratio O is less than the set threshold θ, the gesture is considered as an invalid gesture and is discarded; otherwise, the gesture is retained as a valid gesture.

[0046] Step S4: For the remaining valid gestures, a key point detection algorithm is used to obtain key nodes on each gesture, and the key nodes are abstracted into a graph structure, where the node feature is the local context information corresponding to the position in the image;

[0047] For the remaining valid gestures, the key point detection algorithm f(·) is used to obtain the key point set on each gesture and the feature matrix X = [x1, x2, ..., x N ] T , where the node features contain the contextual information around its position in the image. Then, with the key points as nodes, a graph G = (A, X) is constructed, where A is the weighted adjacency matrix of the graph. The i-th row and j-th column element A ij Represents the weight of the edge connecting nodes i and j, initialized to the cosine similarity A of the node features ij =cos(x i ,x j ).

[0048] Step S5: On the one hand, the graph is input into the graph convolutional neural network (GCN) and the pooling layer to obtain gesture features with local information. On the other hand, the complete gesture image is input into the residual network (ResNet) to obtain gesture features with global information.

[0049] On the one hand, a two-layer graph convolutional neural network (GCN) is used to process the features of each node to generate a high-order feature representation H of the node:

[0050]

[0051] in, is the normalized adjacency matrix, σ(·) is the activation function, and W0 and W1 are both trainable parameters.

[0052] Then, the node features in the graph are averaged through the pooling layer to obtain the gesture feature h with local information local :

[0053] h local =MeanPooling(H)

[0054] On the other hand, the complete gesture bounding box B hand Input to the residual network (ResNet) to obtain the gesture feature h with global information global .

[0055] Step S6: Concatenate the gesture features with local and global information to obtain the final gesture feature h hand , and use the linear classifier g(·) to classify it and get the final predicted label That is, whether the gesture is correct or incorrect.

[0056] In order to verify that the model can effectively distinguish valid gestures and classify them correctly, we use manually annotated datasets for verification. We also use common classification indicators such as precision and recall to evaluate the model effect. The final precision is 0.9678 and the recall is 0.8608. In order to more intuitively demonstrate the model effect, we selected some segmented gesture images and their corresponding classification results and confidence.

[0057] The wrong gesture samples and their model confidence are as follows Figure 2 As shown, the correct gesture samples and their model confidence are as follows Figure 3 shown.

[0058] It can be seen that the model can accurately identify valid gestures (gestures on the piano, that is, whether the hands are playing the piano) and correctly classify them with high confidence (correct gestures and incorrect gestures for playing the piano), so the model provided by the present invention has excellent performance.

[0059] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0060] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0061] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method in any other appropriate manner (e.g., by means of firmware).

[0062] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0063] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0064] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0065] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A piano gesture recognition method for complex scenes, characterized in that: The method comprises the following steps: Step S1, using the YOLO model to identify the piano and gesture positions in the piano performance image, and generate a piano bounding box and a gesture bounding box respectively; Step S2: In each bounding box, use the SAM model to extract the edges of the gesture and the piano, and segment the gesture area and the piano area; Step S3, screening valid gestures according to the segmented gesture area and piano area; Step S4: Use a key point detection algorithm to obtain a key point set and a feature matrix of key points on each valid gesture, and construct a graph structure; Step S5, obtaining local features and global features of the gesture based on the graph and the gesture area data of the valid gesture respectively; Step S6: concatenate the local features and the global features to obtain the final gesture features, and classify the final gesture features to obtain the predicted labels.

2. A piano gesture recognition method for complex scenes according to claim 1, characterized in that: The step S2 specifically includes: in each bounding box, using the SAM model to extract gesture edges and piano edges respectively, the area enclosed by the gesture edge is recorded as the gesture area, and the area enclosed by the piano edge is recorded as the piano area.

3. A piano gesture recognition method for complex scenes according to claim 1, characterized in that: The process of screening out valid gestures includes: calculating the overlap ratio between the gesture area and the piano area, if the overlap ratio is less than a set threshold, the gesture is considered to be an invalid gesture and is removed; otherwise, the gesture is retained as a valid gesture.

4. A piano gesture recognition method for complex scenes according to claim 1, characterized in that: The process of constructing the graph structure includes: for valid gestures, using the key point detection algorithm f(·) to obtain the key point set of each gesture and the feature matrix X of the key points; using the key points as nodes, constructing a graph G = (A, X), where A is the weighted adjacency matrix of the graph.

5. A piano gesture recognition method for complex scenes according to claim 4, characterized in that: The weight of the edge composed of nodes i and j in the weighted adjacency matrix A is initialized to the cosine similarity A of the node features ij =cos(x i ,x j ), where x i and x j are the features of nodes i and j respectively.

6. The piano gesture recognition method for complex scenes according to claim 1, characterized in that: The local and global features of the gesture are obtained as follows: the graph is input into the graph convolutional neural network to obtain gesture features with local information; the gesture area data of the valid gesture is input into the ResNet network to obtain gesture features with global information.

7. A piano gesture recognition method for complex scenes according to claim 1 or 6, characterized in that: The process of obtaining the local features of a gesture includes: Use a two-layer graph convolutional neural network to process the features of each node and generate a high-order feature representation H of the node: in, is the normalized adjacency matrix, σ(·) is the activation function, and W0 and W1 are both trainable parameters; Then, the high-order feature representation H in the image is averaged through the pooling layer to obtain the gesture feature h with local information local : h local =MeanPooling(H) Among them, MeanPooling means average pooling.

8. The piano gesture recognition method for complex scenes according to claim 1, characterized in that: The final gesture feature is classified using a linear classifier g(·) to obtain a final predicted label.

9. The piano gesture recognition method for complex scenes according to claim 1, characterized in that: The final predicted labels include correct gestures and incorrect gestures.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images