A method for recognizing a guzheng playing hand shape based on an attention mechanism

Abstract

The application discloses a guzheng playing hand posture recognition method based on an attention mechanism, belongs to the technical field of computer vision and musical instrument playing, and is characterized in that the method comprises three steps of classification scale establishment, data set construction and recognition network design. Firstly, the guzheng hand posture teaching requirements and the image recognition technology characteristics are fused in the classification scale establishment. Secondly, the image acquisition mode based on a free angle is adopted, and the guzheng hand posture data set is constructed through image data enhancement. Finally, the guzheng playing hand posture recognition network based on the attention mechanism is designed by using the transfer learning and the deep neural network and by introducing the multi-dimensional attention mechanism. The actual effect shows that the guzheng playing hand posture recognition method based on the attention mechanism has the feasibility and the effectiveness, can realize the intelligent recognition of the guzheng playing hand posture, and has the practical guiding role for the improvement of the guzheng player's playing level.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and musical instrument performance, specifically relating to a method for recognizing hand shapes in guzheng playing based on an attention mechanism. Background Technology

[0002] In instrumental music teaching, determining the scientific validity of hand positions and providing timely feedback are crucial. In practice, this largely relies on the teacher's individual knowledge and experience, leading to limitations in guidance and lower timeliness. Therefore, utilizing improved image analysis and recognition technology, combined with instrumental music teaching methods, to achieve the rationality assessment and real-time feedback of basic hand positions in instrumental music performance is essential. This paper explores this topic tentatively in the interdisciplinary field of science and art, introducing a cognitive solution.

[0003] In order to translate the teaching requirements for basic hand positions in instrumental performance into machine language, researchers and art education experts need to conduct in-depth discussions and combine instrumental teaching experience with image recognition technology to develop a relatively reasonable and targeted set of evaluation standards.

[0004] Image recognition technology, a crucial branch of machine vision within artificial intelligence, involves extracting features and classifying objects within images. Traditional methods rely on manual modeling of image features, which has significant limitations when dealing with complex, high-dimensional information. Deep learning-based approaches construct neural networks, training these models on large datasets to extract richer image features, enabling more effective classification and recognition. Several landmark achievements in deep learning-based image recognition include LeNet, AlexNet, VGG, GoogLeNet, and ResNet.

[0005] Fine-grained image recognition classifies subsets of basic categories, characterized by low inter-class variance and high intra-class variance. Solutions mainly fall into the following categories: fine-tuning of conventional image classification networks, methods based on fine-grained feature learning (e.g., Bilinear CNN), methods based on object block detection (e.g., Part R-CNN), and methods based on attention mechanisms.

[0006] Attention mechanisms can determine which part of the input to focus on and allocate limited information processing resources to the important parts. Major research directions include: spatial domain-based attention (e.g., SAM), channel-based attention (e.g., SE-Net, SK-Net), and hybrid attention (e.g., CBAM, DANet). Summary of the Invention

[0007] The purpose of this invention is to use image recognition technology to classify and recognize the hand shapes used in guzheng playing.

[0008] The present invention provides a method for recognizing hand positions in guzheng playing based on an attention mechanism, comprising the following steps:

[0009] 1) Develop a guzheng performance hand shape classification scale that integrates guzheng teaching hand shape requirements with image recognition technology:

[0010] Based on experts' prior knowledge and practical experience in guzheng teaching, and combined with image analysis technology in machine vision, the descriptive language of the basic hand positions for guzheng playing is transformed into a mathematical model, and a classification scale for basic hand positions in guzheng playing is developed, specifically as follows:

[0011] The classification scale for guzheng hand positions includes 9 categories: images of correct guzheng playing hand positions taken from frontal (A), left (B), and right (C) perspectives are classified into three subsets: A1, B1, and C1; images of guzheng playing hand positions taken from frontal (A), left (B), and right (C) perspectives are classified into five subsets: A2, B2, B3, C2, and C3; other non-the above categories are classified into the D subset.

[0012] 2) Construct a dataset of guzheng playing hand shapes based on free-viewpoint acquisition, including the following steps:

[0013] 2.1 Free-viewpoint acquisition: Guided by the established classification scale of basic hand shapes for guzheng, images of guzheng playing hand shapes were acquired using image acquisition equipment based on a free-viewpoint acquisition method, targeting the hand shapes and postures of guzheng players.

[0014] The subjects covered different age groups, with a balanced gender ratio, and their guzheng playing levels basically met the actual needs. The data collection equipment used was a mobile free-moving camera.

[0015] 2.2 Data Augmentation: In view of the needs of complex shooting environments in real-world applications, data augmentation technology is used to enhance and expand the data of the actual captured images by means of rotation, adjustment of brightness, contrast and saturation. Among them, rotation corresponds to the inability of handheld camera to keep it perfectly level, and brightness, contrast and saturation correspond to the complex lighting conditions in reality.

[0016] 2.3 Dataset Construction: Based on the established classification scale for guzheng playing hand shapes that integrates guzheng teaching hand shape requirements and image recognition technology, a guzheng playing hand shape image dataset was constructed using image data collected from a free-viewpoint and augmented with data.

[0017] 3) Attention-based hand shape recognition for guzheng (Chinese zither), including the following steps:

[0018] 3.1 Backbone Network: The ResNet50 image recognition network based on deep neural networks is adopted as the backbone network for the classification and recognition of hand shapes in guzheng performance;

[0019] 3.2 Transfer Learning: An improved secondary transfer learning approach is adopted, transferring the parameters of the pre-trained model to the current network based on the relevance of the target task. The target domain is the constructed Guzheng playing hand shape dataset, the source domain is the ImageNet database, and a publicly available gesture recognition database is introduced as the indirect domain. For the fine-grained Guzheng playing hand shape image dataset, secondary transfer learning is performed through a coarse-grained hand shape image database. Pre-training is performed on the ImageNet database to obtain initial weights, and then secondary training is performed using the gesture recognition database to obtain secondary weights, which are then transferred to the current Guzheng playing hand shape image recognition task. Coarse learning is performed first to better learn the features of the hand, and then fine learning is performed on the Guzheng playing hand shapes.

[0020] 3.3 Attention Mechanism: A multi-dimensional attention mechanism is introduced, which combines spatial attention, channel attention, and feature attention. Spatial pixels, channels, and feature maps are weighted and fused respectively. Feature maps from different layers are extracted and fused and used as a mask to react on the input. For high-dimensional intermediate feature maps, attention weights are trained sequentially along the spatial and channel dimensions, and then multiplied with the original feature map to adaptively adjust the features, thereby improving the accuracy of guzheng playing hand shape classification and recognition.

[0021] The hand shape recognition method for guzheng playing based on attention mechanism proposed in this invention is feasible and effective. It can realize intelligent recognition of guzheng playing hand shape, provide timely feedback and guidance to guzheng learners, and play an important role in improving the playing level and efficiency of guzheng players. Attached Figure Description

[0022] Figure 1 This is a flowchart of an attention-based hand shape recognition method for guzheng playing.

[0023] Figure 2 A schematic diagram of hand shape image acquisition for guzheng playing based on free-viewpoint;

[0024] Figure 3 This is a schematic diagram of the hand shape image acquisition results for guzheng playing based on a free-viewpoint perspective;

[0025] Figure 4 The effect of data augmentation;

[0026] Figure 5 Here is a partial screenshot of the database.

[0027] Figure 6 A flowchart for improved transfer learning;

[0028] Figure 7 This is a structural diagram of the multidimensional attention module. Detailed Implementation

[0029] The present invention will now be described in conjunction with the accompanying drawings.

[0030] The present invention provides a method for recognizing hand positions in guzheng playing based on an attention mechanism, comprising the following steps:

[0031] 1) Develop a guzheng performance hand shape classification scale that integrates guzheng teaching hand shape requirements with image recognition technology:

[0032] Based on experts' prior knowledge and practical experience in guzheng teaching, and combined with image analysis technology in machine vision, the descriptive language of the basic hand positions for guzheng playing was transformed into a mathematical model, and a classification scale for basic hand positions for guzheng playing was developed, as shown in Table 1:

[0033] Table 1. Classification Table of Basic Guzheng Playing Hand Shapes

[0034]

[0035] The guzheng hand position classification scale includes 9 categories: images of correct guzheng playing hand positions are collected from frontal (A), left (B), and right (C) perspectives, and categorized into three subsets: A1, B1, and C1; images of incorrect guzheng playing hand positions are collected from frontal (A), left (B), and right (C) perspectives, and categorized into five subsets: A2, B2, B3, C2, and C3; other categories not mentioned above are categorized into subset D. The explanation of the guzheng playing hand position classification scale is shown in Table 2.

[0036] Table 2. Explanation of the Classification Table of Basic Guzheng Playing Hand Shapes

[0037]

[0038]

[0039] 2) Construct a dataset of guzheng playing hand shapes based on free-viewpoint acquisition, including the following steps:

[0040] 2.1 Free-viewpoint-based acquisition: Guided by the established guzheng basic hand shape classification scale, image acquisition equipment was used to acquire images of the guzheng player's hand shapes and postures using a free-viewpoint-based acquisition method, such as... Figure 2 As shown, obtain the hand shape image for playing the guzheng;

[0041] To ensure the comprehensiveness and richness of the dataset, the subjects included different age groups, with a balanced gender ratio, and the performance level basically met the actual needs. The acquisition device used was a mobile free camera, which improved operability and convenience, and is conducive to the expansion of subsequent practical applications. The free-view acquisition method can present the posture of the playing hands from multiple angles, making the information in the constructed dataset more complete and comprehensive.

[0042] A schematic diagram of hand shape image acquisition for guzheng playing based on free-viewpoint is shown below. Figure 2 As shown, the corresponding collection results are as follows: Figure 3 As shown, a total of 3,272 images of guzheng playing hand shapes were collected from 256 volunteers.

[0043] 2.2 Data Augmentation: Considering the demands of complex shooting environments in real-world applications, data augmentation techniques are used to enhance and expand the data of the captured images through rotation, adjustment of brightness, contrast, and saturation, such as... Figure 4 As shown, rotation corresponds to the inability of a handheld camera to maintain absolute horizontality, and brightness, contrast, and saturation correspond to the complex lighting conditions in reality. After data augmentation, the image data volume reached 19,632 images, thus providing large-scale support for the algorithm and verifying the generalization ability of the network model.

[0044] 2.3 Dataset Construction: Based on the established classification scale for guzheng playing hand shapes that integrates teaching hand shape requirements with image recognition technology, a guzheng playing hand shape image dataset was constructed using image data collected from a free-viewpoint and augmented with data. Some of the resulting images are shown below. Figure 5 As shown;

[0045] 3. Attention-based hand shape recognition for guzheng, including the following steps:

[0046] 3.1 Backbone Network: The ResNet50 image recognition network based on deep neural networks is adopted as the backbone network for the classification and recognition of hand shapes in guzheng performance;

[0047] 3.2 Transfer Learning: An improved secondary transfer learning approach is adopted, transferring the parameters of the pre-trained model to the current network based on the relevance of the target task. The target domain is the constructed Guzheng playing hand shape dataset, the source domain is the ImageNet database, and a publicly available gesture recognition database is introduced as the indirect domain. For the fine-grained Guzheng playing hand shape image dataset, secondary transfer learning is performed through a coarse-grained hand shape image database. Pre-training is performed on the ImageNet database to obtain initial weights, and then secondary training is performed using the gesture recognition database to obtain secondary weights, which are then transferred to the current Guzheng playing hand shape image recognition task. Coarse learning first better learns the features of the hand, and then fine learning of the Guzheng playing hand shape is performed to obtain better model parameters and faster convergence speed.

[0048] 3.3 Attention Mechanism: A multi-dimensional attention mechanism is introduced, which combines spatial attention, channel attention, and feature attention. Spatial pixels, channels, and feature maps are weighted and fused separately. Feature maps from different layers are extracted and fused and then fed back to the input. For high-dimensional intermediate feature maps, attention weights are trained sequentially along the spatial and channel dimensions, and then multiplied with the original feature map to adaptively adjust the features, thereby improving the accuracy of guzheng playing hand shape classification and recognition.

Claims

1.A method for recognizing a playing hand posture of a guzheng based on an attention mechanism, characterized in that Comprising the following steps: 1) Formulate the Guzheng playing hand posture classification scale that fuses the requirements of Guzheng teaching hand postures and image recognition technology: According to the prior knowledge and practical experience of experts in Guzheng teaching, combined with image analysis technology in machine vision, the descriptive language of the specification of basic Guzheng playing hand postures is converted into a mathematical model, and a basic Guzheng playing hand posture classification scale is formulated, which is as follows: The classification scale of Guzheng playing hand postures includes 9 categories: from the front A view, left side B view, right side C view, respectively collect correct Guzheng playing hand posture images, and classify them into A1, B1, C1 three category subsets; From the front A view, left side B view, right side C view, respectively collect incorrect Guzheng playing hand posture images, and classify them into A2, B2, B3, C2, C3 five category subsets; Other categories are classified into D category subset; 2) Build a Guzheng playing hand posture image dataset based on free-angle collection, comprising the following steps: 2.1 Free-angle collection: Under the guidance of the formulated basic Guzheng hand posture classification scale, use image collection equipment to collect Guzheng playing hand posture images based on free-angle collection for the hand posture of Guzheng players; The collected people cover different age groups, with balanced gender ratio, and playing level basically meets the distribution of actual demand; The collection equipment selects a mobile free camera; 2.2 Data enhancement: Combined with the demand of complex shooting environment in real application, use data enhancement technology to rotate, adjust brightness, contrast and saturation to expand the data of the actual image, among which, rotation corresponds to the situation that handheld camera cannot maintain absolute level, brightness, contrast and saturation correspond to the complex situation of light in reality; 2.3 Build dataset: According to the formulated Guzheng playing hand posture classification scale that fuses the requirements of Guzheng teaching hand postures and image recognition technology, use the image data collected based on free-angle and data enhanced to build a Guzheng playing hand posture image dataset; 3) Guzheng playing hand posture recognition based on attention mechanism, comprising the following steps: 3.1 Main network: Use the image recognition network ResNet50 based on deep neural network as the backbone network of Guzheng playing hand posture classification and recognition; 3.2 Transfer learning: Improved twice transfer learning method is adopted, and the parameters of the pre-trained model are transferred to the current network according to the relevance of the target task; The target domain is the constructed Guzheng playing hand posture dataset, the source domain is ImageNet database, and the public gesture recognition database is introduced as an indirect domain; For the fine-grained Guzheng playing hand posture image dataset, secondary transfer learning is carried out through the coarse-grained hand posture image database; Pre-train on ImageNet database to get initial weights, then use gesture recognition database for secondary training to get secondary weights, and migrate them to the current Guzheng playing hand posture image recognition task; First, better learn the coarse learning of hand features, and then fine learning of Guzheng playing hand postures; 3.3 Attention mechanism: Introduce multi-dimensional attention mechanism, adopt the joint mode of spatial attention and channel attention and feature attention, respectively weight and fuse spatial pixels, channels and feature maps, and then react to the input after extracting and fusing different layer feature maps. For high-dimensional intermediate feature maps, attention weights are trained along the space and channel dimensions in turn, and then multiplied with the original feature map to adaptively adjust the features and improve the accuracy of guzheng playing hand posture classification and recognition.

Images