Online playing teaching platform construction method based on artificial intelligence
Through an online performance teaching platform based on artificial intelligence, using the palm as the recognition target, combining feature extractor and neural network model, the problem of insufficient accuracy of gesture recognition in the existing technology is solved, and high accuracy and high real-time performance are achieved. Gesture recognition improves the teaching effect of online music performance.
Patent Information
- Application Number
- CN202411014455.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
The existing music performance system based on gesture action recognition is difficult to detect gestures through visual features under different finger sizes and occlusion, resulting in insufficient recognition accuracy and real-time performance.
Using an online performance teaching platform method based on artificial intelligence, we can use the user's hand video and sound to recognize gestures and build a gesture library, combine feature extractors and neural network models to improve the accuracy of gesture recognition, and use the palm as the recognition target. , reduce data enhancement operations, improve recognition speed and accuracy.
It realizes high accuracy and high real-time gesture recognition in virtual space, reduces the difficulty of finger detection, and improves the accuracy and user experience of online music performance teaching.
Smart Images

Figure CN118968618A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for constructing an online performance teaching platform based on artificial intelligence. Background Art
[0002] Music performance is usually performed in the way of playing physical instruments, while music performance using gestures means achieving the purpose of playing music as accurately and smoothly as playing real instruments only through hand movements and gesture changes.
[0003] There are many kinds of music playing systems based on gesture recognition in the prior art. For example, the Chinese invention patent with application number CN202111495747.5 (application publication number CN114387616A) discloses a music playing system and method based on gesture recognition; another example is the Chinese invention patent with application number CN 201810965724.8 (application publication number CN109166565A) discloses a virtual instrument processing method, device, virtual instrument equipment and storage medium. Although both of the above patents can recognize the gestures of the performers in the video, without carrying keyboard instruments such as physical pianos, the effect of playing music accurately and smoothly as real instruments can be achieved, and the accuracy and real-time performance of remote performance can be improved. However, there are the following deficiencies in the prior art: most of the prior art obtains the gesture information of each finger of the user relative to the virtual instrument keyboard, but on the one hand, due to the large span of the size of the fingers of the same user, and the fingers are blocked and self-blocked; on the other hand, the fingers lack the same high-contrast pattern as the face, so it is relatively difficult to detect gestures only by visual features. Therefore, the prior art needs to be further improved. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for constructing an online performance teaching platform based on artificial intelligence with simple gesture detection in view of the above-mentioned prior art.
[0005] The technical solution adopted by the present invention to solve the above technical problems is: a method for constructing an online performance teaching platform based on artificial intelligence, characterized by comprising the following steps:
[0006] Step 1: Take a video of the user's hands while playing, and record the sound of the performance;
[0007] Step 2: Identify the user's gestures in the video, and segment the gestures and sounds according to the beats to form a gesture library corresponding to the gestures and sounds;
[0008] Step 3: The gestures in the gesture library are marked according to the rhythm, pitch and intonation of the corresponding sounds, and the strength of the gestures is marked in combination with the size of the sounds to obtain labels for each gesture;
[0009] Step 4: construct the gestures identified in step 3 into a first data set;
[0010] Step 5: construct a performance model, and train the constructed performance model using the first data set in batches to obtain a first performance model;
[0011] Step 6: Acquire the gestures and corresponding sounds of the designated performer, construct them into a second data set, and use the second data set in batches to train the first performance model in step 5 to obtain a second performance model;
[0012] Step 7: Arrange the second performance model on the network server, so as to construct an online performance teaching platform.
[0013] As an improvement, the specific process of identifying the gesture in step 2 is as follows:
[0014] The video is divided into multiple video streams according to a preset time T, and multiple frame images are obtained in each video stream;
[0015] Each frame of the image is recognized and it is determined whether the palm can be recognized. If so, the positioning coordinates of the palm and fingers are extracted to obtain gesture data; if not, the palm in the image is relocated.
[0016] Preferably, the specific process of identifying each frame of image is as follows:
[0017] Step a, constructing a palm detector, inputting the first frame of the image to be recognized into the palm detector, obtaining a hand bounding box corresponding to the first frame of the image to be recognized, and locating the palm key points in the hand bounding box corresponding to the first frame of the image to be recognized;
[0018] Step b, using the palm key points of the i-th frame of the image to be recognized to obtain the hand bounding box corresponding to the i+1-th frame of the image to be recognized, and locating the palm key points in the hand bounding box of the i+1-th frame of the image to be recognized; let i be 1, 2, ... M-1 in sequence; M is the total number of image frames in each video stream;
[0019] Step c: perform feature recognition on the image in each hand boundary box through a feature extractor to obtain a recognition result.
[0020] To improve the palm positioning accuracy, the following operations are performed for each hand bounding box:
[0021] Use the non-maximum suppression algorithm to compare the current hand bounding box with other hand bounding boxes to determine whether the overlap rate of the two hand bounding boxes exceeds the preset threshold. If so, only the hand bounding box with the highest confidence is retained; if not, both hand bounding boxes are retained.
[0022] Preferably, the hand bounding box is a square.
[0023] Preferably, the feature extractor in step c is a feature pyramid network.
[0024] In order to further improve the palm recognition speed, preferably, the recognition of each frame of the video stream also includes:
[0025] Determine whether the left hand or the right hand can be recognized. If so, mark the recognized hand, obtain the palm image, and use the palm image to obtain gesture data; if not, re-recognize the palm.
[0026] Preferably, the video stream is cut at a speed of N frames per second to obtain multiple frames of images in each video stream.
[0027] Preferably, the performance model in step 5 adopts a neural network.
[0028] Compared with the prior art, the advantages of the present invention are: firstly, the gesture image sequence of the performer is obtained through the camera, then the gesture is segmented and the basic information of the gesture is obtained through gesture recognition, combined with the construction of a virtual character in the virtual space, to carry out a single-person online teaching mode for the performer. In this method, the human palm is used as the recognition target, which greatly reduces the data enhancement (such as rotation, translation and scaling) operations, can improve the accuracy of gesture recognition, and is easier than finger detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a block diagram of a method for constructing an online performance teaching platform in an embodiment of the present invention. DETAILED DESCRIPTION
[0030] The present invention is further described in detail below with reference to the accompanying drawings.
[0031] There are many kinds of musical instruments in online performance teaching, and each instrument has a unique performance style and performance skills. In this embodiment, the performance methods are divided into four types: string performance, wind instrument performance, percussion instrument performance and keyboard performance. String instruments include violin, cello, harp, etc., wind instruments include woodwind and brass, percussion instruments include bass drum, cymbals, bells, etc., and keyboard instruments include piano, organ, electronic keyboard, etc.
[0032] like Figure 1As shown, the method for constructing an online performance teaching platform based on artificial intelligence in this embodiment includes the following steps:
[0033] Step 1: Take a video of the user's hands while playing, and record the sound of the performance;
[0034] The camera in this embodiment is installed on the musical instrument. When selecting the installation position, it is necessary to ensure that it does not affect the user's performance and can easily capture the user's palm video when playing. For example, the camera is installed on the piano keyboard. The installation position of the camera is kept at a certain distance from the position where the hand acts on the piano keyboard, and the occlusion is minimized.
[0035] Step 2: Identify the user's gestures in the video, and segment the gestures and sounds according to the beats to form a gesture library corresponding to the gestures and sounds;
[0036] The specific process of gesture recognition is as follows:
[0037] The video is divided into multiple video streams according to a preset time T, and multiple frame images are obtained in each video stream;
[0038] Recognize each frame of the image and determine whether the palm can be recognized. If so, extract the location coordinates of the palm and fingers to obtain gesture data; if not, relocate the palm in the image;
[0039] In this embodiment, the video stream is cut at a rate of N frames per second to obtain multiple frames of images in each video stream; and in this embodiment, T is preferably 62.5ms; N=16;
[0040] The specific process of identifying each frame of image in this embodiment is as follows:
[0041] Step a, constructing a palm detector, inputting the first frame of the image to be recognized into the palm detector, obtaining a hand bounding box corresponding to the first frame of the image to be recognized, and locating the palm key points in the hand bounding box corresponding to the first frame of the image to be recognized;
[0042] Step b, using the palm key points of the i-th frame of the image to be recognized to obtain the hand bounding box corresponding to the i+1-th frame of the image to be recognized, and locating the palm key points in the hand bounding box of the i+1-th frame of the image to be recognized; let i be 1, 2, ... M-1 in sequence; M is the total number of image frames in each video stream;
[0043] Step c: Perform feature recognition on the image in each hand boundary box through a feature extractor to obtain a recognition result;
[0044] Additionally, for each hand bounding box, the following operations are performed:
[0045] Use the non-maximum suppression algorithm to compare the current hand bounding box with other hand bounding boxes to determine whether the overlap rate of the two hand bounding boxes exceeds the preset threshold. If so, only the hand bounding box with the highest confidence is retained; if not, both hand bounding boxes are retained;
[0046] Since the palm is shaped like a square, the number of anchor frames of other proportions can be reduced by 3 to 5 times; only a square bounding box is used to model the palm, and the hand bounding box in this embodiment is a square; and the feature extractor in this embodiment is a feature pyramid network; first, feature maps of different scales are generated through convolution and pooling, and then the feature maps of different scales are fused through upsampling and convolution operations to form a feature pyramid, and finally the required features are extracted from the feature pyramid. Since the context needs to be perceived in a larger scene, so that both large objects and small objects can be perceived under features of different scales, it is necessary to use the idea of FPN of YOLOV3 to improve the detection ability of small targets, because the receptive field of the value of an element in a large feature map is relatively small, which is suitable for detecting small objects; the receptive field of each element in a small feature map is large, which is suitable for detecting large objects;
[0047] Step 3: The gestures in the gesture library are marked according to the rhythm, pitch and intonation of the corresponding sounds, and the strength of the gestures is marked in combination with the size of the sounds to obtain labels for each gesture;
[0048] The technology used in this step is the existing technology and will not be described in detail here;
[0049] Step 4: construct the gestures identified in step 3 into a first data set;
[0050] Step 5: construct a performance model, and train the constructed performance model using the first data set in batches to obtain a first performance model;
[0051] The performance model in this embodiment adopts a neural network;
[0052] Step 6: Acquire the gestures and corresponding sounds of the designated performer, construct them into a second data set, and use the second data set in batches to train the first performance model in step 5 to obtain a second performance model;
[0053] Step 7: Arrange the second performance model on the network server, so as to construct an online performance teaching platform.
[0054] In this embodiment, multiple virtual characters can also be constructed in the same virtual space (network server) to achieve online multi-person performance and teaching.
[0055] In addition, the recognition of each frame of the video stream also includes:
[0056] Determine whether the left hand or the right hand can be recognized. If so, mark the recognized hand, obtain the palm image, and use the palm image to obtain gesture data; if not, re-recognize the palm.
[0057] There are several reasons for choosing the palm as the first step in gesture segmentation:
[0058] 1. The size of the hands varies greatly;
[0059] 2. There is occlusion and self-occlusion of the hand;
[0060] 3. Hands lack the same high-contrast patterns as faces, such as relatively distinct features around the mouth and face, so it is relatively difficult to detect hands based solely on visual features;
[0061] In this embodiment, the palm position is located by scanning the entire image and using a directional hand positioning frame. The hand coordinate model is provided with a precisely cropped palm image, which greatly reduces data enhancement (such as rotation, translation and scaling) operations, so that the performance of the model can be used to improve the accuracy of coordinate positioning. In the real-time tracking scenario, the hand positioning frame of the current frame is derived from the hand key point coordinate prediction of the previous frame, which can avoid using the palm detector for each frame. The palm detector is only used in the first frame or when the hand is lost.
[0062] The present invention also protects a computer-readable storage medium on which a computer program is stored. When the computer program is run by a processor, the method for constructing an online performance teaching platform as described above is executed.
[0063] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for constructing an online performance teaching platform based on artificial intelligence, characterized in that The steps include: Step 1: Take a video of the user's hands while playing, and record the sound of the performance; Step 2: Identify the user's gestures in the video, and segment the gestures and sounds according to the beats to form a gesture library corresponding to the gestures and sounds; Step 3: The gestures in the gesture library are marked according to the rhythm, pitch and intonation of the corresponding sounds, and the strength of the gestures is marked in combination with the size of the sounds to obtain labels for each gesture; Step 4: construct the gestures identified in step 3 into a first data set; Step 5: construct a performance model, and train the constructed performance model using the first data set in batches to obtain a first performance model; Step 6: Acquire the gestures and corresponding sounds of the designated performer, construct them into a second data set, and use the second data set in batches to train the first performance model in step 5 to obtain a second performance model; Step 7: Arrange the second performance model on the network server, so as to construct an online performance teaching platform.
2. The method for constructing an online performance teaching platform according to claim 1, characterized in that: The specific process of gesture recognition in step 2 is as follows: The video is divided into multiple video streams according to a preset time T, and multiple frame images are obtained in each video stream; Each frame of the image is recognized and it is determined whether the palm can be recognized. If so, the positioning coordinates of the palm and fingers are extracted to obtain gesture data; if not, the palm in the image is relocated.
3. The method for constructing an online performance teaching platform according to claim 2, characterized in that: The specific process of identifying each frame of image is as follows: Step a, constructing a palm detector, inputting the first frame of the image to be recognized into the palm detector, obtaining a hand bounding box corresponding to the first frame of the image to be recognized, and locating the palm key points in the hand bounding box corresponding to the first frame of the image to be recognized; Step b, using the palm key points of the i-th frame of the image to be recognized to obtain the hand bounding box corresponding to the i+1-th frame of the image to be recognized, and locating the palm key points in the hand bounding box of the i+1-th frame of the image to be recognized; Let i be 1, 2, ...M-1 in sequence; M is the total number of image frames in each video stream; Step c: perform feature recognition on the image in each hand boundary box through a feature extractor to obtain a recognition result.
4. The method for constructing an online performance teaching platform according to claim 3, characterized in that: For each hand bounding box, perform the following operations: Use the non-maximum suppression algorithm to compare the current hand bounding box with other hand bounding boxes to determine whether the overlap rate of the two hand bounding boxes exceeds the preset threshold. If so, only the hand bounding box with the highest confidence is retained; if not, both hand bounding boxes are retained.
5. The method for constructing an online performance teaching platform according to claim 4, characterized in that: The hand bounding box is a square.
6. The method for constructing an online performance teaching platform according to claim 3, characterized in that: The feature extractor in step c is a feature pyramid network.
7. The method for constructing an online performance teaching platform according to claim 2, characterized in that: The recognition of each frame of the video stream also includes: Determine whether the left hand or the right hand can be recognized. If so, mark the recognized hand, obtain the palm image, and use the palm image to obtain gesture data; if not, re-recognize the palm.
8. The method for constructing an online performance teaching platform according to any one of claims 2 to 7, characterized in that: The video stream is cut at a rate of N frames per second to obtain multiple frame images in each video stream.
9. The method for constructing an online performance teaching platform according to any one of claims 1 to 7, characterized in that: The performance model in step 5 adopts a neural network.
