Ocarina fingering real-time error correction system and method based on deep learning

Abstract

The invention provides a deep learning-based ocarina fingering real-time error correction system and method, and the method comprises the steps: carrying out the image preprocessing of a real-time video stream when a target user practices an ocarina, and obtaining a standard image sequence; for each frame of standard image in the standard image sequence, determining the coverage confidence of each fingertip of the target user and the corresponding sound hole in the standard image, and determining the static fingering characteristics of the finger of the target user in the standard image for the coverage state of the corresponding sound hole according to all the coverage confidence; determining a static fingering feature of a target user finger to a corresponding sound hole coverage state in each frame of standard image; performing time sequence context analysis on all the static fingering features to obtain a fingering track fusing fingering conversion spatio-temporal context in the ocarina practicing process of the target user; and outputting a fingering identification result of the target user practicing the ocarina based on the fingering track. By adopting the scheme of the invention, the ocarina fingering dynamic identification fusing the visual space information and the playing time sequence context can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent auxiliary technology for musical instrument performance, and more particularly, to a real-time ocarina fingering error correction system and method based on deep learning. BACKGROUND

[0002] As an easy-to-learn but difficult-to-master wind instrument, the ocarina's performance accuracy highly depends on the standardization and coordination of fingering. Traditional ocarina teaching mainly relies on on-site guidance by teachers or self-study by students, lacking real-time and objective fingering feedback mechanisms, which leads to the formation of incorrect muscle memory by beginners, low learning efficiency and high correction costs. With the development of artificial intelligence and computer vision technology, using intelligent means to assist musical instrument practice has become an important research direction in the field of music education.

[0003] The existing ocarina fingering assistance methods mainly include two categories: one is a follow-along system based on audio analysis, which indirectly judges fingering errors by recognizing the deviation between the played pitch and the standard pitch, but this method cannot distinguish between fingering errors and breath control problems, and cannot identify multiple alternative fingerings for the same pitch; the second is a visual method based on static image recognition, which judges the relative position of fingers and tone holes through single-frame images, but this method is difficult to handle the dynamic continuity during fingering transition, and lacks robustness in complex scenes such as finger occlusion and light changes. In addition, existing methods rely on single modal information and lack collaborative analysis of hand spatial posture, fingering transition timing characteristics, and audio information, resulting in limited correction accuracy and high feedback delay. Therefore, how to realize the dynamic recognition of ocarina fingering by fusing visual spatial information and performance timing context has become a difficult problem in the industry. SUMMARY

[0004] The present application provides a real-time ocarina fingering error correction system and method based on deep learning, which can realize dynamic recognition of ocarina fingering by fusing visual spatial information and performance timing context.

[0005] In a first aspect, the present application provides a real-time ocarina fingering recognition method based on deep learning, which is used for real-time recognition of the fingering of a target user in an ocarina fingering real-time error correction system. The method comprises: Collecting a real-time video stream of the target user practicing the ocarina, and performing image preprocessing on the real-time video stream to obtain a standard image sequence; For each frame of standard image in the standard image sequence, performing joint key point detection of the standard image for the hand and the ocarina to obtain the three-dimensional coordinate distribution of the finger joint nodes and the spatial coordinate distribution of the ocarina tone hole centers during the target user's ocarina practice; performing spatial relationship calibration between the fingertips and the tone holes based on the three-dimensional coordinate distribution and the spatial coordinate distribution to obtain a coverage confidence of each fingertip of the target user in the standard image and a corresponding tone hole, determining a static fingering feature of a coverage state of the fingers of the target user on the corresponding tone holes in the standard image according to all the coverage confidences, and further determining a static fingering feature of a coverage state of the fingers of the target user on the corresponding tone holes in each frame of the standard image; performing timing context analysis on all the static fingering features to obtain a fingering trajectory that fuses a timing context and a spatial context of a fingering conversion in the process in which the target user practices the ocarina; outputting a fingering recognition result of the target user practicing the ocarina based on the fingering trajectory.

[0006] In some embodiments, the image preprocessing on the real-time video stream to obtain the standard image sequence specifically includes: decoding and frame sampling on the real-time video stream to obtain a plurality of original images; color space conversion and filtering processing on each original image to obtain a standard image corresponding to each original image; determining a standard image sequence based on all the standard images.

[0007] In some embodiments, the joint key point detection on the standard image of the hand and the ocarina to obtain a three-dimensional coordinate distribution of the finger joint nodes and a spatial coordinate distribution of the ocarina tone hole centers in the process in which the target user practices the ocarina specifically includes: performing multi-scale feature extraction on the standard image through a feature extraction layer of a hand-ocarina collaborative perception network to obtain a shared feature atlas; enhancing semantic features of a hand region and an ocarina region in the shared feature atlas through parallel spatial attention branches and channel attention branches to obtain attention-enhanced features; performing three-dimensional coordinate regression on the attention-enhanced features to synchronously output a three-dimensional coordinate set of the finger joint nodes and a spatial coordinate set of the ocarina tone hole centers; respectively performing spatiotemporal consistency verification on the three-dimensional coordinate set and the spatial coordinate set to obtain a three-dimensional coordinate distribution of the finger joint nodes and a spatial coordinate distribution of the ocarina tone hole centers in the process in which the target user practices the ocarina.

[0008] In some embodiments, the spatial relationship calibration between the fingertips and the tone holes based on the three-dimensional coordinate distribution and the spatial coordinate distribution to obtain a coverage confidence of each fingertip of the target user in the standard image and a corresponding tone hole includes: establishing a local spatial coordinate system with each tone hole of the ocarina as a center based on the spatial coordinate distribution; According to the three-dimensional coordinate distribution, the Euclidean distance from each fingertip of the target user to the center of the corresponding sound hole, and the angle between the normal vector in the local spatial coordinate system, the spatial geometric parameter is obtained; The spatial geometric parameter is corrected in the coordinate system distortion to obtain a corrected spatial relationship feature; The pixel-level semantic segmentation is performed on the sound hole region in the standard image, and the coverage feature of the pixels of the ocarina sound hole covered by the skin of the fingers is extracted; Based on the spatial relationship feature and the coverage feature, the coverage confidence of each fingertip of the target user in the standard image and the corresponding sound hole is obtained.

[0009] In some embodiments, the static fingering feature of the target user's finger covering state on the corresponding sound hole in the standard image is determined according to all coverage confidences, and specifically includes: All coverage confidences are arranged and combined in the order of the sound holes of the ocarina to form a touch pressure coupling vector; The feature dimension reduction processing is performed on the touch pressure coupling vector to obtain a low-dimensional fingering representation; The hand shape contour feature is extracted from the three-dimensional coordinate distribution of the finger joint nodes in the target user's ocarina practice process; The low-dimensional fingering representation and the hand shape contour feature are fused to obtain a finger hole state tensor; The static fingering feature of the target user's finger covering state on the corresponding sound hole in the standard image is obtained by nonlinear transformation of the finger hole state tensor through a static feature encoder.

[0010] In some embodiments, the static fingering feature is analyzed in the time sequence context to obtain a fingering trajectory that fuses the time and space context of the fingering conversion in the target user's ocarina practice process, and specifically includes: All static fingering features are arranged in time sequence to construct a fingering feature sequence; The bidirectional time sequence dependent features are extracted in the fingering feature sequence; The bidirectional time sequence dependent features are subjected to convolution operation in the time dimension to obtain time sequence enhancement features of fingering conversion mode recognition in the local time window; The time sequence enhancement features are weighted by the attention mechanism to obtain spatio-temporal fusion features; The fingering trajectory that fuses the time and space context of the fingering conversion in the target user's ocarina practice process is reconstructed based on the spatio-temporal fusion features.

[0011] In some embodiments, the fingering recognition result of the target user's ocarina practice is output based on the fingering trajectory, and specifically includes: The finger movement trajectory is dynamically time warping matched with a pre-stored standard finger movement template to obtain a trajectory similarity; A target user's ocarina playing finger movement recognition result is determined based on the trajectory similarity.

[0012] In a second aspect, the present application provides a deep learning-based ocarina finger movement real-time correction system, which comprises a finger movement real-time recognition unit, and the finger movement real-time recognition unit comprises: A collection module is configured to collect a real-time video stream of a target user's ocarina playing, and perform image preprocessing on the real-time video stream to obtain a standard image sequence. A processing module is configured to, for each standard image in the standard image sequence, perform joint key point detection on the standard image to obtain a three-dimensional coordinate distribution of finger joints and a spatial coordinate distribution of ocarina hole centers in the target user's ocarina playing process. The processing module is configured to calibrate the spatial relationship between the fingertips and the holes based on the three-dimensional coordinate distribution and the spatial coordinate distribution to obtain a coverage confidence of each fingertip of the target user in the standard image and a corresponding hole, determine a static finger movement feature of the target user's finger movement in the standard image based on all the coverage confidences, and further determine a static finger movement feature of the target user's finger movement in each standard image. The processing module is configured to perform timing context analysis on all the static finger movement features to obtain a finger movement trajectory that fuses the timing and spatial context of the finger movement conversion in the target user's ocarina playing process. An execution module is configured to output a target user's ocarina playing finger movement recognition result based on the finger movement trajectory.

[0013] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory stores a code, and the processor is configured to acquire the code and execute the above-mentioned deep learning-based ocarina finger movement real-time recognition method.

[0014] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned deep learning-based ocarina finger movement real-time recognition method.

[0015] The technical scheme provided by the embodiments of the present application has the following beneficial effects: The deep learning-based ocarina fingering real-time correction system and method provided in the application first collects a real-time video stream of a target user practicing an ocarina, performs image preprocessing on the real-time video stream, and obtains a standard image sequence; for each standard image in the standard image sequence, joint key point detection of the standard image is performed to obtain a three-dimensional coordinate distribution of finger joints and a spatial coordinate distribution of ocarina hole centers in the process of the target user practicing the ocarina; the spatial relationship between the fingertips and the holes is calibrated based on the three-dimensional coordinate distribution and the spatial coordinate distribution to obtain the coverage confidence of each fingertip of the target user in the standard image and the corresponding hole, the static fingering features of the target user's fingers in the standard image are determined according to all the coverage confidences, and the static fingering features of the target user's fingers in each frame of the standard image are determined; the time sequence context analysis is performed on all the static fingering features to obtain a fingering track that fuses the time and space context of fingering conversion in the process of the target user practicing the ocarina; and the fingering recognition result of the target user practicing the ocarina is output based on the fingering track.

[0016] It can be seen that the present application is based on the fingering trajectory output target user practice dizi fingering recognition result; first, the determination of the coverage confidence can obtain the probability evaluation of the target user covering the tone hole when practicing the dizi, the determination of the coverage confidence can convert the fingering state into a continuous probability value between 0 and 1, realize the fine and quantitative evaluation of the accuracy of the fingering, provide the data basis for distinguishing the fingering error and the breath problem, which makes the system can identify the suspected not to press strictly, part of the coverage and other critical state, provides the data basis for the grading evaluation and early correction of the accuracy of the fingering, thereby effectively solves the problem that the prior art cannot handle the critical state, at the same time, the coverage confidence is calculated by a lightweight evaluation network based on three-dimensional spatial relationship characteristics (such as Euclidean distance and normal vector angle), rather than relying on a single, easily disturbed visual feature, when there is temporary occlusion, light mutation or image noise, it can still give reasonable probability inference based on the available part of the information, thereby significantly improving the stability and reliability of the system in the real and complex playing environment, solving the problem of insufficient robustness in complex scenes in the prior art; then, the determination of the static fingering feature can obtain the final fingering feature representation of the confidence of the finger pressing the tone hole and the actual spatial pose of the hand when the target user practices the dizi, the static fingering feature as a bridge connecting image perception and time series analysis, fuses the multi-dimensional fingering state in the single frame image, including the fingernail-tone hole coverage confidence and the hand shape contour feature, into a unified and regular feature representation, through the joint coding of the coverage confidence and the hand shape contour, the ambiguity problem of single modal information (such as pure audio or pure two-dimensional coordinates) in the fingering judgment in the prior art is overcome, the recognition robustness in complex hand shape and partial occlusion scene is significantly improved, at the same time, the generated standardized static feature provides a stable and consistent input unit for subsequent time series analysis, so that the system can accurately model the dynamic process of fingering conversion on this basis, thereby realizing the leap from isolated frame analysis to continuous trajectory recognition, and laying a reliable data foundation for high-precision real-time correction; finally, the determination of the fingering trajectory can obtain the continuous fingering motion path of the target user in the process of practicing the dizi, the determination of the fingering trajectory can effectively capture the dynamic characteristics in the fingering conversion process, can compensate for instantaneous detection errors through the information of the previous and next frames, still maintains stable recognition performance in complex playing scenes such as partial occlusion and rapid conversion, overcomes the defect that the existing static detection method is easily affected by single frame quality, and can identify abnormal patterns (such as conversion delay, path error, etc.) in the fingering conversion process, provides more rich time dimension information for error correction feedback, significantly improves the accuracy and practicality of the error correction guidance, thereby solving the key problem that the prior art cannot analyze the coherence and conversion accuracy of the fingering; in summary, based on the above scheme, the dynamic recognition of dizi fingering can be realized by fusing visual spatial information and playing time series context. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is an example flow chart of a deep learning-based real-time dizi fingering recognition method according to some embodiments of the present application; Figure 2 is an operation flow chart of determining three-dimensional coordinate distribution and spatial coordinate distribution according to some embodiments of the present application; Figure 3 is an example flow chart of determining coverage confidence according to some embodiments of the present application; Figure 4 is a structural schematic diagram of a fingering real-time recognition unit according to some embodiments of the present application; Figure 5 is an internal structure diagram of a computer device for implementing a deep learning-based real-time dizi fingering recognition method according to some embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to better understand the technical solutions of the present application, the technical solutions of the present application will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0019] Reference Figure 1 The figure is an example flow chart of a deep learning-based real-time dizi fingering recognition method according to some embodiments of the present application, which mainly includes the following steps: In step 101, a real-time video stream of a target user practicing dizi is collected, and image preprocessing is performed on the real-time video stream to obtain a standard image sequence.

[0020] It should be noted that in the present application, the real-time video stream refers to continuous video data containing the target user's hand and dizi when the target user practices dizi. The real-time video stream can continuously capture the dynamic change characteristics in the fingering conversion process, providing a complete data basis for subsequent time series analysis and trajectory reconstruction. When specifically implemented, the real-time video stream of the target user practicing dizi can be collected in the following manner, i.e., the original video data containing the target user's hand and dizi can be continuously collected at a rate of thirty frames per second by a color image sensor deployed in front of the target user as the real-time video stream of the target user practicing dizi. Wherein, the color image sensor is a binocular camera or a depth camera that synchronously collects image depth information, and has automatic exposure control and automatic white balance function to adapt to different lighting environments.

[0021] In some embodiments, the image preprocessing of the real-time video stream to obtain a standard image sequence can be implemented in the following steps: Decoding and frame sampling are performed on the real-time video stream to obtain a plurality of original images; color space conversion and filtering processing is performed on each frame of the original image to obtain a standard image corresponding to each frame of the original image; The standard image sequence is determined based on all the standard images.

[0022] In a specific implementation, the decoding and frame sampling of the real-time video stream to obtain the multiple frames of original images can be implemented in the following manner: first, a video decoder can be invoked to restore the received compressed real-time video stream into an uncompressed original video frame sequence; then, an equal-interval frame extraction algorithm can be used to perform time sequence downsampling on the decoded original video frame sequence according to a preset target frame rate, for example, from 30 frames per second to 15 frames per second, to selectively retain or discard video frames of specified serial numbers, thereby effectively reducing the data throughput while maintaining smoothness, and assigning a unique timestamp to each frame of image, thereby obtaining the multiple frames of original images; wherein the original image frame refers to a single unprocessed image in the real-time video stream, and the original image frame can provide the most original and undistorted data source for subsequent image processing, thereby ensuring that subsequent processing can be based on complete original information.

[0023] In a specific implementation, the color space conversion and filtering processing of each frame of the original image to obtain a standard image corresponding to each frame of the original image can be implemented in the following manner: for each frame of the original image, first, the original image can be converted from a red-green-blue color space to a hue-saturation-value color space, and in the conversion process, the component values of the hue-saturation-value color space, namely hue, saturation, and value, are calculated pixel by pixel through a nonlinear transformation formula; then, the original image after color space conversion is subjected to filtering enhancement processing, that is, a spatial domain filtering algorithm, for example, a nonlinear filtering method based on linear structure features, can be used to effectively remove impulse noise while maintaining the sharpness of the image edges to suppress image noise, and a contrast enhancement technique, for example, a Retinex image enhancement algorithm based on gamma correction, can be used to enhance local detail information by adaptively adjusting the distribution of image gray levels to improve image quality; finally, the high-quality image after color space conversion and filtering processing is taken as the standard image of the original image, and the standard image corresponding to each frame of the original image can be obtained through the above steps; wherein the standard image refers to a normalized single frame of image after color space conversion and filtering processing, and the standard image eliminates light difference and noise interference, and can provide uniform specification and quality-stable input data for subsequent key point detection.

[0024] It should be noted that in the present application, the standard image sequence refers to a set of continuous standard images arranged in chronological order, which maintains the spatiotemporal continuity of the finger movement and can provide complete and coherent visual data basis for time series analysis and dynamic trajectory reconstruction. In specific implementation, the standard image sequence can be determined based on all standard images in the following manner, i.e., the time series consistency of all standard images can be checked, the timestamp information carried by each frame of image is verified to ensure that the frame interval is uniform and consistent with the preset sampling frequency, thereby avoiding the problem of uneven frame sequence caused by time-consuming fluctuation in processing, and then the ordered image frame queue arranged in chronological order according to all standard images passing the time series consistency check is taken as the standard image sequence.

[0025] In step 102, for each frame of standard image in the standard image sequence, joint key point detection of hand and ocarina is performed on the standard image to obtain the three-dimensional coordinate distribution of the finger joint nodes and the spatial coordinate distribution of the ocarina tone hole center in the process of practicing ocarina by the target user.

[0026] In some embodiments, with reference to Figure 2 The figure is an operation flow chart for determining the three-dimensional coordinate distribution and the spatial coordinate distribution according to some embodiments of the present application. In the present application, joint key point detection of hand and ocarina is performed on the standard image to obtain the three-dimensional coordinate distribution of the finger joint nodes and the spatial coordinate distribution of the ocarina tone hole center in the process of practicing ocarina by the target user, which can be implemented in the following steps: Multi-scale feature extraction is performed on the standard image through the feature extraction layer of the hand-ocarina collaborative perception network to obtain a shared feature map; The semantic features of the hand region and the ocarina region in the shared feature map are enhanced through the parallel spatial attention branch and the channel attention branch respectively to obtain attention-enhanced features; Three-dimensional coordinate regression is performed on the attention-enhanced features to synchronously output the three-dimensional coordinate set of the finger joint nodes and the spatial coordinate set of the ocarina tone hole center; Temporal and spatial consistency verification is performed on the three-dimensional coordinate set and the spatial coordinate set respectively to obtain the three-dimensional coordinate distribution of the finger joint nodes and the spatial coordinate distribution of the ocarina tone hole center in the process of practicing ocarina by the target user.

[0027] It should be noted that in the present application, the hand-ocarina collaborative perception network is a deep learning network architecture specially designed for ocarina fingering recognition, and its core function is to realize the synchronous and accurate positioning and feature extraction of the hand and the ocarina; the hand-ocarina collaborative perception network learns the hand posture features and ocarina structure features through a shared backbone feature extraction layer, avoiding the spatial inconsistency problem caused by separate processing, and on this basis, integrates a parallel spatial attention branch and a channel attention branch; the spatial attention mechanism can adaptively focus on the key spatial regions of finger joints and ocarina tone holes in the image, while the channel attention mechanism selects the feature channels that are most discriminative to the two types of targets, thereby collaboratively enhancing the expression ability of related features, and finally outputting the three-dimensional coordinates of the finger joints and the spatial coordinates of the ocarina tone holes through a unified three-dimensional coordinate regression module; fundamentally ensuring the spatio-temporal synchronization and spatial consistency of hand and ocarina key point detection, thereby significantly improving the robustness and accuracy of fingering recognition under complex scenes, partial occlusion and different playing postures, and providing a reliable data basis for subsequent fingering state judgment and correction.

[0028] In specific implementation, the shared feature map can be realized by the following manner, that is, the standard image is subjected to multi-scale feature extraction by the feature extraction layer of the hand-ocarina collaborative perception network, that is, the deep convolutional neural network structure is adopted to process image features of different scales through multiple parallel convolution paths, the shallow path captures local detail features such as edges and textures of the hand and the ocarina, the deep path extracts global semantic features such as the hand contour and the overall shape of the ocarina, and then the multi-scale features are subjected to scale alignment and channel dimension splicing and fusion through upsampling and downsampling operations, forming a shared feature map containing micro details to macro structure; wherein the shared feature map refers to a unified representation of multi-scale visual features of the target user's hand and the ocarina, which integrates the finger joint contour of the hand region and the tone hole position information of the ocarina region, and can provide a feature basis containing rich spatial context information for subsequent key point detection, avoiding the information fragmentation problem when separately extracting features of the hand and the ocarina.

[0029] In a specific implementation, the semantic features of the hand region and the dizi region in the shared feature map are enhanced by the parallel spatial attention branch and the channel attention branch, respectively, to obtain the attention-enhanced features. The attention-enhanced features can be obtained in the following manner: the shared feature map is enhanced by the parallel spatial attention branch and the channel attention branch in the hand-dizi collaborative perception network, that is, the spatial attention branch generates a spatial attention map by calculating the importance weights of different spatial positions in the shared feature map, highlights the key regions related to the hand joint and the dizi tone hole to suppress irrelevant background interference; the channel attention branch calculates a channel weight vector by modeling the dependency between the feature channels in the shared feature map, enhances the feature channels with strong discriminability for hand posture and dizi structure to weaken redundant channels; then the outputs of the spatial attention branch and the channel attention branch are weighted and fused with the shared feature map by element-wise multiplication to obtain the attention-enhanced features that enhance the semantic expression of the hand and dizi regions. The attention-enhanced features are feature maps optimized by the spatial and channel double attention mechanisms, which can significantly improve the feature response intensity of the hand joint and the dizi tone hole region, thereby effectively suppressing background interference and providing enhanced semantic features for accurate key point positioning.

[0030] In a specific implementation, the attention-enhanced features are subjected to three-dimensional coordinate regression, and the three-dimensional coordinate set of the finger joints and the spatial coordinate set of the dizi tone hole centers are simultaneously output in the following manner: the attention-enhanced features are subjected to three-dimensional coordinate regression by a three-dimensional coordinate regression module in the hand-dizi collaborative perception network. The three-dimensional coordinate regression module adopts a fully convolutional network structure and can gradually map the attention-enhanced features to two independent three-dimensional coordinate prediction channels through multiple convolutional layers. One prediction channel regresses the three-dimensional coordinates of the 21 joints of the hand, and the other prediction channel regresses the three-dimensional coordinates of the centers of the tone holes on the dizi. The key point coordinate prediction of the hand and the dizi is simultaneously optimized through end-to-end training, and finally the three-dimensional coordinate set of the finger joints and the spatial coordinate set of the dizi tone hole centers are simultaneously output. The three-dimensional coordinate set refers to the coordinate data set of each joint of the target user's hand in three-dimensional space, which can accurately describe the spatial position information of the finger joints and provide accurate hand posture data for subsequent fingering state analysis. The spatial coordinate set refers to the coordinate data set of the centers of the tone holes of the dizi in three-dimensional space, which can accurately mark the physical positions of the dizi tone holes and provide an accurate reference benchmark for establishing the spatial correspondence between the fingertips and the tone holes.

[0031] It should be noted that in the present application, the three-dimensional coordinate distribution refers to the three-dimensional coordinate dataset of the target user's hand joint after space-time consistency verification, which can provide stable and reliable hand movement position information, and ensure the space-time continuity and accuracy of the hand posture data in the fingering recognition process; the spatial coordinate distribution refers to the spatial coordinate dataset of the ocarina sound hole after space-time consistency verification, which can establish a stable sound hole space reference system, and provide a reliable space reference for continuously and accurately evaluating the relative position relationship between the fingertip and the sound hole; in specific implementation, the three-dimensional coordinate set and the spatial coordinate set are respectively subjected to space-time consistency verification, and the three-dimensional coordinate distribution of the target user's finger joint and the spatial coordinate distribution of the ocarina sound hole center during ocarina practice can be realized in the following manner, that is: firstly, for the hand joint coordinates, the rationality of the length ratio of each knuckle in the three-dimensional coordinate set can be verified by using the physiological constraints of the existing hand skeleton structure, and the abnormal jitter can be eliminated by using time sequence filtering to smooth the joint trajectory through the continuity of finger movement between adjacent frames, so as to obtain the three-dimensional coordinate distribution of the target user's finger joint during ocarina practice; then, for the ocarina sound hole coordinates, whether the relative position relationship between each sound hole in the spatial coordinate set conforms to the physical constraints can be verified according to the fixed geometric structure of the ocarina, and the abnormal points caused by temporary occlusion or detection error can be excluded through multi-frame coordinate stability analysis, so as to optimize the accuracy and reliability of the coordinates, and obtain the spatial coordinate distribution of the ocarina sound hole center during the target user's ocarina practice.

[0032] In step 103, the spatial relationship calibration between the fingertip and the sound hole is performed based on the three-dimensional coordinate distribution and the spatial coordinate distribution, to obtain the coverage confidence of each finger fingertip of the target user and the corresponding sound hole in the standard image, and the static fingering feature of the target user's finger to the corresponding sound hole coverage state in the standard image is determined according to all the coverage confidences, and then the static fingering feature of the target user's finger to the corresponding sound hole coverage state in each frame of the standard image is determined.

[0033] In some embodiments, with reference to Figure 3 The figure is an exemplary flow chart for determining the coverage confidence according to some embodiments of the present application, and the spatial relationship calibration between the fingertip and the sound hole based on the three-dimensional coordinate distribution and the spatial coordinate distribution in the present application can be realized in the following steps: In step 1031, a local spatial coordinate system centered on each sound hole of the ocarina is established based on the spatial coordinate distribution; In step 1032, the Euclidean distance and the normal vector angle between each finger fingertip of the target user and the corresponding sound hole center in the local spatial coordinate system are calculated according to the three-dimensional coordinate distribution, and then the spatial geometric parameters are obtained. In step 1033, the spatial geometric parameters are corrected in the coordinate system, and a corrected spatial relationship feature is obtained. In step 1034, the sound hole region in the standard image is subjected to pixel-level semantic segmentation, and a coverage feature of each sound hole of the ocarina covered by the finger skin pixels is extracted. In step 1035, the feature fusion is performed based on the spatial relationship feature and the coverage feature, and then the coverage confidence of each finger tip of the target user and the corresponding sound hole in the standard image is obtained.

[0034] In a specific implementation, the local spatial coordinate system centered on each sound hole of the ocarina based on the spatial coordinate distribution can be implemented in the following manner, that is, the center point of the sound hole in the spatial coordinate distribution can be taken as the coordinate origin, the Z-axis of the coordinate system can be determined according to the overall orientation vector of the ocarina body, that is, the direction of the mouthpiece or the tail of the ocarina body, the Y-axis can be determined by using the normal vector of the tangent plane of the ocarina fingering area, that is, perpendicular to the outside of the ocarina body, and finally the X-axis can be determined by the right-hand rule to form a complete orthogonal right-hand coordinate system to obtain the local coordinate system of the sound hole. In the establishment process, the three-dimensional space coordinate calibration method is referred to to ensure the unitization and orthogonalization processing of each coordinate axis vector, so as to eliminate the reference inconsistency problem caused by the change of the spatial posture of the ocarina. Each sound hole in the ocarina is independently constructed into a local coordinate system according to the above process, thereby forming a set of local coordinate systems strictly corresponding to the actual spatial structure of the ocarina, and the set of local coordinate systems is taken as the local spatial coordinate system centered on each sound hole of the ocarina. The local spatial coordinate system refers to a set of independent three-dimensional reference systems established with each sound hole of the ocarina as the center. The local spatial coordinate system can provide a unified spatial measurement reference for each sound hole and eliminate the measurement error caused by the change of the spatial posture of the ocarina.

[0035] In a specific implementation, the spatial geometric parameters can be obtained by the following method: for each finger tip of the target user, first, the three-dimensional coordinates of the finger tip key points in the three-dimensional coordinate distribution are converted to the local coordinate system of the corresponding ocarina sound hole through a spatial coordinate transformation, to obtain the coordinates of the finger tip key points in the local coordinate system; then, the Euclidean distance from the coordinates of the finger tip key points in the local coordinate system to the center of the sound hole is calculated to reflect the proximity between the finger and the sound hole, and the spatial angle between the normal vector at the finger tip point and the normal vector at the center of the sound hole is calculated to evaluate the orientation relationship of the finger relative to the sound hole, and the vector composed of the Euclidean distance and the spatial angle is taken as the geometric vector of the finger tip and the corresponding sound hole, so that the geometric vector of each finger tip of the target user and the corresponding sound hole can be obtained through the above steps, and the set of all geometric vectors is taken as the spatial geometric parameters; the spatial geometric parameters are a set of quantitative index vectors representing the proximity and orientation relationship between the finger tip of the target user and the ocarina sound hole, which can accurately represent the spatial relative position relationship between the finger tip of the target user and the ocarina sound hole, and provide quantifiable geometric features for the coverage state evaluation.

[0036] In a specific implementation, the spatial geometric parameters are subjected to coordinate system distortion correction to obtain the corrected spatial relationship features, which can be implemented by the following method: the spatial geometric parameters are input into a distortion correction model for coordinate system distortion correction to eliminate the spatial measurement deviation caused by lens distortion, view angle tilt and three-dimensional reconstruction error of the camera equipment, and through coordinate transformation and parameter compensation calculation, the corrected accurate geometric parameters are output as the corrected spatial relationship features; the distortion correction model can be established based on the system distortion parameters obtained in advance through a calibration board, which compensates the scale change caused by perspective projection and lens distortion through a nonlinear mapping function for the Euclidean distance parameters, and corrects the angle deviation caused by the coordinate system tilt through a rotation matrix transformation for the normal vector angle parameters; the spatial relationship features are a set of quantitative index vectors representing the proximity and orientation relationship between the finger tip of the target user and the ocarina sound hole after the coordinate system distortion correction, which can provide accurate and reliable spatial relationship data, effectively eliminate system measurement errors, and ensure the accuracy of the coverage state judgment.

[0037] In a specific implementation, the pixel-level semantic segmentation is performed on the sound hole region in the standard image, and the coverage feature of the ceramic flute sound hole covered by the finger skin pixels can be extracted in the following manner, that is: first, for each sound hole of the ceramic flute, the corresponding sound hole region can be determined in the standard image based on the spatial coordinate distribution of the sound hole center in the local coordinate system of the sound hole, for example, a circular region with a radius of 1 cm around the sound hole center coordinate, and then a trained segmentation network (such as U-Net architecture) is used to finely segment the sound hole region, output the probability of each pixel in the sound hole region belonging to the finger skin, ceramic flute material or background, and calculate the ratio of the number of pixels in the sound hole region belonging to the finger skin to the total number of pixels in the sound hole region as the coverage rate of the sound hole covered by the finger skin pixels, and through the above steps, the coverage rate of each sound hole of the ceramic flute covered by the finger skin pixels can be obtained; then, the vector composed of all the sound hole coverage rates is taken as the coverage feature representing the visual coverage degree; wherein the coverage feature is a visual appearance feature reflecting the coverage of the ceramic flute sound hole region by the target user's finger skin pixels, and the coverage feature can directly reflect the actual contact between the finger and the sound hole from the image appearance level, providing important visual evidence for the coverage state judgment.

[0038] It should be noted that in the present application, the coverage confidence is an evaluation of the probability of the target user covering the tone hole with the fingertips when practicing the ocarina. The closer the coverage confidence is to 1, the more complete the coverage of the tone hole by the fingertips, and the closer the coverage confidence is to 0, the less complete or not covered the tone hole is covered by the fingertips. The spatial geometric relationship can be converted into a unified probability index to provide a standardized quantitative basis for fingering state judgment. In specific implementation, the coverage confidence of each finger fingertip of the target user in the standard image and the corresponding tone hole can be obtained by fusing the spatial relationship features and the coverage features in the following manner, that is, the existing feature fusion evaluation network can be used to fuse the spatial relationship features and the coverage features and calculate the coverage confidence of each finger fingertip of the target user and the corresponding tone hole. The feature fusion evaluation network first learns the complex mapping relationship from the spatial geometric features to the tone hole coverage state through a neural network structure with multiple hidden layers, that is, the Euclidean distance and the normal vector angle are first subjected to nonlinear transformation to extract high-level abstract features, and the extracted high-level abstract features are mapped to a scalar value between 0 and 1 through a fully connected layer as the spatial coverage of the finger fingertip and the tone hole corresponding to the Euclidean distance and the normal vector angle. Then, all the spatial coverages and all the coverages in the coverage features are standardized and dimensionally aligned, and then the correlation weight between the two feature modalities is calculated through the cross-attention mechanism and the features are spliced and fused through the fully connected layer. Finally, the fused features are mapped to the coverage confidence in the range of 0-1 through the sigmoid activation function, thereby obtaining the coverage confidence of each finger fingertip of the target user in the standard image and the corresponding tone hole. The feature fusion evaluation network can learn how to optimally combine geometric relationship information and visual appearance information through a large number of labeled fingering samples, can accurately determine when the spatial relationship is ambiguous, for example, when the fingers are very close but not actually in contact with the tone hole, and can rely on visual features to maintain stability when the visual features are disturbed, for example, when skin color recognition is difficult due to changes in lighting, thereby significantly improving the robustness and accuracy of coverage state evaluation.

[0039] In some embodiments, determining the static fingering feature of the target user's finger on the corresponding tone hole coverage state in the standard image according to all the coverage confidences can be achieved in the following steps: Arranging and combining all the coverage confidences in the order of the tone holes of the ocarina to form a touch pressure coupling vector; Performing feature dimension reduction processing on the touch pressure coupling vector to obtain a low-dimensional fingering representation; Extracting a hand shape contour feature from the three-dimensional coordinate distribution of the finger joints of the target user during the practice of the ocarina; Fusing the low-dimensional fingering representation and the hand shape contour feature to obtain a tone hole state tensor; The static feature encoder is used for nonlinear transformation of the finger-hole state tensor to obtain a static fingering feature of a corresponding hole covering state of a target user's finger in a standard image.

[0040] In a specific implementation, all of the covering confidence degrees are arranged and combined according to the hole order of the ocarina to form the touch-pressure coupling vector, which can be implemented in the following manner: the covering confidence degrees of the fingertips of each finger and the corresponding holes are standardized arranged according to the physical arrangement order of the holes of the ocarina, for example, the holes are numbered in sequence from the mouthpiece to the tail of the ocarina according to the spatial positions of the holes on the ocarina body, and the confidence degree values are filled in correspondence to form an initial multi-dimensional vector, and the multi-dimensional vector is taken as the touch-pressure coupling vector; wherein the touch-pressure coupling vector refers to a feature vector composed of the probability estimates of the fingertips of the target user covering the holes when practicing the ocarina arranged according to the hole order, and the touch-pressure coupling vector can systematically organize the discrete finger-hole covering information into structured data, and provide a unified input format for subsequent feature analysis.

[0041] In a specific implementation, the touch-pressure coupling vector is subjected to feature dimension reduction processing to obtain a low-dimensional fingering representation, which can be implemented in the following manner: a linear dimension reduction method based on principal component analysis can be used to perform feature dimension reduction on the touch-pressure coupling vector to obtain a low-dimensional fingering representation, that is, the covariance matrix of the touch-pressure coupling vector is calculated and its eigenvalues and eigenvectors are solved as principal components, and the first few principal components with the highest cumulative contribution rate are selected to construct a low-dimensional subspace, and the original touch-pressure coupling vector is projected into the low-dimensional space to be converted into a low-dimensional fingering representation with higher information density and lower dimension; wherein the low-dimensional fingering representation is a compact feature representation of the probability estimates of the fingertips of the target user covering the holes when practicing the ocarina, and the low-dimensional fingering representation can significantly reduce the feature dimension under the premise of retaining key information, improve the processing efficiency and enhance the feature robustness.

[0042] In a specific implementation, the hand shape contour feature can be extracted from the three-dimensional coordinate distribution of the finger joints of the target user practicing the ocarina in the following manner: the existing point cloud boundary extraction technology (such as the alpha shape algorithm) can be used to extract geometric features representing the outer contour of the hand from the three-dimensional coordinate distribution of the finger joints, such as the bounding box size of the hand and the distribution of the contour key points, and the set of extracted geometric features is taken as the hand shape contour feature of the target user; wherein the hand shape contour feature is a geometric morphological feature describing the special hand shape posture of the target user practicing the ocarina, and the hand shape contour feature can quantitatively describe the spatial posture of the hand during playing, and provide important hand shape structural context information for fingering recognition.

[0043] In a specific implementation, the low-dimensional fingering feature and the hand shape contour feature are fused to obtain the finger hole state tensor in the following manner: first, the low-dimensional fingering feature and the hand shape contour feature are respectively normalized by using the maximum and minimum value normalization method to adjust the numerical range to the same interval; then, the low-dimensional fingering feature and the hand shape contour feature after the scale unification are combined into a comprehensive feature vector by using the vector splicing method, and the comprehensive feature vector is rearranged and filled into a two-dimensional matrix of a preset structure according to the correspondence between the hand region and the tone hole region, so as to form a finger hole state tensor containing multi-dimensional information; wherein the finger hole state tensor is a multi-dimensional data structure reflecting the confidence of the fingers of the target user pressing the tone holes and the actual spatial posture of the hand when practicing the ocarina, and the finger hole state tensor can establish a joint representation of the finger hole coverage and the hand posture, and provide a complete static feature basis for comprehensive evaluation of the fingering state.

[0044] It should be noted that in the present application, the static fingering feature is the final fingering feature representation reflecting the confidence of the fingers of the target user pressing the tone holes and the actual spatial posture of the hand when practicing the ocarina, and the static fingering feature can generate a unified and regular feature expression to ensure that the subsequent time series analysis module obtains stable and reliable input features; in a specific implementation, the static fingering feature of the target user's fingers on the corresponding tone hole coverage state in the standard image is obtained by performing nonlinear transformation on the finger hole state tensor by using a static feature encoder in the following manner: the finger hole state tensor is input into the pre-trained static feature encoder, higher-level abstract features in the finger hole state tensor are gradually learned and extracted by using multiple nonlinear transformations such as a ReLU activation function, and a preliminary fingering feature is output through the last layer of the static feature encoder, and then the preliminary fingering feature is standardized by using a Z-score standardization method to obtain the static fingering feature of the target user's fingers on the corresponding tone hole coverage state in the standard image; wherein the static feature encoder is a lightweight feedforward neural network.

[0045] In step 104, all the static fingering features are analyzed in the time series context to obtain a fingering trajectory that fuses the time and space context of fingering conversion during the target user's practice of the ocarina.

[0046] In some embodiments, the time series context analysis of all the static fingering features to obtain a fingering trajectory that fuses the time and space context of fingering conversion during the target user's practice of the ocarina can be implemented in the following steps: All the static fingering features are arranged in chronological order to construct a fingering feature sequence; Bidirectional time series dependent features are extracted in the fingering feature sequence; performing convolution operation on the bidirectional time sequence dependent features in the time dimension to obtain time sequence enhanced features of fingering conversion mode recognition in a local time window; performing key frame weighting on the time sequence enhanced features through an attention mechanism to obtain spatio-temporal fusion features; reconstructing fingering trajectories that fuse the spatio-temporal context of fingering conversion during the target user's practice of the ocarina based on the spatio-temporal fusion features.

[0047] In a specific implementation, all static fingering features are arranged in chronological order, and the fingering feature sequence can be implemented in the following manner, i.e., a feature sequence obtained by sorting all static fingering features in chronological order according to the time stamp is used as the fingering feature sequence; wherein the fingering feature sequence refers to a sequence of static fingering features arranged in chronological order, which retains the time sequence evolution information of the fingering state during the target user's practice of the ocarina, and provides continuous time dimension data basis for analyzing the fingering conversion process.

[0048] In a specific implementation, the bidirectional time sequence dependent features in the forward and backward directions can be extracted from the fingering feature sequence in the following manner, i.e., the bidirectional time sequence dependent features in the forward and backward directions can be extracted from the fingering feature sequence by a bidirectional deep time sequence modeling network, the forward propagation path of the bidirectional deep time sequence modeling network processes the fingering feature sequence in chronological order, learns the fingering evolution rule from history to current, and captures the forward time sequence dependent features; at the same time, the backward propagation path of the bidirectional deep time sequence modeling network processes the fingering feature sequence in reverse chronological order, learns the fingering correlation from future to current, and captures the backward time sequence dependent features, and the set of forward time sequence dependent features and backward time sequence dependent features is used as the bidirectional time sequence dependent features in the forward and backward directions; wherein both propagation paths of the bidirectional deep time sequence modeling network use a recurrent neural network structure with a gating mechanism, effectively model the long-term dependency relationship in the fingering sequence through the synergistic effect of the forget gate, input gate and output gate, and generate a set of hidden state sequences respectively, which respectively encode the fingering evolution information in different time directions, and then output the forward hidden state sequence as the forward bidirectional time sequence dependent features and the backward hidden state sequence as the bidirectional time sequence dependent features in the forward and backward directions; the bidirectional time sequence dependent features are time correlation features representing fingering conversion during the target user's practice of the ocarina, which can capture the historical rules and future trends of fingering conversion, and enhance the model's understanding ability of fingering continuity.

[0049] In a specific implementation, the convolution operation on the bidirectional time-dependent feature in the time dimension to obtain the time sequence enhancement feature for fingering conversion mode recognition in a local time window can be implemented in the following manner: a set of time convolution kernels with different perception domains can be used to perform sliding window calculation on the bidirectional time-dependent feature sequence along the time dimension, each time convolution kernel is responsible for detecting fingering conversion patterns of a specified time scale, shorter time convolution kernels focus on capturing subtle fingering changes, and longer time convolution kernels aim to identify macro fingering conversion trends. During the convolution operation, each convolution kernel extracts the local spatiotemporal pattern of fingering conversion in the local time window it covers through weight sharing mechanism, and introduces the expression ability of the model using a nonlinear activation function. Multiple convolution kernels process the same input feature in parallel to generate a set of rich time sequence features, each time sequence feature focuses on representing a specified type of fingering conversion rule, and finally the set of all time sequence features is used as the time sequence enhancement feature for fingering conversion mode recognition in a local time window.

[0050] In a specific implementation, the key frame weighting of the time sequence enhancement feature through the attention mechanism to obtain the spatiotemporal fusion feature can be implemented in the following manner: first, a content-based attention model can be used to calculate the importance weight of the time sequence feature of the target user's fingering at each time step in the time sequence enhancement feature with respect to the overall fingering sequence; then, according to the calculated importance weight, all time sequence features in the time sequence enhancement feature are weighted and fused to obtain the spatiotemporal fusion feature that highlights the fingering conversion key nodes; wherein the spatiotemporal fusion feature is an enhanced feature that highlights the fingering conversion key nodes of the target user practicing the ocarina, and the spatiotemporal fusion feature focuses on the key nodes in the fingering conversion process, which can optimize the quality and discriminability of the feature representation.

[0051] It should be noted that in the present application, the fingering trajectory refers to the continuous fingering movement path in the process of the target user practicing the ocarina. The fingering trajectory completely describes the dynamic conversion process of the target user's fingering, and can provide intuitive and reliable space-time motion representation for fingering accuracy evaluation. In specific implementation, the fingering trajectory that fuses the space-time context of fingering conversion in the process of the target user practicing the ocarina can be reconstructed based on the space-time fusion feature in the following manner, that is, the trajectory generation network can be used to map the space-time fusion feature to a continuous and smooth fingering trajectory as the fingering trajectory that fuses the space-time context of fingering conversion in the process of the target user practicing the ocarina, that is, the trajectory generation network first performs nonlinear transformation on the input space-time fusion feature through a fully connected layer to project it to a high-dimensional latent space, then uses a deconvolution operation to gradually upsample, restores the time details of the fingering trajectory, and especially considers the physical constraints of fingering conversion in the trajectory generation process, including the natural smoothness of finger movement, the reasonable speed range of fingering conversion, and the accessibility constraint between adjacent fingerings, to ensure that the generated trajectory conforms to the human hand kinematics law, thereby forming a complete fingering movement trajectory that considers the space-time context.

[0052] In step 105, a fingering recognition result of the target user practicing the ocarina is output based on the fingering trajectory.

[0053] In some embodiments, the fingering recognition result of the target user practicing the ocarina can be output based on the fingering trajectory in the following steps: The fingering trajectory is dynamically time warping matched with a pre-stored standard fingering template to obtain a trajectory similarity; The fingering recognition result of the target user practicing the ocarina is determined based on the trajectory similarity.

[0054] In a specific implementation, the trajectory similarity can be obtained by matching the fingering trajectory with a pre-stored standard fingering template using dynamic time warping, which can be achieved in the following manner: first, a standard fingering template trajectory corresponding to the target user's practice piece can be extracted from a standard fingering library; then, a distance matrix of the dynamic time warping algorithm can be constructed based on the standard fingering template trajectory and the fingering trajectory, each element of the distance matrix representing the Euclidean distance between the fingering trajectory and the standard template trajectory at a specified time point, and the cumulative distance of the warping path with the smallest cumulative distance is then found using a dynamic programming method, and the trajectory similarity is obtained by calculating the normalized cumulative distance of the warping path; wherein the warping path can overcome the time axis shift problem caused by the difference in playing speed, and in the path search process, monotonicity and continuity constraints are imposed to ensure that the time alignment relationship conforms to the actual playing logic; the trajectory similarity is a quantitative value of the matching degree between the target user's fingering trajectory and the standard template, with a numerical range of 0 to 1, where 1 indicates that the target user's fingering trajectory completely matches the standard template, and 0 indicates that the target user's fingering trajectory has no similarity with the standard template; the trajectory similarity can provide an objective evaluation index that is resistant to time stretching interference, thereby realizing quantitative measurement of fingering accuracy.

[0055] In a specific implementation, the fingering recognition result of the target user practicing the ocarina can be determined based on the trajectory similarity in the following manner: first, a multi-level decision threshold can be set according to expert experience and statistical learning, including a correct fingering threshold, an acceptable deviation threshold, and an incorrect fingering threshold, forming a hierarchical decision mechanism; then, the trajectory similarity is compared with the correct fingering threshold, the acceptable deviation threshold, and the incorrect fingering threshold, respectively, and when the trajectory similarity is higher than the correct fingering threshold, it is determined to be correct fingering, when the trajectory similarity is within the acceptable deviation interval, it is determined to have a slight deviation but does not need to be corrected immediately, and when the trajectory similarity is lower than the incorrect fingering threshold, it is determined to be incorrect fingering; for the incorrect fingering, error positioning analysis can be started to accurately identify the finger number and time interval where the error occurs, and match the specific error type by combining the fingering error knowledge base, such as incomplete hole coverage and finger sequence error; finally, a structured recognition result including the fingering determination conclusion, error type description, spatial position coordinates, and targeted correction suggestions is synthesized as the fingering recognition result of the target user practicing the ocarina, and the fingering recognition result is fed back to the target user in real time through a visual interface.

[0056] It should be noted that in this application, the fingering recognition result refers to the structured output of the correctness determination, error positioning, and correction suggestions of the target user practicing the ocarina, which can provide specific and operable fingering feedback to the target user, effectively supporting the implementation of real-time error correction function.

[0057] In addition, another aspect of the present application, in some embodiments, the present application provides a deep learning-based ocarina fingering real-time correction system, which comprises a fingering real-time recognition unit, reference Figure 4 The figure is a structural schematic diagram of a fingering real-time recognition unit according to some embodiments of the present application. The fingering real-time recognition unit 400 comprises a collection module 401, a processing module 402 and an execution module 403, which are described as follows: The collection module 401 is mainly used for collecting real-time video stream of the target user practicing ocarina in the present application, and performing image preprocessing on the real-time video stream to obtain a standard image sequence. The processing module 402 is mainly used for joint key point detection of the standard image in each frame of the standard image sequence in the present application, to obtain the three-dimensional coordinate distribution of the finger joint node and the spatial coordinate distribution of the ocarina sound hole center in the process of the target user practicing ocarina. It should be noted that the processing module 402 is also used for spatial relationship calibration of the fingertips and the sound holes based on the three-dimensional coordinate distribution and the spatial coordinate distribution in the present application, to obtain the coverage confidence of each fingertip of the target user and the corresponding sound hole in the standard image, and to determine the static fingering feature of the target user's finger covering state on the corresponding sound hole in the standard image according to all the coverage confidences, and further to determine the static fingering feature of the target user's finger covering state on the corresponding sound hole in each frame of the standard image. In addition, it should be noted that the processing module 402 is also used for timing context analysis of all the static fingering features in the present application, to obtain the fingering trajectory that fuses the temporal and spatial context of fingering conversion in the process of the target user practicing ocarina. The execution module 403 is mainly used for outputting the fingering recognition result of the target user practicing ocarina based on the fingering trajectory in the present application.

[0058] The above-mentioned various modules of the deep learning-based ocarina fingering real-time correction system can be realized by software, hardware and their combinations. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0059] In addition, in one embodiment, the present application provides a computer device, which can be a server, and its internal structure diagram can be as follows: Figure 5As shown in the figure. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data of the deep learning-based ocarina fingering real-time identification method. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program can implement a deep learning-based ocarina fingering real-time identification method when executed by the processor.

[0060] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0061] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above deep learning-based ocarina fingering real-time identification method embodiments.

[0062] In one embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to implement the steps in the above deep learning-based ocarina fingering real-time identification method embodiments.

[0063] In one embodiment, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the above deep learning-based ocarina fingering real-time identification method embodiments.

[0064] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0065] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0066] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A deep learning-based real-time ocarina fingering recognition method, used to recognize the fingering of a target user in a real-time ocarina fingering correction system, characterized in that, The method includes the following steps: Collect real-time video streams of target users practicing the ocarina, and perform image preprocessing on the real-time video streams to obtain standard image sequences; For each frame of the standard image sequence, joint key point detection of the hand and ocarina is performed on the standard image to obtain the three-dimensional coordinate distribution of the finger joints and the spatial coordinate distribution of the ocarina's sound hole center during the target user's ocarina practice. Based on the three-dimensional coordinate distribution and the spatial coordinate distribution, the spatial relationship between the fingertip and the tone hole is calibrated to obtain the coverage confidence of each fingertip of the target user and the corresponding tone hole in the standard image. Based on all the coverage confidence, the static fingering features of the target user's finger covering the corresponding tone hole in the standard image are determined, and then the static fingering features of the target user's finger covering the corresponding tone hole in each frame of the standard image are determined. Temporal context analysis was performed on all static fingering features to obtain the fingering trajectory that incorporates the spatiotemporal context of fingering transitions during the target user's ocarina practice. Based on the fingering trajectory, the fingering recognition results of the target user practicing the ocarina are output.

2. The method as described in claim 1, characterized in that, The process of preprocessing the real-time video stream to obtain a standard image sequence specifically includes: The real-time video stream is decoded and frame sampled to obtain multiple frames of original images; Color space conversion and filtering are performed on each original image frame to obtain the standard image corresponding to each original image frame; A standard image sequence is determined based on all the standard images.

3. The method as described in claim 1, characterized in that, Joint keypoint detection of the hand and ocarina is performed on standard images to obtain the three-dimensional coordinate distribution of finger joints and the spatial coordinate distribution of the ocarina's tone hole center during the target user's ocarina practice. Specifically, this includes: Multi-scale feature extraction is performed on standard images through the feature extraction layer of the hand-ocarina collaborative perception network to obtain a shared feature map; By enhancing the semantic features of the hand region and ocarina region in the shared feature map through parallel spatial attention branches and channel attention branches, attention-enhanced features are obtained. Perform three-dimensional coordinate regression on the attention enhancement features and simultaneously output the three-dimensional coordinate set of the finger joint points and the spatial coordinate set of the ocarina sound hole center; The spatiotemporal consistency of the three-dimensional coordinate set and the spatial coordinate set are checked respectively to obtain the three-dimensional coordinate distribution of the finger joints and the spatial coordinate distribution of the ocarina's sound hole center during the target user's ocarina practice.

4. The method as described in claim 1, characterized in that, Based on the three-dimensional coordinate distribution and the spatial coordinate distribution, the spatial relationship between the fingertip and the tone hole is calibrated to obtain the coverage confidence of each fingertip of the target user and the corresponding tone hole in the standard image. Specifically, this includes: A local spatial coordinate system centered on each tone hole of the ocarina is established based on the aforementioned spatial coordinate distribution. Based on the three-dimensional coordinate distribution and the Euclidean distance and normal angle between the fingertips of the target user and the corresponding sound hole center in the local spatial coordinate system, spatial geometric parameters are obtained. The spatial geometric parameters are subjected to coordinate system distortion correction to obtain the corrected spatial relationship characteristics; Pixel-level semantic segmentation is performed on the tone hole region in the standard image to extract the coverage features of each tone hole of the ocarina by the pixels covered by the finger skin. Feature fusion is performed based on the spatial relationship features and the coverage features to obtain the coverage confidence of each fingertip of the target user and the corresponding tone hole in the standard image.

5. The method as described in claim 1, characterized in that, Based on all coverage confidence scores, the static fingering features of the target user's finger covering the corresponding tone holes in the standard image are determined, specifically including: All coverage confidences are arranged and combined according to the order of the ocarina's tone holes to form a touch-pressure coupling vector; The touch-pressure coupling vector is subjected to feature dimensionality reduction processing to obtain a low-dimensional fingering representation; Hand shape contour features are extracted from the three-dimensional coordinate distribution of finger joints during target users' ocarina practice. The low-dimensional fingering representation is fused with the hand shape contour features to obtain the finger hole state tensor. By performing a nonlinear transformation on the finger hole state tensor using a static feature encoder, static fingering features of the target user's finger covering the corresponding tone hole in the standard image are obtained.

6. The method as described in claim 1, characterized in that, Temporal context analysis was performed on all static fingering features to obtain the fingering trajectory that incorporates the spatiotemporal context of fingering transitions during the target user's ocarina practice. This specifically includes: Arrange all static fingering features in chronological order to construct a fingering feature sequence; Extract forward and backward bidirectional temporal dependency features from the fingering feature sequence; Perform a convolution operation on the bidirectional temporal dependency features in the temporal dimension to obtain temporal enhancement features for finger transfer pattern recognition within a local time window; The temporal enhancement features are weighted by keyframes using an attention mechanism to obtain spatiotemporal fusion features; Based on the aforementioned spatiotemporal fusion features, the fingering trajectory of the spatiotemporal context of fingering transitions during the ocarina practice of the target user is reconstructed.

7. The method as described in claim 1, characterized in that, The fingering recognition results for the target user's ocarina practice, based on the aforementioned fingering trajectory, specifically include: The finger movement trajectory is dynamically time-warped and matched with a pre-stored standard finger movement template to obtain the trajectory similarity. The fingering recognition result of the target user practicing the ocarina is determined based on the trajectory similarity.

8. A real-time fingering correction system for the ocarina based on deep learning, the system comprising a real-time fingering recognition unit, characterized in that, The real-time finger recognition unit includes: The acquisition module is used to acquire real-time video streams of target users practicing the ocarina, and to perform image preprocessing on the real-time video streams to obtain standard image sequences. The processing module is used to perform joint key point detection of the hand and ocarina on each frame of the standard image sequence, so as to obtain the three-dimensional coordinate distribution of the finger joints and the spatial coordinate distribution of the ocarina's sound hole center during the target user's ocarina practice. The processing module is used to calibrate the spatial relationship between the fingertip and the tone hole based on the three-dimensional coordinate distribution and the spatial coordinate distribution, obtain the coverage confidence of each fingertip of the target user and the corresponding tone hole in the standard image, determine the static fingering features of the target user's finger covering the corresponding tone hole in the standard image based on all the coverage confidence, and then determine the static fingering features of the target user's finger covering the corresponding tone hole in each frame of the standard image. The processing module is used to perform temporal context analysis on all static fingering features to obtain the fingering trajectory that integrates the spatiotemporal context of fingering transitions during the target user's ocarina practice. The execution module is used to output the finger recognition results of the target user practicing the ocarina based on the finger trajectory.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the real-time ocarina fingering recognition method based on deep learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time ocarina fingering recognition method based on deep learning as described in any one of claims 1 to 7.

Images