A musical instrument fingering perception and recognition method based on AR technology

Abstract

The present invention relates to the field of augmented reality technology, and in particular to a musical instrument fingering perception and recognition method based on AR technology, comprising: step S1, constructing a user target musical instrument three-dimensional model and a user hand three-dimensional model according to a three-dimensional model building method; step S2, configuring the user target musical instrument three-dimensional model and the user hand three-dimensional model to an AR environment; step S3, correcting the user target musical instrument three-dimensional model, and applying the corrected user target musical instrument three-dimensional model to step S2; step S4, optimizing the process of correcting the user target musical instrument three-dimensional model; step S5, correcting step S4 according to the number of optimizations P within a preset cycle and the preset number of optimizations P0; step S6, real-time acquisition of user fingering data; step S7, identifying the user fingering data, and synchronously rendering the identification result to the user hand three-dimensional model; step S8, outputting according to the user target requirements. The present invention improves the efficiency and accuracy of musical instrument learning.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a musical instrument fingering perception and recognition method based on AR technology. Background Art

[0002] In traditional instrument learning, beginners often find it difficult to master the correct fingering and posture, and face-to-face guidance from teachers is inefficient and costly. Although some existing instrument teaching software can provide fingering diagrams, they lack interactivity and cannot correct learners' incorrect movements in real time. Augmented reality technology can superimpose virtual information into the real environment, providing a new way to solve this problem.

[0003] Chinese Patent Publication No.: CN111679742A discloses an AR-based interactive control method, device, electronic device and storage medium, wherein the method includes: obtaining a real scene image captured by an AR device; when a physical musical instrument is detected in the real scene image, displaying a virtual musical instrument model corresponding to the physical musical instrument in the AR device; receiving a trigger operation for the virtual musical instrument model displayed in the AR device; based on the trigger operation, controlling the AR device to play the sound effect corresponding to the trigger operation. However, this solution can only be used when a physical musical instrument is presented in the real scene image, and it is difficult to improve the efficiency and accuracy of musical instrument learning. Summary of the invention

[0004] To this end, the present invention provides a musical instrument fingering perception and recognition method based on AR technology, so as to overcome the problems of low efficiency and accuracy of musical instrument learning in the prior art.

[0005] To achieve the above object, the present invention provides a musical instrument fingering perception and recognition method based on AR technology, comprising:

[0006] Step S1, constructing a three-dimensional model of a user's target musical instrument and a three-dimensional model of a user's hand according to a three-dimensional model building method;

[0007] Step S2, configuring the user's target instrument 3D model and the user's hand 3D model into the AR environment;

[0008] Step S3, correcting the user target instrument three-dimensional model according to the real-time position information of the user target instrument three-dimensional model, and applying the corrected user target instrument three-dimensional model to step S2;

[0009] Step S4, optimizing the process of correcting the three-dimensional model of the user's target musical instrument according to the fixing condition of the user terminal device, and applying the optimized process of correcting the three-dimensional model of the user's target musical instrument to step S3;

[0010] Step S5, modifying step S4 according to the number of optimizations P within a preset period and the preset number of optimizations P0;

[0011] Step S6, collecting user fingering data in real time;

[0012] Step S7, identifying the user's fingering data, and synchronously rendering the identification result onto the user's hand three-dimensional model;

[0013] Step S8, obtaining user target requirements and outputting according to the user target requirements.

[0014] Furthermore, the three-dimensional model building method includes:

[0015] Step S100, collecting a user hand image and a target musical instrument image to obtain user hand image data and target musical instrument image data;

[0016] Step S101, constructing a three-dimensional model of a user's target musical instrument;

[0017] Step S102: construct a three-dimensional model of the user's hand.

[0018] Furthermore, in the step S100, the user hand image and the target musical instrument image are collected by the user terminal device to obtain the user hand image data and the target musical instrument image data;

[0019] In step S101, when constructing the user target instrument three-dimensional model, the target instrument image data is compared with the instrument model library, and the user target instrument three-dimensional model is output according to the comparison result, wherein:

[0020] When there is preset target instrument image data consistent with the target instrument image data in the instrument model library, the preset target instrument model corresponding to the preset target instrument image data is output as the user target instrument three-dimensional model;

[0021] When there is no preset target instrument image data consistent with the target instrument image data in the instrument model library, a reminder to update the instrument model library is pushed to the instrument model administrator terminal, and the instrument model administrator updates the instrument model library and constructs the user's target instrument three-dimensional model based on the updated instrument model library;

[0022] In the step S101, when constructing the user's target musical instrument three-dimensional model, a personalized model setting window is also pushed to the user, and the user personalizes the user's target musical instrument three-dimensional model in the personalized model setting window.

[0023] Furthermore, in the step S102, when constructing the three-dimensional model of the user's hand, the user's hand image data is processed by the hand image processing model to obtain hand processed image data;

[0024] In the step S102, when constructing the three-dimensional model of the user's hand, the preset hand-model database is also divided into a 70% hand training set, a 20% hand verification set and a 10% hand test set, the hand training set is input into the 3D convolutional neural network model to train the 3D convolutional neural network model, and the hand verification set is input into the trained 3D convolutional neural network model, the parameters of the trained 3D convolutional neural network model are iteratively optimized, and the hand test set is input into the iteratively optimized 3D convolutional neural network model for testing to obtain a test accuracy C, the test accuracy C is compared with the preset test accuracy C0, and the training compliance of the 3D convolutional neural network model is judged according to the comparison result, and output according to the judgment result, wherein:

[0025] When C≥C0, the 3D convolutional neural network model is determined to have met the training standards, the 3D convolutional neural network model is output as a 3D hand model, and the hand processing image data is input into the 3D hand model to obtain a 3D hand model of the user;

[0026] When C<C0, it is determined that the 3D convolutional neural network model training does not meet the standards, the preset hand-model database is updated, and the 3D convolutional neural network model is trained according to the updated preset hand-model database until the 3D convolutional neural network model training meets the standards.

[0027] Furthermore, in the step S2, an AR tool is used to configure the user's target instrument three-dimensional model and the user's hand three-dimensional model into the AR environment, and parameters of the user's target instrument three-dimensional model and the user's hand three-dimensional model are set according to preset rules.

[0028] Furthermore, in step S3, the AR environment image data is obtained, the real-time position information of the user's target instrument three-dimensional model is detected according to the image detection model, the practicality of the user's target instrument three-dimensional model is judged according to the detection result, and the user's target instrument three-dimensional model is corrected according to the judgment result, wherein:

[0029] When the detection result is that the target instrument is at the target position, the practicability of the three-dimensional model of the user's target instrument is determined to be qualified, and the three-dimensional model of the user's target instrument is not corrected;

[0030] When the detection result is that it is not in the target position, the practicality of the user target instrument three-dimensional model is determined to be unqualified, the user target instrument three-dimensional model is corrected, the current three-dimensional coordinates (x, y, z) of the user target instrument three-dimensional model are obtained, the target three-dimensional coordinates are set to (x0, y0, z0), the x-axis offset is Δx=x0-x, the y-axis offset is Δy=y0-y, and the z-axis offset is Δz=z0-z. The three-dimensional coordinates of the corrected user target instrument three-dimensional model are (x1, y1, z1), x1=x+Δx, y1=y+Δy, z1=z+Δz, and the corrected user target instrument three-dimensional model is applied to step S2.

[0031] Furthermore, in the step S4, the fixing condition of the user terminal device is obtained, and the detection stability of the real-time position information of the user target instrument three-dimensional model is judged according to the fixing condition of the user terminal device, and the process of correcting the user target instrument three-dimensional model is optimized according to the judgment result, wherein:

[0032] When the user terminal device is fixed by a bracket, it is determined that the detection stability of the real-time position information of the user target instrument three-dimensional model is up to standard, and the process of correcting the user target instrument three-dimensional model is not optimized;

[0033] When the user terminal device is fixed in the form of being worn on the body, it is determined that the detection stability of the real-time position information of the correction of the user's target instrument three-dimensional model does not meet the standard, and the process of correcting the user's target instrument three-dimensional model is optimized to adjusting the user terminal device, and the optimized process of correcting the user's target instrument three-dimensional model is applied to step S3.

[0034] Furthermore, in step S5, the optimization times P of step S4 within a preset period are obtained, the optimization times P are compared with the preset optimization times P0, the optimization frequency of step S4 is judged according to the comparison result, and step S4 is corrected according to the judgment result, wherein:

[0035] When P<P0, step S4 is determined to be low-frequency optimization, and step S4 is not corrected;

[0036] When P≥P0, step S4 is determined to be high-frequency optimization, the fixing status of the user terminal device is corrected to being worn on the body, and a device fixing reminder is pushed to the user, so that the user fixes the user terminal device.

[0037] Furthermore, in step S7, the support vector machine is trained by pre-setting user fingering data, and the trained support vector machine is output as a fingering recognition model, the user fingering data is input into the fingering recognition model for recognition, and the fingering recognition result is obtained, and the recognition result is synchronously rendered onto the three-dimensional model of the user's hand through the three-dimensional graphics library and the three-dimensional engine.

[0038] Furthermore, in step S8, the user's target demand is acquired by pushing the target demand window to the user, and output is performed according to the user's target demand, wherein:

[0039] When the user's goal is free practice, the target teaching fingering is not provided to the user, and the user's incorrect fingering when learning the target teaching fingering is not collected;

[0040] When the user's goal is to learn a course, the system provides the user with target teaching fingerings, collects the user's incorrect fingerings during the course learning process, and provides real-time feedback on the user's incorrect fingerings and makes teaching suggestions.

[0041] Compared with the prior art, the beneficial effect of the present invention lies in that the method constructs a user target instrument 3D model and a user hand 3D model according to the 3D model establishment method through step S1 to provide a high-precision instrument and hand model, laying a foundation for subsequent AR display; the method displays the 3D model in an AR environment through step S2 to increase interactivity and immersion, and provide a visual learning and performance guidance platform; the method corrects the user target instrument 3D model through step S3 to improve the accuracy and authenticity of the instrument model in the AR environment, and ensures that the model position of the user during operation is consistent with the actual position; the method corrects the process of correcting the user target instrument 3D model through step S4 Optimize, improve the efficiency and accuracy of operation, provide a smoother user experience, reduce delays and errors, the method corrects the correction process through step S5 to further improve accuracy, provide a dynamic adjustment mechanism, and adapt to different usage scenarios. The method collects user fingering data in real time through step S6 to provide data support for subsequent analysis and guidance. The method identifies the user's fingering movements through step S7 and renders them to the user's hand three-dimensional model to provide intuitive feedback, help the user correct incorrect fingering, and improve learning efficiency. The method performs personalized output according to user needs through step S8 to meet the specific requirements of different users, provide diversified functions, and enhance practicality and attractiveness.

[0042] In particular, in step S100, the user hand image and the target instrument image are collected by the user terminal device, which can provide accurate user hand image data and target instrument image data, and significantly improve the interactivity, immersion and personalized customization capabilities of the augmented reality application.

[0043] In particular, in the step S101, the efficiency and accuracy of instrument recognition are improved by quickly matching the user's target instrument image with the preset image data in the model library, and when there is no matching instrument image data in the model library, update reminders are promptly pushed to the administrator to ensure timely updating and expansion of the model library. At the same time, a variety of setting options are provided through the personalized setting window to make the user's target instrument three-dimensional model more flexible and diversified, and able to meet the needs of different users.

[0044] In particular, in step S102, by dividing the preset hand-model database into a training set, a validation set and a test set, the generalization ability of the 3D convolutional neural network model on different data can be ensured to avoid overfitting, and through iterative optimization and the introduction of the validation set, the 3D convolutional neural network model parameters can be continuously adjusted during the training process to improve the performance and accuracy of the 3D convolutional neural network model. The training results of the 3D convolutional neural network model are clearly evaluated through the preset test accuracy C0 to ensure that the output three-dimensional hand model has high accuracy and practicality. At the same time, by updating the preset hand-model database, the 3D convolutional neural network model can be continuously improved to adapt to more diverse user hand image data, thereby improving the robustness and generalization ability of the 3D convolutional neural network model.

[0045] In particular, in step S2, through AR technology, users can interact with virtual musical instruments in real time in the real world, providing a more intuitive and immersive experience, thereby helping learners master the operating skills of musical instruments more quickly.

[0046] In particular, in step S3, through real-time detection and correction, the position of the three-dimensional model of the user's target instrument is ensured to be accurate, and the three-dimensional model of the user's target instrument is ensured to be always in the preset target position, avoiding the inconvenience caused by the position offset of the model, and providing more accurate feedback and guidance for the user's operation. At the same time, the automatic correction mechanism effectively reduces the operational errors caused by position offset and improves reliability and stability.

[0047] In particular, in step S4, when the user terminal device is in a bracket-fixed state, the device has high stability, so the real-time position information detection of the user's target instrument three-dimensional model will be more accurate, avoiding errors caused by device shaking and movement. The optimization process can omit unnecessary correction steps, thereby improving efficiency. When the user terminal device is in a wearable state, the device will move with the user, resulting in substandard detection stability. Timely detection and optimization of the correction process can reduce erroneous detection and improve reliability.

[0048] In particular, in step S5, the optimization strategy can be dynamically adjusted based on the comparison between the real-time optimization times P and the preset optimization times P0. When the optimization times are lower than the preset value, the existing optimization frequency is maintained. When the optimization times reach and exceed the preset optimization times, it is automatically determined as high-frequency optimization and corrected, thereby improving the intelligence level and pushing device fixed reminders to avoid frequent optimization due to unfixed equipment and improve equipment use efficiency.

[0049] In particular, in step S7, the support vector machine is trained by presetting the user's fingering data to obtain a fingering recognition model, so that the fingering recognition model can efficiently and accurately recognize fingering movements, thereby obtaining accurate fingering recognition results.

[0050] In particular, in step S8, through the target demand window, the user's current needs can be accurately understood, thereby providing a personalized learning experience. The user can choose free practice and learning courses according to his or her own learning stage and goals to ensure that the learning content and feedback mechanism meet their needs. In the free practice mode, the user can independently choose the practice content and method without being restricted by the preset teaching process, thereby enhancing the flexibility and autonomy of learning and facilitating the user's self-exploration and creation. In the learning course mode, standard target teaching fingerings are provided to help users master correct movements and techniques. At the same time, by real-time collection and analysis of the user's incorrect fingerings, timely feedback and teaching suggestions can be provided to ensure that the user can quickly correct the errors and improve learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the process of the instrument fingering perception and recognition method based on AR technology in this embodiment;

[0052] Figure 2 It is a schematic diagram of the flow of the three-dimensional model building method of this embodiment. DETAILED DESCRIPTION

[0053] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0055] See also Figure 1 As shown, it is a flow chart of the instrument fingering perception and recognition method based on AR technology in this example, and the method includes:

[0056] Step S1, constructing a three-dimensional model of a user's target musical instrument and a three-dimensional model of a user's hand according to a three-dimensional model building method;

[0057] Step S2, configuring the user's target instrument 3D model and the user's hand 3D model into the AR environment;

[0058] Step S3, correcting the user target instrument three-dimensional model according to the real-time position information of the user target instrument three-dimensional model, and applying the corrected user target instrument three-dimensional model to step S2;

[0059] Step S4, optimizing the process of correcting the three-dimensional model of the user's target musical instrument according to the fixing condition of the user terminal device, and applying the optimized process of correcting the three-dimensional model of the user's target musical instrument to step S3;

[0060] Step S5, modifying step S4 according to the number of optimizations P within a preset period and the preset number of optimizations P0;

[0061] Step S6, collecting user fingering data in real time;

[0062] Step S7, identifying the user's fingering data, and synchronously rendering the identification result onto the user's hand three-dimensional model;

[0063] Step S8, obtaining user target requirements and outputting according to the user target requirements.

[0064] Specifically, the method is applied to music teaching, instrument performance guidance and virtual instrument application scenarios. The method establishes a three-dimensional model of the user's target instrument and hand, and configures the model in an AR environment to form a virtual display. At the same time, the three-dimensional model of the user's target instrument is corrected according to real-time position information to ensure its accuracy in the AR environment. The accuracy and stability of the model are improved by optimizing the correction process, and the correction process is corrected according to the number of optimizations within a preset cycle to further improve the accuracy. The user's fingering data is collected in real time and the collected fingering data is identified, and the identification result is synchronously rendered to the user's hand three-dimensional model, and finally output according to the user's target needs. In particular, the method constructs the user's target instrument three-dimensional model and the user's hand three-dimensional model according to the three-dimensional model establishment method through step S1 to provide a high-precision instrument and hand model, laying a foundation for subsequent AR display. The method displays the three-dimensional model in the AR environment through step S2 to increase interactivity and immersion, and provide A visualized learning and performance guidance platform, wherein the method corrects the three-dimensional model of the user's target instrument through step S3, improves the accuracy and authenticity of the instrument model in the AR environment, and ensures that the model position during the user's operation is consistent with the actual position; the method optimizes the process of correcting the three-dimensional model of the user's target instrument through step S4, improves the efficiency and accuracy of the operation, provides a smoother user experience, and reduces delays and errors; the method corrects the correction process through step S5, further improves the accuracy, provides a dynamic adjustment mechanism, and adapts to different usage scenarios; the method collects user fingering data in real time through step S6, and provides data support for subsequent analysis and guidance; the method identifies the user's fingering movements through step S7, and renders them onto the three-dimensional model of the user's hand, provides intuitive feedback, helps the user correct incorrect fingering, and improves learning efficiency; the method performs personalized output according to user needs through step S8, meets the specific requirements of different users, provides diversified functions, and enhances practicality and attractiveness.

[0065] See also Figure 2 As shown, it is a schematic diagram of the process of the three-dimensional model building method of this embodiment, and the three-dimensional model building method includes:

[0066] Step S100, collecting a user hand image and a target musical instrument image to obtain user hand image data and target musical instrument image data;

[0067] Step S101, constructing a three-dimensional model of a user's target musical instrument;

[0068] Step S102: construct a three-dimensional model of the user's hand.

[0069] Specifically, in step S100, the user's hand image and the target musical instrument image are collected by the user terminal device to obtain the user's hand image data and the target musical instrument image data.

[0070] Specifically, the user terminal device refers to the hardware device used by the user to experience and interact with AR content, such as a smart phone, a tablet computer and AR glasses; the user hand image data refers to the image of the user's hand collected by the sensors and cameras on the user terminal device, including the front image of the hand, the back image of the hand and the side image of the hand; the target musical instrument image data refers to the image information containing the target musical instrument collected by the sensors and cameras on the user terminal device, such as the type, appearance, shape, color and texture of the musical instrument.

[0071] Specifically, in step S100, the user hand image and the target instrument image are collected by the user terminal device, which can provide accurate user hand image data and target instrument image data, and significantly improve the interactivity, immersion and personalized customization capabilities of the augmented reality application.

[0072] Specifically, in step S101, when constructing the user target instrument 3D model, the target instrument image data is compared with the instrument model library, and the user target instrument 3D model is output according to the comparison result, wherein:

[0073] When there is preset target instrument image data consistent with the target instrument image data in the instrument model library, the preset target instrument model corresponding to the preset target instrument image data is output as the user target instrument three-dimensional model;

[0074] When there is no preset target instrument image data consistent with the target instrument image data in the instrument model library, a reminder to update the instrument model library is pushed to the instrument model administrator terminal, and the instrument model administrator updates the instrument model library and constructs the user's target instrument three-dimensional model based on the updated instrument model library;

[0075] In the step S101, when constructing the user's target musical instrument three-dimensional model, a personalized model setting window is also pushed to the user, and the user personalizes the user's target musical instrument three-dimensional model in the personalized model setting window.

[0076] Specifically, the musical instrument model library refers to a pre-set database containing preset target musical instrument image data and preset target musical instrument models corresponding to the preset target musical instrument image data, the preset target musical instrument image data refers to the musical instrument image data pre-stored in the musical instrument model library, the preset target musical instrument model refers to the user target musical instrument three-dimensional model generated according to the preset target musical instrument image data and stored in the musical instrument model library, the musical instrument model administrator terminal refers to a computing device and software interface used by the musical instrument model administrator, the musical instrument model administrator refers to a professional responsible for managing and maintaining the musical instrument model library, this embodiment does not limit the update method of the musical instrument model library, and relevant technical personnel in this field can freely set it according to actual needs, and only need to meet the needs of updating the musical instrument model library, such as setting the update method of the musical instrument model library to update through big data, the personalized model setting window refers to the interactive interface for the user to personalize the target user's target musical instrument three-dimensional model pushed to the user in the process of building the target user's target musical instrument three-dimensional model, the personalized setting refers to the user's custom adjustment to the target user's target musical instrument three-dimensional model through the personalized model setting window, for example, the user can select and modify the color of the musical instrument, change the material of the musical instrument, adjust the size of the musical instrument, add and remove decorations on the musical instrument, and adjust the sound properties of the musical instrument.

[0077] Specifically, in step S101, the efficiency and accuracy of instrument recognition are improved by quickly matching the user's target instrument image with the preset image data in the model library, and when there is no matching instrument image data in the model library, update reminders are promptly pushed to the administrator to ensure timely updating and expansion of the model library. At the same time, a variety of setting options are provided through the personalized setting window to make the user's target instrument three-dimensional model more flexible and diversified, and able to meet the needs of different users.

[0078] Specifically, in step S102, when constructing the three-dimensional model of the user's hand, the user's hand image data is processed by the hand image processing model to obtain hand processed image data;

[0079] In the step S102, when constructing the three-dimensional model of the user's hand, the preset hand-model database is also divided into a 70% hand training set, a 20% hand verification set and a 10% hand test set, the hand training set is input into the 3D convolutional neural network model to train the 3D convolutional neural network model, and the hand verification set is input into the trained 3D convolutional neural network model, the parameters of the trained 3D convolutional neural network model are iteratively optimized, and the hand test set is input into the iteratively optimized 3D convolutional neural network model for testing to obtain a test accuracy C, the test accuracy C is compared with the preset test accuracy C0, and the training compliance of the 3D convolutional neural network model is judged according to the comparison result, and output according to the judgment result, wherein:

[0080] When C≥C0, the 3D convolutional neural network model is determined to have met the training standards, the 3D convolutional neural network model is output as a 3D hand model, and the hand processing image data is input into the 3D hand model to obtain a 3D hand model of the user;

[0081] When C<C0, it is determined that the 3D convolutional neural network model training does not meet the standards, the preset hand-model database is updated, and the 3D convolutional neural network model is trained according to the updated preset hand-model database until the 3D convolutional neural network model training meets the standards.

[0082] Specifically, the hand image processing model refers to a convolutional neural network model that is trained according to a preset image processing database and meets a preset processing accuracy. The convolutional neural network model refers to a deep learning model used to extract features from user hand image data and perform classification and regression. This embodiment does not limit the training process of the convolutional neural network model. Relevant technical personnel in this field can freely set it according to actual needs, and only need to meet the needs of processing user hand image data. For example, the convolutional neural network model can be set to be trained through a preset user hand image processing database. The preset user hand image processing database refers to a user hand image database that has been pre-collected and processed for training, verifying and testing convolutional neural network models. The preset hand-model database refers to a user hand image data and its corresponding The database of three-dimensional models, the hand training set refers to a data set used to train the 3D convolutional neural network model, the hand verification set refers to a data set used to verify the performance of the 3D convolutional neural network model and optimize parameters during the training process, the hand test set refers to a data set for evaluating the final performance of the 3D convolutional neural network model after the training and verification are completed, the test accuracy C refers to the prediction accuracy of the 3D convolutional neural network model on the hand test set, and the preset test accuracy C0 refers to a preset test accuracy value for judging whether the model has been trained to meet the standard. This embodiment does not limit the update method of the preset hand-model database. Relevant technical personnel in this field can freely set it according to actual needs, and only need to meet the needs of training the 3D convolutional neural network model. For example, the update method of the preset hand-model database can be set to update through big data.

[0083] Specifically, in step S102, by dividing the preset hand-model database into a training set, a validation set and a test set, the generalization ability of the 3D convolutional neural network model on different data can be ensured to avoid overfitting, and through iterative optimization and the introduction of the validation set, the 3D convolutional neural network model parameters can be continuously adjusted during the training process to improve the performance and accuracy of the 3D convolutional neural network model. The training results of the 3D convolutional neural network model are clearly evaluated through the preset test accuracy C0 to ensure that the output three-dimensional hand model has high accuracy and practicality. At the same time, by updating the preset hand-model database, the 3D convolutional neural network model can be continuously improved to adapt to more diverse user hand image data, thereby improving the robustness and generalization ability of the 3D convolutional neural network model.

[0084] Specifically, in step S2, the user's target instrument three-dimensional model and the user's hand three-dimensional model are configured into the AR environment using an AR tool, and parameters of the user's target instrument three-dimensional model and the user's hand three-dimensional model are set according to preset rules.

[0085] Specifically, the AR tool refers to software that can introduce three-dimensional models into an augmented reality environment for interaction and display, such as Unity3D and Microsoft HoloLens. The preset rules refer to the standards and specifications required for configuring and operating three-dimensional models in an AR environment, such as position and alignment rules, interaction rules, physical property rules, and animation and motion rules. The AR environment refers to a scene in which virtual information and digital content are superimposed on the real world.

[0086] Specifically, in step S2, through AR technology, users can interact with virtual musical instruments in real time in the real world, providing a more intuitive and immersive experience, thereby helping learners master the operating skills of musical instruments more quickly.

[0087] Specifically, in step S3, the AR environment image data is obtained, the real-time position information of the user's target instrument three-dimensional model is detected according to the image detection model, the practicality of the user's target instrument three-dimensional model is judged according to the detection result, and the user's target instrument three-dimensional model is corrected according to the judgment result, wherein:

[0088] When the detection result is that the target instrument is at the target position, the practicability of the three-dimensional model of the user's target instrument is determined to be qualified, and the three-dimensional model of the user's target instrument is not corrected;

[0089] When the detection result is that it is not in the target position, the practicality of the user target instrument three-dimensional model is determined to be unqualified, the user target instrument three-dimensional model is corrected, the current three-dimensional coordinates (x, y, z) of the user target instrument three-dimensional model are obtained, the target three-dimensional coordinates are set to (x0, y0, z0), the x-axis offset is Δx=x0-x, the y-axis offset is Δy=y0-y, and the z-axis offset is Δz=z0-z. The three-dimensional coordinates of the corrected user target instrument three-dimensional model are (x1, y1, z1), x1=x+Δx, y1=y+Δy, z1=z+Δz, and the corrected user target instrument three-dimensional model is applied to step S2.

[0090] Specifically, the AR environment image data refers to real-time image frames acquired through a user terminal device, the image detection model refers to a computer vision model used to identify and locate musical instruments in an AR environment, such as a YOLO model and a convolutional neural network model, the real-time position information of the user's target musical instrument three-dimensional model refers to the position of the user's target musical instrument three-dimensional model in three-dimensional space, being at the target position means that the real-time position information of the user's target musical instrument three-dimensional model conforms to the target three-dimensional coordinates, not being at the target position means that the real-time position information of the user's target musical instrument three-dimensional model does not conform to the target three-dimensional coordinates, the current three-dimensional coordinates refer to the position of the user's target musical instrument three-dimensional model in the three-dimensional space at the current moment, the target three-dimensional coordinates refer to the preset position of the user's target musical instrument three-dimensional model in the three-dimensional space, and the offset refers to the difference between the current three-dimensional coordinates and the target three-dimensional coordinates.

[0091] Specifically, in step S3, through real-time detection and correction, the position of the three-dimensional model of the user's target instrument is ensured to be accurate, and the three-dimensional model of the user's target instrument is ensured to be always in the preset target position, avoiding the inconvenience caused by the position offset of the model, and providing more accurate feedback and guidance for the user's operation. At the same time, the automatic correction mechanism effectively reduces the operational errors caused by position offset and improves reliability and stability.

[0092] Specifically, in step S4, the fixation status of the user terminal device is obtained, and the detection stability of the real-time position information of the user target instrument three-dimensional model is judged according to the fixation status of the user terminal device, and the process of correcting the user target instrument three-dimensional model is optimized according to the judgment result, wherein:

[0093] When the user terminal device is fixed by a bracket, it is determined that the detection stability of the real-time position information of the user target instrument three-dimensional model is up to standard, and the process of correcting the user target instrument three-dimensional model is not optimized;

[0094] When the user terminal device is fixed in the form of being worn on the body, it is determined that the detection stability of the real-time position information of the correction of the user's target instrument three-dimensional model does not meet the standard, and the process of correcting the user's target instrument three-dimensional model is optimized to adjusting the user terminal device, and the optimized process of correcting the user's target instrument three-dimensional model is applied to step S3.

[0095] Specifically, this embodiment does not limit the method of obtaining the fixing status of the user terminal device. Relevant technical personnel in the field can freely set it according to actual needs, and it is only necessary to meet the need to obtain the fixing status of the user terminal device. For example, it can be set to push a fixing status acquisition window to the user for the user to fill in. This embodiment does not limit the process of adjusting the user terminal device. Relevant technical personnel in the field can freely set it according to actual needs, and it is only necessary to meet the need to optimize the process of correcting the three-dimensional model of the user's target musical instrument. For example, the process of adjusting the user terminal device can be set to adjust the direction of the camera in the user terminal device. The fixing status of the user terminal device refers to the state in which the user terminal device is fixed during use. The bracket fixation refers to fixing the user terminal device at a position and angle by using a bracket. The wearable refers to wearing the user terminal device on the user's body, such as wearing it on the hand and wearing it on the head.

[0096] Specifically, in step S4, when the user terminal device is in a bracket-fixed state, the stability of the device is high, so the real-time position information detection of the three-dimensional model of the user's target instrument will be more accurate, avoiding errors caused by device shaking and movement. The optimization process can omit unnecessary correction steps, thereby improving efficiency. When the user terminal device is in a wearable state, the device will move with the user, resulting in substandard detection stability. Timely detection and optimization of the correction process can reduce erroneous detection and improve reliability.

[0097] Specifically, in step S5, the optimization times P of step S4 within a preset period are obtained, the optimization times P are compared with the preset optimization times P0, the optimization frequency of step S4 is judged according to the comparison result, and step S4 is corrected according to the judgment result, wherein:

[0098] When P<P0, step S4 is determined to be low-frequency optimization, and step S4 is not corrected;

[0099] When P≥P0, step S4 is determined to be high-frequency optimization, the fixing status of the user terminal device is corrected to being worn on the body, and a device fixing reminder is pushed to the user, so that the user fixes the user terminal device.

[0100] Specifically, the preset period refers to a preset time interval for performing correction operations on step S4. The present embodiment does not fix the specific value of the preset period. Relevant technical personnel in the field can freely set it according to actual needs, and it is only necessary to meet the need to judge the optimization frequency of step S4. For example, the preset period can be set to 1 hour. The number of optimizations P refers to the cumulative number of optimization operations performed in step S4 within the preset period. The preset number of optimizations P0 refers to a preset value for judging the optimization frequency of step S4. The present embodiment does not fix the specific value of the preset number of optimizations. Relevant technical personnel in the field can freely set it according to actual needs, and it is only necessary to meet the need to judge the optimization frequency of step S4. For example, the preset number of optimizations P0 can be set to 4. The fixed reminder of the device refers to a notification sent to the user when step S4 is determined to be a high-frequency optimization.

[0101] Specifically, in step S5, the optimization strategy can be dynamically adjusted based on the comparison between the real-time optimization times P and the preset optimization times P0. When the optimization times are lower than the preset value, the existing optimization frequency is maintained. When the optimization times reach and exceed the preset optimization times, it is automatically determined as high-frequency optimization and corrected, thereby improving the intelligence level and pushing device fixed reminders to avoid frequent optimization due to unfixed equipment and improve equipment use efficiency.

[0102] Specifically, in step S6, this embodiment does not limit the real-time collection method of user fingering data. Relevant technical personnel in the field can freely set it according to actual needs, and it is only necessary to meet the demand for real-time collection of user fingering data. For example, the real-time collection method of user fingering data can be set to collect user fingering data through a user terminal. The user fingering data refers to the finger movement and posture data of the user's fingers in the AR environment.

[0103] Specifically, in step S7, the support vector machine is trained by pre-setting user fingering data, and the trained support vector machine is output as a fingering recognition model, the user fingering data is input into the fingering recognition model for recognition, and the fingering recognition result is obtained, and the recognition result is synchronously rendered onto the three-dimensional model of the user's hand through the three-dimensional graphics library and the three-dimensional engine.

[0104] Specifically, the preset user fingering data refers to user finger movement and posture data collected and sorted in advance, the support vector machine refers to a supervised learning model used for classification and regression analysis, the fingering recognition result refers to the result of action recognition obtained after the input user fingering data is recognized by the fingering recognition model, the three-dimensional graphics library refers to a software library for creating and operating three-dimensional graphic objects, the three-dimensional engine refers to a software framework for creating 3D applications, and the synchronous rendering refers to updating the fingering recognition result to the rendering of the user's hand three-dimensional model in real time, so that the user's hand three-dimensional model can display the action in real time.

[0105] Specifically, in step S7, the support vector machine is trained by presetting the user's fingering data to obtain a fingering recognition model, so that the fingering recognition model can efficiently and accurately recognize the fingering movements, thereby obtaining an accurate fingering recognition result.

[0106] Specifically, in step S8, the user's target demand is acquired by pushing the target demand window to the user, and output is performed according to the user's target demand, wherein:

[0107] When the user's goal is free practice, the target teaching fingering is not provided to the user, and the user's incorrect fingering when learning the target teaching fingering is not collected;

[0108] When the user's goal is to learn a course, the system provides the user with target teaching fingerings, collects the user's incorrect fingerings during the course learning process, and provides real-time feedback on the user's incorrect fingerings and makes teaching suggestions.

[0109] Specifically, the target requirement window refers to an interface for interacting with the user, the target teaching fingering refers to the standard fingering movements and gestures provided to the user, the incorrect fingering refers to the deviation and error caused by the user's failure to correctly perform the standard movements in the process of learning the target teaching fingering, and the teaching suggestion refers to the improvement and correction suggestions provided to the user after the user's incorrect fingering is detected, for example, please pay attention to relax your wrist, use the little finger of your right hand to press the C key, and the palm should face forward. This embodiment does not limit the method of real-time feedback. Relevant technical personnel in this field can freely set it according to actual needs, and only need to meet the demand for real-time feedback on the user's incorrect fingering. For example, the real-time feedback method can be set to highlight the error position on the user interface.

[0110] Specifically, in step S8, through the target demand window, the user's current needs can be accurately understood, thereby providing a personalized learning experience. The user can choose free practice and learning courses according to his or her own learning stage and goals to ensure that the learning content and feedback mechanism meet their needs. In the free practice mode, the user can independently choose the practice content and method without being restricted by the preset teaching process, thereby enhancing the flexibility and autonomy of learning and facilitating the user's self-exploration and creation. In the learning course mode, standard target teaching fingerings are provided to help users master correct movements and techniques. At the same time, by real-time collection and analysis of the user's incorrect fingerings, timely feedback and teaching suggestions can be provided to ensure that the user can quickly correct the errors and improve learning efficiency.

[0111] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A musical instrument fingering perception and recognition method based on AR technology, characterized in that: include: Step S1, constructing a three-dimensional model of a user's target musical instrument and a three-dimensional model of a user's hand according to a three-dimensional model building method; Step S2, configuring the user's target instrument 3D model and the user's hand 3D model into the AR environment; Step S3, correcting the user target instrument three-dimensional model according to the real-time position information of the user target instrument three-dimensional model, and applying the corrected user target instrument three-dimensional model to step S2; Step S4, optimizing the process of correcting the three-dimensional model of the user's target musical instrument according to the fixing condition of the user terminal device, and applying the optimized process of correcting the three-dimensional model of the user's target musical instrument to step S3; Step S5, modifying step S4 according to the number of optimizations P within a preset period and the preset number of optimizations P0; Step S6, collecting user fingering data in real time; Step S7, identifying the user's fingering data, and synchronously rendering the identification result onto the user's hand three-dimensional model; Step S8, obtaining user target requirements and outputting according to the user target requirements.

2. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: The three-dimensional model building method comprises: Step S100, collecting a user hand image and a target musical instrument image to obtain user hand image data and target musical instrument image data; Step S101, constructing a three-dimensional model of a user's target musical instrument; Step S102: construct a three-dimensional model of the user's hand.

3. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 2, characterized in that: In the step S100, the user hand image and the target musical instrument image are collected by the user terminal device to obtain the user hand image data and the target musical instrument image data; In step S101, when constructing the user target instrument three-dimensional model, the target instrument image data is compared with the instrument model library, and the user target instrument three-dimensional model is output according to the comparison result, wherein: When there is preset target instrument image data consistent with the target instrument image data in the instrument model library, the preset target instrument model corresponding to the preset target instrument image data is output as the user target instrument three-dimensional model; When there is no preset target instrument image data consistent with the target instrument image data in the instrument model library, a reminder to update the instrument model library is pushed to the instrument model administrator terminal, and the instrument model administrator updates the instrument model library and constructs the user's target instrument three-dimensional model based on the updated instrument model library; In the step S101, when constructing the user's target musical instrument three-dimensional model, a personalized model setting window is also pushed to the user, and the user personalizes the user's target musical instrument three-dimensional model in the personalized model setting window.

4. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 2, characterized in that: In the step S102, when constructing the three-dimensional model of the user's hand, the user's hand image data is processed by the hand image processing model to obtain hand processed image data; In the step S102, when constructing the three-dimensional model of the user's hand, the preset hand-model database is also divided into a 70% hand training set, a 20% hand verification set and a 10% hand test set, the hand training set is input into the 3D convolutional neural network model to train the 3D convolutional neural network model, and the hand verification set is input into the trained 3D convolutional neural network model, the parameters of the trained 3D convolutional neural network model are iteratively optimized, and the hand test set is input into the iteratively optimized 3D convolutional neural network model for testing to obtain a test accuracy C, the test accuracy C is compared with the preset test accuracy C0, and the training compliance of the 3D convolutional neural network model is judged according to the comparison result, and output according to the judgment result, wherein: When C≥C0, the 3D convolutional neural network model is determined to have met the training standards, the 3D convolutional neural network model is output as a 3D hand model, and the hand processing image data is input into the 3D hand model to obtain a 3D hand model of the user; When C<C0, it is determined that the 3D convolutional neural network model training does not meet the standards, the preset hand-model database is updated, and the 3D convolutional neural network model is trained according to the updated preset hand-model database until the 3D convolutional neural network model training meets the standards.

5. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: In the step S2, the user's target instrument three-dimensional model and the user's hand three-dimensional model are configured into the AR environment using an AR tool, and parameters of the user's target instrument three-dimensional model and the user's hand three-dimensional model are set according to preset rules.

6. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: In the step S3, the AR environment image data is obtained, the real-time position information of the user's target instrument three-dimensional model is detected according to the image detection model, the practicality of the user's target instrument three-dimensional model is judged according to the detection result, and the user's target instrument three-dimensional model is corrected according to the judgment result, wherein: When the detection result is that the target instrument is at the target position, the practicability of the three-dimensional model of the user's target instrument is determined to be qualified, and the three-dimensional model of the user's target instrument is not corrected; When the detection result is that it is not in the target position, the practicality of the user target instrument three-dimensional model is determined to be unqualified, the user target instrument three-dimensional model is corrected, the current three-dimensional coordinates (x, y, z) of the user target instrument three-dimensional model are obtained, the target three-dimensional coordinates are set to (x0, y0, z0), the x-axis offset is Δx=x0-x, the y-axis offset is Δy=y0-y, and the z-axis offset is Δz=z0-z. The three-dimensional coordinates of the corrected user target instrument three-dimensional model are (x1, y1, z1), x1=x+Δx, y1=y+Δy, z1=z+Δz, and the corrected user target instrument three-dimensional model is applied to step S2.

7. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: In the step S4, the fixing condition of the user terminal device is obtained, and the detection stability of the real-time position information of the user target instrument three-dimensional model is judged according to the fixing condition of the user terminal device, and the process of correcting the user target instrument three-dimensional model is optimized according to the judgment result, wherein: When the user terminal device is fixed by a bracket, it is determined that the detection stability of the real-time position information of the user target instrument three-dimensional model is up to standard, and the process of correcting the user target instrument three-dimensional model is not optimized; When the user terminal device is fixed in the form of being worn on the body, it is determined that the detection stability of the real-time position information of the correction of the user's target instrument three-dimensional model does not meet the standard, and the process of correcting the user's target instrument three-dimensional model is optimized to adjusting the user terminal device, and the optimized process of correcting the user's target instrument three-dimensional model is applied to step S3.

8. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: In step S5, the optimization times P of step S4 within a preset period are obtained, the optimization times P are compared with the preset optimization times P0, the optimization frequency of step S4 is judged according to the comparison result, and step S4 is corrected according to the judgment result, wherein: When P<P0, step S4 is determined to be low-frequency optimization, and step S4 is not corrected; When P≥P0, step S4 is determined to be high-frequency optimization, the fixing status of the user terminal device is corrected to being worn on the body, and a device fixing reminder is pushed to the user, so that the user fixes the user terminal device.

9. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: In step S7, the support vector machine is trained by presetting the user's fingering data, and the trained support vector machine is output as a fingering recognition model. The user's fingering data is input into the fingering recognition model for recognition to obtain the fingering recognition result, and the recognition result is synchronously rendered onto the user's hand three-dimensional model through the three-dimensional graphics library and the three-dimensional engine.

10. The method for perceiving and identifying musical instrument fingerings based on AR technology according to claim 1, characterized in that: In step S8, the user's target demand is acquired by pushing the target demand window to the user, and output is performed according to the user's target demand, wherein: When the user's goal is free practice, the target teaching fingering is not provided to the user, and the user's incorrect fingering when learning the target teaching fingering is not collected; When the user's goal is to learn a course, the system provides the user with target teaching fingerings, collects the user's incorrect fingerings during the course learning process, and provides real-time feedback on the user's incorrect fingerings and makes teaching suggestions.

Images