An intelligent air instrument based on motion capture and virtual reality technology

Abstract

The present invention belongs to the technical field of intelligent musical instruments, and specifically relates to an intelligent air musical instrument based on motion capture and virtual reality technology. The invention can intelligently correct the user's playing movements through a deviation judgment module and a historical action matching function. Even if the user's movements are not completely standard, the intention can be intelligently identified through historical data matching, greatly reducing performance errors. Due to the intelligent correction and historical data matching functions, users can quickly get started even if they use it for the first time. The action deviation correction function allows beginners to play standard notes quickly, reducing the difficulty of learning music performance, and is suitable for music education promotion. The timbre output module not only converts the target tapping action into the corresponding performance timbre based on the user's hand tapping strength, frequency and other status information, ensuring that high-fidelity virtual musical instrument performance effects are presented in real time through virtual reality headsets and audio equipment.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent musical instruments, and in particular relates to an intelligent air musical instrument based on motion capture and virtual reality technology. Background Art

[0002] With the rapid development of computer technology and information technology, motion capture and virtual reality technology (VR) have gradually penetrated into all aspects of our lives, from game entertainment, education and training to medical rehabilitation. Its wide range of application scenarios continues to expand, bringing people unprecedented immersive experience.

[0003] In the field of musical instrument percussion, traditional musical instrument percussion requires people to directly touch the instrument with their fingers or other body parts to produce sound. However, with the advancement of technology and people's constant pursuit of musical experience, a new intelligent air instrument based on motion capture and virtual reality technology has emerged. The rapid development of motion capture technology and virtual reality technology (VR) has provided new possibilities for the innovation of musical percussion methods. Motion capture technology can accurately capture the movement information of the human body, while virtual reality technology can build a realistic virtual environment, making the user feel as if they are in a real scene. Combining these two technologies can realize the real-time capture and conversion of the percussionist's movements, thereby achieving the goal of percussing music without physical contact with the instrument.

[0004] However, most smart musical instruments on the market currently focus on a single playing experience. Larger instruments often require a larger venue, and the user is required to touch the instrument to play. At the same time, there is a lack of real-time analysis and intelligent correction functions for the user's playing movements. This means that beginners may still produce undesirable sound effects due to non-standard movements when playing, affecting learning outcomes. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent air musical instrument based on motion capture and virtual reality technology, which can realize music performance without physical musical instruments through motion capture and virtual reality technology, providing users with an innovative and highly free performance experience.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An intelligent air musical instrument based on motion capture and virtual reality technology, comprising a percussion action module, a deviation judgment module, a history action module, a percussion action module and a timbre output module;

[0008] A striking action module, used to obtain body motion data of a target user, and obtain multiple musical instrument striking action information from the body motion data;

[0009] The deviation judgment module is used to judge whether the musical instrument striking action information meets the first preset condition. If not, it is judged that the user's striking deviation is present and marked as the musical instrument striking deviation action information.

[0010] A historical action module is used to obtain playing user information, obtain corresponding multiple musical instrument striking historical action information according to the playing user information, and mark it as musical instrument striking corresponding action information;

[0011] A striking action module, used to obtain the striking action similarity between the striking deviation action information of the musical instrument and the striking corresponding action information of each musical instrument, obtain the striking corresponding action information of the musical instrument corresponding to the striking action similarity that meets the second preset condition, and mark it as the target striking action information;

[0012] The timbre output module is used to obtain the hand striking state information in the target striking action information, and obtain the corresponding performance timbre according to the target striking action information and the hand striking state information.

[0013] In a preferred embodiment, the deviation judgment module includes a standard action unit and an action judgment unit;

[0014] Standard action unit, used to obtain standard action information of musical instrument percussion;

[0015] An action judgment unit, used to judge whether the musical instrument striking action information is the same as the musical instrument striking standard action information;

[0016] If the musical instrument striking action information is different from the musical instrument striking standard action information, the user's striking deviation is determined and marked as the musical instrument striking deviation action information;

[0017] If the musical instrument striking action information is the same as the musical instrument striking standard action information, the user's striking standard is determined, and the corresponding performance timbre is output according to the same musical instrument striking standard action information.

[0018] In a preferred embodiment, the historical action module includes a user information unit, a biometric information unit, an information collection unit, and an information identification unit;

[0019] A user information unit, used to obtain information about the performing user;

[0020] A biometric information unit, used to obtain biometric information from the performing user information, wherein the biometric information includes fingerprint information, portrait information, iris information and voice information;

[0021] An information collection unit, used to obtain multiple biometric information in a local database and aggregate them into a biometric collection;

[0022] An information identification unit, used to identify whether the biometric information exists in the biometric identification set;

[0023] If the biometric information exists in the database, the performing user is determined to be an old user;

[0024] If the biometric information does not exist in the database, it is determined that the performing user is a new user.

[0025] In a preferred embodiment, the historical action module further includes a known user unit and an action unit;

[0026] A known user unit is used to obtain, for the old user according to the determination result, a plurality of musical instrument striking history action information corresponding to the old user in a local database;

[0027] The action unit is used to mark each musical instrument striking historical action information as musical instrument striking corresponding action information.

[0028] In a preferred embodiment, the historical action module further includes an end time unit, a standard duration unit, a start time unit, an action summary unit, and a corresponding action unit;

[0029] An end time unit is used to obtain the time of the determination result for the new user according to the determination result and mark it as the end time;

[0030] Standard duration unit, used to obtain standard duration;

[0031] The start time unit is used to obtain the start time based on the standard duration and the end time;

[0032] An action summary unit, used to obtain an action summary period according to a start time and an end time;

[0033] The corresponding action unit is used to obtain the historical action information of multiple musical instruments of all users within the action summary period, and mark it as the corresponding action information of musical instrument percussion.

[0034] In a preferred embodiment, the tapping action module includes a deviation feature unit, a corresponding feature unit and an action similarity unit;

[0035] A deviation feature unit, used for obtaining a plurality of upper limb joint feature vectors corresponding to the deviation action information of the musical instrument striking, and marking them as upper limb joint deviation feature vectors;

[0036] A corresponding feature unit is used to obtain a plurality of upper limb joint feature vectors corresponding to each musical instrument striking corresponding action information, and mark them as upper limb joint corresponding feature vectors;

[0037] The action similarity unit is used to obtain the striking action similarity between the musical instrument striking deviation action information and the striking action corresponding action information of each musical instrument according to multiple upper limb joint deviation feature vectors and multiple upper limb joint corresponding feature vectors.

[0038] In a preferred embodiment, the tapping action module further includes a similarity threshold unit, a similarity judgment unit, a similarity table unit and a target action unit;

[0039] A similarity threshold unit, used to obtain a similarity threshold of a tapping action;

[0040] A similarity judgment unit, used to judge whether the tapping action similarity exceeds a tapping action similarity threshold;

[0041] If the tapping action similarity exceeds the tapping action similarity threshold, the tapping action similarity exceeding the tapping action similarity threshold is marked as the first tapping action similarity;

[0042] A similarity table unit, used for sorting the first similarities of the plurality of tapping actions in descending order to obtain a tapping action similarity table;

[0043] The target action unit is used to obtain the instrument striking action information corresponding to the maximum striking action similarity from the striking action similarity table, and mark it as the target striking action information.

[0044] In a preferred embodiment, the timbre output module includes a striking state unit, a state value unit, an action table unit, a sub-target table unit and a timbre output unit;

[0045] A tapping state unit, used to obtain hand tapping state information in the target tapping action information, wherein the hand tapping state information includes hand tapping speed, hand tapping strength and hand tapping angle;

[0046] A state value unit, used to obtain a tapping state value according to a hand tapping speed, a hand tapping force and a hand tapping angle;

[0047] An action table unit, used for obtaining a timbre action table, wherein the timbre action table includes a plurality of striking action sub-tables and each striking action sub-table includes a plurality of playing timbres corresponding to striking state values;

[0048] A sub-target table unit, used for obtaining a corresponding knocking action sub-table from the timbre action table according to the target knocking action information, and marking it as a knocking action sub-target table;

[0049] The timbre output unit is used to obtain the corresponding performance timbre from the striking action sub-target table according to the striking state value, and output the performance timbre.

[0050] In a preferred embodiment, it also includes a virtual action unit, a virtual sound unit and a virtual reality unit;

[0051] A virtual action unit, used for obtaining a corresponding tapping virtual action according to target tapping action information;

[0052] A sound virtualization unit, used to obtain a corresponding instrument sound virtual effect according to the performance timbre;

[0053] The virtual reality unit is used to obtain the corresponding virtual reality performance scene according to the virtual action of striking and the virtual effect of the sound of the musical instrument, and feed it back to the performing user.

[0054] And, an intelligent air instrument terminal based on motion capture and virtual reality technology, comprising:

[0055] one or more processors;

[0056] a storage device having one or more programs stored thereon;

[0057] When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent air instrument based on motion capture and virtual reality technology.

[0058] The technical effects achieved by the present invention are:

[0059] The present invention can intelligently correct the user's playing movements through the deviation judgment module and the historical action matching function. Even if the user's movements are not completely standard, the intention can be intelligently identified through historical data matching, thereby greatly reducing playing errors. Since it has the intelligent correction and historical data matching functions, the user can quickly get started even if it is the first time to use it. The action deviation correction function allows beginners to play standard notes quickly, reducing the difficulty of learning music performance, and is suitable for music education promotion. The timbre output module not only converts the target tapping action into the corresponding playing timbre based on the user's hand tapping strength, frequency and other status information, ensuring that a high-fidelity virtual instrument playing effect is presented in real time through a virtual reality head display and audio equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a system module diagram provided by the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0064] Secondly, the present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the schematic diagrams are only examples and should not limit the scope of protection of the present invention.

[0065] Please see attached Figure 1 As shown, an intelligent air instrument based on motion capture and virtual reality technology is provided, comprising:

[0066] It includes a knocking action module, a deviation judgment module, a history action module, a knocking action module and a tone output module;

[0067] A striking action module, used to obtain body motion data of a target user, and obtain multiple musical instrument striking action information from the body motion data;

[0068] The deviation judgment module is used to judge whether the musical instrument striking action information meets the first preset condition. If not, it is judged that the user's striking deviation is present and marked as the musical instrument striking deviation action information.

[0069] A historical action module is used to obtain playing user information, obtain corresponding multiple musical instrument striking historical action information according to the playing user information, and mark it as musical instrument striking corresponding action information;

[0070] A striking action module, used to obtain the striking action similarity between the striking deviation action information of the musical instrument and the striking corresponding action information of each musical instrument, obtain the striking corresponding action information of the musical instrument corresponding to the striking action similarity that meets the second preset condition, and mark it as the target striking action information;

[0071] The timbre output module is used to obtain the hand striking state information in the target striking action information, and obtain the corresponding performance timbre according to the target striking action information and the hand striking state information.

[0072] As mentioned above, when percussion music is combined with virtual reality, the percussion music includes drumstick percussion instruments, gong hammer percussion instruments and cymbals. The percussion action module obtains the user's body movement data, especially the "percussion" action of the hand in the air. These data are collected by high-frequency motion capture equipment, including the position information of the user's hand, movement trajectory, etc. Through these data, the percussion action module identifies multiple different instrument percussion action information. The deviation judgment module is responsible for analyzing the accuracy of the percussion action and pre-sets an action standard, namely the "first preset condition", which is used to judge whether the performer's action deviates from the standard action. If the percussion action does not match the preset standard action, the deviation judgment module will judge it as "percussion deviation" and mark these deviation actions as "instrument percussion deviation action information". The historical action module obtains the corresponding historical action data based on the performing user information, and compares the user's current deviation action with the corresponding historical performance record to find the most similar "instrument percussion corresponding action information", that is, the historical action that is closest to the current deviation action. The percussion action module is responsible for calculating the correlation between the current deviation action and the historical action. Similarity, by calculating the "striking action similarity" to evaluate the degree of match between the two, if the similarity meets the preset "second preset condition", the action will be marked as "target tapping action information", which means that the action can be considered as the action that the user wants to express. The timbre output module obtains the "target tapping action information" and "hand tapping state information" to finally generate the corresponding timbre output. The timbre output module not only converts the target tapping action into the corresponding performance timbre based on the user's hand tapping strength, frequency and other state information, ensuring that the virtual reality headset and audio equipment can present high-fidelity virtual instrument performance effects in real time. Through the deviation judgment module and the historical action matching function, the user's performance action can be intelligently corrected. Even if the user's action is not completely standard, the historical data matching can be used to intelligently identify its intention, greatly reducing performance errors. Due to the intelligent correction and historical data matching functions, users can quickly get started even if they use it for the first time. The action deviation correction function allows beginners to play standard notes quickly, reducing the difficulty of learning music performance, which is suitable for music education promotion.

[0073] In a specific embodiment, the deviation judgment module includes a standard action unit and an action judgment unit;

[0074] Standard action unit, used to obtain standard action information of musical instrument percussion;

[0075] An action judgment unit, used to judge whether the musical instrument striking action information is the same as the musical instrument striking standard action information;

[0076] If the musical instrument striking action information is different from the musical instrument striking standard action information, the user's striking deviation is determined and marked as the musical instrument striking deviation action information;

[0077] If the musical instrument striking action information is the same as the musical instrument striking standard action information, the user's striking standard is determined, and the corresponding performance timbre is output according to the same musical instrument striking standard action information.

[0078] As mentioned above, the standard action unit obtains preset standard action information for musical instrument percussion. These standard action information are usually set in the design stage or entered by the user when using it for the first time, and include specific action parameters for correctly playing a certain note or musical instrument, such as the movement trajectory, speed, strength, etc. of the hand. These standard information are used as a benchmark for measuring whether the user's percussion action is correct. The action judgment unit will compare the user's percussion action information with the standard action information to determine whether it meets the preset standards. If the performer's action completely matches the standard action, the user's action will be judged as a correct standard action, and the corresponding performance timbre will be directly output according to the standard action. If the action judgment unit detects that the user's tapping action is inconsistent with the standard action, it will be immediately judged as a tapping deviation. This deviation may come from the deviation of the user's action speed, strength, or action trajectory. The action will be marked as "instrument tapping deviation action information". By accurately comparing the user's tapping action with the preset standard action information, the accuracy of action recognition is greatly improved. It can accurately identify the user's playing action, ensure that the playing tone output is highly matched with the actual action, and generate tone immediately when the action matches the standard, ensuring the real-time response effect. The user can feel feedback similar to real musical instruments when playing, thereby enhancing the presence and immersion of the performance.

[0079] In a specific embodiment, the historical action module includes a user information unit, a biometric information unit, an information collection unit, and an information identification unit;

[0080] A user information unit, used to obtain information about the performing user;

[0081] A biometric information unit, used to obtain biometric information from the performing user information, wherein the biometric information includes fingerprint information, portrait information, iris information and voice information;

[0082] An information collection unit, used to obtain multiple biometric information in a local database and aggregate them into a biometric collection;

[0083] An information identification unit, used to identify whether the biometric information exists in the biometric identification set;

[0084] If the biometric information exists in the database, the performing user is determined to be an old user;

[0085] If the biometric information does not exist in the database, it is determined that the performing user is a new user.

[0086] As mentioned above, the user information unit is responsible for obtaining the basic information of the performing user, which may include the user's name, ID, biometric data, playing habits and other basic data, which provides a basis for subsequent user identity authentication and personalized performance. The biometric information unit extracts the user's biometric data from the user information, such as fingerprint information, portrait information, iris information and voice information. These biometric information are highly unique and personalized data, ensuring accurate identification of the user's identity. Through these biometric information, different users can be more accurately identified and distinguished. The information collection unit extracts the existing user biometric information from the local database and aggregates this information into a "biometric collection", which contains all the user biometric information that has been used for subsequent user identification and matching. Update and expand this collection to ensure the integrity and accuracy of the database information. The information identification unit matches the newly acquired user biometric information to determine whether it exists in the biometric collection. If the match is successful, it means that the user is an old user of the system, and its historical performance data will be directly called to provide a personalized performance experience for the user. If the biometric information does not exist in the database, the user is judged to be a new user, and a new biometric record and performance file will be created for the user to facilitate subsequent tracking of the user's performance habits and preferences. The user's identity can be accurately distinguished, whether it is fingerprints, portraits, irises or voice information, ensuring the high security and accuracy of identity authentication. This multimodal biometric recognition method can effectively prevent identity confusion or misidentification, and protect the user's privacy and the security of personalized data.

[0087] In a specific embodiment, the historical action module further includes a known user unit and an action unit;

[0088] A known user unit is used to obtain, for the old user according to the determination result, a plurality of musical instrument striking history action information corresponding to the old user in a local database;

[0089] The action unit is used to mark each musical instrument striking historical action information as musical instrument striking corresponding action information.

[0090] As mentioned above, when the information recognition unit determines that the user is an old user, the known user unit will extract the user's historical instrument striking action information from the local database. These historical action information include each striking action record when the user wears a virtual reality helmet to use an intelligent air instrument, reflecting the user's playing style, frequency and action habits. The action unit receives these historical action data and marks each historical action information, defining it as "instrument striking corresponding action information". This marking process ensures that the historical data can be effectively called and compared or matched with the current performance. When the user performs a new performance, it can be intelligently compared with the user's current action based on these marked "instrument striking corresponding action information". Through this matching process, it can be identified whether the user is repeating past playing actions or has new ideas, and the output timbre or effect can be adjusted to make the playing experience more personalized and intelligent. For old users, they are already familiar with their past playing habits and do not need to reset or adapt each time. Users can seamlessly continue the content of the last performance, which greatly reduces the learning curve and improves the user's experience.

[0091] In a specific embodiment, the historical action module further includes an end time unit, a standard duration unit, a start time unit, an action summary unit, and a corresponding action unit;

[0092] An end time unit is used to obtain the time of the determination result for the new user according to the determination result and mark it as the end time;

[0093] Standard duration unit, used to obtain standard duration;

[0094] The start time unit is used to obtain the start time based on the standard duration and the end time;

[0095] An action summary unit, used to obtain an action summary period according to a start time and an end time;

[0096] The corresponding action unit is used to obtain the historical action information of multiple musical instruments of all users within the action summary period, and mark it as the corresponding action information of musical instrument percussion.

[0097] As mentioned above, the end time unit records the time of the identification result of the new user and marks it as the "end time". This time point represents the moment when the new user's identity identification is completed. It serves as a key reference point for the subsequent time period calculation. The standard duration unit is responsible for extracting the preset "standard duration". This duration defines the time period range required for reference when analyzing motion data. It is usually set based on factors such as music rhythm and the response speed of motion capture technology to ensure that user motion data can be processed within a reasonable time window. The start time unit calculates the "start time" based on the "standard duration" that has been obtained and the "end time" recorded previously. This process ensures that the historical motion data of all users is collected from a suitable time range. The action summary unit calculates the "start time" based on the "start time" and "end time". " is used to define an "action summary period", which includes all the instrument striking actions of all users within the time range, and all the action data within the period are summarized and processed, and the corresponding action unit obtains the instrument striking historical action information of all users within the "action summary period", ensuring that all action data occurring within the time period can be collected and analyzed. Afterwards, these data are marked as "instrument striking corresponding action information" for comparison with the user's current action or generation of performance output. After determining the identity of the new user, a suitable time range can be accurately defined to capture the performance actions of all users, and the actions of the new user are processed based on the historical actions of all users within this period, avoiding the situation where the new user has no historical actions and improving the recognition accuracy of the new user's actions.

[0098] In a specific embodiment, the tapping action module includes a deviation feature unit, a corresponding feature unit, and an action similarity unit;

[0099] A deviation feature unit, used for obtaining a plurality of upper limb joint feature vectors corresponding to the deviation action information of the musical instrument striking, and marking them as upper limb joint deviation feature vectors;

[0100] A corresponding feature unit is used to obtain a plurality of upper limb joint feature vectors corresponding to each musical instrument striking corresponding action information, and mark them as upper limb joint corresponding feature vectors;

[0101] The action similarity unit is used to obtain the striking action similarity between the musical instrument striking deviation action information and the striking action corresponding action information of each musical instrument according to multiple upper limb joint deviation feature vectors and multiple upper limb joint corresponding feature vectors.

[0102] As mentioned above, the deviation feature unit first obtains the "instrument striking deviation action information" that occurs during the user's performance, which includes the action data of multiple upper limb joints. These data will be extracted and converted into "upper limb joint deviation feature vectors", that is, by capturing the spatial position, motion trajectory and other feature information of each joint of the user's upper limbs (such as shoulder joints, elbow joints, wrists, etc.) when performing actions, thereby forming a vectorized feature that describes the striking deviation. The corresponding feature unit obtains the "upper limb joint corresponding feature vector" related to each instrument striking action from the historical database. These corresponding feature vectors represent the historical actions of the user's past performances. The action similarity unit is responsible for comparing the difference between the "upper limb joint deviation feature vector" and the "upper limb joint corresponding feature vector". By calculating the similarity of multiple joint feature vectors, the similarity between the user's actual action (deviation action) and the standard action or historical action can be determined. The calculation formula for the striking action similarity is: , where X represents the similarity of the tapping action, i represents the number of the upper limb joint deviation feature vectors and the number of the corresponding feature vectors of the upper limb joints, i=1,2,3…n, Represented as the ith upper limb joint deviation feature vector, It is represented as the feature vector corresponding to the i-th upper limb joint, which can accurately capture the subtle deviations of the performer when performing the striking action. Whether it is the movement amplitude, joint angle changes, or slight differences in movement speed, they can all be described in a vectorized way, which helps to deeply analyze the deviations in the action.

[0103] In a specific embodiment, the tapping action module further includes a similarity threshold unit, a similarity judgment unit, a similarity table unit and a target action unit;

[0104] A similarity threshold unit, used to obtain a similarity threshold of a tapping action;

[0105] A similarity judgment unit, used to judge whether the tapping action similarity exceeds a tapping action similarity threshold;

[0106] If the tapping action similarity exceeds the tapping action similarity threshold, the tapping action similarity exceeding the tapping action similarity threshold is marked as the first tapping action similarity;

[0107] A similarity table unit, used for sorting the first similarities of the plurality of tapping actions in descending order to obtain a tapping action similarity table;

[0108] The target action unit is used to obtain the instrument striking action information corresponding to the maximum striking action similarity from the striking action similarity table, and mark it as the target striking action information.

[0109] As mentioned above, the similarity threshold unit first obtains the preset "tapping action similarity threshold", which defines the minimum standard for determining the similarity between two tapping actions. Only when the similarity of the tapping action exceeds this threshold, the current tapping action is considered to have sufficient similarity with the target action, and further processed. The similarity judgment unit receives the tapping action similarity calculated by the action similarity unit and compares it with the preset similarity threshold. If the similarity score exceeds the threshold, the tapping action similarity is marked as "tapping action first similarity" and enters the next step of processing. If the similarity does not exceed the threshold, the action may be ignored or regarded as an invalid action. The similarity form The unit sorts all the "first similarities of tapping actions" in order from large to small, and generates a "tapping action similarity table", which records all action similarities that exceed the threshold. The target action unit selects the action with the highest similarity from the "tapping action similarity table" and marks it as "target tapping action information". This target action represents the standard or historical action that is closest to the tapping action currently performed by the user, ensuring that only tapping actions with high similarity to the standard or historical actions are processed, and those actions with lower similarity are filtered out, thereby improving the accuracy of action recognition, reducing misjudgment, and increasing the tolerance of action recognition, making the performance more tolerant and flexible.

[0110] In a specific embodiment, the timbre output module includes a striking state unit, a state value unit, an action table unit, a sub-target table unit, and a timbre output unit;

[0111] A tapping state unit, used to obtain hand tapping state information in the target tapping action information, wherein the hand tapping state information includes hand tapping speed, hand tapping strength and hand tapping angle;

[0112] A state value unit, used to obtain a tapping state value according to a hand tapping speed, a hand tapping force and a hand tapping angle;

[0113] An action table unit, used for obtaining a timbre action table, wherein the timbre action table includes a plurality of striking action sub-tables and each striking action sub-table includes a plurality of playing timbres corresponding to striking state values;

[0114] A sub-target table unit, used for obtaining a corresponding knocking action sub-table from the timbre action table according to the target knocking action information, and marking it as a knocking action sub-target table;

[0115] The timbre output unit is used to obtain the corresponding performance timbre from the striking action sub-target table according to the striking state value, and output the performance timbre.

[0116] As mentioned above, the tapping state unit first obtains the state information of the user's hand tapping from the target tapping action information, mainly including the hand tapping speed, tapping force and tapping angle. This information represents the dynamic characteristics of the user's tapping and is used to determine the specific nature of the tapping action. The state value unit calculates a "tapping state value" based on the acquired hand tapping speed, tapping force and tapping angle. This state value integrates the hand tapping parameters and provides a comprehensive action feature value to facilitate subsequent timbre matching. The calculation formula of the tapping state value is: , where Q represents the tapping state value, S represents the hand tapping speed, J represents the hand tapping angle, and L represents the hand tapping strength. The action table unit stores a preset "tone action table", which includes different types of tapping action sub-tables, each of which corresponds to multiple tapping state values ​​and their corresponding performance timbres. The timbre action table is equivalent to a library, covering the matching rules of different tapping actions and timbres. The sub-target table unit selects and obtains the tapping action sub-table that best matches the current target tapping action from the timbre action table according to the target tapping action information, marked as "tapping action sub-target table". This sub-table contains the specific correspondence between the tapping state value and the timbre. The timbre output unit obtains the sub-table from the timbre action table according to the calculated tapping state value. The corresponding performance timbre is matched to the "tapping action sub-target table" and output. The final user's tapping action will be presented with the corresponding timbre to achieve real-time music performance effects. By obtaining multi-dimensional status information such as hand tapping speed, strength and angle, and comprehensively calculating the tapping status value, it can accurately reflect the subtle differences in the user's tapping actions, which makes the timbre output more precise. Users can obtain different timbre effects by fine-tuning the tapping method, enhancing the expressiveness of the performance. According to the real-time acquired tapping status value, the sub-table and corresponding timbre in the timbre action table are dynamically selected to ensure that each tapping action of the user can be quickly matched to the most suitable timbre and provide instant feedback, which provides users with a smooth playing experience without waiting or delay.

[0117] In a specific embodiment, it also includes a virtual action unit, a sound virtual unit and a virtual reality unit;

[0118] A virtual action unit, used for obtaining a corresponding tapping virtual action according to target tapping action information;

[0119] A sound virtualization unit, used to obtain a corresponding instrument sound virtual effect according to the performance timbre;

[0120] The virtual reality unit is used to obtain the corresponding virtual reality performance scene according to the virtual action of striking and the virtual effect of the sound of the musical instrument, and feed it back to the performing user.

[0121] As described above, the virtual action unit generates corresponding virtual tapping actions based on the target tapping action information. These virtual actions are simulations of the user's real actions and can be performed in a virtual environment. Through this process, each actual tapping action of the user can be converted into a corresponding virtual tapping action, thereby showing a more vivid performance effect in the virtual environment. The sound virtual unit generates a corresponding instrument sound virtual effect based on the user's performance timbre. This effect simulates the timbre characteristics of a real instrument, so that the sound emitted in the virtual environment is consistent with the sound of the actual instrument, maintaining the realism of the instrument performance. The virtual reality unit combines virtual actions and instrument sound virtual effects to create a corresponding virtual reality performance scene. This scene can not only show the user's virtual actions, but also create an immersive performance experience for the user through real-time feedback of sound effects. Finally, the constructed virtual reality performance scene will be fed back to the performing user, who can experience the virtual reality scene by wearing a VR helmet, so that they can see and hear the effect of their own performance in the virtual environment. This feedback mechanism ensures that users can feel the results of their performance in real time, thereby enhancing their sense of participation and immersion.

[0122] And, an intelligent air instrument terminal based on motion capture and virtual reality technology, comprising:

[0123] one or more processors;

[0124] a storage device having one or more programs stored thereon;

[0125] When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent air instrument based on motion capture and virtual reality technology.

[0126] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. An intelligent air instrument based on motion capture and virtual reality technology, characterized in that: It includes a knocking action module, a deviation judgment module, a history action module, a knocking action module and a tone output module; A striking action module, used to obtain body motion data of a target user, and obtain multiple musical instrument striking action information from the body motion data; The deviation judgment module is used to judge whether the musical instrument striking action information meets the first preset condition. If not, it is judged that the user's striking deviation is present and marked as the musical instrument striking deviation action information. A historical action module is used to obtain playing user information, obtain corresponding multiple musical instrument striking historical action information according to the playing user information, and mark it as musical instrument striking corresponding action information; A striking action module, used to obtain the striking action similarity between the striking deviation action information of the musical instrument and the striking corresponding action information of each musical instrument, obtain the striking corresponding action information of the musical instrument corresponding to the striking action similarity that meets the second preset condition, and mark it as the target striking action information; A timbre output module is used to obtain the hand striking state information in the target striking action information, and obtain the corresponding performance timbre according to the target striking action information and the hand striking state information; The deviation judgment module includes a standard action unit and an action judgment unit; Standard action unit, used to obtain standard action information of musical instrument percussion; An action judgment unit, used to judge whether the musical instrument striking action information is the same as the musical instrument striking standard action information; If the musical instrument striking action information is different from the musical instrument striking standard action information, the user's striking deviation is determined and marked as the musical instrument striking deviation action information; If the musical instrument striking action information is the same as the musical instrument striking standard action information, the user's striking standard is determined, and the corresponding performance timbre is output according to the same musical instrument striking standard action information; The historical action module includes a user information unit, a biometric information unit, an information collection unit, and an information identification unit; A user information unit, used to obtain information about the performing user; A biometric information unit, used to obtain biometric information from the performing user information, wherein the biometric information includes fingerprint information, portrait information, iris information and voice information; An information collection unit, used to obtain multiple biometric information in a local database and aggregate them into a biometric collection; An information identification unit, used to identify whether the biometric information exists in the biometric identification set; If the biometric information exists in the database, the performing user is determined to be an old user; If the biological information does not exist in the database, the performing user is determined to be a new user; The historical action module also includes an end time unit, a standard duration unit, a start time unit, an action summary unit, and a corresponding action unit; An end time unit is used to obtain the time of the determination result for the new user according to the determination result and mark it as the end time; Standard duration unit, used to obtain standard duration; The start time unit is used to obtain the start time based on the standard duration and the end time; An action summary unit, used to obtain an action summary period according to a start time and an end time; The corresponding action unit is used to obtain the historical action information of multiple musical instruments of all users within the action summary period, and mark it as the corresponding action information of musical instrument percussion; The tapping action module includes a deviation feature unit, a corresponding feature unit and an action similarity unit; A deviation feature unit, used for obtaining a plurality of upper limb joint feature vectors corresponding to the deviation action information of the musical instrument striking, and marking them as upper limb joint deviation feature vectors; A corresponding feature unit is used to obtain a plurality of upper limb joint feature vectors corresponding to each musical instrument striking corresponding action information, and mark them as upper limb joint corresponding feature vectors; The action similarity unit is used to obtain the similarity of the musical instrument percussion deviation action information and the corresponding action information of each musical instrument percussion according to the multiple upper limb joint deviation feature vectors and the multiple upper limb joint corresponding feature vectors. The calculation formula of the percussion action similarity is: , where X represents the similarity of the tapping action, i represents the number of the upper limb joint deviation feature vectors and the number of the corresponding feature vectors of the upper limb joints, i=1,2,3…n, Represented as the ith upper limb joint deviation feature vector, It is represented as the feature vector corresponding to the i-th upper limb joint; The timbre output module includes a striking state unit, a state value unit, an action table unit, a sub-target table unit and a timbre output unit; A tapping state unit, used to obtain hand tapping state information in the target tapping action information, wherein the hand tapping state information includes hand tapping speed, hand tapping strength and hand tapping angle; The state value unit is used to obtain the tapping state value according to the hand tapping speed, hand tapping strength and hand tapping angle. The calculation formula of the tapping state value is: , where Q represents the tapping state value, S represents the tapping speed of the hand, J represents the tapping angle of the hand, and L represents the tapping strength of the hand; An action table unit, used for obtaining a timbre action table, wherein the timbre action table includes a plurality of striking action sub-tables and each striking action sub-table includes a plurality of playing timbres corresponding to striking state values; A sub-target table unit, used for obtaining a corresponding knocking action sub-table from the timbre action table according to the target knocking action information, and marking it as a knocking action sub-target table; The timbre output unit is used to obtain the corresponding performance timbre from the striking action sub-target table according to the striking state value, and output the performance timbre.

2. The intelligent air instrument based on motion capture and virtual reality technology according to claim 1 is characterized in that: The historical action module also includes known user units and action units; A known user unit is used to obtain, for the old user according to the determination result, a plurality of musical instrument striking history action information corresponding to the old user in a local database; The action unit is used to mark each musical instrument striking historical action information as musical instrument striking corresponding action information.

3. The intelligent air instrument based on motion capture and virtual reality technology according to claim 1 is characterized in that: The tapping action module also includes a similarity threshold unit, a similarity judgment unit, a similarity table unit and a target action unit; A similarity threshold unit, used to obtain a similarity threshold of a tapping action; A similarity judgment unit, used to judge whether the tapping action similarity exceeds a tapping action similarity threshold; If the tapping action similarity exceeds the tapping action similarity threshold, the tapping action similarity exceeding the tapping action similarity threshold is marked as the first tapping action similarity; A similarity table unit, used for sorting the first similarities of the plurality of tapping actions in descending order to obtain a tapping action similarity table; The target action unit is used to obtain the instrument striking action information corresponding to the maximum striking action similarity from the striking action similarity table, and mark it as the target striking action information.

4. The intelligent air instrument based on motion capture and virtual reality technology according to claim 1 is characterized in that: It also includes virtual action units, vocalization virtual units, and virtual reality units; A virtual action unit, used for obtaining a corresponding tapping virtual action according to target tapping action information; A sound virtualization unit, used to obtain a corresponding instrument sound virtual effect according to the performance timbre; The virtual reality unit is used to obtain the corresponding virtual reality performance scene according to the virtual action of striking and the virtual effect of the sound of the musical instrument, and feed it back to the performing user.

5. An intelligent air instrument terminal based on motion capture and virtual reality technology, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When one or more programs are executed by one or more processors, the one or more processors can implement the intelligent air instrument based on motion capture and virtual reality technology as described in any one of claims 1 to 4.

Images