Intelligent Reasoning Method for Virtual Musical Instrument Playing Control Based on Technique Family Intent Mapping

By constructing a knowledge graph of technique families and reasoning mechanisms, and combining the context of the music with the user's historical preferences, the problem of lack of context awareness in virtual musical instrument performance systems is solved, enabling more coherent and personalized performance control and improving the system's adaptability and expressiveness.

CN123069003APending Publication Date: 2026-11-10BEIJING SOUND PULSE TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611225279.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-13
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing virtual musical instrument performance systems lack the ability to actively perceive the temporal context and musical score structure information of the performance, resulting in a lack of consistency and structural expressiveness in the performance control parameters. The static mapping architecture also shields the differences in the performers' operating habits.

Method used

A knowledge graph of technique families is constructed. User performance action data is collected and preprocessed through motion capture equipment. Technique family classifiers and inference mechanisms are used to identify technique family categories. Intent mapping is performed by combining the context of the music and the user's historical preferences to generate multi-dimensional performance control parameters. Parameter mutations are suppressed through smoothing processing.

Benefits of technology

It has improved the continuity and expressiveness of the performance, adapted to different musical phrases and user habits, and enhanced the system's personalization and versatility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069003A_ABST
    Figure CN123069003A_ABST
Patent Text Reader

Abstract

This invention relates to the field of intelligent musical instrument control, specifically to an intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping. The method includes the following steps: S1: Constructing a technique family knowledge graph; S2: Collecting and preprocessing user performance action data; S3: Performing technique family intent reasoning; S4: Performing the mapping from technique families to performance intent; S5: Generating multi-dimensional performance control parameters; S6: Outputting performance control commands. This invention, by introducing a technique family knowledge graph and a three-level progressive reasoning mechanism—direct judgment, contextual temporal reasoning, and knowledge graph similarity inference—solves the deficiency of existing static mapping methods that lack context awareness. The system processes actions hierarchically based on confidence levels, combining historical recognition sequences and musical context to correct fuzzy actions, enabling the same performance action to adaptively map to intents and parameters consistent with musical logic under different musical phrases, significantly improving the coherence and expressiveness of the performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent musical instrument control, specifically to an intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping. Background Technology

[0002] With the rapid development of digital music technology and human-computer interaction technology, virtual instrument playing systems have been widely used in many scenarios such as music creation, remote music education, stage performance special effects, game audio interaction, and rehabilitation training. This system captures the performer's body movements and converts them into sound control signals for virtual instruments, allowing users to obtain a near-realistic playing experience without physical instruments. This is of great significance for lowering the threshold of music creation and enriching the forms of music expression.

[0003] Existing systems typically consist of motion capture devices such as data gloves, inertial sensors, or depth cameras, along with a matching mapping engine. Their operating principle is as follows: the sensors collect motion data such as hand position, angle, speed, and force in real time, and the software system directly converts the physical signals into MIDI or audio synthesis parameters according to preset deterministic rules. Some commercial tools use static key mapping tables to fix specific actions to preset playing techniques. The overall control logic relies on the static and real-time correspondence between sensor signals and output parameters.

[0004] The aforementioned mapping architecture has an inherent technical flaw—a lack of proactive perception of performance temporal context and musical score structure information. Existing systems are all single-point instantaneous mappings of "signal → parameter," where the same action input triggers the same control parameter output at any time and any musical phrase position. However, musical performance has objective temporal logic and syntactic structure. This contextual information directly determines the actual sound effect that the same underlying action should produce. Existing static mapping architectures completely shield such information at the physical layer, resulting in the output performance control parameters lacking musical phrase coherence and structural expressiveness. This flaw is caused by the "contextual blind spot" of the mapping logic itself and is unrelated to the performer's operating habits. Therefore, to address the above problem, a virtual musical instrument performance control intelligent reasoning method based on technique family intention mapping is proposed. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies and solve at least one of the technical problems mentioned in the background art, this invention proposes an intelligent reasoning method for virtual musical instrument performance control based on technique family intention mapping.

[0006] The technical solution adopted by this invention to solve its technical problem is: a virtual musical instrument playing control intelligent reasoning method based on technique family intent mapping, comprising the following steps: S1: Construct a knowledge graph of technique families, obtain data on the performance technique system of the target virtual musical instrument, cluster the performance techniques according to three dimensions: action mode, acoustic effect and performance function, form several technique families, and construct a knowledge graph of technique families based on the clustering results of the technique families; S2: Collect and preprocess user performance action data. Collect user performance action data in three-dimensional space in real time through motion capture device, and preprocess the performance action data. The preprocessing includes noise reduction filtering, posture normalization, action segmentation and feature extraction. S3: Perform technique family intent reasoning, input the preprocessed performance action data into the technique family classifier, output the technique family category to which the current performance action belongs and its confidence score, when the confidence score is lower than the first preset threshold, start the context reasoning mechanism for correction; when the corrected confidence score of the context reasoning mechanism is still lower than the second preset threshold, start the technique family speculation mechanism for correction. S4: Perform the mapping from technique family to performance intention. Based on the determined technique family identification result, retrieve the corresponding performance intention set from the technique family-intention mapping library. When the performance intention set contains multiple performance intentions, perform intention disambiguation processing to determine the current performance intention. S5: Generate multi-dimensional performance control parameters. Based on the determined performance intention, retrieve the corresponding multi-dimensional performance control parameter set from the intention-parameter mapping library, and perform parameter smoothing processing on the multi-dimensional performance control parameters. S6: Output performance control commands. Encapsulate the smoothed multi-dimensional performance control parameters according to the control protocol format of the target virtual instrument, generate performance control commands, and send them to the virtual instrument playback engine.

[0007] Preferably, in step S1, the technique family knowledge graph uses technique families as nodes, similarity relationships, derivation relationships, mutual exclusion relationships, and temporal relationships between technique families as edges, and the specific performance techniques and their attribute information contained in each technique family as node attributes.

[0008] Preferably, the technique family classifier is built based on a deep learning model and adopts a multi-task learning architecture. While the main task is technique family classification, auxiliary tasks are set as performance action reconstruction and performance dynamics regression.

[0009] Preferably, in step S3, the context reasoning mechanism includes the following processing: obtaining a sequence of technique family recognition results for N consecutive moments prior to the current performance moment; obtaining context information of the currently performed piece of music, wherein the context information of the piece of music includes at least the position information of the current note in the musical phrase structure; inputting the sequence of technique family recognition results and the context information of the piece of music into the temporal reasoning model, and outputting the technique family recognition result after context correction.

[0010] Preferably, in step S3, the technique family inference mechanism includes the following processing: extracting the motion feature vector of the current performance action; calculating the similarity between the motion feature vector and the typical motion feature vector corresponding to each technique family node in the technique family knowledge graph; and selecting the technique family with the highest similarity as the inference result.

[0011] Preferably, in step S4, the technique family-intention mapping library is constructed using the following method: for each technique family, determine one or more performance intentions that the technique family can achieve under different performance context conditions; establish a many-to-many mapping relationship between technique family identifiers and performance intention identifiers; and configure context condition parameters and priority weights for each mapping relationship.

[0012] Preferably, in step S4, the intent disambiguation processing includes the following steps: obtaining the context information of the currently played music, the context information including at least the position number of the current note in the musical phrase, the total length of the musical phrase, the interval relationship between the preceding and following notes, and the time value ratio; obtaining the user's historical performance style preference data; calculating the conditional probability of each candidate performance intent based on the music context information and the historical performance style preference data; and selecting the performance intent with the highest conditional probability as the current performance intent output.

[0013] Preferably, in step S5, the parameter smoothing process includes the following steps: obtaining the performance control parameter value generated at the previous moment; using the performance control parameter value at the previous moment as a benchmark, performing amplitude-limited smoothing on the performance control parameter value generated at the current moment to ensure that the parameter change between adjacent moments does not exceed a preset maximum change rate threshold.

[0014] Preferably, the following steps are also included: Performance evaluation: After step S6, the actual audio signal output by the virtual instrument is collected, and the performance effect of the audio signal is evaluated. The evaluation results are fed back to the technique family reasoning process in step S3 and the intention mapping process in step S4 to achieve closed-loop adaptive optimization.

[0015] Preferably, the following steps are also included: Multi-instrument switching: Obtain the target instrument type selected by the user, and load the corresponding technique family knowledge graph, technique family-intent mapping library and intent-parameter mapping library according to the target instrument type.

[0016] The advantages of this invention are: 1. This invention introduces a technique family knowledge graph and a three-level progressive reasoning mechanism (direct judgment, contextual temporal reasoning, and knowledge graph similarity inference). It can process in layers based on confidence level, and correct fuzzy actions by combining historical recognition sequences and musical context. This enables the same performance action to be adaptively mapped to intentions and parameters that conform to musical logic under different musical phrases, thereby overcoming the deficiency of existing static mapping that lacks context awareness and improving the coherence and expressiveness of the performance.

[0017] 2. This invention uses a hierarchical mapping architecture from technique families to performance intentions, and combines musical context information with users' historical performance style preferences for intention disambiguation. This achieves the conversion from low-level action features to high-level musical semantics, which can match the personalized performance habits of different users, reduce the risk of limited expression caused by a single mapping rule, and improve the system's personalized adaptability.

[0018] 3. This invention generates multi-dimensional performance control parameters through intent-parameter mapping and performs smoothing processing with the maximum rate of change threshold between adjacent time moments as a constraint. This can suppress parameter abrupt changes caused by motion jitter. After being encapsulated by a standard protocol, the output command drives the virtual instrument to produce sound, which has good versatility and portability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The following is in conjunction with the appendix Figure 1 This application will be described in further detail. Example 1: Virtual Violin Playing Control This embodiment uses virtual violin playing control as an example to describe in detail the specific implementation methods of each step.

[0023] Implementation of S1: Obtain data on violin playing techniques, which include: legato, detaché, spiccato, staccato, vibrato, glissando, harmonics, double stops, etc.

[0024] Technique families are formed by clustering techniques based on three dimensions: movement pattern, acoustic effect, and performance function. For example: Bow Techniques: Includes techniques related to right-hand bowing, such as continuous bowing, split bowing, spiccato bowing, and plosive bowing. Elegance techniques: include techniques related to left-hand embellishment such as vibrato and glissando; Harmony family: includes techniques related to multi-note performance such as double stops, chords, and harmonics.

[0025] Construct a knowledge graph of technique families, with bowing technique family, embellishment family, and harmony family as nodes, and establish the relationships between the nodes (such as bowing technique family and embellishment family can occur simultaneously, bowing technique family and harmony family can occur simultaneously, etc.). The specific techniques contained in each technique family and their attributes (such as motion feature vector, acoustic parameter range, etc.) are used as node attributes.

[0026] Implementation of S2: The data glove (with built-in inertial and bending sensors) collects the performer's hand movement data in real time. The data glove collects the bending angle of the left hand fingers (for identifying fingering and vibrato), the rotation angle and movement trajectory of the right wrist (for identifying bowing movements), and the spatial coordinates of both hands.

[0027] The collected raw data is subjected to noise reduction filtering (e.g., using Kalman filtering or low-pass filtering), posture normalization (converting hand posture to a standard reference coordinate system), motion segmentation (detecting the start and end points of motion based on motion speed thresholds), and feature extraction (extracting statistical features within each time window, such as mean, variance, peak value, slope, etc.).

[0028] Implementation of S3: The extracted action feature vectors are input into a pre-trained technique family classifier. This classifier uses a three-layer fully connected neural network structure. The input layer receives the action feature vectors, the hidden layer uses the ReLU activation function, and the output layer uses the Softmax function to output the probability distribution of each technique family.

[0029] For example, assuming that the feature vector of the current playing action is input into the classifier and the output results are: 0.85 confidence for the bowing family, 0.10 confidence for the embellishment family, and 0.05 confidence for the harmony family, and the highest confidence of 0.85 is greater than the first threshold (for example, it can be set to 0.70), then the current technique family is directly determined to be the bowing family.

[0030] If the current performance action is relatively ambiguous, the output results are: bowing family confidence 0.55, embellishment family confidence 0.30, and harmony family confidence 0.15. If the highest confidence of 0.55 is less than the first threshold (e.g., 0.70) but greater than the second threshold (e.g., 0.40), the contextual reasoning mechanism is activated. This mechanism obtains the technique family identification sequence (bowing family, bowing family, bowing family, bowing family, bowing family) from the previous 5 moments and the current musical context (the current note is a quarter note, the dynamic marking is f, the tempo marking is Allegro, and the note is in the crescendo rising section of the phrase). The temporal reasoning model corrects the identification result to bowing family based on this information (the confidence is increased to 0.72 after correction).

[0031] If the confidence score after contextual reasoning is still lower than the second threshold (e.g., lower than 0.40), the technique family inference mechanism is activated. By calculating the similarity between the current action feature vector and the typical feature vectors of each technique family in the knowledge graph, the technique family with the highest similarity is selected as the inference result.

[0032] Implementation of S4: Based on a defined technique family (bow technique family), the corresponding set of performance intentions is retrieved from the technique family-intention mapping library. The performance intentions corresponding to the bow technique family under different contextual conditions include: continuous intention (corresponding to legato bowing), articulated intention (corresponding to separate bowing), flexible intention (corresponding to spiccato bowing), and short intention (corresponding to staccato bowing), etc.

[0033] Combining the current musical context (note duration is a quarter note, dynamic is f, tempo is Allegro, and the note is in the crescendo rising section of the phrase) and the user's historical preferences (the user previously preferred to use legato bowing under similar conditions), intention disambiguation is performed, and the conditional probability of each candidate intention is calculated. For example, the coherent intention has the highest probability (0.65), and is then determined as the current performance intention.

[0034] Implementation of S5: Based on the defined performance intention (coherence intention), the corresponding multi-dimensional performance control parameter set is retrieved from the intention-parameter mapping library. The parameters corresponding to the coherence intention include: timbre control parameters (the filter cutoff frequency is set to a low value to produce a warm timbre), dynamic control parameters (the onset time is set to a long value to achieve a smooth onset), and time value control parameters (the overlap coefficient between notes is set to a positive value to achieve a legato effect), etc.

[0035] The generated parameters are smoothed to ensure that the change between the parameter value and the previous time step does not exceed a preset threshold (such as the maximum rate of change threshold) to avoid parameter abrupt changes.

[0036] Implementation of S6: The generated multi-dimensional performance control parameters are encapsulated into MIDI control change messages and sent to a virtual violin sound source (such as the SWAM violin engine) to drive it to produce sound.

[0037] Example 2: Virtual Piano Performance Control This embodiment uses virtual piano performance control as an example to further illustrate the technical solution of the present invention.

[0038] Piano playing techniques include: staccato, non-legato, legato, staccato, arpeggio, trill, glissando, etc.

[0039] Techniques are grouped into families based on their movement patterns, acoustic effects, and performance functions. For example, the touch family (which includes staccato, non-legato, legato, staccato, and other techniques related to finger touch) and the ornamentation family (which includes vibrato, arpeggio, glissando, and other techniques related to note ornamentation).

[0040] Playing motion data is collected using key sensors on keyboard instruments (which detect key speed, force, and release speed) and hand motion capture devices.

[0041] The technique family classifier identifies the key-pressing method based on features such as key speed, force, and release speed. When the identification result is a key-pressing family, it combines the context of the music (such as the position in the musical phrase, the time value relationship between the preceding and following notes) to perform intention disambiguation, determine the specific performance intention (such as whether a smooth legato effect or a crisp staccato effect is needed), and then generate the corresponding timbre and dynamic control parameters.

[0042] The remaining steps are similar to those in Example 1, and will not be repeated here.

[0043] Example 3: System Architecture This embodiment describes the system architecture of the method of the present invention, such as... Figure 1 As shown, the system includes the following functional modules, and the connection relationships and data flow between the modules are as follows: Motion capture module: Used to execute S2, including motion capture equipment and data preprocessing unit. This module outputs the preprocessed motion data to the technique family inference engine.

[0044] Technique Family Knowledge Graph Module: Used to store and manage the technique family knowledge graph built by S1, providing techniques family query, relationship reasoning and similarity calculation functions. This module provides data support for the technique family reasoning engine and intent mapping module.

[0045] Technique family inference engine: Used to execute S3, including technique family classifier, context inference unit and technique family inference unit. The engine receives the feature vector output by the action acquisition module, combines it with the knowledge graph module and music context information, and outputs the technique family recognition result to the intent mapping module.

[0046] Intent Mapping Module: Used to execute S4, including a technique family-intent mapping library and an intent disambiguation unit. This module receives the technique family identification results, music context, and user preferences, and outputs the determined performance intent to the parameter generation module.

[0047] Parameter generation module: Used to execute S5, including an intent-parameter mapping library and a parameter smoothing processing unit. This module receives the performance intent and generates smoothed multi-dimensional control parameters for the instruction output module.

[0048] Command output module: Used to execute S6, including protocol encapsulation unit and command sending unit, which encapsulates parameters into standard protocol commands and sends them to the virtual musical instrument playback engine.

[0049] Feedback Optimization Module: Used to perform performance effect evaluation and closed-loop adaptive optimization. This module collects the audio signal output by the virtual instrument, evaluates it, and then sends the feedback information back to the technique family inference engine and intent mapping module to adjust the inference parameters.

[0050] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for intelligent reasoning in virtual musical instrument performance control based on technique family intent mapping, characterized in that: Includes the following steps: S1: Construct a knowledge graph of technique families, obtain data on the performance technique system of the target virtual musical instrument, cluster the performance techniques according to three dimensions: action mode, acoustic effect and performance function, form several technique families, and construct a knowledge graph of technique families based on the clustering results of the technique families; S2: Collect and preprocess user performance action data. Collect user performance action data in three-dimensional space in real time through motion capture device, and preprocess the performance action data. The preprocessing includes noise reduction filtering, posture normalization, action segmentation and feature extraction. S3: Perform technique family intent reasoning, input the preprocessed performance action data into the technique family classifier, output the technique family category to which the current performance action belongs and its confidence score, when the confidence score is lower than the first preset threshold, start the context reasoning mechanism for correction; when the corrected confidence score of the context reasoning mechanism is still lower than the second preset threshold, start the technique family speculation mechanism for correction. S4: Perform the mapping from technique family to performance intention. Based on the determined technique family identification result, retrieve the corresponding performance intention set from the technique family-intention mapping library. When the performance intention set contains multiple performance intentions, perform intention disambiguation processing to determine the current performance intention. S5: Generate multi-dimensional performance control parameters. Based on the determined performance intention, retrieve the corresponding multi-dimensional performance control parameter set from the intention-parameter mapping library, and perform parameter smoothing processing on the multi-dimensional performance control parameters. S6: Output performance control commands. Encapsulate the smoothed multi-dimensional performance control parameters according to the control protocol format of the target virtual instrument, generate performance control commands, and send them to the virtual instrument playback engine.

2. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: In S1, the technique family knowledge graph uses technique families as nodes, similarity relationships, derivation relationships, mutual exclusion relationships and temporal relationships between technique families as edges, and the specific performance techniques and their attribute information contained in each technique family as node attributes.

3. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: The technique family classifier is built based on a deep learning model and adopts a multi-task learning architecture. While the main task is technique family classification, auxiliary tasks are set up to reconstruct performance movements and regress performance dynamics.

4. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: In S3, the context reasoning mechanism includes the following processing: Obtain the sequence of technique family identification results for N consecutive moments preceding the current performance moment; Obtain the context information of the currently played music, which includes at least the position information of the current note in the musical phrase structure; The technique family identification result sequence and the music context information are input into the temporal reasoning model, and the technique family identification result after context correction is output.

5. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: In S3, the technique family prediction mechanism includes the following processing: Extract the motion feature vector of the current performance action; Calculate the similarity between the action feature vector and the typical action feature vector corresponding to each technique family node in the technique family knowledge graph; The technique family with the highest similarity was selected as the inference result.

6. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: In step S4, the technique family-intent mapping library is constructed using the following method: For each family of techniques, determine one or more performance intentions that the family of techniques can achieve under different performance contexts; Establish a many-to-many mapping relationship between technique family identifiers and performance intention identifiers; Configure context condition parameters and priority weights for each mapping relationship.

7. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: In step S4, the intent disambiguation process includes the following steps: Obtain the context information of the currently played music, which includes at least the position number of the current note in the musical phrase, the total length of the musical phrase, the interval relationship between the notes before and after it, and the time value ratio; Obtain users' historical performance style preference data; Based on the contextual information of the music and the historical performance style preference data, the conditional probability of each candidate performance intention is calculated; The performance intention with the highest conditional probability is selected as the current performance intention output.

8. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: In step S5, the parameter smoothing process includes the following steps: Retrieve the performance control parameter values ​​generated in the previous moment; Based on the performance control parameter value of the previous moment, the performance control parameter value generated at the current moment is subjected to amplitude limiting smoothing to ensure that the parameter change between adjacent moments does not exceed the preset maximum change rate threshold.

9. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: It also includes the following steps: Performance evaluation: After step S6, the actual audio signal output by the virtual instrument is collected, and the performance effect of the audio signal is evaluated. The evaluation results are fed back to the technique family reasoning process in step S3 and the intention mapping process in step S4 to achieve closed-loop adaptive optimization.

10. The intelligent reasoning method for virtual musical instrument performance control based on technique family intent mapping according to claim 1, characterized in that: It also includes the following steps: Multi-instrument switching: Obtain the target instrument type selected by the user, and load the corresponding technique family knowledge graph, technique family-intent mapping library and intent-parameter mapping library according to the target instrument type.