A method, data structure and performance system for generating multi-dimensional performance control parameters based on affective semantics
By generating multidimensional performance control parameters through an emotion semantic encoder and instrument response feature vectors, the problem that existing automatic performance systems cannot adapt to the physical characteristics of instruments and emotional expression is solved, and high-quality, personalized automatic performance effects are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 刘晋恺
- Filing Date
- 2026-04-22
- Publication Date
- 2026-06-05
AI Technical Summary
Existing automatic playing systems lack the ability to describe the dynamic envelope curve of keystroke force, the smoothness of transitions between notes, and emotional expression. They are unable to adaptively adjust to the physical characteristics of different acoustic instruments, resulting in poor performance quality and consistency.
By introducing an emotion semantic encoder and instrument response feature vectors, multi-dimensional performance control parameters are generated through deep neural networks, including velocity envelope, note connection smoothness, and rhythmic stretching offset. Combined with a personalized physical response model for the instrument, performance instructions adapted to the specific instrument are generated.
It achieves end-to-end personalized mapping from emotional descriptions to specific instrument physical actions, improving the naturalness and consistency of automatic performance and simplifying the user interaction process.
Smart Images
Figure CN122157622A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent musical instrument control and music information processing technology, specifically to a method, data format, storage medium, and method and system for driving the actuators of acoustic musical instruments to perform, based on emotional semantics to generate multi-dimensional performance control parameters. Background Technology
[0002] Existing automated playing systems typically rely on the MIDI protocol to control acoustic instruments. The MIDI protocol is essentially a set of discrete event instructions, containing only "note on / off" states and a single "keystroke velocity" value. It lacks the ability to describe the following continuous physical control dimensions: the dynamic envelope curve of keystroke velocity, the smoothness of connection between adjacent notes, and the rhythmic flexibility based on emotional expression. Therefore, MIDI-based automated playing often exhibits mechanical characteristics, lacking the natural expressiveness of human playing.
[0003] Furthermore, there are significant differences in the mechanical and physical characteristics between different acoustic instruments, and even between different pianos of the same model. For example, the static damping of the keys, the relationship between hammer travel and strike point response, and the nonlinear mapping function of the sustain pedal are all different between pianos. Existing automatic playing systems typically ignore the physical differences between individual instruments when generating control commands, resulting in the same set of playing data producing drastically different auditory effects on different instruments, which seriously restricts the quality and consistency of automatic playing.
[0004] In terms of emotional expression, existing software tools usually require users to manually adjust parameters such as the dynamics, duration, and pedal depth of each note one by one. This operation is cumbersome and highly dependent on user experience. It cannot directly and automatically convert natural language emotional descriptions such as "sadness" and "excitement" into continuous control parameters that are adapted to the physical characteristics of specific instruments.
[0005] Therefore, there is an urgent need for a systematic technical solution that can directly map emotional semantics into multi-dimensional continuous performance parameters and can adaptively generate parameters based on the physical characteristics of a specific target instrument. Summary of the Invention
[0006] The purpose of this invention is to provide a method, data structure, and performance system for generating multi-dimensional performance control parameters based on emotional semantics, in order to solve the problems in the prior art where emotional expression relies on manual adjustment, control parameters are of a single dimension, and personalized adaptation cannot be performed for specific instruments.
[0007] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a method for generating multidimensional performance control parameters based on emotional semantics is provided, including the following steps: Obtain the sequence of notes to be played and the emotional description information input by the user; The emotional description information is mapped into an emotional feature vector through an emotional semantic encoder; Obtain the instrument response feature vector of the target instrument; the instrument response feature vector is used to characterize the physical response properties of the target instrument, and is constructed based on at least one or more of the key depth-force response characteristics, pedal depth-sustain effect mapping characteristics, and key bed mechanical damping characteristics of the target instrument. The emotional feature vector, the context feature vector of the note sequence, and the instrument response feature vector are fused and input into the performance parameter decoder; The performance parameter decoder generates multi-dimensional performance control parameters for each note based on the fused features. The multi-dimensional performance control parameters include at least velocity envelope parameters, note connection smoothness parameters, and rhythm stretching offset parameters, and the velocity envelope parameters include attack time and decay rate.
[0008] The core logic of this method is to use the personalized physical response model of the target instrument as an implicit constraint in the parameter generation process, so that the final generated playing instructions can accurately drive the specific instrument to produce a physical timbre that matches the emotional description.
[0009] Secondly, a data structure for multi-dimensional performance data is provided for controlling the actuator of an intelligent musical instrument to drive the acoustic instrument to produce physical performance actions. After being read and parsed by the processing unit, the data structure directly controls at least one physical quantity among the motor torque, speed, or rotation angle of the actuator. The data structure includes: Note sequence data, including the pitch identifier, start timestamp, and duration of each note; Velocity control data, associated with each note, includes velocity peak and velocity envelope parameters, the velocity envelope parameters including attack time and decay rate; Connect smoothness data to define the overlap duration or staccato interval between adjacent notes; Rhythm stretch data is used to define the time offset of the start time of each note relative to the reference beat; Pedal control data, including an envelope sequence of pedal depth changes over time.
[0010] Thirdly, a method for driving an acoustic instrument actuator to play is provided, including: Obtain the note sequence and emotional description information to be played, and use the method described in the first aspect above to generate multi-dimensional performance control parameters for each note; The multi-dimensional performance control parameters are encapsulated into the data structure described in the second aspect above; The encapsulated data packet is sent to the adaptive control device, which converts the data packet into a sequence of physical execution instructions adapted to the mechanical characteristics of the target instrument based on a pre-built instrument response feature vector. The physical execution instruction sequence is output to the intelligent musical instrument actuator, which drives the intelligent musical instrument actuator to perform physical performance actions on the target musical instrument that correspond to the emotional description information.
[0011] Compared with existing technologies, the core innovation of this invention lies in the fact that it introduces "instrument response feature vectors" as an independent input branch into the emotion mapping model for the first time, realizing end-to-end personalized mapping from "semantic emotion" to "specific instrument physical action parameters". Users only need to input natural language emotion descriptions on the front end, and the back end automatically completes complex physical modeling and parameter adaptation calculations, balancing the simplicity of interaction with the professional accuracy of performance results.
[0012] Compared with existing technologies, this invention introduces the instrument response feature vector as a constraint condition for parameter generation, enabling the same emotional description to produce a highly consistent auditory emotional expression effect on instruments with different physical characteristics, effectively overcoming the influence of individual instrument differences on the expressiveness of automatic performance. Attached Figure Description
[0013] Figure 1 This is a flowchart of the method for generating multi-dimensional performance control parameters based on emotional semantics in an embodiment of the present invention.
[0014] Figure 2 This is a schematic diagram of the multidimensional performance data structure in an embodiment of the present invention.
[0015] Figure 3 This is a schematic diagram of the neural network structure of the emotion mapping model in an embodiment of the present invention.
[0016] Figure 4 This is a system architecture diagram for driving the acoustic instrument actuator to play in an embodiment of the present invention. Detailed Implementation
[0017] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described herein are only for explaining the present invention and are not intended to limit the scope of protection of the present invention.
[0018] Example 1: An Emotion Mapping Method Integrating Personalized Physical Models of Musical Instruments like Figure 1 As shown, this embodiment provides a method for generating multi-dimensional performance control parameters based on emotional semantics, including the following steps:
[0019] S101: Obtain the note sequence to be played and the emotional description information input by the user. The note sequence can be derived from a MusicXML file, a MIDI file, or real-time recorded score data; the emotional description information is natural language text input by the user, such as "sad and slow" or "exhilarating and powerful".
[0020] S102: The sentiment description information is mapped into sentiment feature vectors through a sentiment semantic encoder. In one implementation, the sentiment semantic encoder can use a text encoding network based on a pre-trained language model to extract the semantic embedding vector of the sentiment description text. This vector encodes the activation, valence, and dynamic trend information implicit in the sentiment description.
[0021] S103: Obtain the instrument response feature vector of the target instrument. This step is a key innovative aspect of this invention.
[0022] The instrument response feature vector is constructed as follows: A standardized automatic calibration process is performed before the target instrument leaves the factory or upon its first use. Specifically, the intelligent instrument actuator triggers the pitch triggering mechanism of the target instrument using a preset standard drive sequence, simultaneously collecting sensor data reflecting the instrument's mechanical characteristics and acoustic response. The data is fitted using a parameter identification algorithm to extract feature parameters characterizing the instrument's unique physical properties, including but not limited to: dynamic response curve parameters reflecting the relationship between key travel and sound intensity, pedal response function parameters reflecting the mapping relationship between pedal travel and sustain effect, and damping coefficients reflecting the energy dissipation characteristics during key bed reset. These parameters are normalized and encoded into a fixed-dimensional vector, forming the instrument response feature vector of the target instrument, and stored in the non-volatile memory of the adaptive control device. This calibration process is transparent to the user and is automatically completed only during system initialization, requiring no user intervention.
[0023] S104: Fusing the emotion feature vector, the context feature vector of the note sequence, and the instrument response feature vector. The context feature vector may contain pitch information, duration information, interval relationship with adjacent notes, and beat position information for each note. The fusion method may employ vector concatenation or other feature fusion techniques.
[0024] S105: Input the fused features into the performance parameter decoder, which simultaneously generates multi-dimensional performance control parameters for each note.
[0025] like Figure 3 As shown, the emotion mapping model 300 includes an emotion semantic encoder 310, a context feature extractor 320, an instrument response feature encoder 330, and a performance parameter decoder 340.
[0026] The performance parameter decoder 340 is a regression model built on a deep neural network. Its input is the fused joint feature vector, and its output is multi-dimensional performance control parameters, including at least: - Velocity envelope parameters: including velocity peak, attack time (the time it takes for the velocity to rise from zero to the peak) and decay rate (the rate at which the velocity decays from the peak), together define the dynamic profile of the velocity of each note; - Note connection smoothness parameter: used to characterize the degree of overlap or staccato interval between adjacent notes, directly corresponding to the timing overlap control of multi-drive unit actions; - Rhythm Stretch Offset Parameter: Represents the offset of the actual trigger time of a note relative to the theoretical beat position.
[0027] Since the input to the decoder contains the instrument response feature vector, the model has learned the following mapping rules during the training phase: under the physical constraints of a specific instrument, how to adjust parameters such as attack time and connection smoothness to compensate for the auditory differences caused by mechanical characteristics, so that the physical timbre of the final output is consistent with the target emotional expression.
[0028] The training data for the performance parameter decoder can come from performance records of performers on various brands and models of acoustic instruments. During data acquisition, the actual dynamic range curves, the duration of connections between notes, the trigger moment offset, and the response characteristic data of the specific instrument used are recorded simultaneously. Through supervised learning, the model can autonomously learn the complex mapping relationship between "emotional intent—instrument physical constraints—final timbre effect".
[0029] Example 2: Decoupling of Minimalist User Interaction Design from Complex Backend Calculations This embodiment further illustrates the design of separating the user interaction process from the background calculation process.
[0030] The user operation process is as follows: 1. Import the sheet music file into the graphical interface of the performance data generation terminal; 2. Enter a natural language emotion description in the emotion input box, such as "sad and slow"; 3. Click to confirm and generate performance data.
[0031] After that, users do not need to make any additional parameter adjustments or technical settings.
[0032] The background automatic execution process is as follows: 1. The system automatically identifies the currently connected target musical instrument and retrieves the corresponding musical instrument response feature vector from the memory of the adaptive control device; 2. The sentiment semantic encoder parses the sentiment description words input by the user and generates sentiment feature vectors; 3. For each note in the note sequence, extract its context feature vector; 4. The performance parameter decoder generates multi-dimensional control parameters for each note based on fused feature vectors; 5. The system encapsulates the generated multidimensional parameters into a data structure conforming to a predefined format and sends it to the adaptive control device; 6. The adaptive control device converts the parameters in the data packet into a sequence of physical execution instructions adapted to the target instrument based on the physical parameters in the instrument's response feature vector; 7. The intelligent musical instrument actuator receives a sequence of physical execution instructions and drives the drive unit to perform physical performance on the target musical instrument.
[0033] Because the background calculations include adaptive compensation based on the specific physical characteristics of the instrument, the underlying motor control commands generated on different pianos for the same piece of music under the same emotional tag will be different, but the final auditory emotional expression will be highly consistent. Users can obtain a professional-grade automatic playing experience without needing to know the above complex physical modeling and compensation process.
[0034] Example 3: Specific Definition of Multidimensional Performance Data Structure like Figure 2 As shown, the multidimensional performance data structure 200 is a control signal encapsulation with specific physical semantics, which contains the following fields: - Note sequence data 210: Records the pitch identifier, start timestamp, and duration of each note; - Velocity control data 220: Associated with each note, including velocity peak, attack time, decay rate, and optional release time; - Connection Smoothness Data 230: Records the degree of connection between adjacent notes. The value range can be preset and is used to convert the overlap duration of motor actions or staccato intervals. - Rhythm Stretch Data 240: Records the offset of the start time of each note relative to the reference beat grid; - Pedal control data 250: Contains an envelope sequence of pedal depth changes over time.
[0035] This data structure can be serialized into JSON, Protocol Buffers, or a custom binary format and stored in a computer-readable storage medium. Essentially, this data structure is a control signal with specific physical meaning used to control mechanical actuators. When this data structure is read and parsed by the processing unit of the intelligent musical instrument actuator, its field values are directly converted into physical control quantities such as motor torque, speed, angle, or solenoid valve parameters.
[0036] Example 4: Parameter mapping across instrument types The technical solution of this invention is not only applicable to acoustic pianos, but can also be extended to other acoustic musical instruments. For different types of target instruments, the multi-dimensional performance control parameters have an adaptive physical mapping relationship: - When the target instrument is an acoustic piano, the velocity envelope parameter is mapped to the keystroke velocity curve and the hammer acceleration envelope; the connection smoothness parameter is mapped to the overlap time of adjacent key release and pressing actions; the pedal control data is mapped to the angle command of the sustain pedal motor.
[0037] - When the target instrument is a stringed instrument, the velocity envelope parameter is mapped to the envelope curve of bow speed and bow pressure; the onset time is mapped to the acceleration of the initial contact between the bow and string; the connection smoothness parameter is mapped to the smoothness of bow change; and the rhythmic stretching offset parameter is mapped to the delay of vibrato initiation or a slight variation in bow speed.
[0038] - When the target instrument is a wind instrument, the velocity envelope parameter is mapped to the air supply pressure envelope; the attack time is mapped to the speed of airflow establishment; the connection smoothness parameter is mapped to the softness and hardness of the articulation; and the rhythmic stretch offset parameter is mapped to the timing offset of the breath opening.
[0039] In all the cross-instrument applications mentioned above, the instrument response feature vectors are individually constructed for each specific target instrument, thereby ensuring the accuracy of parameter mapping and the consistency of performance.
[0040] The above description is merely a preferred embodiment of the present invention and does not limit the scope of the patent. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of patent protection of the present invention.
Claims
1. A method for generating multidimensional performance control parameters based on emotional semantics, characterized in that, Includes the following steps: Obtain the sequence of notes to be played and the emotional description information input by the user; The emotional description information is mapped into an emotional feature vector through an emotional semantic encoder; Obtain a context feature vector extracted based on the note sequence; the context feature vector is constructed based on at least one of the following: pitch information, duration information, interval relationship with adjacent notes, and beat position information of each note; Obtain the instrument response feature vector of the target instrument; The instrument response feature vector is used to characterize the physical response properties of the target instrument. Its construction method includes: triggering the pitch triggering mechanism of the target instrument with a preset standard driving sequence through a driving device, collecting sensor data and extracting feature parameters through a parameter identification algorithm, and encoding the feature parameters into a fixed-dimensional vector after normalization; the feature parameters are based on at least one or more of the following: key depth-force response characteristics, pedal depth-sustain effect mapping characteristics, and key bed mechanical damping characteristics. The emotion feature vector, the context feature vector, and the instrument response feature vector are fused and input into the performance parameter decoder; The performance parameter decoder generates multi-dimensional performance control parameters for each note based on the fused features. The multidimensional performance control parameters include at least velocity envelope parameters, note connection smoothness parameters, and rhythm stretching offset parameters, and the velocity envelope parameters include onset time and decay rate.
2. The method according to claim 1, characterized in that, The multidimensional performance control parameters also include the dynamic peak value; the note connection smoothness parameter is used to characterize the degree of overlap or staccato interval between adjacent notes; the rhythm stretch offset parameter is used to represent the time offset of the actual trigger time of the note relative to the reference beat position.
3. The method according to claim 1, characterized in that, The emotion semantic encoder uses a text encoding network based on a pre-trained language model; the performance parameter decoder is a regression model built on a deep neural network.
4. The method according to any one of claims 1 to 3, characterized in that, The method also includes a parameter mapping step across instrument types: When the target instrument is an acoustic piano, the force envelope parameter is mapped to the keystroke velocity curve and the hammer acceleration envelope, and the note connection smoothness parameter is mapped to the overlap time of adjacent key release and pressing actions. When the target instrument is a stringed instrument, the dynamic envelope parameter is mapped to the envelope curve of bow speed and bow pressure, the onset time is mapped to the acceleration of the initial contact between the bow and the string, and the note connection smoothness parameter is mapped to the bow transition smoothness. When the target instrument is a wind instrument, the dynamic envelope parameter is mapped to the air pressure envelope, the onset time is mapped to the speed of airflow establishment, and the note connection smoothness parameter is mapped to the softness or hardness of articulation.
5. A data structure for multidimensional performance data, characterized in that, After being read and parsed by the processing unit, the data structure is used to directly control at least one physical quantity among the torque, speed, or rotation angle of the motor of the intelligent musical instrument actuator. The data structure includes: Note sequence data, including the pitch identifier, start timestamp, and duration of each note; Velocity control data, associated with each note, includes velocity peak and velocity envelope parameters, the velocity envelope parameters including attack time and decay rate; Connect smoothness data to define the overlap duration or staccato interval between adjacent notes; Rhythm stretch data is used to define the time offset of the start time of each note relative to the reference beat; Pedal control data, including an envelope sequence of pedal depth changes over time.
6. The data structure according to claim 5, characterized in that, The data structure can be serialized into JSON, Protocol Buffers, or a custom binary format and stored in a computer-readable storage medium.
7. A method for driving the performance of an acoustic instrument actuator based on emotional semantics, characterized in that, include: Obtain the sequence of notes to be played and the emotional description information input by the user; The method described in any one of claims 1 to 4 is used to generate multidimensional performance control parameters for each note; The multi-dimensional performance control parameters are encapsulated into the data structure described in claim 5 or 6; The encapsulated data packet is sent to the adaptive control device, which converts the data packet into a sequence of physical execution instructions adapted to the mechanical characteristics of the target instrument based on a pre-built instrument response feature vector. The physical execution instruction sequence is output to the intelligent musical instrument actuator, which drives the intelligent musical instrument actuator to perform physical performance actions on the target musical instrument that correspond to the emotional description information.
8. A system for generating multi-dimensional performance control parameters based on emotional semantics, characterized in that, include: The data acquisition module is used to acquire the sequence of notes to be played and the emotional description information input by the user; An emotion semantic encoder is used to map the emotion description information into emotion feature vectors; The context extraction module is used to obtain a context feature vector extracted based on the note sequence; the context feature vector is constructed based on at least one of the following: pitch information, duration information, interval relationship with adjacent notes, and beat position information of each note; The instrument feature acquisition module is used to acquire the instrument response feature vector of the target instrument; The instrument response feature vector is used to characterize the physical response properties of the target instrument. Its construction method includes: triggering the pitch triggering mechanism of the target instrument with a preset standard driving sequence through a driving device, collecting sensor data and extracting feature parameters through a parameter identification algorithm, and encoding the feature parameters into a fixed-dimensional vector after normalization; the feature parameters are based on at least one or more of the following: key depth-force response characteristics, pedal depth-sustain effect mapping characteristics, and key bed mechanical damping characteristics. The feature fusion module is used to fuse the emotion feature vector, the context feature vector, and the instrument response feature vector; A performance parameter decoder is used to generate multi-dimensional performance control parameters for each note based on the fused features. The multi-dimensional performance control parameters include at least velocity envelope parameters, note connection smoothness parameters, and rhythm stretching offset parameters, and the velocity envelope parameters include attack time and decay rate.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4, or the steps of the method as described in claim 7.
10. A smart musical instrument playing device, characterized in that, include: Memory, used to store computer programs; A processor, configured to, when executing the computer program, implement the steps of the method as described in any one of claims 1 to 4, or implement the steps of the method as described in claim 7; The communication interface is used to send the generated multi-dimensional performance control parameters or physical execution command sequence to the external actuator.