Electronic guzheng playing method and system based on multi-touch interaction

Abstract

The application relates to the technical field of electronic musical instruments, and particularly discloses an electronic guzheng playing method and system based on multi-touch interaction. Based on a multi-touch sensor array, multi-dimensional playing data of a current player during playing is collected and identified to obtain playing preferences; a timbre recognition model corresponding to the playing preferences is determined based on a playing parameter library, the multi-dimensional playing data is processed to obtain timbre parameters, playing audio is generated, and the playing audio is output based on an audio output device. The application can identify the playing preferences of the current player according to the multi-dimensional playing data when the current player is playing, and then obtain the timbre recognition model corresponding to the playing preferences to perform timbre recognition, so that the timbre parameters corresponding to different playing habits of different players can be accurately identified, and the accuracy of the playing audio is improved. In addition, the multi-touch sensor array can quickly and accurately detect the playing data of the player, and the sensitivity and audio accuracy of the electronic guzheng are improved.

Description

Technical Field

[0001] This application relates to the field of electronic musical instrument technology, and in particular to an electronic guzheng playing method and system based on multi-touch interaction. Background Technology

[0002] With the development of technology, the electronic guzheng, as a product combining traditional musical instruments with modern technology, has gradually attracted people's attention. However, the traditional electronic guzheng achieves playing through simple touch sensing. Due to the limitations of touch sensing, the traditional electronic guzheng often cannot quickly and accurately capture the performer's playing details, such as the position, speed, and force of the touch on the strings, resulting in low sensitivity and audio accuracy. Therefore, how to improve the sensitivity and audio accuracy of the electronic guzheng has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a method and system for playing electronic guzheng based on multi-touch interaction, in order to improve the sensitivity and audio accuracy of electronic guzheng.

[0004] Firstly, this application provides an electronic guzheng playing method based on multi-touch interaction, the method comprising:

[0005] Based on a multi-touch sensor array, multi-dimensional performance data of the current performer is collected, and the multi-dimensional performance data is identified to obtain the performance preferences of the current performer.

[0006] A timbre recognition model corresponding to the performance preference is determined based on a preset performance parameter library;

[0007] Based on the timbre recognition model, the multidimensional performance data is processed to obtain the timbre parameters corresponding to the multidimensional performance data;

[0008] Based on the timbre parameters, a performance audio is generated, and the performance audio is output based on the audio output device corresponding to the electronic guzheng.

[0009] Secondly, this application also provides an electronic guzheng playing system based on multi-touch interaction, the system comprising:

[0010] The performance preference recognition module is used to collect multi-dimensional performance data of the current performer based on a multi-touch sensor array, and to identify the performance preference of the current performer by recognizing the multi-dimensional performance data.

[0011] The recognition model determination module is used to determine the timbre recognition model corresponding to the performance preference based on a preset performance parameter library;

[0012] The timbre parameter acquisition module is used to process the multidimensional performance data based on the timbre recognition model to obtain the timbre parameters corresponding to the multidimensional performance data.

[0013] The performance audio output module is used to generate performance audio based on the timbre parameters, and output the performance audio based on the audio output device corresponding to the electronic guzheng.

[0014] Thirdly, this application also provides an electronic guzheng, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the electronic guzheng playing method based on multi-touch interaction as described above.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the above-described electronic guzheng playing method based on multi-touch interaction.

[0016] This application discloses an electronic guzheng playing method and system based on multi-touch interaction. Based on a multi-touch sensor array, it collects multi-dimensional performance data from the current performer and identifies the performance data to obtain the performer's playing preferences. A timbre recognition model corresponding to the playing preferences is determined based on a preset performance parameter library. Based on the timbre recognition model, the multi-dimensional performance data is processed to obtain timbre parameters corresponding to the multi-dimensional performance data. Based on the timbre parameters, performance audio is generated and output through the audio output device corresponding to the electronic guzheng. This application can identify the current performer's playing preferences based on multi-dimensional performance data during performance, and then obtain a timbre recognition model corresponding to the playing preferences for timbre recognition. This can accurately identify the timbre parameters corresponding to different performers' different playing habits, improving the accuracy of the performance audio. Furthermore, the multi-touch sensor array can quickly and accurately detect the performer's performance data, improving the sensitivity and audio accuracy of the electronic guzheng. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 is a schematic flowchart of an electronic guzheng playing method based on multi-touch interaction provided in the first embodiment of this application;

[0019] Figure 2 is a schematic flowchart of an electronic guzheng playing method based on multi-touch interaction provided in the second embodiment of this application;

[0020] Figure 3 is a schematic flowchart of an electronic guzheng playing method based on multi-touch interaction provided in the third embodiment of this application;

[0021] Figure 4 is a schematic block diagram of an electronic guzheng playing system based on multi-touch interaction provided in an embodiment of this application;

[0022] Figure 5 is a schematic block diagram of the structure of an electronic guzheng provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0025] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] This application provides an electronic guzheng playing method and system based on multi-touch interaction. The multi-touch interaction-based electronic guzheng playing method can be applied to an electronic guzheng. When the current player is playing, the method can identify the player's playing preferences based on multi-dimensional performance data, and then obtain a timbre recognition model corresponding to the playing preferences for timbre recognition. This can accurately identify the timbre parameters corresponding to different playing habits of different players, improving the accuracy of the played audio. Secondly, the multi-touch sensor array can quickly and accurately detect the player's performance data, improving the sensitivity and audio accuracy of the electronic guzheng.

[0028] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a multi-touch interactive electronic guzheng playing method provided in an embodiment of this application. This multi-touch interactive electronic guzheng playing method can be applied to an electronic guzheng, enabling the identification of the current performer's playing preferences based on multi-dimensional performance data during performance. This allows for the acquisition of a timbre recognition model corresponding to the playing preferences, accurately identifying timbre parameters corresponding to different performers' different playing habits, thus improving the accuracy of the played audio. Furthermore, the multi-touch sensor array can quickly and accurately detect the performer's performance data, improving the sensitivity and audio accuracy of the electronic guzheng.

[0030] like Figure 1 As shown, the electronic guzheng playing method based on multi-touch interaction specifically includes steps S101 to S104.

[0031] S101. Based on a multi-touch sensor array, collect multi-dimensional performance data of the current performer, and identify the multi-dimensional performance data to obtain the performance preferences of the current performer.

[0032] In one embodiment, the multi-contact sensor array includes piezoelectric sensors, capacitive sensors, piezoresistive sensors, optical sensors, etc. The multi-contact sensor array can be installed below the display screen of an electronic guzheng, where virtual strings are displayed. The multi-contact sensor array can also be installed in the bridges of the electronic guzheng.

[0033] By analyzing the sensor data from the multi-contact sensor array, multi-dimensional performance data of the current performer can be obtained.

[0034] By analyzing the statistical distribution and patterns of multidimensional data, the performer's stable habits are extracted. Specifically, key features are extracted from the multidimensional performance data, such as the statistical characteristics (mean, variance, maximum, minimum, etc.) of string contact position, force, speed, and duration. Based on these statistical characteristics, the performer's current playing preferences are determined.

[0035] Further, before determining the timbre recognition model corresponding to the performance preference based on the preset performance parameter library, the following steps are also included: obtaining at least one target music score input by the performer, analyzing the at least one target music score to obtain the target fingering and target timbre parameters corresponding to each target music score; collecting multi-dimensional training data when the performer performs according to each target music score based on the multi-touch sensor array; analyzing the multi-dimensional training data, the target fingering and the target timbre parameters corresponding to each target music score to obtain the first mapping curve between the performance preference information of the performer and the target fingering, and the second mapping curve between the multi-dimensional training data, the target fingering and the target timbre parameters; constructing the pre-training recognition model based on the first mapping curve and the second mapping curve, and training the pre-training recognition model based on the multi-dimensional training data and the target timbre parameters to obtain the timbre parameter recognition model; generating the performance parameter library based on the performance preference information and the timbre recognition model.

[0036] In one embodiment, the target music score can be a picture obtained by scanning and recognizing a paper music score, or a MIDI (Musical Instrument Digital Interface, an industry standard for communication between electronic music devices) file (directly reading note information) or a digital music score.

[0037] Analyzing the target music score includes note and rhythm analysis, fingering analysis and timbre analysis to obtain the target fingering and target timbre parameters in the target music score. Specifically, pitch (such as C4, D4), duration (such as quarter note, eighth note), rhythm pattern (such as dotted rhythm, syncopation) are extracted; fingering symbols in the music score are recognized (such as "﹂" indicating tuo, "↖" indicating upward glide, "彡" indicating shaking finger, etc.); dynamic markings (such as p for soft, f for forte), ornaments (such as trill, mordent), special effects (such as harmonics, strumming) are analyzed, corresponding to timbre parameters (such as volume, harmonic distribution, reverb depth).

[0038] In one embodiment, multi-dimensional training data (such as string contact position, string contact force, string contact speed and string contact trajectory, etc.) when the performer performs according to the target music score is collected.

[0039] In one embodiment, the collected multi-dimensional training data is associated with the target fingering and timbre parameters of the music score to form a "data-fingering-timbre" triple.

[0040] Further, the step of analyzing the multidimensional training data, target fingering, and target timbre parameters corresponding to each of the target scores to obtain a first mapping curve between the performer's performance preference information and the target fingering, and a second mapping curve between the multidimensional training data, the target fingering, and the target timbre parameters, includes: aligning the multidimensional training data, the target fingering, and the target timbre parameters in time to obtain a training data time series, a fingering time series, and a timbre parameter time series; identifying preferences in the training data time series, the fingering time series, and the fingering time series based on a preset preference recognition model to obtain the performer's performance preference information, wherein the performance preference information includes preference performance data corresponding to each of the target fingerings and target fingering and preference performance data corresponding to each of the target timbre parameters; generating the first mapping curve based on the preference performance data corresponding to each of the target fingerings, and generating the second mapping curve based on the target fingering and preference performance data corresponding to each of the target timbre parameters.

[0041] In one embodiment, the multidimensional training data, the target fingering, and the target timbre parameters are time-aligned to unify the sensor data, fingering annotations, and timbre parameters onto the same time axis (e.g., based on the sensor's timestamp), forming an aligned time series.

[0042] In one embodiment, the training data time series, fingering time series, and timbre parameter time series are used as input parameters for the preference recognition model. The preference recognition model identifies the preferred performance data corresponding to each target fingering and the target fingering and preferred performance data corresponding to the timbre parameters (e.g., a volume of 80dB corresponds to the "striking" fingering and the performance data is string touch force = 3.0 Newtons and string touch speed = 12cm / s).

[0043] In one embodiment, the performer's stable habits are extracted by statistically aligning the characteristics (mean, variance, frequency distribution) of the time series. Specifically, the mean and variance of the performance data for each fingering technique in the time series are calculated to reflect the performer's dynamic preferences and stability; the distribution of the performance data is statistically analyzed (e.g., the glissando speed v follows a normal distribution N(5,12)) to reflect the performer's movement habits; and the joint probability of timbre parameters with fingering and performance data is calculated (e.g., ... This reflects the influence of "fingering + performance data" on timbre.

[0044] In another embodiment, for preferences involving time-series dependencies (such as the gradual change in speed of glissando or the frequency fluctuation of tremolo), an LSTM (Long Short-Term Memory) network is used to capture the long-term dependencies of the time series. Specifically, the training data time series, fingering time series, and timbre parameter time series are received, and the time series patterns (such as the "position increment + speed stability" pattern of glissando) are captured through memory units to obtain the distribution parameters (such as mean and variance) of the preference performance data.

[0045] In one embodiment, based on the performance preference information output by the preference recognition model, two core mapping curves are generated to achieve a quantitative correlation between "fingering-preference performance data" and "performance data + fingering-timbre parameters".

[0046] In one embodiment, the first mapping curve is a correlation curve between fingering and preferred playing data, which can be used to determine fingering based on the player's string-touching behavior.

[0047] In one embodiment, the second mapping curve is the correlation curve between (target fingering + preferred playing data) and the target timbre parameters, which can be used to determine the final timbre based on the performer's playing data and current fingering. Specifically, for linear relationships (such as a positive correlation between velocity F and volume), multiple linear regression is used for fitting: SPL (Sound Pressure Level, used to represent the strength or volume of a sound). For one-hot encoding of finger placement, These are the weighting coefficients. For non-linear relationships (such as the non-linear correlation between vibrato depth and finger tremor frequency), a neural network is used to fit the mapping relationship.

[0048] In one embodiment, a pre-trained recognition model is constructed based on two mapping curves and optimized using training data to finally obtain a timbre parameter recognition model. Specifically, the architecture of the pre-trained model includes an input layer for receiving sensor data and fingering codes (such as one-hot encoding to represent "upward glissando" and "tremolo"); a feature fusion layer that fuses multimodal features through fully connected layers or convolutional layers; and an output layer that outputs each timbre parameter.

[0049] In one embodiment, the "data-fingering-timbre" triplet is divided into a training set (70%), a validation set (20%), and a test set (10%). Mean squared error is used to measure the difference between the predicted timbre parameters and the target parameters. The Adam optimizer is used with a learning rate of 0.001. The parameters are dynamically adjusted to minimize the loss. The model weights are updated through backpropagation until the validation set loss converges (e.g., the loss does not decrease for 10 consecutive rounds).

[0050] In one embodiment, after training is completed, a performance parameter library is generated by combining performance preference information with a timbre parameter recognition model to store the performer's personalized settings.

[0051] In the above embodiments, the performer's personalized habits are transformed into a quantifiable and callable parameter library, which can clearly identify the different performance habits of different performers. When different performance habits lead to the same multi-dimensional performance data, the different timbre parameters desired by different performers can also be accurately identified, thereby improving the audio accuracy of the electronic guzheng.

[0052] S102. Determine the timbre recognition model corresponding to the performance preference based on the preset performance parameter library;

[0053] In one embodiment, the performance parameter library stores timbre recognition models corresponding to different preferences. The performance parameter library is matched according to the current performer's performance preferences, and the timbre recognition model that matches the performance preferences is obtained.

[0054] The matching process specifically includes: direct matching: if the preference completely matches a preference in the parameter library, the timbre recognition model corresponding to that preference in the performance parameter library is directly called; and blending adjustment: if the preference is between two preferences in the performance parameter library, the parameters of the models corresponding to the two preferences are blended through linear interpolation to obtain a new timbre recognition model.

[0055] Because different performers have different preferences regarding finger length, string pressing force, etc., it's possible for different performers to have the same multidimensional performance data but different desired timbres. In such cases, using only a uniform timbre recognition model cannot accurately identify the different timbre parameters. Therefore, in this embodiment, the current performer's performance preferences are identified in real time based on their multidimensional performance data, and a timbre recognition model is matched according to these preferences. This allows the timbre recognition model to more accurately identify the current performer's target timbre parameters.

[0056] S103. Based on the timbre recognition model, process the multi-dimensional performance data to obtain the timbre parameters corresponding to the multi-dimensional performance data;

[0057] In one embodiment, the timbre recognition model is trained using historical data and is used to process multidimensional performance data to identify the timbre parameters corresponding to the multidimensional performance data. The multidimensional performance data is input into the timbre recognition model for processing to obtain the timbre parameters.

[0058] Specifically, the timbre recognition model performs feature classification on multi-dimensional performance data to obtain fingering recognition features and timbre recognition features; it then identifies the fingering recognition features based on the fingering classifier to determine the current performer's fingering; finally, it analyzes the fingering and timbre recognition features to obtain timbre parameters.

[0059] S104. Generate performance audio based on the timbre parameters, and output the performance audio based on the audio output device corresponding to the electronic guzheng.

[0060] Furthermore, generating the performance audio based on the timbre parameters includes: constructing a discrete spectrum and a time-domain envelope function based on the timbre parameters; obtaining the target audio effect of the current performer; and synthesizing the discrete spectrum and the envelope function based on the target audio effect to generate the performance audio.

[0061] In one embodiment, the fundamental frequency is obtained from the timbre parameters. Then determine the overtone frequency. Overtones are usually integer multiples of the fundamental frequency, that is, the frequency of the nth overtone is... .

[0062] Based on the amplitude proportions of each overtone in the timbre parameters, determine the corresponding amplitude value assigned to each overtone. .

[0063] The fundamental frequency and harmonic frequencies, along with their corresponding amplitude values, are filled into the spectrum array. Specifically, for each frequency point f, if f is the fundamental frequency or the frequency of a certain overtone, the corresponding amplitude value is... Fill in the array; otherwise, the amplitude value at that frequency point is 0.

[0064] In one embodiment, the envelope function typically includes four phases: attack, decay, sustain, and release. Specifically, the time parameters for each phase are obtained from the timbre parameters, namely the attack time, decay time, sustain time, and release time. The starting gain and peak gain for the attack phase, the starting gain and sustain gain for the decay phase, and the starting gain and ending gain for the release phase are determined. Based on the time and gain parameters for each phase, a time-domain envelope function is generated.

[0065] In one embodiment, parameters are determined based on the user-selected target audio effect. These parameters may include reverberation time, delay time, feedback amount, and depth.

[0066] In one embodiment, based on the parameters and timbre parameters corresponding to the target audio, the discrete spectrum is converted into a time-domain signal using an inverse Fourier transform. A time-domain envelope function is applied to the time-domain signal by multiplying the envelope function array element-wise with the time-domain signal array, ensuring that the signal amplitude changes with time in accordance with the definition of the envelope function. The synthesized audio signal is then processed according to the target audio effect. Finally, the processed audio signal is converted into a playable audio format.

[0067] In one embodiment, the audio output device can be a built-in speaker of the electronic guzheng, or an external output device that communicates with the electronic guzheng.

[0068] The above embodiments provide an electronic guzheng playing method and system based on multi-touch interaction. Based on a multi-touch sensor array, it collects multi-dimensional performance data from the current performer, identifies the performance data to obtain the performer's playing preferences, determines a timbre recognition model corresponding to the playing preferences based on a preset performance parameter library, processes the multi-dimensional performance data based on the timbre recognition model to obtain timbre parameters corresponding to the multi-dimensional performance data, generates performance audio based on the timbre parameters, and outputs the performance audio through the audio output device corresponding to the electronic guzheng. This application can identify the current performer's playing preferences based on multi-dimensional performance data during performance, and then obtain a timbre recognition model corresponding to the playing preferences for timbre recognition. This can accurately identify the timbre parameters corresponding to different performers' different playing habits, improving the accuracy of the performance audio. Furthermore, the multi-touch sensor array can quickly and accurately detect the performer's performance data, improving the sensitivity and audio accuracy of the electronic guzheng.

[0069] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating an electronic guzheng playing method based on multi-touch interaction, provided by an embodiment of this application. This multi-touch interaction-based electronic guzheng playing method can be applied to an electronic guzheng, whereby touch data is collected via a multi-touch sensor array, processed by a feature fusion module to obtain effective touch features, and then accurately identified using a performance data recognition model to obtain precise multi-dimensional performance data.

[0070] like Figure 2 As shown, step S101 of the electronic guzheng playing method based on multi-touch interaction specifically includes steps S201 to S205.

[0071] S201. Based on the multi-touch sensor array, collect the touch data generated when the current performer plays based on the virtual strings in the panel display of the electronic guzheng, wherein the multi-touch sensor array is installed under the panel display of the electronic guzheng.

[0072] In this embodiment, the multi-contact sensor array includes piezoelectric sensors, capacitive sensors, piezoresistive sensors, optical sensors, etc. The multi-contact sensor array is attached to the bottom of the display screen, which displays virtual strings.

[0073] When a performer plays on the virtual strings displayed on the panel, a multi-touch sensor array can simultaneously collect multi-dimensional data, including position and trajectory data. Through the coordinate positioning function of the capacitive sensor, it detects the contact point between the finger and the virtual string, continuously samples the movement path of the finger on the virtual string, records the sequence of position changes over time, and obtains the sliding trajectory; time and force data, through the high sampling rate of the sensor to record the start time, duration, and end time of the string touch, and through the pressure-sensitive element to detect the pressure of the finger on the virtual string; multi-touch collaborative data, etc. For compound fingerings (such as large plucks and strumming), the multi-touch sensor can simultaneously detect the touch status of multiple fingers, including identifying the effective touch points at the same time (such as when 5 virtual strings are triggered simultaneously during strumming), and calculating the distance between multiple touch points in the string direction and string length direction (such as the distance between the thumb and middle finger during a large pluck).

[0074] S202. Based on the feature fusion module in the performance data recognition model, perform spatiotemporal feature fusion on the touch data to obtain effective touch features;

[0075] Among them, the effective touch features ,in, This represents the characteristic value at spatial location (x, y) and time t, where x is the position of the virtual string along its length and y is the virtual string's number. This represents the eigenvalue at spatial offset (i, j) and time offset k, where i is the positional offset in the string length direction, j is the offset of the string number, and k is the time offset. This represents the weight coefficients of the 3D convolution kernel at spatial offset (i, j) and temporal offset k.

[0076] In one embodiment, The input feature map is a structured representation of the raw data collected by the multi-contact sensor array after preprocessing. The dimensions include the spatial dimension H: the position of the virtual string along its length (e.g., 1-100cm, x-axis), with each position corresponding to a detection point of the capacitive sensor; the spatial dimension W: the number of the virtual string (e.g., string 1-21, y-axis), with each string corresponding to a detection channel of the piezoresistive / piezoelectric sensor; and the time dimension T: the sampling time step of the sensor (e.g., t=1,2,...,5, representing the last 5 sampling times).

[0077] The preprocessing steps include: data calibration, dynamic baseline calibration of the capacitive sensor (to eliminate ambient temperature and humidity drift), and pressure value correction of the piezoresistive sensor; feature enhancement, which improves the saliency of key features through operations such as neighborhood difference (highlighting the edge of the contact string) and normalization (unifying the dimensions); and multimodal fusion, which weights and fuses multi-source data such as position, force, and velocity to form an input feature map.

[0078] Where i∈[−1,1] indicates that the string length direction covers the left, middle, and right 3 units of the current position; j∈[-1, 1] indicates that the string number covers the front, middle, and back 3 strings of the current string; k∈[−2,2] indicates that the time covers the 2 frames before the current time, the current frame, and the 2 frames after the current time, for a total of 5 time steps.

[0079] In one embodiment, the 3D convolutional kernel traverses the input feature map using a sliding window. Each window center corresponds to an output position (x, y, t). For each window, the points in the neighborhood of the input feature map are calculated. and corresponding weights The sum of the products yields the output features. That is, effective touch features.

[0080] S203. The effective touch features are respectively transmitted to the sub-pixel positioning module, speed calculation module and force calculation module in the performance data recognition model for processing to obtain the string touch position, string touch speed and string touch force.

[0081] Further, step S203 includes: based on the sub-pixel positioning module, extracting pressure peak points and pressure gradient values ​​of pixels surrounding the pressure peak points from the effective touch features, and correcting the position of the pressure peak points based on the pressure gradient values ​​to obtain the touch string position; based on the speed calculation module, extracting the position difference and time difference between adjacent frames from the effective touch features, and performing differential calculation based on the position difference and time difference to obtain the touch string speed; based on the force calculation module, extracting pressure peak data, pressure change rate, and contact area from the effective touch features, and obtaining the touch string pressure based on the pressure peak data, pressure change rate, and contact area; wherein, the touch string pressure... Where A represents the contact area. For peak pressure data, The rate of change of pressure, These are the weighting coefficients for contact area, peak pressure data, and pressure change rate, respectively.

[0082] In one embodiment, the effective touch feature map is traversed to find local maxima (i.e., pressure peaks) within each connected region. This is done by retrieving the peaks from integer pixel coordinates. The correction is made to the sub-pixel level, using the pressure gradient of pixels around the peak point (reflecting the direction and rate of pressure change) for interpolation.

[0083] Specifically, the pressure gradient in the four neighborhoods (upper, lower, left, and right) around the peak point is calculated:

[0084]

[0085]

[0086] in, The pressure gradient is in the x-direction (chord length direction). The pressure gradient is in the y-direction (the direction of the string number).

[0087] Subpixel position correction: Assume the pressure distribution near the peak point is a quadratic surface. Then the subpixel position It can be calculated using the ratio of the gradient to the second derivative:

[0088]

[0089]

[0090] in, and The second derivatives are calculated in the x and y directions through the neighborhood pixels.

[0091] In one embodiment, the speed calculation module aims to calculate the real-time speed of the string-touching action based on the position and time differences between adjacent frames. Specifically, the input is the string-touching position sequence output by the sub-pixel positioning module. (Time steps t=1,2,...,N), the timestamp of each time step is... .

[0092] The position difference is obtained by calculating the position change over consecutive time steps, and the time difference is obtained by calculating the interval between timestamps. The string contact velocity is obtained by differentiating the position difference and the time difference.

[0093] In one embodiment, the goal of the force calculation module is to accurately reflect the force applied to the touch by integrating peak pressure, rate of change of pressure, and contact area. Specifically, pressure-related features, including peak pressure data, are extracted from effective touch characteristics. : Feature values ​​of peak points in the sub-pixel positioning module (reflecting maximum pressure); pressure change rate : Rate of change of pressure value over time; Contact area A: Number of pixels in the effective contact area. Through The contact pressure is calculated.

[0094] Among them, contact area A: the larger the contact area between the finger and the string, the more uniform the pressure distribution, and the more stable the actual string-touching force. Peak pressure : Directly reflects the maximum pressure at the moment of string contact; rate of change of pressure : Reflects the "speed" of the string contact (such as strumming) (Larger, stronger perception of force).

[0095] In this embodiment, the subpixel positioning, speed calculation, and force calculation modules work together to provide key data for fingering recognition and tone generation of the electronic guzheng. Specifically, subpixel positioning provides high-precision string-touching position to ensure the accuracy of pitch calculation; speed calculation provides string-touching speed to distinguish between fingerings such as glissando (slow) and strumming (fast); and force calculation provides string-touching pressure to control volume and harmonic distribution (e.g., higher harmonics are richer when the string is touched hard).

[0096] In another instance, the subpixel positioning, speed calculation, and force calculation modules can be processed simultaneously and in parallel, directly extracting relevant data from effective touch features for processing to obtain the string touch position, string touch speed, and string touch force, thereby improving the efficiency of timbre recognition.

[0097] S204. Based on the trajectory recognition module in the performance data recognition model, perform trajectory recognition on the string-touching position to obtain the string-touching trajectory;

[0098] In one embodiment, the input to the trajectory recognition module is a high-precision string position sequence output by the sub-pixel positioning module, with each time step t corresponding to a two-dimensional coordinate. Geometric and dynamic features are extracted from the two-dimensional coordinates corresponding to each time step. Geometric features include trajectory direction, trajectory length, and curvature. The trajectory direction is calculated by estimating the vector between the starting and ending points, reflecting the overall direction of trajectory movement. Dynamic features include the rate of change of velocity, frequency characteristics, and the number of contact points.

[0099] Trajectory matching is performed using the DTW algorithm to determine the string-touching trajectory. Specifically, dynamic programming is used to calculate the minimum matching distance between the geometric and dynamic features of the input trajectory and the geometric and dynamic features of the template trajectory. The smaller the distance, the higher the similarity. The template trajectory with the smallest distance is selected as the string-touching trajectory. The template trajectory is obtained during model training.

[0100] In another embodiment, for complex trajectories (such as the high-frequency reciprocation of finger shaking or the cross-string movement of strumming), machine learning algorithms are used for identification. Specifically, time dependencies are captured through LSTM network layers and classified through fully connected layers.

[0101] S205. The string-touching position, string-touching speed, string-touching force, and string-touching trajectory are used as the multi-dimensional performance data.

[0102] In one embodiment, the string contact position, string contact speed, string contact force, and string contact trajectory constitute multi-dimensional performance data.

[0103] In the above embodiments, touch data is collected by a multi-touch sensor array, processed by a feature fusion module to obtain effective touch features, and then accurately identified by a performance data recognition model to obtain accurate multi-dimensional performance data.

[0104] Further, step S101 also includes: obtaining sensor data when the current performer touches the strings based on the multi-touch sensor array, wherein the multi-touch sensor array is installed in the bridge of the electronic guzheng; grouping the sensor data in each bridge according to a preset string number, and analyzing the sensor data in each group to obtain multi-dimensional performance sub-data corresponding to the strings in each group; and analyzing and integrating the multi-dimensional performance sub-data to obtain the multi-dimensional performance data of the current performer.

[0105] In one embodiment, the bridge is an important component on the guzheng used to fix the strings and transmit vibrations. Installing a sensor array here can effectively capture the vibration and contact information of the strings.

[0106] In this embodiment, the multi-contact sensor array includes pressure sensors and vibration sensors, etc.

[0107] The sensor data from the bridge is grouped according to the preset string number. Each string of the electronic guzheng has a corresponding string number, and the sensor data from the bridge corresponding to the same string are grouped together.

[0108] In a specific embodiment, sensor data within a group is identified using a performance data recognition model to obtain multi-dimensional performance sub-data for each group. Among these, the string-touching position... Determined by the effective vibrating length of the string, and related to the vibration frequency. Satisfies the string vibration equation:

[0109]

[0110] in, For string tension, Let be the linear density of the chord.

[0111] Extracting vibration frequency from vibration sensor Based on the string vibration equation, the position of contact with the string can be deduced. .

[0112] In one embodiment, the force of the string strike Data from the pressure sensor at the bottom of the bridge It needs to be calculated based on the contact area A:

[0113]

[0114] in, These are the weighting coefficients.

[0115] In one embodiment, the geometric and dynamic characteristics of the time series of the string-touching trajectory position include: trajectory type (matched by DTW template, such as the linear trajectory of a glissando or the high-frequency undulating trajectory of a tremolo); trajectory length; and trajectory direction (the sign of the difference between the start and end positions, positive for an upward glissando and negative for a downward glissando).

[0116] In one embodiment, the multidimensional performance sub-data of each group is integrated to form multidimensional performance data reflecting the performer's overall movements. Specifically, the multidimensional performance sub-data of all groups is synchronized in time and then integrated to obtain multidimensional performance data.

[0117] In the above embodiments, the sensor data obtained by the multi-contact sensor is grouped and processed in parallel, which can improve the efficiency of obtaining multi-dimensional performance data and thus improve the sensitivity of the electronic guzheng.

[0118] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a multi-touch interactive electronic guzheng playing method provided in an embodiment of this application. This multi-touch interactive electronic guzheng playing method can be applied to electronic guzheng to identify timbre parameters in conjunction with fingering techniques. It can comprehensively capture performance details, accurately grasp the performer's intentions, and significantly improve the accuracy of timbre recognition results.

[0119] like Figure 3 As shown, the electronic guzheng playing method based on multi-touch interaction specifically includes steps S301 to S303.

[0120] S301. Based on the timbre recognition model, perform feature classification on the multi-dimensional performance data to obtain fingering recognition features and timbre recognition features;

[0121] S302. Identify the fingering recognition features to determine the fingering of the current performer;

[0122] S303. Analyze the fingering and the timbre recognition features to obtain the timbre parameters.

[0123] In one embodiment, the multidimensional performance data is generated by the performer during the performance and contains a wealth of information, such as the position of the string touch, the force of the string touch, the speed of the string touch, and the trajectory of the string touch.

[0124] The timbre recognition model analyzes multi-dimensional performance data, dividing it into fingering recognition features and timbre recognition features. Fingering recognition features are used to identify the fingerings used by the performer, such as thumb strikes, index finger flicks, and middle finger hooks, mainly involving features related to the playing action, such as the position, force, and speed of the string contact. Timbre recognition features are used to determine the timbre parameters produced by the performance, such as frequency components and dynamic characteristics, mainly involving features related to timbre.

[0125] In one embodiment, the fingering classifier in the timbre recognition model is used to identify the extracted fingering recognition features.

[0126] Finger type classifiers are pre-trained based on a large amount of training data and can learn patterns in the feature vectors of different finger types. Common algorithms include support vector machines, decision trees, and neural networks.

[0127] The fingering classifier outputs the fingering category most likely used by the current performer, such as identifying that the performer is using the thumb-pike fingering technique.

[0128] In one embodiment, timbre parameters are determined based on fingering and timbre recognition features. Specifically, based on fingering and timbre recognition features, volume variations are analyzed to determine envelope parameters (attack, attenuation, sustain, release). The presence of modulation effects such as vibrato and glissando is identified, and parameters such as the modulation frequency and depth of vibrato, the starting frequency and target frequency of glissando, and the glissando duration are determined.

[0129] In the above embodiments, combining fingering to identify timbre parameters can comprehensively capture performance details, accurately grasp the performer's performance intentions, and significantly improve the accuracy of timbre identification results.

[0130] Please see Figure 4 , Figure 4 This application provides a schematic block diagram of an electronic guzheng playing system based on multi-touch interaction, which is used to execute the aforementioned electronic guzheng playing method based on multi-touch interaction. The electronic guzheng playing system based on multi-touch interaction can be configured on an electronic guzheng.

[0131] like Figure 4 As shown, the electronic guzheng playing system 400 based on multi-touch interaction includes:

[0132] The performance preference recognition module 401 is used to collect multi-dimensional performance data of the current performer based on a multi-touch sensor array, and to identify the performance preference of the current performer by recognizing the multi-dimensional performance data.

[0133] The recognition model determination module 402 is used to determine the timbre recognition model corresponding to the performance preference based on a preset performance parameter library;

[0134] The timbre parameter acquisition module 403 is used to process the multidimensional performance data based on the timbre recognition model to obtain the timbre parameters corresponding to the multidimensional performance data.

[0135] The performance audio output module 404 is used to generate performance audio based on the timbre parameters and output the performance audio based on the audio output device corresponding to the electronic guzheng.

[0136] Furthermore, the performance preference recognition module 401 includes:

[0137] A touch data acquisition unit is used to acquire touch data generated when the current performer plays based on the virtual strings on the panel display of the electronic guzheng, based on the multi-touch sensor array, wherein the multi-touch sensor array is installed under the panel display of the electronic guzheng.

[0138] The spatiotemporal feature fusion unit is used to perform spatiotemporal feature fusion on the touch data based on the feature fusion module in the performance data recognition model to obtain effective touch features;

[0139] The model processing unit is used to transmit the effective touch features to the sub-pixel positioning module, speed calculation module and force calculation module in the performance data recognition model for processing, so as to obtain the string touch position, string touch speed and string touch force.

[0140] The string-touching trajectory acquisition unit is used to perform trajectory recognition on the string-touching position based on the trajectory recognition module in the performance data recognition model, and obtain the string-touching trajectory.

[0141] A multi-dimensional performance data determination unit is used to use the string-touching position, the string-touching speed, the string-touching force, and the string-touching trajectory as the multi-dimensional performance data;

[0142] Among them, the effective touch features ,in, This represents the characteristic value at spatial location (x, y) and time t, where x is the position of the virtual string along its length and y is the virtual string's number. This represents the eigenvalue at spatial offset (i, j) and time offset k, where i is the positional offset in the string length direction, j is the offset of the string number, and k is the time offset. This represents the weight coefficients of the 3D convolution kernel at spatial offset (i, j) and temporal offset k.

[0143] Furthermore, the model processing unit includes:

[0144] The touch string position acquisition sub-unit is used to extract the pressure peak point and the pressure gradient value of the pixels around the pressure peak point from the effective touch features based on the sub-pixel positioning module, and to perform position correction on the pressure peak point based on the pressure gradient value to obtain the touch string position.

[0145] The string touch speed acquisition subunit is used to extract the position difference and time difference of adjacent frames from the effective touch features based on the speed calculation module, and perform differential calculation based on the position difference and the time difference to obtain the string touch speed;

[0146] The string pressure acquisition subunit is used to extract pressure peak data, pressure change rate and contact area from the effective touch features based on the force calculation module, and to obtain the string pressure based on the pressure peak data, the pressure change rate and the contact area;

[0147] Among them, the string pressure Where A represents the contact area. For peak pressure data, The rate of change of pressure, These are the weighting coefficients for contact area, peak pressure data, and pressure change rate, respectively.

[0148] Furthermore, the performance preference recognition module 401 also includes:

[0149] A sensor data acquisition unit is used to acquire sensor data when the current player touches the strings based on the multi-touch sensor array, wherein the multi-touch sensor array is installed in the bridge of the electronic guzheng;

[0150] The grouped data analysis unit is used to group the sensor data in each bridge according to the preset string number, and analyze the sensor data in each group to obtain the multi-dimensional performance sub-data corresponding to the strings in each group.

[0151] The sub-data integration unit is used to analyze and integrate the multi-dimensional performance sub-data to obtain the multi-dimensional performance data of the current performer.

[0152] Furthermore, the timbre parameter acquisition module 403 includes:

[0153] The feature classification unit is used to perform feature classification on the multidimensional performance data based on the timbre recognition model to obtain fingering recognition features and timbre recognition features;

[0154] The fingering recognition unit is used to recognize the fingering recognition features and determine the fingering of the current performer;

[0155] The timbre parameter acquisition unit is used to analyze the fingering and the timbre recognition features to obtain the timbre parameters.

[0156] Furthermore, the performance audio output module 404 includes:

[0157] A discrete spectrum construction unit is used to construct a discrete spectrum and a time-domain envelope function based on the timbre parameters.

[0158] The performance audio generation unit is used to obtain the target audio effect of the current performer, and synthesize the discrete spectrum and the envelope function based on the target audio effect to generate the performance audio.

[0159] Furthermore, the electronic guzheng playing device 400 based on multi-touch interaction also includes a performance parameter library generation module, which includes:

[0160] The target data acquisition unit is used to acquire at least one target musical score input by the performer, and to analyze at least one of the target musical scores to obtain the target fingering and target timbre parameters corresponding to each target musical score;

[0161] The training data acquisition unit is used to acquire multi-dimensional training data of the performer when playing according to each of the target scores, based on the multi-touch sensor array.

[0162] The mapping curve acquisition unit is used to analyze the multidimensional training data corresponding to each of the target scores, the target fingering and the target timbre parameters, and obtain the first mapping curve between the performer's performance preference information and the target fingering, as well as the second mapping curve between the multidimensional training data, the target fingering and the target timbre parameters;

[0163] The model training unit is used to construct the pre-trained recognition model based on the first mapping curve and the second mapping curve, and to train the pre-trained recognition model based on the multi-dimensional training data and the target timbre parameters to obtain the timbre parameter recognition model.

[0164] The performance parameter library generation unit is used to generate the performance parameter library based on the performance preference information and the timbre recognition model.

[0165] Furthermore, the mapping curve obtaining unit includes:

[0166] The time alignment subunit is used to time-align the multidimensional training data, the target fingering, and the target timbre parameters to obtain the training data time series, the fingering time series, and the timbre parameter time series.

[0167] The performance preference information acquisition subunit is used to identify the performance preference of the performer based on the preset preference recognition model, the training data time series, the fingering time series and the fingering time series preference recognition, wherein the performance preference information includes the preference performance data corresponding to each of the target fingerings and the target fingering and preference performance data corresponding to each of the target timbre parameters;

[0168] The mapping curve generation subunit is used to generate the first mapping curve based on the preference performance data corresponding to each of the target fingerings, and to generate the second mapping curve based on the target fingerings and preference performance data corresponding to each of the target timbre parameters.

[0169] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the system and each module described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] The above-described system can be implemented as a computer program, which can be used in, for example... Figure 5 The electronic guzheng shown is running.

[0171] Please see Figure 5 , Figure 5 This is a schematic block diagram of the structure of an electronic guzheng provided in an embodiment of this application.

[0172] See Figure 5 The electronic guzheng includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0173] The non-volatile storage medium can store the operating system and computer program. The computer program includes program instructions that, when executed, cause the processor to perform any electronic guzheng playing method based on multi-touch interaction.

[0174] The processor provides computing and control capabilities to support the operation of the entire electronic guzheng.

[0175] The internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to perform any electronic guzheng playing method based on multi-touch interaction.

[0176] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic guzheng to which the present application is applied. A specific electronic guzheng may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0178] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0179] Based on a multi-touch sensor array, multi-dimensional performance data of the current performer is collected, and the multi-dimensional performance data is identified to obtain the performance preferences of the current performer.

[0180] A timbre recognition model corresponding to the performance preference is determined based on a preset performance parameter library;

[0181] Based on the timbre recognition model, the multidimensional performance data is processed to obtain the timbre parameters corresponding to the multidimensional performance data;

[0182] Based on the timbre parameters, a performance audio is generated, and the performance audio is output based on the audio output device corresponding to the electronic guzheng.

[0183] In one embodiment, when the processor collects multi-dimensional performance data of the current performer based on a multi-touch sensor array, it is used to:

[0184] Based on the multi-touch sensor array, touch data generated by the current performer when playing based on the virtual strings on the panel display of the electronic guzheng is collected, wherein the multi-touch sensor array is installed under the panel display of the electronic guzheng;

[0185] Based on the feature fusion module in the performance data recognition model, spatiotemporal feature fusion is performed on the touch data to obtain effective touch features;

[0186] The effective touch features are respectively transmitted to the sub-pixel positioning module, speed calculation module and force calculation module in the performance data recognition model for processing to obtain the string touch position, string touch speed and string touch force.

[0187] Based on the trajectory recognition module in the performance data recognition model, the string-touching position is identified to obtain the string-touching trajectory.

[0188] The string-touching position, string-touching speed, string-touching force, and string-touching trajectory are used as the multi-dimensional performance data;

[0189] Among them, the effective touch features ,in, This represents the characteristic value at spatial location (x, y) and time t, where x is the position of the virtual string along its length and y is the virtual string's number. This represents the eigenvalue at spatial offset (i, j) and time offset k, where i is the positional offset in the string length direction, j is the offset of the string number, and k is the time offset. This represents the weight coefficients of the 3D convolution kernel at spatial offset (i, j) and temporal offset k.

[0190] In one embodiment, when the processor transmits the effective touch features to the sub-pixel positioning module, speed calculation module, and force calculation module in the performance data recognition model for processing to obtain the string touch position, string touch speed, and string touch force, it is used to achieve the following:

[0191] Based on the subpixel positioning module, pressure peak points and pressure gradient values ​​of pixels surrounding the pressure peak points are extracted from the effective touch features, and the position of the pressure peak points is corrected based on the pressure gradient values ​​to obtain the touch string position.

[0192] Based on the speed calculation module, the position difference and time difference between adjacent frames are extracted from the effective touch features, and the differential calculation is performed based on the position difference and the time difference to obtain the string touch speed;

[0193] Based on the force calculation module, pressure peak data, pressure change rate, and contact area are extracted from the effective touch features, and the touch string pressure is obtained based on the pressure peak data, the pressure change rate, and the contact area.

[0194] Among them, the string pressure Where A represents the contact area. For peak pressure data, The rate of change of pressure, These are the weighting coefficients for contact area, peak pressure data, and pressure change rate, respectively.

[0195] In one embodiment, the processor, in addition to implementing the acquisition of multi-dimensional performance data during the current performance by the performer based on a multi-touch sensor array, is also used to implement:

[0196] Based on the multi-touch sensor array, sensor data is obtained when the current performer touches the strings, wherein the multi-touch sensor array is installed in the bridge of the electronic guzheng;

[0197] The sensor data in each bridge is grouped according to the preset string number, and the sensor data in each group is analyzed to obtain the multi-dimensional performance sub-data corresponding to the strings in each group.

[0198] The multidimensional performance sub-data is analyzed and integrated to obtain the multidimensional performance data of the current performer.

[0199] In one embodiment, when the processor processes the multidimensional performance data based on the timbre recognition model to obtain the timbre parameters corresponding to the multidimensional performance data, it is configured to:

[0200] Based on the timbre recognition model, the multidimensional performance data is classified to obtain fingering recognition features and timbre recognition features.

[0201] The fingering recognition features are identified to determine the fingering of the current performer;

[0202] The fingering and the timbre recognition features are analyzed to obtain the timbre parameters.

[0203] In one embodiment, when the processor generates performance audio based on the timbre parameters, it is configured to:

[0204] Based on the timbre parameters, a discrete spectrum and a time-domain envelope function are constructed;

[0205] The target audio effect of the current performer is obtained, and the discrete spectrum and the envelope function are synthesized based on the target audio effect to generate the performance audio.

[0206] In one embodiment, before the processor implements the determination of the timbre recognition model corresponding to the performance preference based on a preset performance parameter library, it is further configured to implement:

[0207] Obtain at least one target musical score input by the performer, and analyze at least one of the target musical scores to obtain the target fingering and target timbre parameters corresponding to each target musical score;

[0208] Based on the multi-touch sensor array, multi-dimensional training data is collected when the performer plays according to each of the target scores;

[0209] The multidimensional training data, target fingering, and target timbre parameters corresponding to each target musical score are analyzed to obtain a first mapping curve between the performer's performance preference information and the target fingering, as well as a second mapping curve between the multidimensional training data, target fingering, and target timbre parameters.

[0210] Based on the first mapping curve and the second mapping curve, the pre-trained recognition model is constructed, and the pre-trained recognition model is trained based on the multi-dimensional training data and the target timbre parameters to obtain the timbre parameter recognition model;

[0211] Based on the performance preference information and the timbre recognition model, the performance parameter library is generated.

[0212] In one embodiment, when the processor analyzes the multidimensional training data corresponding to each of the target musical scores, the target fingering, and the target timbre parameters to obtain a first mapping curve between the performer's performance preference information and the target fingering, and a second mapping curve between the multidimensional training data, the target fingering, and the target timbre parameters, it is configured to:

[0213] The multidimensional training data, the target fingering, and the target timbre parameters are time-aligned to obtain the training data time series, the fingering time series, and the timbre parameter time series.

[0214] Based on a preset preference recognition model, the performance preference information of the performer is obtained by identifying the preference of the training data time series, the fingering time series, and the fingering time series preferences. The performance preference information includes the preference performance data corresponding to each target fingering and the target fingering and preference performance data corresponding to each target timbre parameter.

[0215] Based on the preferred playing data corresponding to each of the target fingerings, the first mapping curve is generated, and based on the target fingerings and preferred playing data corresponding to each of the target timbre parameters, the second mapping curve is generated.

[0216] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the electronic guzheng playing methods based on multi-touch interaction provided in the embodiments of this application.

[0217] The computer-readable storage medium can be the internal storage unit of the electronic guzheng as described in the foregoing embodiments, such as the hard drive or memory of the electronic guzheng. The computer-readable storage medium can also be an external storage device of the electronic guzheng, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic guzheng.

[0218] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for playing an electronic guzheng based on multi-touch interaction, characterized in that, include: Based on a multi-touch sensor array, multi-dimensional performance data of the current performer is collected, and the multi-dimensional performance data is identified to obtain the performance preferences of the current performer. A timbre recognition model corresponding to the performance preference is determined based on a preset performance parameter library; Based on the timbre recognition model, the multidimensional performance data is processed to obtain the timbre parameters corresponding to the multidimensional performance data; Based on the timbre parameters, a performance audio is generated, and the performance audio is output based on the audio output device corresponding to the electronic guzheng.

2. The electronic guzheng playing method based on multi-touch interaction according to claim 1, characterized in that, The multi-dimensional performance data collected by the multi-contact sensor array during the current performance includes: Based on the multi-touch sensor array, touch data generated by the current performer when playing based on the virtual strings on the panel display of the electronic guzheng is collected, wherein the multi-touch sensor array is installed under the panel display of the electronic guzheng; Based on the feature fusion module in the performance data recognition model, spatiotemporal feature fusion is performed on the touch data to obtain effective touch features; The effective touch features are respectively transmitted to the sub-pixel positioning module, speed calculation module and force calculation module in the performance data recognition model for processing to obtain the string touch position, string touch speed and string touch force. Based on the trajectory recognition module in the performance data recognition model, the string-touching position is identified to obtain the string-touching trajectory. The string-touching position, string-touching speed, string-touching force, and string-touching trajectory are used as the multi-dimensional performance data; Among them, the effective touch features ,in, This represents the characteristic value at spatial location (x, y) and time t, where x is the position of the virtual string along its length and y is the virtual string's number. This represents the eigenvalue at spatial offset (i, j) and time offset k, where i is the positional offset in the string length direction, j is the offset of the string number, and k is the time offset. This represents the weight coefficients of the 3D convolution kernel at spatial offset (i, j) and temporal offset k.

3. The electronic guzheng playing method based on multi-touch interaction according to claim 2, characterized in that, The step of transmitting the effective touch features to the sub-pixel positioning module, speed calculation module, and force calculation module in the performance data recognition model for processing to obtain the string touch position, string touch speed, and string touch force includes: Based on the subpixel positioning module, pressure peak points and pressure gradient values ​​of pixels surrounding the pressure peak points are extracted from the effective touch features, and the position of the pressure peak points is corrected based on the pressure gradient values ​​to obtain the touch string position. Based on the speed calculation module, the position difference and time difference between adjacent frames are extracted from the effective touch features, and the differential calculation is performed based on the position difference and the time difference to obtain the string touch speed; Based on the force calculation module, pressure peak data, pressure change rate, and contact area are extracted from the effective touch features, and the touch string pressure is obtained based on the pressure peak data, the pressure change rate, and the contact area. Among them, the string pressure Where A represents the contact area. For peak pressure data, The rate of change of pressure, These are the weighting coefficients for contact area, peak pressure data, and pressure change rate, respectively.

4. The electronic guzheng playing method based on multi-touch interaction according to claim 1, characterized in that, The method of collecting multi-dimensional performance data based on a multi-touch sensor array during the current performance also includes: Based on the multi-touch sensor array, sensor data is obtained when the current performer touches the strings, wherein the multi-touch sensor array is installed in the bridge of the electronic guzheng; The sensor data in each bridge is grouped according to the preset string number, and the sensor data in each group is analyzed to obtain the multi-dimensional performance sub-data corresponding to the strings in each group. The multidimensional performance sub-data is analyzed and integrated to obtain the multidimensional performance data of the current performer.

5. The electronic guzheng playing method based on multi-touch interaction according to claim 1, characterized in that, The process of processing the multidimensional performance data based on the timbre recognition model to obtain the timbre parameters corresponding to the multidimensional performance data includes: Based on the timbre recognition model, the multidimensional performance data is classified to obtain fingering recognition features and timbre recognition features. The fingering recognition features are identified to determine the fingering of the current performer; The fingering and timbre recognition features are analyzed to obtain the timbre parameters.

6. The electronic guzheng playing method based on multi-touch interaction according to claim 1, characterized in that, The step of generating performance audio based on the timbre parameters includes: Based on the timbre parameters, a discrete spectrum and a time-domain envelope function are constructed; The target audio effect of the current performer is obtained, and the discrete spectrum and the envelope function are synthesized based on the target audio effect to generate the performance audio.

7. The electronic guzheng playing method based on multi-touch interaction according to claim 1, characterized in that, Before determining the timbre recognition model corresponding to the performance preference based on a preset performance parameter library, the method further includes: Obtain at least one target musical score input by the performer, and analyze at least one of the target musical scores to obtain the target fingering and target timbre parameters corresponding to each target musical score; Based on the multi-touch sensor array, multi-dimensional training data is collected when the performer plays according to each of the target scores; The multidimensional training data, target fingering, and target timbre parameters corresponding to each target musical score are analyzed to obtain a first mapping curve between the performer's performance preference information and the target fingering, as well as a second mapping curve between the multidimensional training data, target fingering, and target timbre parameters. Based on the first mapping curve and the second mapping curve, the pre-trained recognition model is constructed, and the pre-trained recognition model is trained based on the multi-dimensional training data and the target timbre parameters to obtain the timbre parameter recognition model; Based on the performance preference information and the timbre recognition model, the performance parameter library is generated.

8. The electronic guzheng playing method based on multi-touch interaction according to claim 7, characterized in that, The step of analyzing the multidimensional training data corresponding to each of the target scores, the target fingering, and the target timbre parameters to obtain the first mapping curve between the performer's performance preference information and the target fingering, and the second mapping curve between the multidimensional training data, the target fingering, and the target timbre parameters, includes: The multidimensional training data, the target fingering, and the target timbre parameters are time-aligned to obtain the training data time series, the fingering time series, and the timbre parameter time series. Based on a preset preference recognition model, the performance preference information of the performer is obtained by identifying the preference of the training data time series, the fingering time series, and the fingering time series preferences. The performance preference information includes the preference performance data corresponding to each target fingering and the target fingering and preference performance data corresponding to each target timbre parameter. Based on the preferred playing data corresponding to each of the target fingerings, the first mapping curve is generated, and based on the target fingerings and preferred playing data corresponding to each of the target timbre parameters, the second mapping curve is generated.

9. An electronic guzheng playing system based on multi-touch interaction, characterized in that, include: The performance preference recognition module is used to collect multi-dimensional performance data of the current performer based on a multi-touch sensor array, and to identify the performance preference of the current performer by recognizing the multi-dimensional performance data. The recognition model determination module is used to determine the timbre recognition model corresponding to the performance preference based on a preset performance parameter library; The timbre parameter acquisition module is used to process the multidimensional performance data based on the timbre recognition model to obtain the timbre parameters corresponding to the multidimensional performance data. The performance audio output module is used to generate performance audio based on the timbre parameters, and output the performance audio based on the audio output device corresponding to the electronic guzheng.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the electronic guzheng playing method based on multi-touch interaction as described in any one of claims 1 to 7.

Images