Performance data visualization method and system

Abstract

The invention relates to the field of playing data visualization, in particular to a playing data visualization method and system. The method comprises the steps that at least one section of pedal data of at least one pedal is obtained, and the pedal data comprises a pedal value and a time stamp of the pedal value; generating at least one section of labeling track according to the pedal data; wherein the labeling track comprises a first simulation track and a second simulation track; correspondingly, the method comprises the steps that when the pedal value is larger than a first pedal threshold value, a first simulation track is generated according to the pedal value, and the first simulation track is displayed in a first labeling form; when the pedal value is larger than a second pedal threshold value and smaller than or equal to the first pedal threshold value, a second simulation track is generated according to the pedal value, and the second simulation track is displayed in a second labeling form; and generating a combined track according to the at least one section of labeled track, and displaying the combined track on the music score. According to the invention, the smoothness and integrity of pedal data display can be improved, and the user experience is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of performance data visualization, in particular to a performance data visualization method and system. BACKGROUND

[0002] In the early stage of learning piano, some users press the right pedal for a long time without releasing it in order to play loudly, which results in poor performance. Some users practice pedal operation in the piano room, but they are in chaos when they perform on stage. These improper pedal operations are largely related to the proficiency of pedal technology. These problems need to be solved in learning so that users can gradually reach a higher level of piano performance.

[0003] Currently, the prior art also attempts to propose some precise reminding methods for piano pedal movements.

[0004] For example, patent application CN120510757A discloses a piano pedal movement amplitude indicating device for music teaching, which comprises a placement plate and an indicating structure arranged on the placement plate. The bottom of the placement plate is provided with a suction cup. The indicating structure comprises an indicating frame. The front surface of the indicating frame is provided with a soft pedal display screen, a mute pedal display screen and a sustain pedal display screen from left to right. A reminding member for reminding pedal switching is arranged on the indicating frame. The soft pedal display screen is provided with a first indicating member for displaying soft pedal movement. Through the arrangement of the indicating structure, the movement and amplitude of the piano pedal can be displayed, thereby realizing precise reminding of the piano pedal movement. The movement amplitude of the piano pedal can be precisely controlled by the practice personnel, which facilitates the teaching and display of the piano, improves the teaching effect of the piano, and is beneficial to the rapid learning of the piano.

[0005] For another example, patent application CN120412345A provides an intelligent piano teaching projection device and method based on computer vision. The device comprises a data acquisition and projection module, a calibration module, a data processing module, a side projection module and an auxiliary module. The intelligent piano teaching projection method based on the device comprises segmenting the overhead image of the performer in the piano key area collected by the data acquisition and projection module using the Mask2former image segmentation model. The finger skeleton modeling algorithm is used to construct the finger skeleton of the performer, to determine whether the current performance is correct. If the performance is incorrect, the light projection position of the data acquisition and projection module remains unchanged, prompting the performer to press the keys again. If the performance is correct, the next time the keys and pedals need to be pressed are projected to guide the performer. The present application does not need to modify the piano. By fixing the device to the piano, the piano guidance can be realized through projection.

[0006] However, the applicant notices that it is still difficult for the user to autonomously determine the gap between the actual playing and the expected playing effect. SUMMARY

[0007] The present application aims to provide a playing data visualization method, which partially solves or alleviates the above-mentioned deficiencies in the prior art, and can improve the fluency and integrity of the foot pedal data display.

[0008] In order to solve the above-mentioned technical problems, the present application specifically adopts the following technical solutions: The first aspect of the present application is to provide a playing data visualization method, comprising the steps of: S101, obtaining at least one foot pedal data of at least one foot pedal, and the foot pedal data comprising: a foot pedal value, and a time mark of the foot pedal value; S102, generating at least one annotation track according to the foot pedal data; wherein the annotation track comprises: a first simulation track, a second simulation track; correspondingly, S102 comprises the steps of: When the foot pedal value is greater than a first foot pedal threshold value, a first simulation track is generated according to the foot pedal value, and the first simulation track is displayed in a first annotation form; When the foot pedal value is greater than a second foot pedal threshold value and less than or equal to the first foot pedal threshold value, a second simulation track is generated according to the foot pedal value, and the second simulation track is displayed in a second annotation form; S103, generating a merged track according to at least one of the annotation tracks, and displaying the merged track on a score.

[0009] In some embodiments, the annotation track further comprises: a virtual track; correspondingly, S102 further comprises the steps of: When the foot pedal value is less than or equal to the second foot pedal threshold value, the foot pedal value is identified as a virtual target; Selecting a neighboring foot pedal value adjacent to the virtual target; wherein the difference between the time mark of the neighboring foot pedal value and the time mark of the virtual target is less than a preset time difference value; Generating a virtual track according to the neighboring foot pedal value and the virtual target, wherein the virtual track is displayed in a third annotation form.

[0010] In some embodiments, the step of generating a virtual track according to the neighboring foot pedal value and the virtual target comprises: (1) obtaining a neighboring foot pedal data group, and the neighboring foot pedal data group comprising: a first neighboring foot pedal data and a second neighboring foot pedal data, the first neighboring foot pedal data and the second neighboring foot pedal data being respectively the neighboring foot pedal value before and after the virtual target; (2) generating at least two predicted trajectories according to the adjacent foot pedal data set by using a virtual prediction method; wherein, step (2) comprises: generating a virtual point by using the adjacent foot pedal data set, the virtual point being used to define a starting height of a predicted trajectory; calculating a change trajectory of the adjacent foot pedal data set, and generating a virtual change trajectory according to the change trajectory; generating a predicted trajectory according to the starting height and the virtual change trajectory; (3) generating the virtual trajectory according to the at least two predicted trajectories.

[0011] In some embodiments, step (3) comprises: fusing the at least two predicted trajectories by using a trajectory fusion method to generate the virtual trajectory.

[0012] In some embodiments, the trajectory fusion method comprises one or more of the following: a direct connection method, a smooth transition method, and a trajectory alignment method based on an optimization method.

[0013] In some embodiments, the second adjacent foot pedal data is obtained from an external data source, the external data source is collected by a score database, and the external data source records a standard foot pedal value of at least one note node and a time label of the standard foot pedal value.

[0014] In some embodiments, the method further comprises the following steps: generating a first auxiliary trajectory according to the first foot pedal threshold, the first auxiliary trajectory being displayed in a fourth annotation form; and / or, generating a second auxiliary trajectory according to the second foot pedal threshold, the second auxiliary trajectory being displayed in a fifth annotation form.

[0015] In some embodiments, the method further comprises the following steps: obtaining a plurality of pieces of first foot pedal data in a first time period; generating a foot pedal score according to the plurality of pieces of first foot pedal data and an external data source; wherein, the external data source comprises: a standard foot pedal value, and a time label of the standard foot pedal value; judging whether the foot pedal score is less than a first preset score, if yes, generating a guide trajectory according to the external data source, and the guide trajectory being displayed in a sixth annotation form.

[0016] In some embodiments, the step of generating a guide trajectory according to the external data source comprises: obtaining the corresponding external data source under a second time period; generating a first guide trajectory according to the corresponding external data source; acquire a marked trajectory under the second time period; adopt a trajectory fusion method to generate the guide trajectory according to the marked trajectory and the first guide trajectory.

[0017] In a second aspect, the present application provides a performance data visualization system, comprising: a pedal data acquisition module configured to acquire at least one piece of pedal data of at least one pedal, wherein the pedal data comprises a pedal value and a time mark of the pedal value; a marked trajectory generation module configured to generate at least one piece of marked trajectory according to the pedal data, wherein the marked trajectory comprises a first simulation trajectory and a second simulation trajectory, and the marked trajectory generation module is further configured to: when the pedal value is greater than a first pedal threshold, generate the first simulation trajectory according to the pedal value, and display the first simulation trajectory in a first marked form; when the pedal value is greater than a second pedal threshold and less than or equal to the first pedal threshold, generate the second simulation trajectory according to the pedal value, and display the second simulation trajectory in a second marked form; a merged trajectory display module configured to generate a merged trajectory according to the at least one piece of marked trajectory, and display the merged trajectory on a musical score.

[0018] Beneficial technical effects: The pedal data display scheme is particularly suitable for piano teaching scenarios, and specifically: 1) The second simulation trajectory corresponding to invalid data and the first simulation trajectory corresponding to valid data are merged, which has at least the following effects in terms of visual display fluency: 1) The user's pedal action is regarded as a group of continuous action curves linked to the musical score, rather than isolated action points, and continuous display helps to associate muscle memory and reduce understanding difficulty. 2) Further, by observing invalid data, it can be determined whether the performer's control of the pedal is professional enough, for example, if the marked trajectory repeatedly jumps at the second pedal threshold, it indicates that the performer's foot strength control is unstable or the pedal position is not appropriate and needs to be adjusted. Therefore, displaying the process of changing the pedal from invalid value to valid value helps to understand the complete change path, acceleration and other pedal usage habits of the pedal data.

[0019] 2) By means of the fusion mechanism of the simulation pedal trajectory and the virtual pedal trajectory, a virtual trajectory that best meets the actual situation can be generated, which effectively avoids the sudden disconnection of the marked trajectory caused by blank pedal data, thereby ensuring that the pedal is complete and smooth in the pedal data visualization display process. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. In all the drawings, similar elements or parts are generally identified by similar reference signs. In the drawings, each element or part is not necessarily drawn according to the actual proportion. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 A flowchart of a performance data visualization method provided by an embodiment of the present application; Figure 2 An example diagram of a performance data visualization method provided by an embodiment of the present application; Figure 3 A structural schematic diagram of a performance data visualization system provided by an embodiment of the present application; Figure 4 A structural schematic diagram of an exemplary piano pedal system applicable to the present application; Figure 5 Another structural schematic diagram of an exemplary piano pedal system applicable to the present application; Figure 6 Another structural schematic diagram of an exemplary piano pedal system applicable to the present application.

[0022] Reference signs: 111, mounting plate; 112, mounting block; 113, external left pedal; 114, external middle pedal; 115, external right pedal; 211, mounting frame; 212, stepping motor; 213, connecting rod; 221, bottom plate; 222, connecting rod mechanism; 223, shock absorbing spring; 2221, transmission connecting rod; 2222, hinged support; 2223, connecting bolt; 3, piano pedal. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0024] Herein, the suffixes such as "module", "part", or "unit" used for an element of the present application are merely intended for facilitating explanation of the present application, and have no special meaning and do not constrict the meaning of an element. Therefore, "module", "part", or "unit" can be mixedly used.

[0025] Herein, the terms "upper", "lower", "inner", "outer", "front", "rear", "one side", "the other side", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are merely intended for convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and thus cannot be construed as limiting the present application. In addition, the terms "first", "second" are used for descriptive purposes only, and cannot be construed as indicating or implying relative importance.

[0026] Herein, unless explicitly defined and limited otherwise, the terms "mount", "provided with", "connected", and the like should be broadly understood, for example, "connected" can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, can be directly connected, or indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0027] Herein, "and / or" includes any and all combinations of one or more of the listed items.

[0028] Herein, "a plurality of" means two or more, that is, it includes two, three, four, five, etc.

[0029] In the present specification, the term "about" typically means + / - 5% of the stated value, more typically + / - 4% of the stated value, more typically + / - 3% of the stated value, more typically + / - 2% of the stated value, even more typically + / - 1% of the stated value, and even more typically + / - 0.5% of the stated value.

[0030] In the present specification, certain embodiments can be disclosed in a format that is a range. It is to be understood that such a "range" format is merely used for convenience and brevity and should be interpreted in a flexible manner to include not only the numerical values explicitly recited as the limits of the range, but also to include all the individual numerical values or sub-ranges within that range as if each numerical value and sub-range is explicitly recited. For example, a range of 1-6 should be interpreted to include the explicitly recited ranges of, for example, from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, and the like, as well as the individual numbers 1, 2, 3, 4, 5, and 6. This same principle applies to ranges derived from two or more ranges, for example, a range of 1-6 and a range of 4-8 can be interpreted as a range of 1-8.

[0031] Embodiment one: Please refer to Figure 1 The present application provides a performance data visualization method, in some embodiments, comprising the steps of: S101, obtaining at least one foot pedal data of at least one foot pedal, and the foot pedal data comprises: foot pedal value, and time mark of the foot pedal value; S102, generating at least one annotation track according to the foot pedal data; wherein the annotation track comprises: first simulation track, second simulation track; correspondingly, S102 comprises the steps of: When the foot pedal value is greater than the first foot pedal threshold value, generating the first simulation track according to the foot pedal value, and the first simulation track is displayed in the first annotation form; When the foot pedal value is greater than the second foot pedal threshold value and less than or equal to the first foot pedal threshold value, generating the second simulation track according to the foot pedal value, and the second simulation track is displayed in the second annotation form; S103, generating a merged track according to at least one of the annotation tracks, and displaying the merged track on the score.

[0032] It should be understood that the foot pedal data mentioned in the present application is not simply a foot pedal value, but a set of time series data with accurate time marks.

[0033] In some embodiments, the foot pedal value can refer to the depth value, angle value or corresponding force value of the foot pedal changed by the user's down action, which is obtained by the foot pedal sensor (for example, it can be an external position sensor or a position sensor). The time mark can be the precise time of collecting the foot pedal value, or the elapsed time since the start of the performance. In some embodiments, the present application preferably aligns the time marks of the foot pedal data and the playing data, and greatly improves the performance effect through the coordinated display of the two, and strengthens the effect of the pedal on the performance of the harmony, the melody, the rhythm or the style.

[0034] In some embodiments, generating an annotation track according to the foot pedal data comprises the steps of: converting the corresponding foot pedal value into coordinates in the foot pedal display diagram according to the time mark; connecting the coordinates adjacent to each other into line segments; and generating a continuous annotation track according to the plurality of adjacent line segments.

[0035] Preferably, the present application sets the first foot pedal threshold value and the second foot pedal threshold value, and displays the foot pedal values in different intervals in different annotation forms, which can improve the readability of the foot pedal data visualization display by distinguishing the intervals of different foot pedal values, and help users intuitively see whether the foot pedal data meets the angle or force requirements of the foot pedal, thereby improving the performance effect.

[0036] In some embodiments, the first pedal threshold generally refers to an effective threshold capable of triggering the weak tone, soft tone, and delay effect, that is, a pedal value greater than the first pedal threshold can be regarded as valid data, corresponding to the first simulation track, and displayed in the first annotation form.

[0037] In some embodiments, the second pedal threshold generally refers to an invalid threshold that can be collected by a sensor (for example, an external position sensor or an additional position sensor), but is not enough to trigger the weak tone, soft tone, and delay effect, that is, a pedal value greater than the second pedal threshold and less than or equal to the first pedal threshold can be regarded as invalid data, corresponding to the second simulation track, and displayed in the second annotation form.

[0038] It should be understood that the present application preferably combines the second simulation track corresponding to the invalid data and the first simulation track corresponding to the valid data, which has at least the following effects in terms of visual display fluency: 1) the present application regards the user's pedal action as a set of continuous action curves linked to the score, rather than a number of isolated action points, and continuous display helps to associate muscle memory and reduce understanding difficulty. 2) Further, by observing the invalid data, it can be determined whether the performer's control of the pedal is professional enough, for example, if the annotated track repeatedly jumps at the second pedal threshold, it indicates that the performer's foot strength control is unstable, or the pedal position is not appropriate, and needs to be adjusted. It can be seen that displaying the process of the pedal changing from invalid value to valid value helps the performer understand the complete change path, acceleration and other pedal usage habits of the pedal data.

[0039] In some embodiments, generating a merged track according to at least one of the annotated tracks can include the step of connecting multiple annotated tracks into a continuous merged track in the order of time markers.

[0040] In some embodiments, the first annotation form and the second annotation form can adopt different visual effects. For example, in order to facilitate the user to distinguish, the colors, transparencies and other visual effects adopted by the first annotation form and the second annotation form can be different.

[0041] It should be noted that the annotation forms mentioned in the present application can all be different display forms.

[0042] In some embodiments, the display form can be a static shape, a dynamic visual effect, or a combination of both, which can be color change (e.g., highlighting, gradient color), flashing effect (e.g., periodic flashing, pulsing light effect, water ripple effect), arrow indication (e.g., direction guide, dynamic path), icon and symbol, text annotation (e.g., brief description, numerical label), shadow and outline (e.g., projection effect, halo design, bold outline), animation effect (e.g., explosion, floating bubble), line thickening or highlighting, etc., without limitation.

[0043] Preferably, the first annotation form is different from the display form of the second annotation form.

[0044] In some embodiments, the annotation track further comprises a virtual track, and correspondingly, S102 further comprises the steps of: When the foot pedal value is less than or equal to the second foot pedal threshold value, the foot pedal value is identified as a virtual target; An adjacent foot pedal value adjacent to the virtual target is selected, wherein the difference between the time mark of the adjacent foot pedal value and the time mark of the virtual target is less than a preset time difference value; A virtual track is generated according to the adjacent foot pedal value and the virtual target, wherein the virtual track is displayed in a third annotation form.

[0045] In some embodiments, when the foot pedal value is less than or equal to the second foot pedal threshold value (which can be due to various reasons, such as the user not stepping on the foot pedal or stepping on the foot pedal with very small force, and the foot pedal value corresponding to the time mark is blank), the foot pedal value less than or equal to the second foot pedal threshold value can be identified as a virtual target.

[0046] That is, in the present embodiment, the foot pedal data is continuously displayed by classification according to the effectiveness, which can form a smooth guide line (i.e., a combined track) on the one hand. The smoothness of the guide line has a certain correlation with the music rhythm, so it is more convenient for the user to intuitively understand the actual foot pedal data. On the other hand, the guide line can be displayed differently based on the double classification or triple classification, so that the foot pedal playing state can be marked differently (i.e., the user can intuitively focus on the overall effect of the foot pedal).

[0047] It should be understood that the present application further improves the continuity of the annotation track by combining virtual and real. In the following, the virtual and real combination will be further introduced.

[0048] In some embodiments, an adjacent foot pedal value adjacent to the virtual target can be selected, wherein the difference between the time mark of the adjacent foot pedal value and the time mark of the virtual target is less than a preset time difference value.

[0049] Preferably, one or more foot pedal values closest in time to the virtual target can be selected as the adjacent foot pedal values in a foot pedal value range less than or equal to the second foot pedal threshold.

[0050] It should be understood that the present application determines the selection range of the adjacent foot pedal values within the preset time difference range, which ensures that the generation of the virtual trajectory conforms to the maximum possibility of the actual situation. For example, in a short foot pedal blank value (e.g., quickly lifting up between two down pedal actions), the use of the adjacent foot pedal values to construct the virtual trajectory can make the final labeled trajectory (i.e., which can include the first simulation trajectory, the second simulation trajectory, and the virtual trajectory) look smooth and continuously transitioned, rather than harshly truncated and starting from zero.

[0051] In some embodiments, the virtual trajectory can be generated according to the adjacent foot pedal values and the virtual target, including the steps of: (1) obtaining an adjacent foot pedal data set, and the adjacent foot pedal data set includes: first adjacent foot pedal data and second adjacent foot pedal data, the first adjacent foot pedal data and the second adjacent foot pedal data are respectively the adjacent foot pedal values located before and after the virtual target; (2) generating at least two predicted trajectories according to the adjacent foot pedal data set using a virtual prediction method; wherein step (2) includes: generating a virtual point using the adjacent foot pedal data set, the virtual point is used to define the starting height of the predicted trajectory; calculating the change trajectory of the adjacent foot pedal data set, and generating a virtual change trajectory according to the change trajectory; generating a predicted trajectory according to the starting height and the virtual change trajectory; (3) generating the virtual trajectory according to the at least two predicted trajectories.

[0052] Wherein, the virtual trajectory can be displayed in a third labeling form.

[0053] In some embodiments, the obtained adjacent foot pedal data set (or can be referred to as adjacent foot pedal data) can include first adjacent foot pedal data located before the virtual target and second adjacent foot pedal data located after the virtual target.

[0054] In some embodiments, a virtual point can be determined according to the speed or acceleration of the adjacent foot pedal data. It should be understood that the virtual point is very close to the foot pedal value of the adjacent foot pedal data, that is, the virtual point can be an approximate value or a predicted value of the adjacent foot pedal data.

[0055] Preferably, one adjacent foot pedal data can be directly used as a virtual point.

[0056] For example, in the pedal data less than or equal to the second pedal threshold, the maximum adjacent pedal data is taken as the virtual point.

[0057] In some embodiments, the virtual point is used to define the starting height of the predicted trajectory, i.e., to determine the starting point of the predicted trajectory.

[0058] In some embodiments, the change trajectory of the adjacent pedal data can be taken as the virtual change trajectory. Specifically, the direction of the change trajectory of the adjacent pedal data can be taken as the direction of the virtual change trajectory.

[0059] In some embodiments, taking the first adjacent pedal data as an example, calculating the change trajectory of the adjacent pedal data can include: 1) two first adjacent pedal data (time stamps T1, T2, corresponding pedal values V1, V2) can be selected from a plurality of first adjacent pedal data; 2) the change speed r (i.e., the slope of the change trajectory) is calculated based on the two first adjacent pedal data as V2-V1 / T2-T1.

[0060] In some embodiments, the adjacent pedal data can be determined according to the current collected real data, or can be determined according to the pre-stored historical data, or can be selected according to the current data and the historical data, as long as the adjacent pedal data meeting the confidence requirement can be determined, which is not limited here.

[0061] It should be understood that the change speed r calculated based on the preceding method is the trend of the pedal value changing with time.

[0062] In some embodiments, the slope of the change trajectory can be determined according to the change speed r between the adjacent pedal data.

[0063] In some embodiments, if multiple first adjacent pedal data are selected, a mathematical method such as linear regression or moving average can be used to calculate a change speed r that is more consistent with the actual situation.

[0064] In some embodiments, the virtual change trajectory can be generated according to the change trajectory, or the slope of the change trajectory can be taken as the slope of the virtual change trajectory.

[0065] It should be understood that the virtual change trajectory is a direction reference starting from the virtual point based on the change speed r of the adjacent pedal data. For example, if it is a linear prediction (i.e., the change speed r is constant), the virtual change trajectory is a straight line. If the change speed r itself is changing, a higher-order mathematical model can be used for prediction, which is not limited here.

[0066] In some embodiments, the predicted trajectory is generated according to the starting height (i.e., the virtual point) and the virtual change trajectory (direction reference).

[0067] Preferably, the present application can connect two pieces of predicted trajectory (the predicted trajectory before the virtual target is the first predicted trajectory; the predicted trajectory after the virtual target is the second predicted trajectory) to generate a virtual trajectory.

[0068] In some embodiments, the intersection of the two pieces of predicted trajectory can be identified as a virtual target.

[0069] In some embodiments, step (3) comprises: The trajectory fusion method is adopted to fuse at least two predicted trajectories to generate the virtual trajectory.

[0070] In some embodiments, the trajectory fusion method that can be taken includes one or more of the following: direct connection method, smooth transition method, trajectory alignment method based on optimization method.

[0071] In some embodiments, other existing technologies can also be used to fuse the predicted trajectories as long as the effect of smoothing the virtual trajectory is achieved, which is not limited here.

[0072] In some embodiments, the second adjacent foot pedal data is obtained from an external data source, the external data source is collected through a music score database, and the external data source records the standard foot pedal value of at least one note node and the time label of the standard foot pedal value.

[0073] Preferably, the external data source can be the standard foot pedal data corresponding to each music score.

[0074] In some embodiments, the second adjacent foot pedal data after the virtual target can be determined according to the external data source; further, the second predicted trajectory can be generated according to the second adjacent foot pedal data and the virtual target, and the specific generation method can refer to the first predicted trajectory.

[0075] In some embodiments, the present application preferably uses high-precision sensors (such as position sensors, external position sensors, force feedback sensors, etc.) devices, which can collect and transmit more accurate foot pedal data in real time, and at the same time through the display mode of virtual and real combination, so that the visual display of foot pedal data not only conforms to the actual situation, but also has a more smooth visual effect.

[0076] In summary, the present application can generate a virtual trajectory that conforms to the actual situation to the greatest extent through the virtual and real combination mode, i.e. through the fusion mechanism of simulated foot pedal data and virtual foot pedal data, effectively avoids the sudden disconnection of the labeled trajectory caused by blank foot pedal data, and ensures that the foot pedal is complete and smooth in the process of visual display of foot pedal data.

[0077] In some embodiments, the method further comprises the step of: generate a first auxiliary track according to the first pedal threshold, the first auxiliary track being displayed in a fourth annotation form; And / or, generate a second auxiliary track according to the second pedal threshold, the second auxiliary track being displayed in a fifth annotation form.

[0078] Preferably, the fourth annotation form is different from the display form of the fifth annotation form. Please refer to Figure 2 Different auxiliary tracks and corresponding annotation forms can be used to display different pedal thresholds, which can help users intuitively observe the interval of pedal data and help them correct the foot strength to the normal range without professional guidance.

[0079] In some embodiments, the method further comprises the steps of: obtaining a plurality of first pedal data in a first time period; generating a pedal score according to the plurality of first pedal data and an external data source; wherein the external data source comprises a standard pedal value and a time label of the standard pedal value; determining whether the pedal score is less than a first preset score, and if so, generating a guide track according to the external data source, and the guide track being displayed in a sixth annotation form.

[0080] In some embodiments, generating a pedal score according to the plurality of first pedal data and an external data source comprises the steps of: 1) calculating the difference degree (for example, which can be represented by a ratio) between the first pedal data and the standard pedal value (obtained according to the external data source); for example, if the first pedal data reflects that the user's pedal depth is 10% and the corresponding standard pedal depth is about 20%, the difference degree between the first pedal data and the standard pedal value can be calculated as 50%, and the score can be 50 points. 2) calculating the average difference degree of a plurality of difference degrees in the first time period (for example, one difference degree can give one score), 3) the average difference degree (i.e. the average of a plurality of scores) can be taken as the pedal score.

[0081] In some embodiments, the pedal score can also be obtained according to a preset score table, wherein the preset table includes the difference degree between the first pedal data and the standard pedal value and the corresponding pedal score.

[0082] In some embodiments, if the pedal score is less than the first preset score, a guide track can be generated according to the external data source.

[0083] Specifically, the generation method of the guide track can be: aligning the time label of the standard external data source with the actual pedal data, and generating a corresponding annotation track according to the standard pedal value. Preferably, the guide track is displayed in a sixth annotation form.

[0084] In some embodiments, the guiding track can be displayed synchronously with the merging track, so as to facilitate comparison by the user and guide the user.

[0085] In some embodiments, the step of generating the guiding track according to the external data source comprises: acquiring the external data source corresponding to the second time period; generating a first guiding track according to the corresponding external data source; acquiring a labeled track in a time period adjacent to the second time period; generating the guiding track by fusing the labeled track and the first guiding track by using a track fusion method.

[0086] Preferably, in some embodiments, when the foot pedal score is less than a first preset score and greater than a second preset score, a first guiding track of a first length is generated; when the foot pedal score is less than or equal to the second preset score, a first guiding track of a second length is generated, and the first length is less than the second length.

[0087] In this embodiment, guiding tracks of different lengths are provided for different foot pedal states, so as to provide guidance to the user in time when the user's foot pedal state is not good, and at the same time, through flexible adjustment of the length, the user can be guided while avoiding or reducing excessive external interference, so that the user can more autonomously practice the pedal.

[0088] In some embodiments, the track fusion method can be used to generate the guiding track by fusing the labeled track and the first guiding track.

[0089] In some embodiments, generating the first guiding track according to the corresponding external data source can comprise the step of aligning the standard external data source with the time label of the actual foot pedal data, and generating the corresponding first guiding labeled track according to the standard foot pedal value.

[0090] It should be understood that the guiding track generated by fusing the real data (labeled track) and the standard data (first guiding labeled track) is a dynamic track based on the current level of the user and gradually approaching the standard, rather than a static track based on the standard data, so as to play a guiding role and the difficulty for the user is not too great.

[0091] The track fusion can be achieved by using existing technical means such as interpolation algorithm.

[0092] In some embodiments, the first time period and the second time period can be a period of performance, for example, ten minutes.

[0093] See Figure 3The application further provides a performance data visualization system, which comprises: a foot pedal data acquisition module configured to acquire at least one piece of foot pedal data of at least one foot pedal, and the foot pedal data comprises a foot pedal value and a time mark of the foot pedal value; a marked trajectory generation module configured to generate at least one piece of marked trajectory according to the foot pedal data, wherein the marked trajectory comprises a first simulation trajectory and a second simulation trajectory, and the marked trajectory generation module is further configured to: when the foot pedal value is greater than a first foot pedal threshold value, generate the first simulation trajectory according to the foot pedal value, and the first simulation trajectory is displayed in a first marked form; when the foot pedal value is greater than a second foot pedal threshold value and less than or equal to the first foot pedal threshold value, generate the second simulation trajectory according to the foot pedal value, and the second simulation trajectory is displayed in a second marked form; a merged trajectory display module configured to generate a merged trajectory according to the at least one piece of marked trajectory, and display the merged trajectory on a musical score.

[0094] It should be understood that the system can also be used to perform the steps described in any embodiment of the application.

[0095] In order to more clearly introduce the application scenario, an exemplary piano foot pedal system (see Figure 4 , Figure 5 , Figure 6 ) to which the application can be applied is introduced as follows: The piano foot pedal system comprises an external unit, an additional unit and a control unit, the external unit is independently arranged outside the piano, the additional unit is arranged on the piano, and the external unit and the additional unit are electrically connected with the control unit.

[0096] Specifically, the control unit adopts a microcontroller unit (MCU), which can be independently arranged or integrated in the external unit or the additional unit.

[0097] The external unit comprises an external pedal module and a data acquisition module, the external pedal module comprises at least one external pedal, the data acquisition module is configured to acquire the stepping data of the external pedal, and the control unit is configured to control the additional unit to step the corresponding piano pedal according to the stepping data.

[0098] The external pedal module further comprises a mounting plate 111 and a mounting block 112 fixedly connected to the top of the mounting plate 111, and the external pedal comprises an external left pedal 113, an external middle pedal 114 and an external right pedal 115 which are hingedly arranged side by side on one side of the mounting block 112.

[0099] In some embodiments, a buffer spring is connected between each of the outer left pedal 113, the outer middle pedal 114, the outer right pedal 115 and the mounting plate 111. The buffer spring not only buffers the user's stepping force on the outer pedal to avoid collision between the outer pedal and the mounting plate 111, but also helps the outer pedal to reset after the user releases the stepping force on the outer pedal.

[0100] The data acquisition module includes an outer position sensor electrically connected with the control unit. The outer position sensor is configured to acquire the stepping height data (also referred to as stepping value or foot stepping value in some other embodiments of the present application) of the outer left pedal 113 and the outer right pedal 115, and the control unit controls the installation unit to step the corresponding piano pedal 3 to the corresponding height according to the stepping height data.

[0101] Alternatively, in some embodiments, the data acquisition module includes a force feedback sensor instead, which is configured to acquire the user's stepping force (i.e. stepping value or foot stepping value).

[0102] Although the upright piano and the grand piano both include left, middle and right pedals, only the piano left pedal (soft pedal) and the piano right pedal (sostenuto pedal) need to be controlled in gray scale, and the piano middle pedal only needs to be controlled in on / off. Therefore, in the present embodiment, the installation unit includes two stepping driving modules, which are respectively arranged for the piano left pedal and the piano right pedal to step the piano left pedal and the piano right pedal according to the stepping height data of the outer left pedal 113 and the outer right pedal 115.

[0103] The stepping driving module includes a mounting frame 211 and a stepping motor 212. The mounting frame 211 is fixedly connected to the piano, and the stepping motor 212 is vertically fixedly connected to the mounting frame 211 and electrically connected with the control unit. The stepping motor 212 is a linear motor including a stator and a mover. The stator is in a cylindrical shape, and the mover is vertically slidingly inserted into the stator. The stator is fixedly connected to the mounting frame 211. The bottom end of the mover is in transmission connection with the corresponding piano pedal 3 to drive the corresponding piano pedal 3 to move downward, thereby achieving the stepping of the piano pedal 3.

[0104] The present embodiment discards the traditional driving mode of rotary motor + gear set / screw rod mechanism, and adopts a linear motor as the stepping motor 212. The stator and the mover of the linear motor directly generate thrust through magnetic field interaction, so that the mover slides in the stator in the vertical direction without any intermediate mechanical reversing or deceleration link. This change eliminates the periodic noise and transmission gap caused by the meshing of the gear set or the friction of the screw rod. Therefore, when the stepping driving module steps the corresponding piano pedal according to the stepping height data, the mover can keep silent and respond sensitively during the high-frequency reciprocating motion in the stator, from the original tens of milliseconds to sub-milliseconds.

[0105] The top end of the mover is coaxially fixedly connected with a connecting rod 213, the top end of the connecting rod 213 is fixedly connected with an added position sensor, and the added position sensor is electrically connected with the control unit. The added position sensor is used to detect the moving distance of the mover. If the detection result shows that the moving distance of the mover is the same as the pedal height data, the control unit controls the pedal motor 212 to be closed, and the pedal operation on the piano pedal 3 is stopped.

[0106] In the embodiment, the external position sensor and the added position sensor can adopt a proximity switch (such as an inductive or capacitive non-contact switch) or a swing head switch (also known as a lever type limit switch or a micro switch) to realize a position detection accuracy of ±0.1 mm: 1) The system can accurately restore the formation details of the user's pedal operation, and realize fine control of the pedal effects such as sustain and soft; 2) The pedal driving response of the added unit is synchronized with the user's operation height, which improves the real-time performance and authenticity of the performance. In addition, the added position sensor is directly fixedly connected to the top of the mover through the connecting rod 213. Compared with the external or separated sensor, this integrated transmission + sensing structure not only shortens the signal transmission path and reduces the wiring complexity, but also avoids the response delay and data drift caused by component misalignment or secondary installation errors, thereby ensuring the detection efficiency and accuracy.

[0107] In some embodiments, the height sensor, the external position sensor and the added position sensor can also adopt a high-resolution linear encoder.

[0108] In some embodiments, the external position sensor and the added position sensor can be replaced by a force feedback sensor, or both can be used together.

[0109] In some embodiments, the added unit further comprises two connecting rod transmission modules, the connecting rod transmission modules are one-to-one arranged with the pedal driving modules, and the connecting rod transmission modules are connected between the output ends of the pedal driving modules and the corresponding piano pedals 3.

[0110] Specifically, the connecting rod transmission module comprises a bottom plate 221 and a connecting rod mechanism 222 arranged on the bottom plate 221, the connecting rod mechanism 222 comprises a transmission connecting rod 2221, a hinged support 2222 and a connecting bolt 2223, the hinged support 2222 is fixedly connected to the top of the bottom plate 221, and the non-end of the transmission connecting rod 2221 is hinged to the hinged support 2222. One end of the transmission connecting rod 2221 is fixedly connected to the bottom end of the straight guide rod through the connecting bolt 2223, and the other end of the transmission connecting rod 2221 is fixedly connected to the corresponding piano pedal 3 through the connecting bolt 2223.

[0111] The treading motor 212 is used to drive one end of the transmission connecting rod 2221 to move upward, so that the other end of the transmission connecting rod 2221 drives the corresponding piano pedal 3 to move downward, thereby realizing the treading of the piano pedal 3.

[0112] In some embodiments, the connecting rod transmission module further comprises a damping spring 223 connected between the transmission connecting rod 2221 and the bottom plate 221, and the damping spring 223 is located between the hinge support 2222 and the piano pedal 3. The damping spring 223 not only can buffer the force applied by the transmission connecting rod 2221 to the piano pedal 3, so as to avoid that the structure is damaged due to the too strong force, but also can assist the transmission connecting rod 2221 to reset after the force applied by the transmission connecting rod 2221 to the piano pedal 3 is ended.

[0113] In some embodiments, the cross section of the transmission connecting rod 2221 is square, the bottom end of the straight guide rod is fixedly connected with a flange plate, a square hole is opened on the flange plate, the end of the transmission connecting rod 2221 away from the piano pedal 3 is inserted into the square hole, and the transmission connecting rod 2221 and the flange plate are fixedly connected by bolts. The size of the square hole matches the cross section size of the transmission connecting rod 2221, that is, the inner wall of the square hole is in clearance fit with the outer wall of the transmission connecting rod 2221, so as to limit the transmission connecting rod 2221 and avoid that the transmission connecting rod 2221 rotates in the use process, thereby ensuring the accuracy of force transmission.

[0114] It should be noted that, in this document, the terms "comprising", "containing", or any other similar term are intended to encompass non-exclusive inclusion, so that a process, method, article or device that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0115] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and a general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a computer terminal (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0116] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection of the present application.

Claims

1. A performance data visualization method, characterized by, Including steps: S101, obtaining at least one segment of pedaling data of at least one pedal, wherein the pedaling data includes: a pedaling value and a time stamp of the pedaling value; S102, generating at least one labeled trajectory according to the pedaling data; wherein the labeled trajectory includes: a first simulation trajectory and a second simulation trajectory; correspondingly, S102 includes the steps of: When the pedaling value is greater than a first pedaling threshold, a first simulation trajectory is generated according to the pedaling value, and the first simulation trajectory is displayed in a first annotation form; When the pedaling value is greater than a second pedaling threshold value and less than or equal to the first pedaling threshold value, a second simulation trajectory is generated according to the pedaling value, and the second simulation trajectory is displayed in a second annotation form; S103: Generate a merged trajectory based on at least one segment of the marked trajectory, and display the merged trajectory on the music score.

2. The method of claim 1, wherein, The marked track also includes: a virtual track; correspondingly, S102 also includes the steps of: When the pedal value is less than or equal to the second pedal threshold, the pedal value is identified as a virtual target; Selecting a neighboring step value that is adjacent to the virtual target; wherein a difference between a time stamp of the neighboring step value and a time stamp of the virtual target is less than a preset time difference; A virtual track is generated according to the adjacent pedal values ​​and the virtual target, wherein the virtual track is displayed in a third annotation format.

3. The method of claim 2, wherein, The step of generating a virtual track according to the adjacent footstep value and the virtual target comprises: (1) Acquiring a neighboring pedaling data group, wherein the neighboring pedaling data group includes: first neighboring pedaling data and second neighboring pedaling data, wherein the first neighboring pedaling data and the second neighboring pedaling data are the neighboring pedaling values ​​located before and after the virtual target, respectively; (2) generating at least two predicted trajectories based on the adjacent pedaling data sets using a virtual prediction method; wherein step (2) includes: generating a virtual point using the adjacent pedal data set, wherein the virtual point is used to define a starting height of a predicted trajectory; Calculating a change trajectory of the adjacent pedal data group, and generating a virtual change trajectory based on the change trajectory prediction; generating a predicted trajectory according to the starting height and the virtual change trajectory; (3) Generating the virtual trajectory according to at least two predicted trajectories.

4. The method of claim 3, wherein, Step (3) includes: A trajectory fusion method is adopted to fuse at least two predicted trajectories to generate the virtual trajectory.

5. The method of claim 4, wherein, The trajectory fusion method includes one or more of the following: a direct connection method, a smooth transition method, and a trajectory alignment method based on an optimization method.

6. The method of claim 3, wherein, The second adjacent pedal data is obtained from an external data source, which is collected through a music score database, and the external data source records a standard pedal value of at least one note node and a time tag of the standard pedal value.

7. The method of claim 1, wherein, Also includes the steps: generating a first auxiliary trajectory according to the first stepping threshold, wherein the first auxiliary trajectory is displayed in a fourth annotation format; And / or, a second auxiliary trajectory is generated according to the second footstep threshold, and the second auxiliary trajectory is displayed in a fifth annotation format.

8. The method of claim 1, wherein, Also includes the steps: Obtaining a plurality of pieces of first pedal data in a first time period; Generating a pedal score according to the plurality of pieces of first pedal data and an external data source, wherein the external data source comprises a standard pedal value and a time label of the standard pedal value; If the pedal score is less than a first preset score, generating a guide track according to the external data source, and displaying the guide track in a sixth annotation form.

9. The method of claim 8, wherein, The step of generating a guide track according to the external data source comprises: Obtaining the corresponding external data source in a second time period; Generating a first guide track according to the corresponding external data source; Obtaining an annotation track in the second time period; Fusing the annotation track and the first guide track to generate the guide track by using a track fusion method.

10. A performance data visualization system, characterized by, Comprise: A pedal data acquisition module, configured to obtain at least one piece of pedal data of at least one pedal, and the pedal data comprises a pedal value and a time label of the pedal value; An annotation track generation module, configured to generate at least one piece of annotation track according to the pedal data, wherein the annotation track comprises a first simulation track and a second simulation track, and the annotation track generation module is further configured to: When the pedal value is greater than a first pedal threshold, generating the first simulation track according to the pedal value, and displaying the first simulation track in a first annotation form; When the pedal value is greater than a second pedal threshold and less than or equal to the first pedal threshold, generating the second simulation track according to the pedal value, and displaying the second simulation track in a second annotation form; A merged track display module, configured to generate a merged track according to at least one piece of the annotation track, and display the merged track on a score.

Images