Display method of playing keys and smart musical instrument

Abstract

The application relates to the field of digital music instruments, and particularly to a display method of playing keys and an intelligent music instrument. The method comprises the following steps: acquiring a plurality of groups of motion capture results, each group of motion capture results comprising the identification of a key, key motion data, and the motion capture result being associated with a corresponding third timestamp; identifying the key touch dynamic characteristics of a player according to the plurality of groups of motion capture results in a preset time period to match the playing skill data in the preset time period; generating a plurality of groups of key pressing images and corresponding first playing prompt icons according to the plurality of groups of key motion data; generating a plurality of second playing prompt icons according to the playing skill data; and superimposing the key pressing images and the first and second playing prompt icons based on the third timestamp and the identification of the key to generate a continuous playing animation. The depth and individualization level of the visual feedback in the playing animation are enhanced, so that the viewers can more quickly and clearly acquire and understand the playing behavior.

Description

[0001] Priority application This application claims priority to Chinese invention patent application filed on January 20, 2025 [Application No. 2025100899468], entitled "Method for displaying piano keys and intelligent musical instrument", which is incorporated herein by reference in its entirety. Technical Field

[0002] This application relates to the field of musical instrument digitization, and in particular to a method for displaying piano keys and a smart musical instrument. Background Technology

[0003] With the development of technology, the digital expression of music and art is becoming increasingly diverse, with broad application prospects in education, entertainment, virtual reality, and other fields. Among them, the motion capture of musical instrument performances and its transformation into animation is a highly promising technology. It combines knowledge from multiple disciplines such as music, computer science, and visual arts, providing viewers with an unprecedented sense of immersion and engagement.

[0004] For example, Chinese invention patent application CN104516670 A discloses a tactile piano key, relating to the field of manufacturing technology for simulated electronic musical instruments. It includes a central processing unit (CPU), connected to which are a display screen and a touch screen. The CPU contains a comprehensive processing and operating system module, connected to which are a setting module, a contact area recognition module, a playing force analysis module, a note recognition module, a display module, and a data transmission and communication module. This solves the problems of traditional piano keys requiring fixed positions, large size, difficult manufacturing, and high cost. Another example is Chinese invention patent application CN111259863A, which discloses a method for detecting / displaying playing hand shapes, a medium, a piano, a terminal, and a server. The method for detecting playing hand shapes includes: acquiring the player's playing hand shape data; processing the player's playing hand shape data to obtain playing hand shape information. The described hand position detection method can automatically acquire the player's hand position data and thus obtain hand position information, which helps the player maintain the correct hand position and develop good playing habits.

[0005] However, existing digital instrument performance methods struggle to cope with the complex scenarios in piano performance, especially when dealing with complex musical works or advanced performance techniques. Their flexibility and adaptability are low, resulting in significant limitations in terms of intelligence and practicality. Summary of the Invention

[0006] The main objective of this application is to provide a method for displaying piano keys and a smart musical instrument. To solve the aforementioned technical problems, this application specifically adopts the following technical solution: The first aspect of this application is to provide a method for displaying an animation of playing piano keys, comprising the steps of: S101. Obtain multiple sets of motion capture results. Each set of motion capture results includes: the identifier of the piano key, the piano key motion data, the piano key motion data including: force, and / or displacement, and the motion capture result is associated with a corresponding third timestamp. S102. Based on multiple sets of motion capture results within a preset time period, identify the touch dynamic features of the performer within the preset time period, and match the performance skill data within the preset time period based on the touch dynamic features. The performance skill data is associated with multiple third timestamps corresponding to the preset time period. S103. Generate multiple sets of piano key pressing images and multiple sets of first performance prompt icons corresponding to the piano key pressing images based on the multiple sets of piano key movement data, wherein the first performance prompt icons include at least one of pressing pressure icon, pressing displacement icon, and pressing speed icon; generate a plurality of second performance prompt icons based on the performance skill data; S104. Based on the third timestamp and the key identifier, the key pressing image, the first performance prompt icon, and the second performance prompt icon are superimposed to generate a continuous performance animation.

[0007] In some embodiments, the method further includes: monitoring the total number of the first performance prompt icon and the second performance prompt icon in the performance animation; when the total number of icons is greater than the first density number, setting the display priority of each of the first performance prompt icon and the second performance prompt icon according to a preset display weight rule; adding corresponding visual emphasis elements to some of the first performance prompt icons and / or some of the second performance prompt icons according to the display priority, and rendering and displaying the visual emphasis elements.

[0008] In some embodiments, the display priority of the second performance prompt icon is higher than that of the first performance prompt icon.

[0009] In some embodiments, the method further includes: when the total number of icons is greater than the second density number, hiding a portion of the first performance prompt icon according to the display priority, so that the total number of icons is less than or equal to the second density number, wherein the second density number is greater than the first density number.

[0010] In some embodiments, the method further includes: determining the key pressing force based on the key movement data at time t; when the key pressing force is less than a first preset force, adding and rendering a first visual emphasis element to mark the first performance prompt icon corresponding to the key pressing force; and / or, when the key pressing force is greater than a second preset force, adding and rendering a second visual emphasis element to mark the first performance prompt icon corresponding to the key pressing force; wherein the second preset force is greater than the first preset force.

[0011] In some embodiments, the method further includes: when the pressure applied to the piano key is less than the first preset pressure, and / or when the pressure applied to the piano key is greater than the second preset pressure, updating the first performance prompt icon corresponding to the pressure applied to the piano key to a third performance prompt icon, wherein the display priority of the third performance prompt icon is higher than the display priority of the first performance prompt icon.

[0012] In some embodiments, S104 includes: displaying the first performance prompt icon corresponding to the key press image within a first preset range of the key press image; and / or acquiring multiple key movement data over a preset time period corresponding to the performance skill data as a type of key movement data; analyzing the distribution density of the first performance prompt icon corresponding to the type of key movement data, determining the dense distribution area with the highest distribution density, and displaying the second performance prompt icon corresponding to the performance skill data within a second preset range from the dense distribution area.

[0013] In some embodiments, S102 includes: counting the number of motion captures in the motion capture results within the preset time period, and calculating the data fluctuation value of the piano key motion data within the preset time period; when the touch dynamic feature identifies that the number of motion captures is greater than a preset number and the data fluctuation value is less than a preset fluctuation value, the playing technique data within the preset time period is matched as a legato technique; and / or, calculating the span distance of the piano keys based on the piano key identifiers and a third timestamp in multiple sets of motion capture results within the preset time period; when the touch dynamic feature identifies that the span distance is greater than a preset distance, the playing technique data within the preset time period is matched as a large-span jump technique.

[0014] A second aspect of this application is to provide an intelligent musical instrument, the intelligent musical instrument comprising: piano keys; Motion capture device, used to acquire multiple sets of motion capture results; Memory is used to store computer programs; A processor is configured to execute the computer program and, while executing the computer program, implement the steps of the method for displaying piano keys as provided in any embodiment of this application.

[0015] Beneficial effects: This application proposes a method for displaying piano key playing, which uses motion capture results to reproduce the animation effect of key pressing and generates two types of prompt icons with different functional characteristics. The differentiated display of the icons realizes multi-dimensional performance information feedback, which significantly enhances the depth and personalization of visual feedback in the performance animation, enabling viewers to obtain and understand the performer's performance behavior more quickly and clearly.

[0016] Clear and easy-to-understand performance information (such as the first performance prompt icon) meets the needs of beginners for immediate feedback. Professional skill guidance (such as the second performance prompt icon) supports advanced learners in gaining a deeper understanding of complex performance techniques and artistic expression. Simultaneously, corresponding visual emphasis elements are added to the rendering of different performance behaviors, differentiating icons of varying importance and connotations. This allows users to intuitively identify each pressing action and performance technique, and quickly obtain key information. Therefore, it effectively assists viewers at different levels in understanding and mastering performance details, comprehensively covering user needs, improving the viewing experience, learning effectiveness, and the accuracy of professional assessments, thereby enhancing the practicality of the performance animation.

[0017] This application also provides a dynamic multi-level icon display management mechanism, which formulates different display strategies for two types of icons, and further adopts different processing strategies dynamically under different density thresholds according to the total number of icons, and introduces the display priority of icons to implement the processing strategies, so that the display of multiple icons in the performance animation is orderly, such as visual enhancement strategy and icon hiding strategy, which optimizes the efficiency of information presentation and ensures that viewers can always obtain the most relevant and important performance information in the performance animation, thus highly adapting to the complex scenes in piano performance. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of this application; for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is a schematic diagram of the time axis correlation of collected data provided in an embodiment of this application; Figure 2 This is a schematic flowchart illustrating a method for displaying piano keys according to an embodiment of this application; Figure 3 This is a display screen of a performance animation provided in an embodiment of this application; Figure 4 This is another display screen of the performance animation provided in the embodiments of this application; Figure 5 This is yet another display screen for a performance animation provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] In this document, suffixes such as “module,” “part,” or “unit” used to denote elements are used only for illustrative purposes and have no specific meaning in themselves. Therefore, “module,” “part,” or “unit” may be used interchangeably.

[0022] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," and "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0024] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0025] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0026] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0027] With the development of technology, the digital expression of music and art has become increasingly rich, showing broad application prospects in education, entertainment, virtual reality and other fields. However, existing technologies are still insufficient when dealing with piano performance scenarios, especially when handling complex musical works or advanced performance techniques. The flexibility and adaptability of digital expression are low, the actual application effect is poor, and it cannot meet the user's needs.

[0028] Based on this, this application proposes a method for displaying piano key playing, which uses motion capture results to reproduce the animation effect of key pressing and generates two types of prompt icons with different functional characteristics. The differentiated display of the icons realizes multi-dimensional performance information feedback, which significantly enhances the depth and personalization of visual feedback in the performance animation, enabling viewers to obtain and understand the performer's performance behavior more quickly and clearly.

[0029] In the field of musical instrument performance, motion capture technology aims to record the performer's body movements and convert this data into a digital format to create realistic animations or provide feedback to improve playing skills. This technology relies on sensors and algorithms to identify parameters such as the performer's finger position, force, and speed.

[0030] For example, advanced motion capture devices are used to acquire multiple sets of motion capture results to record the hand postures and key movements of a real performer.

[0031] For example, this application provides a motion capture method, including the steps of: S1, acquire multiple sets of motion data for at least one first piano key, the motion data including: force, and / or displacement, the motion data being associated with a first timestamp; Specifically, the force here can be the pressure exerted on the piano key, which is used to provide feedback on the user's finger pressure; the displacement can be the pressing depth of the key, or the rotation angle of the key at a certain time. For example, in some embodiments, a corresponding sensor is provided below the key, which can detect the displacement of the corresponding key in real time.

[0032] S2, when the motion data indicates that the first key is pressed between the first time t1 and the second time t2, multiple sets of first hand images of the object are collected between the third time t3 and the fourth time t4; wherein the third time t3 is earlier than the first time t1 and the fourth time t4 is later than the second time t2; the first hand images are associated with a second timestamp. S3, perform a pressing action or a first hand with a pressing tendency based on the first hand image recognition; In some embodiments, at least one image acquisition module (such as a camera) is used to acquire an image of the hand of an object (or user).

[0033] In some embodiments, the camera may include any suitable device capable of acquiring images and converting them into electrical signals. The camera may include, but is not limited to, a video photometer, an infrared scanner, a video camera, etc.

[0034] In some embodiments, one or more cameras may be configured to capture images of the hand. For example, in some embodiments, a first camera may be configured to capture lateral images of the hand, capable of identifying the degree of finger flexion. As another example, in some embodiments, a second camera may be configured above the piano keys to identify the degree of pressure applied by each finger.

[0035] In some embodiments, the collected data may also be one or more video data consisting of multiple sets of hand images.

[0036] S4, identify the first finger in the first hand that performs the pressing action or has the pressing tendency; When an object plays a piece of music, the processor can acquire hand image information from the camera, recognize the hand image, extract user gesture information, that is, recognize the hand (such as distinguishing between the left and right hands), and then recognize the pressing action or the first finger with the pressing tendency based on the degree of bending and pressing of each finger in the hand.

[0037] In some embodiments, when the degree of bending of a finger is large, it can be preliminarily determined that the finger has made a pressing action; or, in some embodiments, when the degree of bending of a finger is relatively small, it can be determined that it may produce or perform a pressing action, and therefore it is preliminarily determined that the finger has a pressing tendency.

[0038] In some embodiments, when multiple people are involved in a medley, the two objects (or even multiple objects) can be initially identified by using hand features (such as hand size, skin texture, wrist joint features, etc.). That is, the objects are first associated with hand images, and then left and right hands and fingers are distinguished.

[0039] S5, identify the second piano key associated with the first finger; In some embodiments, the camera can calculate or infer the actual position of the first finger at the current moment from the hand image. Matching the actual position with the position of the piano keys can identify the second key actually operated by the first finger, such as by obtaining the identifier of the second key. The identifier can be a key number (ID), name, or other data used to identify the key.

[0040] For example, in some embodiments, the information about the piano keys is predetermined, such as the coordinates of the keys or the relative positions of multiple keys, or the positional relationships of the keys can be recorded using images (e.g., taking one or more images including parts or all of the keys). Correspondingly, when it is necessary to locate the key played by a finger, the position information can be determined by comparing the position of the key observed in the image with the predetermined position, or the real-time image observed by the camera can be compared with the recorded image to identify the key.

[0041] S6, determine whether the identifier of the second key matches that of the first key. If so, generate at least one set of motion capture results between the first moment and the second moment. The motion capture results include: the first hand and the first finger used to perform the pressing action, the identifier of the key, and the corresponding motion data. The motion capture results are associated with a corresponding third timestamp.

[0042] In some embodiments, the method further includes the steps of: acquiring multiple sets of second hand images, the second hand images being associated with a fourth timestamp; identifying a hand performing a pressing action or having a pressing tendency based on the multiple sets of second hand images; identifying a finger from the hand performing the pressing action or having the pressing tendency; identifying a third piano key associated with the finger, and expected data for the third piano key, the expected data including: pressing time, and / or pressing force; acquiring the motion data corresponding to the third piano key and the fourth timestamp; determining whether the expected data and the motion data match; if not, correcting the motion capture result.

[0043] In other words, this embodiment further provides an error identification and correction function. For example, if hand image analysis indicates that a finger has pressed or tended to press a piano key (i.e., the third key) for an extended period, but the expected data is not acquired for that key (e.g., only a brief signal length is acquired, which differs significantly from the prediction result of the hand image), then it is considered that there may be a data acquisition or analysis error, and the data processing process needs to be corrected.

[0044] For example, in some embodiments, when the first finger corresponding to the first key cannot be identified, the method further includes the step of: determining whether the first hand and the second hand intersect based on the third hand image; if not, identifying at least one second finger in the first hand that failed to generate the motion capture result; wherein, when only one second finger is acquired, the second finger is associated with the first key.

[0045] In this embodiment, when it is detected that the first key has been pressed by an object, but no corresponding finger (i.e., the first finger) is associated with it, an elimination method can be used to query for potentially associated fingers. Specifically, in this embodiment, when there are unassociated fingers in the object's left hand, the system first uses image retrieval to determine whether the object's left and right hands are crossed. If they are not crossed, it checks whether the other fingers of the left hand have been successfully matched and generated corresponding motion capture results. If so, it queries for fingers that have not successfully generated motion capture results. When there is a finger that has not successfully generated motion capture results, it is recommended to associate that finger with the first key.

[0046] In some embodiments, when the result of determining whether the first hand and the second hand intersect based on the third hand image is yes, the method further includes the steps of: acquiring multiple sets of fourth hand images between a fifth time t5 and a sixth time t6; wherein the fifth time t5 is later than the third time t3, and the sixth time t6 is later than the fourth time t4; the fourth hand images are associated with a fifth timestamp; identifying first positions of the first hand and the second hand at at least two first time nodes based on the fourth hand images, the first time nodes being located between t5 and t3; predicting second positions of the first hand and the second hand at a second time node based on the positions of the at least two first time nodes, the second time nodes being located between t3 and t4; predicting a third position of the first finger based on the second position; and querying multiple sets of associated motion data based on the third position.

[0047] This embodiment describes another error correction method when hands are crossed. For example, when an object's left and right hands are crossed, the hand position during the crossing period is predicted using at least one segment of image data before and after the crossing. Furthermore, the position of the fingers can be predicted based on the image data before and after the crossing. The prediction result is then matched with the actual detected sensor data (i.e., motion data). If the match is successful, the current result is used to complete the correction.

[0048] For a clearer description of the technical solution adopted in this application, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic diagram of the time axis correlation of collected data provided in an embodiment of this application, such as... Figure 1As shown, the motion data and the time axis of the first and fourth hand images in this embodiment are depicted exemplarily.

[0049] In some embodiments, at least one sensor is provided under at least one piano key, and the sensor is configured to detect one or more of the following data: displacement, velocity, and acceleration of the piano key.

[0050] In some embodiments, the sensor may be a velocity sensor, a displacement sensor, or an acceleration sensor.

[0051] Furthermore, in some embodiments, the speed sensor includes, but is not limited to, one or more of the following: magnetoelectric speed sensor, Hall effect speed sensor, photoelectric speed sensor, and magnetoelectric speed sensor.

[0052] Furthermore, in some embodiments, the displacement sensor includes, but is not limited to, one or more of the following: potentiometer-type displacement sensor, inductive displacement sensor, capacitive displacement sensor, magnetostrictive displacement sensor, optical sensor, ultrasonic sensor, Hall effect displacement sensor, oscillator-type displacement sensor, photoelectric encoder, contact sensor, and non-contact sensor, such as eddy current displacement sensor, laser displacement sensor, ultrasonic displacement sensor, etc., which do not directly contact the object being measured.

[0053] It is worth noting that this application can be flexibly applied to multiple scenarios: for example, when a piano teacher needs to conduct remote online teaching, the motion capture data can be fed back to the teacher's end in real time, assisting the teacher in completing the remote teaching task. Another example is that when students practice after class, they can use motion capture data for self-correction. Alternatively, parents can also use the self-analysis function of motion capture data to supervise their children's practice process.

[0054] The following detailed description, in conjunction with the accompanying drawings, outlines some embodiments of this application. Unless otherwise specified, the following embodiments and features described herein can be combined with each other. Please refer to... Figure 2 , Figure 2 This is a schematic flowchart illustrating a method for displaying piano keys according to an embodiment of this application, such as... Figure 2 As shown in the figure, this application embodiment provides a method for displaying piano keys, the method including S101 to S104.

[0055] S101. Obtain multiple sets of motion capture results. Each set of motion capture results includes: the identifier of the piano key, the piano key motion data, the piano key motion data including: force, and / or displacement, and the motion capture result is associated with a corresponding third timestamp.

[0056] The key markings are used to distinguish the individual keys on the instrument keyboard. Specifically, they can be the unique number or position information of each key, but this is not limited here.

[0057] Key motion data is used to record the state changes of each key on the instrument keyboard. Force can be the pressure applied to the key, providing feedback on the user's finger pressure; displacement can be the depth of key press or the rotation angle of a key at a given time. Key motion data may also include the speed at which each key is pressed or released, etc., but this is not limited here.

[0058] The third timestamp refers to the time stamp that is precisely associated with each key motion data in the motion capture result. It is used to arrange and process all data in the order of actual occurrence, ensuring the continuity and realism of the key movements in the subsequently generated performance animation.

[0059] Specifically, motion capture devices are used to acquire multiple sets of motion capture results, which are used to record the actual piano key movements during the performer's performance.

[0060] S102. Based on multiple sets of motion capture results within a preset time period, identify the touch dynamic features of the performer within the preset time period, and match the performance skill data within the preset time period based on the touch dynamic features. The performance skill data is associated with multiple third timestamps corresponding to the preset time period.

[0061] The preset time period is a predefined time interval used for subsequent analysis of the performer's continuous playing movements during the performance. The specific value can be set according to different analysis purposes or application scenarios, and can also be flexibly adjusted according to factors such as playing speed, repertoire difficulty, or teaching objectives. There are no limitations here. For example, the preset time period can be set to 0.5 seconds to capture the dynamic characteristics of rapid key touches; it can also be set to a relatively long 5 seconds or 10 seconds to analyze longer musical phrases or complex technical expressions.

[0062] Among them, touch dynamic characteristics refer to the results obtained by quantitatively analyzing continuous playing actions within a preset time period, reflecting the speed, frequency, and pattern of the player's finger-key interaction within the preset time period. For example, touch dynamic characteristics can be reflected in rhythm density, that is, the density of key pressing within a certain period of time; dynamic variation, that is, the gradual increase or decrease in the intensity of each key being pressed within a certain period of time; and repetition rate, that is, the rate at which the same note or a group of notes is repeatedly played in a short period of time.

[0063] Among them, performance technique data refers to the technical methods or expressive intentions of the performer matched based on the dynamic characteristics of touch. Performance technique data can be standard technique types defined by professional musicians and widely accepted, such as legato, staccato, staccato, staccato, and ornamentation, or it can be highly personal performance techniques of the performer, such as the performer's personalized treatment of specific types of musical phrases, rhythms, or timbres, or performance styles such as classical style and baroque style.

[0064] It should be understood that there is a relationship between feature recognition and technique matching between touch dynamics and performance technique data. For example, a pre-defined algorithm or machine learning model can be used to identify touch dynamics and match them with specific performance technique data.

[0065] Specifically, by capturing multiple sets of motion results within a preset time period, the dynamic characteristics of the performer's key touch are identified, and corresponding performance skill data is further matched. Each performance skill data is associated with multiple third timestamps corresponding to the preset time period, so that the performance skill data can be displayed in the performance animation within the corresponding time period. Each performance skill data can also be associated with specific piano key icons, so that the subsequently generated second performance prompt icons can be displayed at specific times and positions in the performance animation. Through meticulous analysis of performance behavior, data support is provided for multi-dimensional performance information feedback in subsequent performance animations.

[0066] S103. Generate multiple sets of piano key pressing images and multiple sets of first performance prompt icons corresponding to the piano key pressing images based on the multiple sets of piano key movement data, wherein the first performance prompt icons include at least one of pressing pressure icon, pressing displacement icon, and pressing speed icon; generate a plurality of second performance prompt icons based on the performance skill data; The first performance prompt icon is a visual element directly related to the physical characteristics of pressing the piano keys. It is used to provide immediate basic feedback, presenting raw data that is either unprocessed or only simply processed, such as information on force, displacement, and speed.

[0067] The pressure and displacement icons are used to represent the force or intensity of pressing the keys during performance, helping viewers intuitively understand the performer's dynamic control. For example, color gradients, shadow depths, or direct numerical labels can be used to represent changes in force; darker colors or shadows indicate stronger touch force, while lighter colors or shadows indicate lighter touch force.

[0068] The pressing speed icon indicates the speed at which a finger approaches and ultimately presses down on a key, providing feedback on the rhythmic relationship between each note. For example, different symbols or animation effects (such as varying light effects) can be used to represent pressing speed, helping viewers intuitively understand the performer's sense of rhythm.

[0069] The second performance cue icon is a visual element related to specific performance techniques and musical expression. It is used to provide guidance for specific musical expressions and presents performance technique data obtained through in-depth processing of motion capture results, such as legato icons and staccato icons. It should be understood that different performance technique data can correspond to different second performance cue icons.

[0070] Specifically, the physical movements of the piano keys are simulated based on each set of key motion data. Furthermore, the smoothness and realism of the performance animation can be enhanced by simulating other data such as pressing speed and pressure within the key motion data. Thus, each set of key motion data is converted into one or more images displaying the pressing or releasing of one or more keys at a specific point in time—i.e., key pressing images—achieving digital visualization of the key states during performance. Simultaneously, corresponding first performance prompt icons are generated, such as pressure intensity icons, pressing displacement icons, and pressing speed icons, resulting in multiple sets of key pressing images and their corresponding first performance prompt icons. Further, second performance prompt icons are generated based on each performance technique data, corresponding to specific techniques, resulting in several second performance prompt icons.

[0071] It should be understood that since the piano key movement data is associated with a corresponding third timestamp, the multiple sets of piano key pressing images and the first performance prompt icons corresponding to the multiple sets of piano key pressing images are also associated with a corresponding third timestamp. Furthermore, each performance skill data is associated with multiple third timestamps corresponding to a preset time period. Therefore, several second performance prompt icons are also associated with their respective multiple third timestamps.

[0072] S104. Based on the third timestamp and the key identifier, the key pressing image, the first performance prompt icon, and the second performance prompt icon are superimposed to generate a continuous performance animation.

[0073] Specifically, based on the third timestamp and the key identifiers, the generated key press images can be positioned at specific locations on the keyboard for display. The corresponding first performance prompt icons can be displayed near their respective key press images, enabling learners to quickly and accurately obtain the key movement data corresponding to the press images. It also allows all first performance prompt icons at a certain moment to be displayed uniformly in a specific area, making the performance animation interface clearer and providing viewers with a better visual experience.

[0074] The second performance prompt icon can also be displayed in the performance animation within a preset time period based on multiple third timestamps associated with the corresponding performance skill data. The display position can be the location pointed to by multiple key markers in multiple sets of motion capture results within the corresponding preset time period. It should be understood that each key marker points to the position of that key on the keyboard. When there are multiple key markers, the center point pointed to by the multiple key markers can be selected as the display position of the second performance prompt icon.

[0075] This allows for the precise overlay of the key press image, the first performance prompt icon, and the second performance prompt icon at each moment, ensuring that each key press image, the first performance prompt icon, and the second performance prompt icon accurately corresponds to the specific time point and key position for display, ultimately forming a coherent video stream, which is the performance animation.

[0076] In some embodiments, S104 includes: displaying the first performance prompt icon corresponding to the key press image within a first preset range of the key press image.

[0077] Specifically, a first performance prompt icon corresponding to the current key press image is displayed within a first preset range. The first preset range is usually set to a small area around the key press image, such as within a vertical distance of 1 cm from the key press image, so that the first performance prompt icon is close to its corresponding key press image but does not obscure the key press image, allowing the viewer to accurately match the two.

[0078] In some embodiments, S104 includes: acquiring multiple key movement data over a preset time period corresponding to the performance skill data, as a type of key movement data; analyzing the distribution density of the first performance prompt icon corresponding to the type of key movement data, determining the dense distribution area with the highest distribution density, and displaying the second performance prompt icon corresponding to the performance skill data within a second preset range from the dense distribution area.

[0079] Specifically, the densely distributed area can be a region where the total number of the first performance prompt icons is greater than the third density, or it can be the region with the largest total number of the first performance prompt icons. The size of the second preset range can be determined based on the size of the second performance prompt icons to avoid the densely distributed area of ​​the first performance prompt icons and display the corresponding second performance prompt icons near the densely distributed area. This minimizes the occlusion between the first and second performance prompt icons and creates good correlation feedback, thereby optimizing the distribution density of the performance prompts, ensuring that viewers receive balanced visual prompt information, and improving the efficiency of users obtaining performance feedback information.

[0080] It should be understood that, based on the overlapping divisions of preset time periods and the richness of playing techniques in different pieces, multiple secondary performance prompt icons may be displayed simultaneously. For example, if the first preset time period includes the third timestamps t1, t2, and t3, a secondary performance prompt icon A will be generated accordingly; if the first preset time period includes the third timestamps t2, t3, and t4, a secondary performance prompt icon B will be generated accordingly. At times t2 and t3, both secondary performance prompt icons A and B will be displayed simultaneously. Displaying the corresponding secondary performance prompt icons near densely distributed areas allows each secondary performance prompt icon to be displayed in a specific location. It also allows for differentiated display of multiple secondary performance prompt icons at the same time, improving the accuracy of performance feedback information display and the user's viewing experience.

[0081] In some embodiments, the second performance prompt icon is continuously displayed in the performance animation within multiple third timestamps corresponding to a preset time period.

[0082] In some embodiments, when generating a performance animation, an animation effect is created at a preset speed and preset angle, causing the first performance prompt icon to move away from the key-pressing image. When the distance between the first performance prompt icon and the key-pressing image reaches a preset distance, the first performance prompt icon is not displayed. Thus, over time, the first performance prompt icon can expand outwards from the key-pressing image in the performance animation until it disappears, ensuring the timeliness of the icon display.

[0083] In some embodiments, the preset speed can be determined based on the key pressing speed in the key motion data, and a linear mapping relationship is established between the two. That is, the greater the key pressing speed, the greater the preset speed, thereby ensuring that the animation effect matches the strength and rhythm of the actual playing action, enhancing the user's immersion and the realism of the feedback.

[0084] For example, taking a piano as an intelligent musical instrument, please refer to [link to relevant documentation]. Figures 3 to 5 , Figure 3 This is a display screen of a performance animation provided in an embodiment of this application; Figure 4 This is another display screen of the performance animation provided in the embodiments of this application; Figure 5 This is yet another display screen for a performance animation provided in the embodiments of this application.

[0085] When not being played, the key press images can appear as follows: Figure 3 The keyboard shown has no keys that are pressed, and neither the first nor the second performance prompt icon is displayed.

[0086] After the performance begins, the key press images can be displayed as follows: Figure 4The keyboard shown can have either a full keyboard image or a partial image of the key press area 10. Simultaneously, multiple first performance prompt icons 20 are displayed, uniformly positioned in a specific area at the top of the keyboard. As time progresses, these first performance prompt icons 20 gradually move away from the keyboard at a certain upward speed in the animation; that is, the earlier the timestamp, the farther away the first performance prompt icon 20 is from the keyboard, until it disappears from the performance animation.

[0087] When performing legato techniques, the key pressing images can be presented as follows: Figure 5 The keyboard shown can have either a full keyboard image or a partial image of the key press area 10. Since legato playing is characterized by rapid and even key presses, the performance animation will contain multiple first performance prompt icons 20 and second performance prompt icons 30 corresponding to legato techniques. The second performance prompt icons 30 will appear next to the densely distributed area of ​​the first performance prompt icons 20.

[0088] It should be understood that the performance animation shows the dynamic pressing process of the piano keys as they are played, and uses clear and easy-to-understand performance information (such as the first performance prompt icon) to meet the needs of beginners for immediate feedback. It also supports advanced learners to gain a deeper understanding of complex performance techniques and artistic expression through professional skill guidance (such as the second performance prompt icon).

[0089] For example, the piano key motion data in the motion capture results are suitable motion instructions to drive the 3D model, creating a high-precision 3D model of the piano keys to simulate the geometry, texture, and other physical properties of the actual object. Furthermore, by combining 3D modeling, rendering, and data processing technologies, the generated piano key press images, first performance prompt icons, and second performance prompt icons can all be in three-dimensional form, and the final synthesized performance animation can also present a 3D effect.

[0090] Furthermore, this application also provides a dynamic multi-level icon display management mechanism, which dynamically adopts different processing strategies under different density thresholds based on the total number of icons, and introduces the display priority of icons to implement processing strategies, such as visual enhancement strategies and icon hiding strategies, thereby optimizing the efficiency of information presentation and ensuring that viewers can always obtain the most relevant and important feedback information in the performance animation, thus highly adapting to the complex scenarios in piano performance.

[0091] In some embodiments, the method further includes: monitoring the total number of the first performance prompt icon and the second performance prompt icon in the performance animation; when the total number of icons is greater than the first density number, setting the display priority of each of the first performance prompt icon and the second performance prompt icon according to a preset display weight rule; adding corresponding visual emphasis elements to some of the first performance prompt icons and / or some of the second performance prompt icons according to the display priority, and rendering and displaying the visual emphasis elements.

[0092] The preset display weight rule refers to the pre-defined numerical or hierarchical rules used to evaluate the importance of different types of icons. Specifically, these rules can be set according to the importance of each icon to the viewer's understanding of the performance, and are not limited here. Furthermore, the display priority is a numerical or hierarchical representation of icon importance, used to determine which icons are prioritized or emphasized in the performance animation.

[0093] For example, the second performance prompt icon has a higher display priority than the first performance prompt icon. Because of the importance of the second performance prompt icon in guiding performance techniques, it is given a higher display priority, ensuring that this type of icon is displayed preferentially or emphasized in complex scenarios.

[0094] For example, the first performance prompt icon and the second performance prompt icon can also be set with different display priorities. For instance, the display priority of the pressing pressure icon in the first performance prompt icon is higher than the display priority of the pressing speed icon; the display priority of the staccato icon in the second performance prompt icon is higher than the display priority of the ornament.

[0095] Visual emphasis elements refer to various visual elements used to highlight specific information, guide viewers' attention, or enhance the viewing experience, thereby enabling viewers to obtain key information through intuitive visual cues.

[0096] For example, visual emphasis elements can be static graphic markers, dynamic visual effects, or a combination of both. Specifically, they can be color changes (e.g., highlighting, gradient colors), flashing effects (e.g., periodic flashing, pulse light effects, water ripple effects), arrow indicators (e.g., directional guidance, dynamic paths), icons and symbols, text labels (e.g., brief descriptions, numerical labels), shadows and outlines (e.g., projection effects, halo design, bold outlines), animation effects (e.g., explosions, floating bubbles), thickened or highlighted lines, etc., without limitation.

[0097] Specifically, the system monitors the total number of first and second performance prompt icons in each area of ​​the performance animation in real time. When the total number of icons exceeds a preset first density, the display priority of each icon is set according to a preset display weight. Based on the display priority, visual emphasis elements, such as highlighted borders, magnification effects, or dynamic effects, are selectively added to some first and / or second performance prompt icons. Therefore, even with a large number of icons, viewers can clearly obtain the most relevant and important performance information, avoiding information overload in the performance animation and improving the efficiency of performance information presentation.

[0098] In some embodiments, the method further includes: when the total number of icons is greater than the second density number, hiding a portion of the first performance prompt icon according to the display priority, so that the total number of icons is less than or equal to the second density number, wherein the second density number is greater than the first density number.

[0099] The first density quantity is used to trigger the mechanism for adjusting display priority and adding visually emphasized elements; the second density quantity is used to trigger the mechanism for hiding some low-priority icons. The second density quantity is set to be greater than the first density quantity to dynamically adopt different processing strategies at different density thresholds based on the total number of icons.

[0100] Specifically, when the total number of icons exceeds the second density, some of the first performance prompt icons will be hidden according to the display priority to ensure that the total number of icons does not exceed the second density, so as to keep the interface clear and not cause visual confusion due to too many icons, improve the efficiency of performance information presentation, and be highly adaptable to the complex scenarios of piano performance.

[0101] In some embodiments, the method further includes: determining the key pressing force based on the key motion data at time t; adding a corresponding visual emphasis element based on the key pressing force; and rendering and displaying the visual emphasis element to mark the first performance prompt icon corresponding to the key pressing force.

[0102] Specifically, based on the key motion data at time t, the actual force of each key being pressed is accurately calculated. Based on the key pressing force, specific visual emphasis elements are added to the corresponding first performance prompt icon, and these visual emphasis elements are rendered and displayed to intuitively mark the key touch behavior of different forces.

[0103] For example, when the key motion data includes force, the force is used as the key pressing force; when the key motion data includes displacement, the key pressing force is calculated from the displacement; when the key motion data includes both force and displacement, the key pressing force can be calculated by combining the two data to provide the accuracy of the pressing force data.

[0104] In some embodiments, the method further includes: when the pressure applied to the piano key is less than a first preset pressure, adding and rendering a first visual emphasis element to mark the first performance prompt icon corresponding to the pressure applied to the piano key; and / or, when the pressure applied to the piano key is greater than a second preset pressure, adding and rendering a second visual emphasis element to mark the first performance prompt icon corresponding to the pressure applied to the piano key; wherein the second preset pressure is greater than the first preset pressure.

[0105] Specifically, the first preset pressure is used to identify lighter key presses. When the pressure applied to the key is below this value, the key press is considered to be in the light pressure range, and a corresponding first visual emphasis element is added to remind the user to pay attention. Conversely, the second preset pressure is used to identify heavier key presses. When the pressure applied to the key exceeds this value, the key press is considered to be in the heavy pressure range, and a corresponding second visual emphasis element is added to remind the user to pay attention and avoid excessive force.

[0106] The second preset dynamic range is set to be greater than the first, providing layered visual emphasis based on different dynamic ranges. This makes the performance animation more intuitive and efficient, with a clear focus on information display, enhancing the viewing experience. The specific dynamic range value can be set according to the instrument type and user needs, and is not limited here.

[0107] like Figure 5 As shown, if the pressing force of a key is less than the first preset force, it is determined to be a light touch. A bluish-green font is added and displayed to mark the first performance prompt icon 20 corresponding to the key pressing force, indicating a light touch. If the pressing force of a key is between the first and second preset forces, the original icon style of the first performance prompt icon is maintained, and the first performance prompt icon 20 is displayed in a normal green font, indicating a moderate touch. If the pressing force of a key is greater than the second preset force, it is determined to be a strong touch. A yellowish-green font is added and displayed to mark the first performance prompt icon 20 corresponding to the key pressing force, indicating a high touch. Furthermore, as the force increases, the color of the first performance prompt icon 20 can change from blue to green and then to yellow through different visual emphasis elements.

[0108] In some embodiments, the method further includes: when the pressure applied to the piano key is less than the first preset pressure, and / or when the pressure applied to the piano key is greater than the second preset pressure, updating the first performance prompt icon corresponding to the pressure applied to the piano key to a third performance prompt icon, wherein the display priority of the third performance prompt icon is higher than the display priority of the first performance prompt icon.

[0109] Specifically, different display priorities can be set within the first performance prompt icon. When the key touch is in the light force range and / or in the strong force range, the corresponding first performance prompt icon will be updated to the third performance prompt icon. The third performance prompt icon has a higher display priority, so that the performance behavior of key touch with special force is displayed first or emphasized in the performance animation, avoiding being hidden when the total number of icons is greater than the second density number, ensuring the full display of key performance information, and improving the practicality and viewing experience of the performance animation.

[0110] In some embodiments, the performance skill data includes legato techniques. S102 includes: counting the number of motion captures in the motion capture results within the preset time period, and calculating the data fluctuation value of the piano key motion data within the preset time period; when the touch dynamic feature identifies that the number of motion captures is greater than a preset number and the data fluctuation value is less than a preset fluctuation value, the performance skill data within the preset time period is matched as a legato technique.

[0111] Each motion capture result represents one interaction between a finger and the piano keys. The number of motion captures refers to the total number of times motion capture results are captured within a preset time period, used to evaluate the frequency and density of the performer's playing movements. The corresponding touch dynamics are reflected in rhythm density. Correspondingly, the preset number is used to determine whether the number of motion captures has reached the standard for triggering the recognition of a specific playing technique. The specific value can be flexibly set according to actual needs and is not limited here.

[0112] The data fluctuation value refers to the range of change in key movement data (such as force, displacement, and pressing speed) within a preset time period, used to assess the stability and consistency of the performance. Correspondingly, the preset fluctuation value is used to determine whether the data fluctuation value has reached the standard for triggering the recognition of a specific playing technique. The specific value can be flexibly set according to actual needs and is not limited here.

[0113] The characteristic of legato playing is the smooth connection between notes without obvious pauses or intervals. It is usually manifested as a fast and even touch, and the corresponding dynamic characteristics of the touch can be reflected in rhythmic density, dynamic variations, etc. It should be understood that the preset time period can be set relatively long, such as 5 seconds, to fully capture the dynamic characteristics of the touch corresponding to a series of fast and even playing movements, thereby improving the accuracy of playing technique recognition.

[0114] When the touch dynamic features within a preset time period are identified as having a greater number of motion captures than a preset number and a data fluctuation value less than a preset fluctuation value, the performer's touch behavior conforms to the characteristics of being coherent and stable, and the performance technique data within the preset time period is matched as legato technique.

[0115] In some embodiments, the performance skill data includes large-span jump skills. S102 includes: calculating the span distance of the piano keys based on the key identifiers and third timestamps in multiple sets of motion capture results within the preset time period; when the touch dynamic features identify that the span distance is greater than the preset distance, the performance skill data within the preset time period is matched as a large-span jump skill.

[0116] The "crossing distance" refers to the straight-line or actual path length between the key icons in multiple sets of motion capture results within a preset time period. Assuming the key icons correspond to the C and G keys within the preset time period, the distance between them is calculated as the crossing distance. Correspondingly, the preset distance is used to determine whether the crossing distance meets the standard for triggering the recognition of a specific playing technique. The specific value can be flexibly set according to actual needs and is not limited here.

[0117] The characteristic of span-jumping techniques is that the fingers move rapidly over a long distance on the keyboard, typically spanning multiple octaves or more. The corresponding touch dynamics can be reflected in the playing distance, etc. It should be understood that the preset time period is set relatively short, such as 0.5 seconds, to capture the touch dynamics corresponding to large and rapid playing movements within a short period of time, thereby improving the accuracy of playing technique recognition.

[0118] When the dynamic features of the key touch within a preset time period are identified as spanning a distance greater than a preset distance, the performer's key touch behavior conforms to the characteristics of large-scale rapid movement in a short period of time, and the performance skill data within the preset time period is matched as a large-span jump technique.

[0119] In some embodiments, each set of motion capture results further includes: hand posture data of the performer's hand and fingers. The hand posture data records the precise position, orientation, and shape of the performer's hand and fingers in three-dimensional space, such as the three-dimensional coordinates and rotation angle of the hand, as well as the position and angle of each finger joint, ensuring the realistic reproduction of complex fingering movements. For example, during a performer's performance, hand image information can be acquired through an image acquisition device, and the position of the hand and each finger in the image, the degree of bending and pressing of each finger, can be identified to generate hand posture data.

[0120] For example, the third timestamp is also used to synchronize the timing information of hand posture data and piano key motion data, assigning a precise timestamp to each set of motion capture results, so that all data can be arranged and processed in the order of actual occurrence.

[0121] In some embodiments, the touch dynamic features include playing fingering features. Playing fingering features refer to the results obtained by quantitatively analyzing continuous playing actions corresponding to finger-key interactions within a preset time period, and also reflect the speed, frequency, and pattern of the performer's finger-key interaction.

[0122] In some embodiments, the method further includes: identifying the performance fingering characteristics of the performer within the preset time period based on the hand posture data in multiple sets of motion capture results within the preset time period; and matching the performance fingering characteristics with performance skill data within the preset time period.

[0123] Specifically, by using hand posture data over a period of time, the fingering of the performer can be determined within that period, thereby identifying the fingering characteristics of the performer within a preset time period. This allows for the analysis of the performer's touch dynamics from a fingering perspective and the matching of corresponding performance technique data.

[0124] For example, hand posture data over a period of time reflects the movement trajectory of multiple fingers. When the playing fingering feature is identified as a finger moving from one note to another, farther note, with the thumb passing under the palm between other fingers to reach a new position, the playing technique data within a preset time period is matched based on the playing fingering feature as a thumb crossing technique. As another example, when the playing fingering feature is identified as a rapid change of different fingers on the same key while maintaining the same hand position, the playing technique data within a preset time period is matched based on the playing fingering feature as a fingering transition technique. In addition, playing fingering features can also be used to identify and match extended fingering techniques, contracted fingering techniques, rotating fingering techniques, alternating fingering techniques, large-span jump techniques, and so on.

[0125] It should be understood that key motion data and hand posture data are manifestations of performance actions from different angles. By matching touch dynamic features with performance technique data from these two angles, the accuracy and comprehensiveness of performance technique recognition can be effectively improved, thereby achieving multi-dimensional performance information feedback and significantly enhancing the depth and personalization of visual feedback in performance animations.

[0126] For example, in this embodiment of the application, the user can be either a performer or a viewer of the performance animation. The performer can generate corresponding performance animations using a motion capture device during the performance; while the viewer can learn and appreciate the performance details through the animations. It should be noted that the performer and the viewer of the performance animation can be the same person, meaning the performer can watch their own performance recordings and optimize their performance skills based on real-time feedback.

[0127] For example, performance animations can be generated in real time, that is, created and displayed synchronously during the performance using real-time motion capture results, providing immediate feedback; or they can be generated after all data has been captured, that is, complete performance data is recorded first, and then analyzed and animated, suitable for subsequent review and detailed evaluation.

[0128] For example, the above method can be implemented as a computer program that can run on a smart musical instrument. The instrument has keys arranged in a fixed order, and each key has a fixed pitch register. Examples include a piano, organ, accordion, or electronic keyboard. These keys can be arranged into a keyboard.

[0129] The intelligent musical instrument includes piano keys; a motion capture device for acquiring multiple sets of motion capture results; and a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0130] Piano keys are used for direct interaction with the performer to play music. They can function like a traditional piano or integrate additional technologies, such as sensors, to detect key pressure, speed, and other parameters.

[0131] Motion capture devices are used to monitor and record the performer's limb movements, such as the position and movement path of the fingers. They can also be used to monitor and record the movement of the piano keys, thereby obtaining multiple sets of motion capture results and receiving detailed feedback on the performance technique.

[0132] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any method of displaying the piano keys.

[0133] The processor provides computing and control capabilities to support the operation of the entire intelligent musical instrument.

[0134] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to execute any method of displaying the piano keys.

[0135] This network interface is used for network communication, such as sending assigned tasks.

[0136] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0137] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: S101. Obtain multiple sets of motion capture results. Each set of motion capture results includes: the identifier of the piano key, the piano key motion data, the piano key motion data including: force, and / or displacement, and the motion capture result is associated with a corresponding third timestamp. S102. Based on multiple sets of motion capture results within a preset time period, identify the touch dynamic features of the performer within the preset time period, and match the performance skill data within the preset time period based on the touch dynamic features. The performance skill data is associated with multiple third timestamps corresponding to the preset time period. S103. Generate multiple sets of piano key pressing images and multiple sets of first performance prompt icons corresponding to the piano key pressing images based on the multiple sets of piano key movement data, wherein the first performance prompt icons include at least one of pressing pressure icon, pressing displacement icon, and pressing speed icon; generate a plurality of second performance prompt icons based on the performance skill data; S104. Based on the third timestamp and the key identifier, the key pressing image, the first performance prompt icon, and the second performance prompt icon are superimposed to generate a continuous performance animation.

[0138] For example, the processor is used to run a computer program stored in the memory, and is also used to implement the steps of the method for displaying piano keys provided in any embodiment of this application, which will not be repeated here.

[0139] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement the steps of the method for displaying piano keys provided in any of the embodiments of this application.

[0140] The computer-readable storage medium can be the internal storage unit of the smart piano described in the foregoing embodiments, such as the hard drive or memory of the smart piano. Alternatively, the computer-readable storage medium can be an external storage device of the smart piano, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the smart piano.

[0141] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for displaying piano keys, characterized in that, Including the following steps: S101. Obtain multiple sets of motion capture results. Each set of motion capture results includes: the identifier of the piano key, the piano key motion data, the piano key motion data including: force, and / or displacement, and the motion capture result is associated with a corresponding third timestamp. S102. Based on multiple sets of motion capture results within a preset time period, identify the touch dynamic features of the performer within the preset time period, and match the performance skill data within the preset time period based on the touch dynamic features. The performance skill data is associated with multiple third timestamps corresponding to the preset time period. S103. Generate multiple sets of piano key pressing images and multiple sets of first performance prompt icons corresponding to the piano key pressing images based on the multiple sets of piano key movement data, wherein the first performance prompt icons include at least one of pressing pressure icon, pressing displacement icon, and pressing speed icon; generate a plurality of second performance prompt icons based on the performance skill data; S104. Based on the third timestamp and the key identifier, the key pressing image, the first performance prompt icon, and the second performance prompt icon are superimposed to generate a continuous performance animation.

2. The method according to claim 1, characterized in that, The method further includes: Monitor the total number of the first performance prompt icon and the second performance prompt icon within the performance animation; When the total number of icons is greater than the first density number, the display priority of each of the first performance prompt icons and the second performance prompt icons is set according to the preset display weight rules; According to the display priority, add corresponding visual emphasis elements to some of the first performance prompt icons and / or some of the second performance prompt icons, and render and display the visual emphasis elements.

3. The method according to claim 2, characterized in that, The second performance prompt icon has a higher display priority than the first performance prompt icon.

4. The method according to claim 2, characterized in that, The method further includes: When the total number of icons is greater than the second density number, a portion of the first performance prompt icon is hidden according to the display priority, so that the total number of icons is less than or equal to the second density number, wherein the second density number is greater than the first density number.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Based on the key motion data at time t, determine the key pressing force. When the pressure applied to the piano key is less than a first preset pressure, a first visual emphasis element is added and rendered to mark the first performance prompt icon corresponding to the pressure applied to the piano key; and / or, When the pressure applied to the piano key is greater than the second preset pressure, a second visual emphasis element is added and rendered to mark the first performance prompt icon corresponding to the pressure applied to the piano key; wherein, the second preset pressure is greater than the first preset pressure.

6. The method according to claim 5, characterized in that, The method further includes: When the pressure applied to the piano key is less than the first preset pressure, and / or when the pressure applied to the piano key is greater than the second preset pressure, the first performance prompt icon corresponding to the pressure applied to the piano key is updated to a third performance prompt icon, and the display priority of the third performance prompt icon is higher than the display priority of the first performance prompt icon.

7. The method according to claim 1, characterized in that, S104 includes: Within a first preset range of the piano key press images, display the first performance prompt icon corresponding to the piano key press images; and / or, Acquire multiple key movement data within a preset time period corresponding to the performance skill data, and use them as a type of key movement data; analyze the distribution density of the first performance prompt icon corresponding to the type of key movement data, determine the dense distribution area with the highest distribution density, and display the second performance prompt icon corresponding to the performance skill data within a second preset range from the dense distribution area.

8. The method according to claim 1, characterized in that, S102 includes: The number of motion captures within the preset time period is counted, and the data fluctuation value of the piano key movement data within the preset time period is calculated; when the touch dynamic feature identifies that the number of motion captures is greater than a preset number, and the data fluctuation value is less than a preset fluctuation value, the playing technique data within the preset time period is matched as legato technique; and / or, The span distance of the piano keys is calculated based on the key identifiers and third timestamps in multiple sets of motion capture results within the preset time period; when the touch dynamic features identify that the span distance is greater than the preset distance, the playing technique data within the preset time period is matched as a large-span jump technique.

9. The method according to claim 1, characterized in that, Each set of motion capture results also includes: hand posture data of the performer's hand and fingers, and the touch dynamic features include playing fingering features; The method further includes: identifying the performance fingering characteristics of the performer within the preset time period based on the hand posture data in multiple sets of motion capture results within the preset time period; and matching the performance fingering characteristics with performance skill data within the preset time period.

10. A smart musical instrument, characterized in that, The intelligent musical instrument includes: piano keys; Motion capture device, used to acquire multiple sets of motion capture results; Memory is used to store computer programs; A processor is configured to execute the computer program and, in executing the computer program, implement the method for displaying the piano keys as described in any one of claims 1 to 9.

Images