Multi-dimensional data display method, system and program product

Abstract

The invention relates to the piano online teaching technology, in particular to a multi-dimensional data display method and system and a program product, and the method comprises the steps: obtaining a first data set collected by a first device end of a first user; transmitting the first data set to a second device end of at least one second user; generating a music score interface according to the music score propelling data; responding to a visualization signal sent by the first user and / or the second user, and adaptively generating a visualization rule; responding to the visualization rule and the first data set to generate a first visualization interface; and displaying the first visual interface and the music score interface on the second display device. According to the invention, a video and music score synchronous display scheme is provided for piano online teaching, the scheme has relatively high real-time performance and interactivity, and a recommended display scheme can be set for teachers or students in a targeted manner.

Description

[0001] Divisional application The present application is a divisional application of the invention patent application with the application date of September 12, 2025, the invention name of a performance data transmission method, system and program product, and the application number of 202511301115.9. TECHNICAL FIELD

[0002] The present application relates to the technical field of online piano teaching, in particular to a multi-dimensional data display method, system and program product. BACKGROUND

[0003] At present, mainstream piano learning needs to rely on real-time guidance of a live teacher, resulting in a very high cost of learning piano.

[0004] In this regard, the prior art also attempts to provide an online teaching scheme: such as patent application 202211311948.X discloses a one-to-many piano online practice method, which comprises the following steps: S1: selecting an online cloud and adjusting offline equipment; S2: installing real-time audio and video transmission equipment and starting a real-time audio converter; S3: the system collects student audio and video and transmits student audio and video to the teacher's mobile phone in real time; S4: the student is explained through real-time live broadcast, and a virtual music keyboard is displayed on the display screen; S5: the teacher evaluates and initiates a personal voice call for guidance; S6: the student's playing process is analyzed by a video recorder, and the system gives an evaluation in combination with the music piece.

[0005] For example, the prior art also attempts to provide a musical instrument auxiliary teaching method based on AR technology: such as patent application 202210512685.2 discloses a musical instrument auxiliary teaching method and system based on AR technology, which comprises a collection step, a teaching video is played through an AR device, and actual video data and actual audio data during playing are synchronously acquired; a judgment step, a loudness peak of a tone in the actual audio data is acquired to determine an actual rhythm point, an audio at the actual rhythm point is acquired to determine whether a playing error occurs, and an actual fingering picture in the actual video data is extracted through the actual rhythm point to determine whether a fingering error occurs; a correction step, a playing error or a fingering error at the actual rhythm point is acquired, and correct audio data or correct fingering pictures corresponding to the actual rhythm point are called, and teaching is guided through audio comparison or picture comparison.

[0006] However, the applicant believes that for the special application of piano education, traditional live teaching or AR technology cannot solve the application pain points.

[0007] Therefore, there is an urgent need for a method for live teaching that can be applied to the piano education scene which is relatively professional. SUMMARY

[0008] The purpose of this invention is to provide a method, system, and program product for transmitting performance data, which partially solves or alleviates the above-mentioned deficiencies in the prior art, can dynamically adjust the visualization rules of video data, and alleviate or reduce the interference to the user's practice under multi-dimensional data switching display.

[0009] To solve the aforementioned technical problems, the present invention specifically adopts the following technical solution: A first aspect of the present invention is to provide a method for transmitting performance data, comprising the steps of: S101, Obtain a first dataset collected by a first device terminal of a first user, wherein the first device terminal includes: a first musical instrument, a first image device set for the first musical instrument, and the first dataset includes: image data collected by the first image device at at least one shooting angle. S102, the first dataset is transmitted to a second device of at least one second user, the second device including: a second musical instrument and a second display device; S103, Generate a music score interface based on the music score advancement data; the music score advancement data is determined by the positioning signal input by the first user or the second user, or the music score advancement data is generated based on a second dataset, the second dataset including: second hardware data and / or second video data, and the second hardware data is the fingering data of the second user, and the second video data is image data acquired by the second image device at the second device end at at least one shooting angle; S104, in response to a visualization signal emitted by the first user and / or the second user, adaptively generate visualization rules, wherein the visualization signal includes: the first dataset or the second dataset; the visualization rules include: The data range includes at least one of the following recommended display intervals: the time scale of the image data to be displayed, the shooting angle, and the spectral amplitude of the image data; wherein the time scale is determined by the score progression data; S105, in response to the visualization rules and the first dataset, a first visualization interface is generated; S106, the first visualization interface and the music score interface are displayed on the second display device.

[0010] In some embodiments, the first user is a music teacher and the second user is a student; or, the first user is a student and the second user is a music teacher.

[0011] In some embodiments, the first instrument and the second instrument are pianos.

[0012] In some embodiments, S104 includes: S1041, Generate or update performance evaluation results based on the second dataset, wherein the performance evaluation results include at least one of the following evaluation indicators: type of fingering error, type of upper limb error, type of body error, error frequency of at least one type of error, and the evaluation indicator is pre-associated with a recommended display interval of at least one video data, wherein the video data is first video data or second video data; S1042, Generate visualization rules based on the performance evaluation results and the data progress of the first video data, wherein the data progress refers to the parsing progress and / or reception level of the first video data.

[0013] In some embodiments, the steps further include: Store the first dataset and / or the second dataset into the storage unit.

[0014] In some embodiments, the first device includes: a first display device; correspondingly, the method further includes: The first display device displays at least one third visual interface, and each of the at least one third visual interface is generated from at least one of the second video data; Wherein, at least one of the enhanced display forms of the third visualization interface is set by the performance evaluation results, and the enhanced display form is used to define at least one of the following enhanced display attributes: display position, display brightness, display area, and display markers.

[0015] In some embodiments, the visualized signal includes: The first visual signal input by the first user includes one or more of the following information: shooting angle, viewing angle range, and display area; And / or, a second visual signal input by a second user, the second visual signal including one or more of the following information: shooting angle, viewing angle range, display area.

[0016] In some embodiments, the second device further includes: a second broadcasting device, and correspondingly, the step of: The sound effects of the first user's performance are reproduced in the second musical instrument or the second broadcasting device based on the first dataset.

[0017] This invention provides a system for transmitting performance data, comprising: The first data acquisition module is used to acquire a first dataset obtained by a first user's first device terminal. The first device terminal includes: a first musical instrument, a first image device set corresponding to the first musical instrument, and the first dataset includes: image data acquired by the first image device at at least one shooting angle. A first data transmission module is configured to transmit the first dataset to a second device terminal of at least one second user, the second device terminal including: a second musical instrument and a second display device; The second interface generation module is used to generate a music score interface based on the music score progression data; the music score progression data is determined by the positioning signal input by the first user or the second user, or the music score progression data is generated based on a second dataset, the second dataset including: second hardware data and / or second video data, and the second hardware data is the fingering data of the second user, and the second video data is image data acquired by the second image device at the second device end at at least one shooting angle; A visualization signal parsing module is used to adaptively generate visualization rules in response to visualization signals emitted by the first user and / or the second user, wherein the visualization signals include: the first dataset or the second dataset; and the visualization rules include: The data range includes at least one of the following recommended display intervals: the time scale of the image data to be displayed, the shooting angle, and the spectral amplitude of the image data; wherein the time scale is determined by the score progression data; The first interface generation module is used to generate a first visualization interface in response to the visualization rules and the first dataset. The collaborative display module is used to display the first visualization interface and the sheet music interface on the second display device.

[0018] The present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method as described in any of the embodiments.

[0019] Beneficial technical effects: This invention provides a multi-dimensional data display solution with strong interactivity and collaboration, which can alleviate the sense of distance in online piano teaching to a certain extent, while reducing the pressure on users to understand multi-dimensional data display.

[0020] Furthermore, this invention focuses on using the second dataset as a guide to optimize the visualization rules. This allows for the use of video data based on the student's performance progress and performance status, enabling students to intuitively understand external guidance (such as video feedback) while maintaining performance continuity.

[0021] In other words, this invention allows for flexible adjustments to the collaborative visualization of videos and scores based on students' performance evaluation results. It can enhance the display of key data to reduce the difficulty for students to understand the data, making it easier for them to maintain a high level of focus and thus maintain an immersive performance state. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0023] Figure 1 This is a schematic diagram of the method flow in an exemplary embodiment of the present invention; Figure 2 This is a flowchart illustrating the first visual interface in an example embodiment of the present invention; Figure 3 This is a schematic diagram of the module structure of a video data display system in an exemplary embodiment of the present invention; Figure 4 This is a schematic diagram of a synchronized video and sheet music display interface in an exemplary embodiment of the present invention; Figure 5 This is a schematic diagram of a video and sheet music synchronized display interface in another exemplary embodiment of the present invention; Figure 6 This is a flowchart illustrating a video display method in yet another exemplary embodiment of the present invention; Figure 7 This is a flowchart illustrating the video display method in another embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0025] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0026] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0027] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0028] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0029] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0030] As used in this specification, the term "about" typically means + / -5% of the value, more typically + / -4% of the value, more typically + / -3% of the value, more typically + / -2% of the value, even more typically + / -1% of the value, and even more typically + / -0.5% of the value.

[0031] In this specification, certain embodiments may be disclosed in a range-bound format. It should be understood that this "range-bound" description is merely for convenience and brevity and should not be construed as a rigid limitation on the disclosed range. Therefore, the description of a range should be considered as having specifically disclosed all possible subranges and the individual numerical values ​​within those ranges. For example, a description of the range 1-6 should be considered as having specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the individual numbers within those ranges, such as 1, 2, 3, 4, 5, and 6. This rule applies regardless of the breadth of the range.

[0032] In this article, "viewing angle amplitude" refers to the range of angles that a video frame can cover. It determines how "wide" a scene a user can see through the video. For example, taking a top-down view as an example, a larger viewing angle amplitude in a top-down view means a wider range of image content is displayed, such as being able to see relatively complete piano keys and hand movements. Conversely, if the viewing angle amplitude is smaller, the number of piano keys may be smaller, meaning the focus is on displaying hand movements.

[0033] Example 1: It should be noted that traditional guidance schemes mainly use electronic sheet music (with performance results marked on it) and video file playback display schemes (such as the method for synchronous display of performance process and sheet music disclosed in patent application CN202110507335.2). However, the guidance application of this review scheme is relatively limited, that is, it is difficult to provide timely feedback during the user's performance.

[0034] In contrast to video review modes, this invention provides a solution for real-time, immersive, collaborative display of video data. Specifically, this invention focuses on dynamically adjusting the video visualization rules by incorporating factors such as the user's performance status and the progress of the video data. This allows for synchronized and real-time display of the video and sheet music, while adjusting the focus on specific information (such as the video's viewing angle and display area) to ensure that users can intuitively capture key information from the collaboratively displayed video data, thus facilitating quick and autonomous adjustments to their current performance status.

[0035] In other words, the real-time collaborative display solution for video and sheet music provided by this invention can reduce the difficulty for users to receive complex information (such as video and sheet music displayed simultaneously on the display interface, with progress markers, error markers, etc.) while providing timely and significant feedback to users. That is to say, this invention provides a data feedback solution that facilitates immersive performance, which can reduce the impact on the user's focus on the performance to a certain extent.

[0036] See Figures 1-2 As shown, the present invention provides a method for displaying video data, including the following steps: S201, receive video data and hardware data of the user during the performance, wherein the video data and the hardware data are associated with a timestamp; wherein, the video data includes: image data of the user and the performance device at at least one shooting angle, and the hardware data includes: fingering data generated by the performance device during the performance. S202, generating a first visualization interface based on the video data; wherein, S202 includes: S2021, Generate or update performance evaluation results based on the hardware data and / or the video data, wherein the performance evaluation results include at least one of the following evaluation indicators: error type of fingering error, error type of upper limb error (specifically, hand shape error, elbow posture error), error type of body error (such as hunchback), error frequency of at least one error type, and the evaluation indicators are pre-associated with a recommended display range of at least one video data. For example, in some embodiments, the type of typing error may include one or more of the following: Finger errors (i.e., using the wrong finger), finger bending, using the entire thumb to touch the key (such as using the entire side of the thumb or even the flat pad of the thumb to touch the key), weak finger independence (such as severe adhesion between the 3rd, 4th, and 5th fingers, so that moving one finger will affect the movement of the other finger), etc.

[0037] For example, in some embodiments, the type of hand gesture error may include one or more of the following: Slumped palm (such as the middle of the palm (metacarpal joint) sinking, the whole hand shape losing its natural arch support), wrist too high / wrist too low (for example, the wrist is stiffly arched upward or pressed down too low, unable to maintain a natural relaxed state basically level with the keys, thus causing the dynamics or note value to deviate), stiff finger turning (finger crossing / finger straddling) (the thumb moves too much when passing under other fingers (or the 3rd or 4th finger crosses the thumb), excessive wrist twisting or causing the body to sway).

[0038] Taking the piano as an example, the thumb, index finger, middle finger, ring finger, and little finger are usually defined as the 1st to 5th fingers.

[0039] S2022, Generate visualization rules based on the performance evaluation results and the data progress of the video data, wherein the data progress refers to the parsing progress and / or reception level of the video data, and the visualization rules include: The data range includes at least one of the following recommended display intervals: the time scale of the image data to be displayed, the shooting angle, and the viewing angle range of the image data; For example, in some embodiments, the shooting angle and viewing angle can be selected based on the performance evaluation results.

[0040] For example, in some embodiments, the time scale refers to the set time that the current visualization rule maintains. For example, in an exemplary embodiment, the set time has a default value preset by the user, that is, it is preferable to retain a certain set time after each visualization rule switch to limit the frequency of visualization rule switching during the performance and reduce the data capture pressure or interference to the user caused by the first visualization interface update.

[0041] Alternatively, in some embodiments, the time scale refers to the length of time of video data to be parsed / read at the current moment, based on the current visualization rules.

[0042] Understandably, multiple video data points captured by an imaging device can be sent to the display device (such as a mobile phone or tablet) in the form of data packets, and a single data packet can encapsulate view information for a given time period. Correspondingly, the time scale can be the duration of one or more data packets. Based on the visualization rules, the length of the data packets to be read and parsed within a given time period can be defined.

[0043] In some embodiments, the length of the time period corresponding to the data packet can be a preset default value.

[0044] Alternatively, in some embodiments, the length of the time period corresponding to the data packet can be dynamically set according to the actual communication environment. For example, when the communication environment is good, the time period corresponding to the data packet can be relatively shorter, that is, the transmission frequency can be more frequent; conversely, when the communication environment is poor, the time period corresponding to the data packet can be relatively longer.

[0045] S2023, Generate a first visualization interface based on the visualization rules and the video data; S203, the first visualization interface is displayed in the music score interface.

[0046] Preferably, in this embodiment, the sheet music interface is a real-time display interface, that is, the sheet music interface is displayed in real time according to the user's actual playing progress.

[0047] In some embodiments, the sheet music interface displays sheet music information (such as piano staff notation), and at least one type of annotation information is displayed on the sheet music information. This annotation information may include progress information, error information, technique information, etc. For example, progress information refers to setting a progress icon to mark the position of the sheet music (such as a specific note) according to the user's playing progress. Error information may include error type, error severity, etc., and technique information may include technique icons, technique annotation explanations, etc. This annotation information can be displayed dynamically or statically.

[0048] The applicant noted that piano performance demands a very high level of concentration, especially for performers who are not very familiar with the sheet music (such as beginners or those just starting to practice the piece). They may find it difficult to keep up with the flow of the performance while simultaneously capturing and paying attention to the information in the electronic sheet music or video files.

[0049] In response, this embodiment proposes a scheme to dynamically adjust the first visualization interface based on the performance status and the data progress of the video data, so as to recommend a suitable display angle or amplitude to the user in combination with the current status, so that the user can focus on the core issues.

[0050] Furthermore, the collaborative display scheme in this invention can reduce the additional performance requirements on the user's configured hardware devices, thereby mitigating latency issues caused by network or hardware performance limitations and reducing the adverse effects of video stuttering on users. In other words, this invention can also reduce the implementation cost of collaborative display to a certain extent.

[0051] Preferably, in this embodiment, the sheet music interface refers to a display interface that simultaneously displays sheet music and video data.

[0052] In some embodiments, the sheet music interface may have default visualization rules when displayed. During the display process, the visualization rules can be dynamically updated based on the user's performance or data reception.

[0053] For example, in some embodiments, the default visualization rules can be user-preset visualization rules.

[0054] Alternatively, in some embodiments, the default visualization rule can be a rule preferred based on user habits, such as the visualization rule that is used most frequently in a historical period.

[0055] Alternatively, in some embodiments, the default visualization rule can be the visualization rule used when the sheet music interface was last displayed.

[0056] For example, in some embodiments, the video data may be video captured by an imaging device (such as a camera) connected to a display device (such as a user's mobile phone or tablet) to display the video synchronously on the display device.

[0057] For example, in some embodiments, one or more cameras may be installed on either side of the piano or above the piano to capture video data, and the video data is timestamped.

[0058] For example, in some embodiments, hardware data refers to the user's fingering data (also known as motion capture data, or simply motion capture data), including: hand information (such as left or right hand) and finger information (such as recording which finger was pressed) used to perform the pressing action (i.e., pressing a piano key), the identifier of the piano key (i.e., which specific key it is), and the corresponding motion data. The fingering data is also timestamped.

[0059] For example, in some embodiments, motion data can be force, and / or displacement.

[0060] Specifically, the force here can be the pressure exerted on the piano key, which is used to provide feedback on the user's finger pressure; the displacement can be the pressing depth of the piano key, or the rotation angle of the piano key at a certain time.

[0061] For example, in some embodiments, a speed sensor, displacement sensor or acceleration sensor is provided below the piano keys, which can detect the speed, displacement or acceleration of the corresponding piano keys in real time, and thus indirectly calculate the force on the piano keys.

[0062] In some embodiments, the steps further include: A progress marker is generated based on the hardware data, and the progress marker is associated with a musical note; Correspondingly, in S203, the first visualization interface and the progress marker are displayed synchronously in the music score interface.

[0063] For example, in some embodiments, standard data of the score (such as the playing order (or time) of each note, playing dynamics, playing pitch, etc.) can be obtained in advance for the piece of music to be played, and the note currently played by the user can be determined by the matching relationship between the hardware data and the standard data, so as to mark the playing progress on the score on the music interface.

[0064] For example, in some embodiments, a music score interface is generated based on progress markers and music score data. The dynamic updating of visualization rules in this invention actually involves dynamically adjusting the collaborative display method (such as display position) of the video and music score.

[0065] In some embodiments, the video data includes: first image data acquired from a top-down perspective; and second image data acquired from a side-view perspective. The top-down angle is set to capture the user's finger posture, and the side-down angle is set to capture the user's upper limb and / or body posture.

[0066] For example, in some embodiments, at least one camera can be placed above the piano to collect first image data, which can be used to capture a complete image of the piano keys and the playing state of the user's hands. At least one camera can be placed on the side of the piano to capture the user's upper limbs and body (such as the sitting posture of the upper body and the pedaling posture of the legs).

[0067] In some embodiments, step S2022 includes: S20221, Select the recommended display range based on the performance evaluation results; S20222, Select recommended video data from the video data according to the recommended display range; S20223, when the content integrity of the recommended video data is greater than the preset integrity threshold, the visualization rules are updated according to the recommended display range.

[0068] In some embodiments, a fingering error or hand shape error has at least one recommended display area.

[0069] For example, in some embodiments, the recommended display range can be a recommended shooting angle (or display view), such as a top view or a side view.

[0070] For example, when the type of fingering error is finger misalignment or weak finger independence, the recommended display area is a top-down view. When the type of fingering error is finger bending or thumb-based keystrokes, the recommended display area is a side view to facilitate observation of the finger bending state.

[0071] For example, when the type of hand shape error is palm collapse or wrist too high / too low, the recommended display range can be a side view. When the type of hand shape error is stiff finger turning, the recommended display range can be a top view.

[0072] For example, in some embodiments, the recommended display range can also be a recommended display viewing angle. For instance, for errors that are relatively easy to observe, a larger viewing angle can be recommended to allow users to make a more comprehensive observation. Conversely, for errors that are more difficult to observe (especially those that beginners tend to overlook), a smaller viewing angle can be recommended for targeted display.

[0073] For example, for fingering errors, the recommended viewing angle can be marked as the first viewing angle. For fingering errors with weak finger independence, or hand shape errors such as stiff finger turning, the recommended viewing angle can be marked as the second viewing angle, and the first viewing angle is greater than the second viewing angle.

[0074] In some embodiments, the recommended viewing angle can be a viewing angle range.

[0075] Understandably, different error types can be labeled with different recommended perspective ranges.

[0076] Furthermore, in some embodiments, when multiple different error types exist, the recommended viewing angle amplitude can be selected by combining the error frequencies of the error types.

[0077] For example, in some embodiments, selecting the recommended display range based on the performance evaluation results includes the following steps: 1. Obtain at least two recommended display ranges based on multiple error types; For example, in some embodiments, when the recommended display ranges for two error types are the same, a single recommended display range is generated.

[0078] 2. Generate a suggestion level for the recommended display interval based on the error frequency of the error type; For example, the higher the frequency of errors, the higher the level of recommendation.

[0079] For example, in some embodiments, the more error types corresponding to the recommended display range, the higher the recommendation level. It is understood that the specific rules for the recommendation level can be customized or selected based on the actual application scenario (such as the target user group or performance scenario requirements).

[0080] 3. Select the final recommended display range according to the suggested procedure.

[0081] For example, the recommended display interval with the highest level of suggestion is selected as the final recommended display interval. In some embodiments, selecting the recommended interval based on the performance evaluation results includes the following steps: Obtain the score category of the score, the score category is used to define the priority performance features of the score, and the priority performance features are associated with performance posture indicators, and the performance posture includes one or more of the following: finger performance posture (such as specific fingering, hand shape posture), upper limb performance posture, upper body performance posture (such as whether the back is hunched), and lower limb performance posture (such as whether the pedal is stepped correctly). Preferably, the above different posture information can be obtained by capturing images using one or more cameras.

[0082] The shooting angle is selected based on the performance evaluation results and the music score category.

[0083] For example, in some embodiments, the corresponding association of the music score category has at least one performance attitude index, and when there are at least two performance attitude indices, different performance attitude indices can also be marked with performance priority.

[0084] For example, in some embodiments, the music score category can be one or more of the following: polyphonic music, fast-paced technical music (which mainly focuses on technical training and is very helpful for improving fingering skills), allegro movements (which often require finger dexterity and evenness), etc.

[0085] For example, when the score is classified as polyphonic music, allegro movement, or fast-paced technical music, the recommended performance posture index can be the finger posture.

[0086] For example, in some embodiments, when the score is a late Romantic work, the recommended posture index can be the posture of the upper limbs and / or the posture of the upper body.

[0087] For late Romantic works, it is often necessary to produce a rich, resonant, and penetrating sound. This sound requires the weight of the entire body and coordinated effort (from the waist, back, shoulders, and arms down to the fingertips).

[0088] For example, in some embodiments, when the music score is of a wide range, continuous chords and octaves, or impressionistic music, the recommended performance posture index can be the performance posture of the upper limbs and / or the performance posture of the upper body.

[0089] For example, in some embodiments, when the musical score is a late Romantic work, the recommended performance posture index can be the performance posture of the lower limbs.

[0090] For example, in some embodiments, when there are at least two shooting angles, the display priority of the shooting angle can be defined according to the music score category, and the music score with higher display priority has a larger display interface.

[0091] For example, in some embodiments, when at least two recommended posture metrics exist, the display priority of the shooting angle can be generated based on the recommendation priority of the posture metrics. For instance, the higher the recommendation priority of the posture metric, the higher the display priority of the corresponding shooting angle.

[0092] For example, in some embodiments, a camera can capture one or more shooting angles, such as the camera's shooting angle being automatically adjusted according to real-time needs (i.e., the recommended shooting angle).

[0093] For example, beginners may make many mistakes in the early stages of practice, and their ability to understand or correct these mistakes is relatively limited. In response, this embodiment provides a scheme to select the shooting angle based on the performance evaluation results and the type of sheet music, so as to highlight the content that should be corrected.

[0094] For example, in some embodiments, when beginners encounter a variety of problems, the scores can be filtered and displayed according to their categories to guide them to focus on correcting errors that have a significant impact on their performance, thereby improving their performance level relatively efficiently.

[0095] This centralized guidance approach helps users quickly correct core errors and increases their interest in practice. It avoids situations where excessively difficult practice can weaken users' interest or motivation, thus defeating the purpose of guidance and correction.

[0096] In some embodiments, the steps further include: (1) Obtain the second visualization interface from the previous moment; (2) Calculate the degree of image difference between the first visualization interface and the second visualization interface; For example, in some embodiments, the degree of image difference may refer to the difference in the amplitude of the viewing angle between the two.

[0097] (3) When the difference in the image is less than the preset difference level, the first visualization interface is allowed to be updated.

[0098] In this embodiment, the updating of the visualization interface is further limited by the switching range. This is to reduce the excessive demands on the user's data reception ability by co-displaying video and sheet music while highlighting the key points of the display as much as possible. In other words, it reduces the interference of switching on the user's performance to a certain extent, so as to help maintain / guide the user's attention to the performance itself.

[0099] In some embodiments, S2022 includes the step of: Determine whether the data progress (such as parsing progress or receiving progress) of the video data has reached a preset progress threshold; If so, the lower limit boundary of the stated view amplitude will be updated or maintained as the first lower limit boundary; If not, the lower limit boundary of the stated view amplitude is updated or maintained as the second lower limit boundary; Wherein, the first lower limit boundary is smaller than the second lower limit boundary, and the view amplitude is used to define the range of the displayed scene.

[0100] For example, in some embodiments, the recommended display range can also be a recommended display viewing angle range, which can be a viewing angle range range. The size of the viewing angle range range is defined by its lower limit boundary. The larger the lower limit boundary, the smaller the range of selectable viewing angle ranges. The smaller the lower limit boundary, the larger the range of viewing angle ranges.

[0101] For example, in some embodiments, the video data parsing progress refers to the parsing progress of the received data packets (such as parsing speed). A larger parsing progress may indicate a relatively good current communication environment, thus allowing for high-definition magnification of local viewpoints. Conversely, when the communication environment is relatively poor, a larger viewing angle is preferred for a more complete display, in order to reduce the perceived stuttering of the video to some extent.

[0102] For example, in some embodiments, the video data reception progress refers to the data packet reception progress. When a complete data packet is received within a specified time, it can also indicate that the current communication environment is better.

[0103] In this embodiment, the data progress of the video data is selected based on the range of the viewing angle. This can minimize the stuttering sensation in poor communication environments while ensuring that the key points of the displayed content are highlighted as much as possible, thus maintaining a relatively smooth display environment and reducing interference with the user's performance.

[0104] Furthermore, the guidance scheme in this invention, which facilitates the centralized and rapid correction of errors, works in conjunction with restrictive switching schemes (such as limiting the switching range or time). On the one hand, it guides users to improve their playing level (i.e., the overall state of their performance) relatively efficiently through limited error correction, guiding them gradually into a better state while avoiding excessive disruption to their playing process. This gradual improvement in playing level, combined with the low-interference characteristics of the visual interface, further helps guide users to maintain or enter an immersive practice state, thereby effectively enhancing the user's application experience.

[0105] In some embodiments, the visualization rules further include: visual range: the display area of ​​the sheet music interface, and the display priority of different shooting angles.

[0106] For example, in some embodiments, the display area of ​​the sheet music interface refers to the display area of ​​the first visualization interface on the sheet music interface, that is, the display size.

[0107] The display area of ​​the sheet music interface can have a default set display area size.

[0108] Alternatively, the display area of ​​the sheet music interface can be selected by the user-input adjustment signal.

[0109] Furthermore, when there are at least two display views, each display view has a display area, and the size of the display area can be determined according to priority (such as display priority). For example, the higher the priority, the larger the display area.

[0110] Furthermore, in some embodiments, it also includes: Obtain user input adjustment signals; The visualization rules can be adjusted based on user input signals.

[0111] For example, users can manually adjust the display position. Figure 4 , Figure 5 As shown, the first visualization interface can be displayed above or below the sheet music page.

[0112] like Figure 4 As shown, the first visualization interface allows one or more perspectives (such as a perspective display icon), and users can manually switch the display perspective.Figure 4 This is an example of a display interface diagram, in which a video can be displayed at the top and a musical score can be displayed at the bottom.

[0113] In some embodiments, the video data may be an actual performance video of the user, or it may be a demonstration video of a teacher, or a performance / demonstration video of a professional performer.

[0114] It should be noted that the video and sheet music data collaborative display solution provided by this invention has technical advantages in the following multiple application scenarios: 1) Online teaching via live streaming: If both the teacher and students are online, the teacher's performance data (such as video data and hardware data) can be transmitted to the student's device via the network. For example, the teacher's performance video can be played on the student's tablet, or the performance video can be played on the student's tablet while the performance sound is reproduced on the student's piano.

[0115] 2. Synchronously transmit video data and sheet music data online: During the live stream, either a student or a teacher can play the piano, and the other party can see the performance data displayed on the staff on their local device (such as a tablet). This data includes real-time dynamics, note values, pedal strokes, and other information. The data is aligned with the staff, making it easy to understand and synchronized with the video, ensuring that the progress of the staff on the screen matches the finger movements in the video demonstration. 3. Video playback Demonstration videos recorded by teachers or practice videos recorded by students can be saved locally or to the cloud for easy playback. For example, teachers or students can manually input location signals (such as clicking on any position on the staff that has been played to find and play the video at that position).

[0116] If the performance is completed and a follow-up session is supported immediately, both the teacher and student can quickly view the video, performance data, and audio recordings to create a comprehensive teaching experience. This method of piano instruction can be even better than having the teacher physically present.

[0117] The beneficial technical effects of Example 1 are as follows: First, this invention provides a solution for real-time, immersive collaborative display of video data. This collaborative display solution can reduce the burden on users when receiving complex information, while providing timely and significant feedback on user issues.

[0118] Specifically, this invention can comprehensively generate an optimized video interface by considering both the user's performance level / state and the quality of the video data. Furthermore, this optimized video interface can provide key prompts to the user while simultaneously displaying multi-dimensional data in sync with the sheet music interface. This allows the user to quickly grasp crucial information at the current moment, preventing the multi-dimensional data display from interfering with the user's immersive performance.

[0119] Example 2 See Figure 3 As shown, the present invention also provides a video data display system 200, comprising: The data receiving module 201 is used to receive video data and hardware data of the user during the performance, and the video data and the hardware data are associated with timestamps; wherein, the video data includes: image data of the user and the playing device at at least one shooting angle, and the hardware data includes: fingering data generated by the playing device during the performance. A visual interface module 202 is used to generate a first visual interface based on the video data; wherein, the visual interface module includes: The performance evaluation unit 2021 is used to generate or update performance evaluation results based on the hardware data and / or the video data, wherein the performance evaluation results include at least one of the following evaluation indicators: type of fingering error, type of upper limb error, type of body error, error frequency of at least one type of error, and the evaluation indicators are pre-associated with a recommended display range of at least one video data. The rule generation unit 2022 is used to generate visualization rules based on the performance evaluation results and the data progress of the video data, wherein the data progress refers to the parsing progress and / or reception level of the video data, and the visualization rules include: The data range includes at least one of the following recommended display intervals: the time scale of the image data to be displayed, the shooting angle, and the viewing angle range of the image data; The interface generation unit 2023 is used to generate a first visualization interface based on the visualization rules and the video data; Display module 203 is used to display the first visualization interface in the music score interface.

[0120] It is understood that the system in this embodiment can also be used to perform the methods or steps in any of the exemplary embodiments described above.

[0121] Example 3 While existing technologies have attempted to provide an online teaching solution (such as patent application CN202211311948.X, which discloses a one-to-many online piano tutoring method), the applicant notes that, unlike traditional online teaching, piano teaching involves a high degree of interactivity and concentration. Therefore, existing online tutoring methods are insufficient to meet the specific requirements of piano teaching.

[0122] For example, during a student's practice, the teacher needs to pay close attention to their fingering, hand position, posture, and the final performance, and provide timely feedback (such as suggestions for adjustment) to the student's performance. Alternatively, from the student's perspective, especially in the early stages of learning or when their familiarity with the score is low, they not only need to focus on reading the score but also need to concentrate highly on their fingering and body movement. Therefore, their ability to adjust to or receive feedback from external guidance may be relatively limited.

[0123] In this context, especially for students, the distance created by online teaching further diminishes the efficiency and effectiveness of teacher guidance. Therefore, traditional online teaching methods struggle to meet the specific needs for interactivity and focus required in the unique setting of piano instruction.

[0124] To address this, the present invention provides a method for collaboratively displaying video and sheet music interfaces. This collaborative display method can intelligently recommend and display videos for piano teaching / practice scenarios such as teacher-student interaction or student self-practice. Alternatively, it can provide teachers or students with a method for synchronously displaying multi-dimensional data from videos and sheet music, highlighting key information, thus enabling users (such as students) to maintain high self-focus while quickly capturing key external feedback.

[0125] In other words, this invention provides a multi-dimensional data display solution with strong interactivity and collaboration, which can alleviate the sense of distance in online piano teaching to a certain extent, while reducing the pressure on users to understand multi-dimensional data displays.

[0126] See also Figure 6 As shown, the present invention also provides a method for transmitting performance data, including the following steps: S101, Obtain a first dataset collected by a first device terminal of a first user, wherein the first device terminal includes: a first musical instrument, a first image device (such as a camera) set for the first musical instrument, and the first dataset includes: image data collected by the first image device at at least one shooting angle. S102, the first dataset is transmitted to a second device of at least one second user, the second device including: a second musical instrument and / or a second display device; S103, Generate the music score interface based on the music score advancement data; the music score advancement data is determined by the positioning signal input by the first user or the second user, or the music score advancement data is generated based on a second dataset, the second dataset including: second hardware data and / or second video data, and the second hardware data is the fingering data of the second user, and the second video data is image data acquired by the second image device at the second device end at at least one shooting angle; In some embodiments, score progression data refers to the current performance progress of the score, such as its use to define the position of the note that needs to be played or is being played (it can also be referred to as score positioning data).

[0127] S104, in response to a visualization signal emitted by the first user and / or the second user, adaptively generate visualization rules, wherein the visualization signal includes: the first dataset or the second dataset; the visualization rules include: The data range includes at least one of the following recommended display intervals: the time scale of the image data to be displayed, the shooting angle, and the spectral amplitude of the image data; wherein the time scale is determined by the score progression data; S105, in response to the visualization rules and the first dataset, a first visualization interface (the first visualization interface is a video interface) is generated. S106, the sheet music interface and the first visualization interface are displayed on the second display device.

[0128] This embodiment provides a scheme to optimize the display method of the video and sheet music display interface of the second user port based on the interaction relationship between the first user and the second user (such as their respective performance data).

[0129] In some embodiments, at least one device may include: a musical instrument (also referred to as a playing device), a display device, and an image device. The image device is configured to correspond to the musical instrument and to capture and record the user's motion data while the user is playing the instrument.

[0130] Preferably, the at least one musical instrument may also be equipped with a data acquisition module, which is used to acquire fingering data.

[0131] Preferably, the at least one device may further include a sound acquisition module for acquiring sound data generated during the performance.

[0132] Correspondingly, the dataset collected by a device may include one or more of the following: video data, fingering data, and audio data.

[0133] Furthermore, at least one device may further include a display device for providing a data display interface, such as a first visualization interface and a fourth visualization interface (i.e., a sheet music interface). Different visualization interfaces can be displayed through one or more devices. For example, a single display device can simultaneously play the first and fourth visualization interfaces, or two sub-display devices can be used to display the first and fourth visualization interfaces respectively. In some embodiments, the first user is a music teacher and the second user is a student; or, in some embodiments, the first user is a student and the second user is a music teacher.

[0134] In some embodiments, the first instrument and the second instrument are pianos.

[0135] It is understood that the visualization rules in this embodiment are mainly used to optimize the display of the video and sheet music interface according to the real-time needs of the first user / second user. In particular, the video display can be adjusted based on the visualization rules.

[0136] Among them, the visualization rules mainly involve adjusting the displayed content (such as perspective and amplitude), display area, display position on the interface, and displayed musical score segments (such as the displayed time period) of the video interface.

[0137] The following section introduces the generation of visualization rules through exemplary solutions in different scenarios: In some embodiments, the time scale determined by the score progression data refers to: based on the performance time node determined by the score progression data, calling performance data for at least a set time period (i.e., time scale) after the corresponding performance time node.

[0138] In some embodiments, the specific process by which the score progression data (or score positioning data) is determined by the positioning signal input by the first user or the second user can be as follows: For example, displaying the sheet music interface on a display device (such as...) Figure 4 (The sheet music described below) At this time, the user can click to select a note (equivalent to the user triggering or inputting a positioning signal), and then the note (or the performance time node corresponding to the note, or the performance position) is used as the sheet music progression data, so as to select the video data corresponding to the sheet music progression data (such as playing the performance video starting from the performance time node).

[0139] For example, when the user selects the second section, the video of the second section being played will begin.

[0140] For example, students can choose to review a specific musical excerpt based on their own practice needs. Alternatively, teachers can select excerpts that students need to focus on practicing, taking into account the overall teaching situation.

[0141] Alternatively, in some embodiments, the score progression data can be predicted and generated using a score alignment algorithm from datasets (such as a first dataset and a second dataset).

[0142] As mentioned above, the generation of score progression data based on the second dataset refers to the use of a score alignment algorithm to predict the current performance time node based on the second dataset.

[0143] For example, a display device can analyze continuous performance actions (such as pitch, dynamics, and duration) based on received hardware data, and dynamically match the current position in the local sheet music corresponding to the current hardware data to obtain sheet music positioning data. This sheet music positioning data is information about the specific position of the current performance within a preset sheet music, and may include information such as bar number and beat. It can also be used to drive automatic page turning and highlighting of the sheet music.

[0144] Furthermore, the sheet music location data can be bound to the timestamp of the current hardware data. Based on the timestamp, when the hardware data is played and presented on the instrument, the corresponding sheet music location data can be called and displayed synchronously on the display device.

[0145] Furthermore, the generation process of the score positioning data is completed locally (e.g., when displayed on a second display device, the data processing unit of the second display device can be used for data processing), without relying on remote data, avoiding network transmission delays, and realizing time-delay-free score display synchronized with hardware data.

[0146] Specifically, based on a preset score alignment algorithm, the system matches and analyzes the continuous hardware data received within a preset time period before and after the current moment with the locally stored performance score, thereby accurately calculating the score position corresponding to the current hardware data and generating score positioning data with a timestamp. Then, based on the score positioning data, the system can determine the score segment (or score fragment) that should be displayed at the current moment and control the display device to display the corresponding score fragment in real time, i.e., play the performance video.

[0147] The preset time period is a time window centered on the current moment of the current hardware data, extended before and after by preset time intervals. This is used to generate accurate score positioning data based on the context information of the current hardware data. The preset performance score is pre-stored locally or downloaded score data, containing information such as note sequences and rhythms.

[0148] The preset score alignment algorithm refers to an algorithm used to match real-time performance data (such as pitch, time value, etc. corresponding to hardware data) with standard scores. For example, the score alignment algorithm can adopt an audio-score matching method disclosed in CN2025102799337, or it can adopt other existing alignment methods. This invention does not limit this.

[0149] It is understood that the video and sheet music collaborative display method in this invention is preferably applied to live piano teaching scenarios.

[0150] For example, in application scenario 1, students practice piano at home while teachers teach online from a different location. Teachers can demonstrate playing the piano on their local piano, and the corresponding video of the demonstration can be simultaneously transmitted to the student's tablet (i.e., display device). Students can then practice synchronously by imitating the teacher's demonstrated fingering.

[0151] Furthermore, teachers can also implement one-to-many teaching based on online teaching, that is, multiple students can practice simultaneously.

[0152] For example, in application scenario 2, students practice piano at home. Their practice videos can be simultaneously transmitted to the teacher's tablet. The teacher can then observe the students' fingering and posture to ensure accuracy and provide timely guidance. Furthermore, multiple students' practice videos can be transmitted to the teacher's device for simultaneous instruction.

[0153] For example, in some embodiments, the first or second dataset may be pre-recorded data, such as performance data (e.g., video data or hardware data) that teachers or students can store during a performance, and the teachers or students can then replay the performance data afterwards.

[0154] In some embodiments, S104 includes: S1041, Generate or update performance evaluation results based on the second dataset, wherein the performance evaluation results include at least one of the following evaluation indicators: type of fingering error, type of upper limb error, type of body error, error frequency of at least one type of error, and the evaluation indicators are pre-associated with a recommended display interval of at least one video data; In some embodiments, the performance evaluation results can be obtained by analyzing fingering data or video data in a second dataset.

[0155] S1042, Generate visualization rules based on the performance evaluation results and the data progress of the video data, wherein the data progress refers to the parsing progress and / or reception level of the video data.

[0156] Preferably, taking a scenario where a teacher is demonstrating and teaching as an example, the video data used in S1042 is the performance video of the teacher (such as the first user), also referred to as the first video data.

[0157] In other words, in this embodiment, the recommended display content will be generated by combining the student's performance and the quality of the video data from the teacher's end.

[0158] In this embodiment, the second dataset is used as a guide to optimize the visualization rules. The video data is called accordingly based on the student's performance progress and performance status, so that the student can intuitively understand the external guidance (such as video feedback) while maintaining the continuity of the performance.

[0159] In other words, this embodiment flexibly adjusts the collaborative visualization of videos and scores based on students' performance evaluation results, which can enhance the display of key data to reduce the difficulty for students to understand the data and help them maintain a high level of focus in order to maintain an immersive performance state.

[0160] It is understood that the specific method for generating the first visual interface in this embodiment can also refer to the method section in Embodiment 1, and will not be repeated here.

[0161] In some embodiments, the steps further include: Store the first dataset and / or the second dataset into the storage unit.

[0162] For example, in some embodiments, the storage unit can be a user's smart device (such as a display device), or it can be a network cloud drive connected to the user's smart device to support data revisit functionality.

[0163] Therefore, in some embodiments, the first dataset or the second dataset can be retrieved from the storage unit. In some embodiments, the first device includes: a first display device; correspondingly, the method further includes: At least one third visual interface is displayed on the first display device (such as a tablet or computer for teachers), and each of the at least one third visual interface is generated by the second video data corresponding to at least one second user. The display format of at least one of the third visualization interfaces is set by the performance evaluation results. The display format is used to define at least one of the following enhanced display attributes: display position, display brightness, display area, and display markers (such as adding error prompts or error type icons to the video interface).

[0164] In this embodiment, the third visualization interface is preferably used to define the video interface generated based on the student's performance video. Preferably, the teacher can simultaneously receive video interfaces transmitted by one or more students, and can provide different reinforcement reminders to the teacher based on the different demonstration states of different students.

[0165] For example, the specific steps for setting the display format of at least one of the third visualization interfaces based on the performance evaluation results can be as follows: When a student’s performance evaluation result falls into the preset first performance state, add or update the first enhanced display attribute for the corresponding third visualization interface. When a student’s performance evaluation result falls into the predetermined second performance state, add or update the second enhanced display attribute for the corresponding third visualization interface. Among them, the visual emphasis effect of the first enhanced display attribute is greater than that of the second enhanced display attribute, such as the display position of the first enhanced display attribute being before the display position of the second enhanced display attribute, or the display area / brightness of the first enhanced display attribute being greater than the display area / brightness of the second enhanced display attribute, or the number / size of the display markers of the first enhanced display attribute being greater than the number / size of the display markers of the second enhanced display attribute.

[0166] For example, a performance evaluation result belonging to the first performance state may mean that the number of error types belongs to a large first numerical range, or that the error frequency of at least one error is greater than a preset frequency threshold.

[0167] For example, a performance evaluation result belonging to the second performance state may mean that the number of error types falls within a smaller second numerical range, or that the error frequency of at least one error is less than or equal to a preset frequency threshold. In some embodiments, the visualized signal includes: The first visual signal input by the first user includes one or more of the following information: shooting angle, viewing angle range, and display area.

[0168] For example, the first user could be a teacher. In this case, the teacher can freely choose the information they want students to focus on, based on teaching needs, such as freely switching the shooting angle display perspective in the visualization rules, or adjusting the display area, etc.

[0169] In some embodiments, the visualized signal includes: The second visual signal input by the second user includes one or more of the following information: shooting angle, viewing angle range, and display area.

[0170] For example, the second user can be a student, who can independently choose the information displayed to flexibly customize settings according to their own needs. Students can also freely adjust parameters such as the viewing angle and size of the display interface on their own tablets to suit the varying learning needs in a piano learning setting.

[0171] Furthermore, in some embodiments, the first visualization signal or the second visualization signal may also include at least one enhanced display attribute of the shooting angle.

[0172] For example, in some embodiments, when multiple shooting angles are displayed simultaneously, one or more shooting angles can be highlighted, such as by displaying the shooting angle in front or by adding enhanced display elements. Enhanced display elements can include brightness values, border displays, etc. For instance, if the enhanced display element is a brightness value, the display brightness of the highlighted shooting angle can be appropriately increased, or a display border can be added to that shooting angle (or the display border can be thickened, brightened, etc.).

[0173] In some embodiments, the second device further includes: a second broadcasting device, and correspondingly, the step of: The sound effects of the first user's performance are reproduced in the second musical instrument or the second broadcasting device based on the dataset.

[0174] For example, in some embodiments, the dataset can be a first dataset, which includes not only the teacher's performance video but also the teacher's performance sound effects transmitted synchronously to the student's end.

[0175] For example, the synchronous transmission of performance sound effects can be achieved by directly reproducing the teacher's performance effect on the smart piano when the student is using it, based on the teacher's fingering data (which can be converted from hardware data).

[0176] For example, the synchronous transmission of performance sound effects can be achieved by converting the teacher's fingering data (i.e., the pitch, dynamics, duration, etc. of each note used for musical attributes) into MIDI data, which can then be played through the user's smart devices (such as mobile phones, tablets, and other audio playback devices) to achieve synchronous transmission of visual and audio information. The beneficial technical effects of this embodiment are: This embodiment leverages the high interactivity and attention requirements of piano education scenarios to provide teachers or students with a method for synchronously displaying multi-dimensional data of video and sheet music, while simultaneously highlighting key aspects of this multi-dimensional data. This allows users (such as students) to maintain a high level of self-focus while quickly capturing key external feedback information.

[0177] Preferably, this embodiment includes an optimization scheme for the display method of the video and sheet music display interface of the second user port based on the interaction relationship between the first user and the second user (such as their respective performance data).

[0178] Specifically, this embodiment flexibly adjusts the collaborative visualization of videos and sheet music based on students' performance evaluation results and the completeness of data on the teacher's end. This allows for enhanced display of key data while ensuring the quality of video display. Furthermore, by providing multi-dimensional data display, it reduces the difficulty for students to understand the data, enabling them to maintain a high level of focus and thus sustain an immersive performance state.

[0179] Example 4 See Figure 7 As shown, the present invention provides a system for transmitting performance data, comprising: The first data acquisition module 101 is used to acquire a first dataset obtained by the first user's first device terminal. The first device terminal includes: a first musical instrument, a first image device set corresponding to the first musical instrument, and the first dataset includes: image data acquired by the first image device at at least one shooting angle. The first data transmission module 102 is configured to transmit the first dataset to a second device terminal of at least one second user, wherein the second device terminal includes: a second musical instrument and a second display device; The second interface generation module 103 is used to generate a music score interface based on the music score progression data; the music score progression data is determined by the positioning signal input by the first user or the second user, or the music score progression data is generated based on a second dataset, the second dataset including: second hardware data and / or second video data, and the second hardware data is the fingering data of the second user, and the second video data is image data acquired by the second image device at at least one shooting angle through the second device terminal; The visualization signal parsing module 104 is configured to adaptively generate visualization rules in response to visualization signals emitted by the first user and / or the second user, wherein the visualization signals include: the first dataset or the second dataset; and the visualization rules include: The data range includes at least one of the following recommended display intervals: the time scale of the image data to be displayed, the shooting angle, and the spectral amplitude of the image data; wherein the time scale is determined by the score progression data; The first interface generation module 105 is used to generate a first visualization interface in response to the visualization rules and the first dataset. The collaborative display module 106 is used to display the first visualization interface and the sheet music interface on the second display device.

[0180] It is understood that the system in this embodiment can also be used to implement the methods or steps in any of the above embodiments, which will not be repeated here.

[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a computer terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0183] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multi-dimensional data display method characterized by comprising: The method comprises the steps of: S101, acquiring a first data set collected by a first device end of a first user, the first device end comprising a first musical instrument and a first image device corresponding to the first musical instrument, and the first data set comprising image data collected by the first image device at at least one shooting angle; S102, transmitting the first data set to a second device end of at least one second user, the second device end comprising a second musical instrument and a second display device; S103, generating a score interface according to score advancing data, the score advancing data being determined by positioning signals input by the first user or the second user, or the score advancing data being generated according to second data set, the second data set comprising second hardware data and / or second video data, the second hardware data being fingering data of the second user, and the second video data being image data collected by a second image device of the second device end at at least one shooting angle; S104, adaptively generating a visualization rule in response to a visualization signal issued by the first user and / or the second user, wherein the visualization signal comprises the first data set or the second data set, and the visualization rule comprises: a data range, and the data range comprising at least one recommended display interval of the image data, the shooting angle, and the visual angle amplitude of the image data; wherein the time scale is determined by the score advancing data; S105, generating a first visualization interface in response to the visualization rule and the first data set; S106, displaying the first visualization interface and the score interface on the second display device.

2. The method of claim 1, wherein, The first user is a musical instrument teacher, and the second user is a student; or the first user is a student, and the second user is a musical instrument teacher.

3. The method of claim 1, wherein, The first musical instrument and the second musical instrument are pianos.

4. The method of claim 1, wherein, S104 comprises: S1041, generating or updating a performance evaluation result according to the second data set, wherein the performance evaluation result comprises at least one evaluation index of a type of fingering error, a type of upper limb error, a type of body error, and a frequency of at least one type of error, and the evaluation index is pre-associated with at least one recommended display interval of video data, the video data being first video data or second video data; S1042, generating a visualization rule according to the performance evaluation result and a data progress of the first video data, wherein the data progress is an analysis progress and / or a receiving degree of the first video data.

5. The method of claim 1, wherein, The method further comprises the step of: storing the first data set and / or the second data set to a storage unit.

6. The method of claim 4, wherein, The first device end comprises a first display device, and correspondingly, the method further comprises: displaying at least one third visualization interface on the first display device, the at least one third visualization interface being respectively generated by at least one second video data. The reinforced display form of at least one of the third visual interfaces is set by the performance evaluation result, and the reinforced display form is used to define at least one of the following reinforced display attributes: display position, display brightness, display area, and display mark.

7. The method of claim 1, wherein, The visual signal includes: A first visual signal input by a first user, the first visual signal including one or more of the following information: shooting angle, visual angle amplitude, and display area; And / or a second visual signal input by a second user, the second visual signal including one or more of the following information: shooting angle, visual angle amplitude, and display area.

8. The method of claim 1, wherein, The second device end further includes a second broadcasting device, and correspondingly further includes the following steps: Reproducing the performance sound effect of the first user in the second musical instrument or the second broadcasting device according to the first data set.

9. A multidimensional data display system, characterized by The method includes: A first data acquisition module is configured to acquire a first data set collected by a first device end of a first user, the first device end including a first musical instrument and a first image device arranged for the first musical instrument, and the first data set including image data collected by the first image device at at least one shooting angle; A first data transmission module is configured to transmit the first data set to a second device end of at least one second user, the second device end including a second musical instrument and a second display device; A second interface generation module is configured to generate a score interface according to score advancing data, the score advancing data being determined by a positioning signal input by the first user or the second user, or the score advancing data being generated according to a second data set, the second data set including second hardware data and / or second video data, the second hardware data being fingering data of the second user, and the second video data being image data collected by a second image device of the second device end at at least one shooting angle; A visual signal analysis module is configured to adaptively generate a visual rule in response to a visual signal issued by the first user and / or the second user, the visual signal including the first data set or the second data set, and the visual rule including: A data range, and the data range including at least one recommended display interval: time scale of image data to be displayed, shooting angle, and visual angle amplitude of image data, wherein the time scale is determined by the score advancing data; A first interface generation module is configured to generate a first visual interface in response to the visual rule and the first data set; A cooperative display module is configured to display the first visual interface and the score interface on the second display device.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the method of any one of claims 1-8.

Images