Performance data transmission method, system and program product

Abstract

The invention relates to the piano online teaching technology, in particular to a playing data transmission method and system and a program product, and the method comprises the steps: obtaining a first data set collected by a first device end of a first user; transmitting the first data set to a second device end of at least one second user; generating a music score interface according to the music score propelling data; responding to a visualization signal sent by the first user and / or the second user, and adaptively generating a visualization rule; responding to the visualization rule and the first data set to generate a first visualization interface; and displaying the first visual interface and the music score interface on the second display device. According to the invention, a video and music score synchronous display scheme is provided for piano online teaching, the scheme has relatively high real-time performance and interactivity, and a recommended display scheme can be set for teachers or students in a targeted manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of online piano teaching, and particularly relates to a performance data transmission method, system and program product. BACKGROUND

[0002] At present, mainstream piano learning needs to rely on real-time guidance of a real teacher, resulting in a very high learning cost of the piano.

[0003] In this regard, the prior art also attempts to provide an online teaching scheme: for example, patent application 202211311948.X discloses a one-to-many piano online accompaniment method, which comprises the following steps: S1: selecting an online cloud and adjusting offline equipment; S2: installing a real-time audio and video transmission device and starting a real-time audio converter; S3: the system collects student audio and video and transmits the student audio and video to the teacher's mobile phone in real time; S4: the student is explained through real-time live broadcast, and a display screen synchronously displays a virtual music keyboard; S5: the teacher evaluates and initiates a personal voice call for guidance; and S6: the student's playing process is analyzed by a video recorder, and the system gives an evaluation in combination with the music piece.

[0004] For another example, the prior art also attempts to provide a musical instrument auxiliary teaching method based on AR technology: for example, patent application 202210512685.2 discloses a musical instrument auxiliary teaching method and system based on AR technology, which comprises a collection step, a teaching video is played through an AR device, and actual video data and actual audio data during playing are synchronously acquired; a judgment step, a loudness peak of a tone in the actual audio data is acquired to determine an actual rhythm point, audio at the actual rhythm point is acquired to determine whether a playing error occurs, and actual fingering pictures in the actual video data are extracted through the actual rhythm point to determine whether a fingering error occurs; and a correction step, a playing error or a fingering error of the actual rhythm point is acquired, and correct audio data or correct fingering pictures corresponding to the actual rhythm point are called, and teaching is guided through audio comparison or picture comparison.

[0005] However, the applicant believes that for the special application of piano education, the traditional live teaching or AR technology cannot solve the application pain points.

[0006] Therefore, there is an urgent need for a method for live teaching that can be applied to the piano education scene which is relatively professional. SUMMARY

[0007] The present application aims to provide a performance data transmission method, system and program product, which partially solves or alleviates the above-mentioned deficiencies in the prior art, can dynamically adjust the visualization rules of video data, and alleviates or reduces the interference of multi-dimensional data switching display on the user's piano practice.

[0008] In order to solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions: A first aspect of the present invention is to provide a method for transmitting performance data, comprising the steps of: S101, obtaining a first data set collected by a first device of a first user, where the first device includes: a first musical instrument, a first imaging device configured for the first musical instrument, and the first data set includes: image data collected by the first imaging device at at least one shooting angle; S102, transmitting the first data set to a second device of at least one second user, where the second device includes: a second musical instrument and a second display device; S103: Generate a music score interface based on the music score advancement data; the music score advancement data is determined by a positioning signal input by the first user or the second user, or the music score advancement data is generated based on a second data set, the second data set including: second hardware data and / or second video data, wherein the second hardware data is fingering data of the second user, and the second video data is image data captured by a second imaging device on the second device end at at least one shooting angle; S104: Adaptively generate visualization rules in response to a visualization signal sent by the first user and / or the second user, wherein the visualization signal includes: the first data set or the second data set; and the visualization rules include: a data range, wherein the data range includes at least one of the following recommended display intervals: a time scale of the image data to be displayed, a shooting angle, and a viewing angle amplitude of the image data; wherein the time scale is determined by the music score advancement data; S105 , generating a first visualization interface in response to the visualization rule and the first data set; S106: Display the first visualization interface and the music score interface on the second display device.

[0009] In some embodiments, the first user is a musical instrument teacher and the second user is a student; or, the first user is a student and the second user is a musical instrument teacher.

[0010] In some embodiments, the first musical instrument and the second musical instrument are pianos.

[0011] In some embodiments, S104 includes: S1041, generating or updating a performance evaluation result according to the second data set, wherein the performance evaluation result comprises at least one of the following evaluation indexes: a type of fingering error, a type of upper limb error, a type of body error, and a frequency of at least one type of error, and the evaluation indexes are previously associated with at least one recommended display interval of video data, wherein the video data is the first video data or the second video data; S1042, generating a visualization rule according to the performance evaluation result and a data progress of the first video data, wherein the data progress is an analysis progress and / or a receiving degree of the first video data.

[0012] In some embodiments, the method further comprises the step of: storing the first data set and / or the second data set to a storage unit.

[0013] In some embodiments, the first device end comprises a first display device, and correspondingly, the method further comprises the steps of: displaying, by the first display device, at least one third visualization interface generated by at least one second video data; wherein an enhanced display form of the at least one third visualization interface is set by the performance evaluation result, and the enhanced display form is used to define at least one of the following enhanced display attributes: display position, display brightness, display area, and display mark.

[0014] In some embodiments, the visualization signal comprises: a first visualization signal input by a first user, wherein the first visualization signal comprises one or more of the following information: shooting angle, angle amplitude, and display area; and / or, a second visualization signal input by a second user, wherein the second visualization signal comprises one or more of the following information: shooting angle, angle amplitude, and display area.

[0015] In some embodiments, the second device end further comprises a second broadcasting device, and correspondingly, the method further comprises the steps of: reproducing, by the second musical instrument or the second broadcasting device, a performance sound effect of the first user according to the first data set.

[0016] The present application provides a performance data transmission system, comprising: a first data acquisition module, configured to acquire a first data set collected by a first device end of a first user, wherein the first device end comprises a first musical instrument, a first image device arranged in correspondence with the first musical instrument, and the first data set comprises image data collected by the first image device at at least one shooting angle. The first data transmission module is configured to transmit the first data set to a second device end of at least one second user, wherein the second device end comprises a second musical instrument and a second display device. The second interface generation module is configured to generate a score interface according to score advancing data, wherein the score advancing data is determined by a positioning signal input by the first user or the second user, or the score advancing data is generated according to a second data set, the second data set comprises second hardware data and / or second video data, and the second hardware data is fingering data of the second user, and the second video data is image data collected by a second image device of the second device end at at least one shooting angle. The visual signal analysis module is configured to adaptively generate a visual rule in response to a visual signal input by the first user and / or the second user, wherein the visual signal comprises the first data set or the second data set, and the visual rule comprises: a data range, and the data range comprises at least one recommended display interval, such as a time scale of image data to be displayed, a shooting angle, and a visual angle amplitude of image data, wherein the time scale is determined by the score advancing data. The first interface generation module is configured to generate a first visual interface in response to the visual rule and the first data set. The collaborative display module is configured to display the first visual interface and the score interface on the second display device.

[0017] The present application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of the embodiments.

[0018] Beneficial technical effects: The present application provides a multi-dimensional data display scheme with strong interactivity and collaboration, which can alleviate the distance feeling of online piano teaching to a certain extent, and reduce the understanding pressure of users on multi-dimensional data display.

[0019] Further, in the present application, the second data set is used as a guide to optimize the visual rule, so that the student's performance progress and performance state are mainly used to call the video data correspondingly, so that the student can intuitively understand the external guidance (such as video feedback) while maintaining the coherence of the performance.

[0020] In other words, the student's performance evaluation results are used to flexibly adjust the collaborative visualization of video and score in the present application, which can strengthen the display of key data to reduce the understanding difficulty of students, and facilitate students to maintain a high degree of concentration to maintain an immersive performance state. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. In all the drawings, similar elements or parts are generally identified by similar reference signs. In the drawings, each element or part is not necessarily drawn according to the actual proportion. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 Flowchart of the method in an exemplary embodiment of the present application; Figure 2 Flowchart of the first visualization interface in an exemplary embodiment of the present application; Figure 3 Module structure diagram of the display system of the video data in an exemplary embodiment of the present application; Figure 4 Interface diagram of the video and score synchronous display in an exemplary embodiment of the present application; Figure 5 Interface diagram of the video and score synchronous display in another exemplary embodiment of the present application; Figure 6 Flowchart of the video display method in still another exemplary embodiment of the present application; Figure 7 Flowchart of the video display method in still another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0024] In this paper, the suffix such as "module", "component" or "unit" used to represent elements is only for the convenience of the description of the present application, and has no specific meaning. Therefore, "module", "component" or "unit" can be used mixedly.

[0025] In this document, the terms "upper", "lower", "inner", "outer", "front", "back", "one end", "the other end" and the like indicate relative positions or orientation of the illustrated directions based on the orientation or position shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0026] In this document, unless otherwise explicitly specified and limited, the terms "mount", "provided with", "connected" and the like should be understood broadly, for example, "connected" can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be directly connected, or indirectly connected through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0027] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0028] In this document, "a plurality of" means two or more, that is, it includes two, three, four, five, etc.

[0029] In this specification, the term "about" typically means + / - 5% of the stated value, more typically + / - 4% of the stated value, more typically + / - 3% of the stated value, more typically + / - 2% of the stated value, even more typically + / - 1% of the stated value, even more typically + / - 0.5% of the stated value.

[0030] In this specification, certain embodiments can be disclosed in a format that is a range. It is to be understood that such a "range" format is merely used for convenience and brevity and should be interpreted in the context of the description unless otherwise indicated. Therefore, the description of a range should be considered to have specifically disclosed all possible sub-ranges as well as individual numerical values within that range. For example, a description of a range 1-6 should be considered to have specifically disclosed sub-ranges like 1-3, 1-4, 1-5, 2-4, 2-6, 3-6, etc., as well as individual numbers 1, 2, 3, 4, 5, and 6 within that range. This same principle applies to ranges having a minimum and / or maximum value.

[0031] "angle of view", herein, the angle of view refers to the angle range that the video picture can cover. It determines how "wide" the scene can be seen by the user through the video. For example, taking the overhead view as an example, the larger the angle of view under the overhead view, the wider the displayed image content, such as the relatively complete piano keys and hand movements. Conversely, if the angle of view is smaller, the number of displayed piano keys may be smaller, i.e. focusing on the display of hand movements.

[0032] Embodiment one: It should be noted that in the traditional guidance scheme, mainly the playback display scheme of the electronic score (on which the performance result is marked) and the video file (such as the performance process and score synchronous display method disclosed in patent application CN202110507335.2) is adopted, but the guidance application of this review scheme is relatively limited, i.e. it is difficult to give timely feedback during user performance.

[0033] To this end, unlike the video review mode, the present application provides a scheme for real-time and immersive collaborative display of video data. Specifically, the present application dynamically adjusts the visualization rules of the video by focusing on the user's performance state, the data progress of the video data and other factors, thereby adjusting the display attention of local information (such as the angle of view of the video, the display area) on the basis of synchronous and real-time display of the video and the score, to ensure that the user can relatively intuitively capture the key information of the video data under collaborative display, and then facilitate the autonomous adjustment of the current performance state.

[0034] In other words, the real-time collaborative display scheme of the video and the score provided by the present application can reduce the difficulty of the user in receiving complex information (such as the video and the score displayed synchronously on the display interface, and the progress mark, error mark, etc. can also be displayed on the score) on the basis of timely and significant problem feedback to the user. That is to say, the present application provides a data feedback scheme for facilitating immersive performance of the user, which can reduce the influence on the user's performance attention to a certain extent.

[0035] Referring to Figures 1-2 The present application provides a display method of video data, comprising the steps of: S201, receiving video data and hardware data of a user in a performance process, and the video data and the hardware data are associated with a time stamp; wherein the video data comprises image data collected by the user and a performance device under at least one shooting angle, and the hardware data comprises fingering data generated by the performance device in the performance process; S202, generating a first visualization interface according to the video data; wherein S202 comprises: S2021, generating or updating a performance evaluation result according to the hardware data and / or the video data, wherein the performance evaluation result comprises at least one of the following evaluation indexes: an error type of a fingering error, an error type of an upper limb error (specifically, a hand shape error, an elbow posture error), an error type of a body error (e.g., a humpback), and a frequency of at least one error type, and the evaluation indexes are pre-associated with at least one recommended display interval of the video data; For example, in some embodiments, the type of fingering error can include one or more of the following: a finger error (i.e., using a wrong finger), a folded finger, a thumb touching a key as a whole (e.g., using the whole side of the thumb or even the flat palm to touch a key), a weak finger independence (e.g., the fingers 3, 4, and 5 are seriously connected, and moving one finger will drag the other fingers to move together), and the like.

[0036] For example, in some embodiments, the type of hand shape error can include one or more of the following: a collapsed palm (e.g., the middle of the palm (metacarpophalangeal joint) is sunken, and the whole hand shape loses the natural arch support), a wrist too high / wrist too low (e.g., the wrist is rigidly arched upward or pressed too low, and cannot maintain a natural relaxed state basically level with the keys, thereby causing a force or time value deviation), and a stiff finger crossing (e.g., when the thumb crosses under the other fingers (or fingers 3, 4, and 5 cross the thumb), the movement is too large, the wrist is twisted too much, or the body is shaken).

[0037] For example, in the case of a piano, the thumb, index finger, middle finger, ring finger, and little finger are usually defined as fingers 1 to 5.

[0038] S2022, generating a visualization rule according to the performance evaluation result and a data progress of the video data, wherein the data progress is an analysis progress and / or a receiving degree of the video data, and the visualization rule comprises: a data range, and the data range comprises at least one of the following recommended display intervals: a time scale of the image data to be displayed, a shooting angle, and a perspective amplitude of the image data. For example, in some embodiments, the shooting angle and the perspective amplitude can be selected according to the performance evaluation result.

[0039] For example, in some embodiments, the time scale refers to a set time maintained by the current visualization rule. For example, in an exemplary embodiment, the set time has a default value pre-set by the user, i.e., preferably, after each visualization rule switching, a certain set time is recommended to be maintained, so as to limit the switching frequency of the visualization rule during the performance process, and reduce the data capture pressure or interference on the user caused by the update of the first visualization interface.

[0040] Or, in some embodiments, the time scale refers to the time length of the video data to be parsed / read at the current time based on the current visualization rule.

[0041] It can be understood that the multiple video data collected by the image device can be sent to the display end (such as a mobile phone or a tablet) in the form of a data packet, and one data packet can package the view information in a time period. Correspondingly, the time scale can be the time length of one or more data packets. In this regard, according to the visualization rule, the length of the data packet to be read and parsed in the next time period can be defined.

[0042] In some embodiments, the length of the time period corresponding to the data packet can be a preset default value.

[0043] Or, in some embodiments, the length of the time period corresponding to the data packet can be dynamically set according to the actual communication environment. For example, when the communication environment is good, the time period corresponding to the data packet can be relatively shortened, that is, the transmission frequency can be more frequent; on the contrary, when the communication environment is poor, the time period corresponding to the data packet can be relatively prolonged.

[0044] S2023, generating a first visualization interface according to the visualization rule and the video data; S203, displaying the first visualization interface in the score interface.

[0045] Preferably, the score interface in the present embodiment is a real-time display interface, that is, the score interface is displayed in real time according to the actual performance progress of the user.

[0046] In some embodiments, the score interface displays score information (such as piano staff), and at least one annotation information is displayed on the score information. The annotation information can include progress information, error information, technique information, etc. For example, the progress information refers to that a progress icon can be set to mark the score position (such as a certain note) according to the performance progress of the user. The error information can be error type, error severity, etc., and the technique information can be skill icon, skill annotation analysis, etc. The annotation information can be dynamically or statically displayed.

[0047] The applicant notes that in piano performance, the requirement for performance concentration is very high, especially for performers who are not familiar with the score (such as piano beginners (such as piano students), or performers who just start to practice the score). They may have difficulty in keeping the performance smooth while capturing and focusing on the information of the electronic score and the video file in time.

[0048] To this end, the embodiment proposes a scheme of dynamically adjusting the first visualization interface according to the performance state and the data progress of the video data, so as to recommend a suitable display perspective or amplitude to the user in combination with the current state, so as to facilitate the user to focus on the core problem.

[0049] In addition, the cooperative display scheme in the present application can also reduce the additional performance requirements of the hardware device configured by the user, so as to alleviate the delay problem caused by the network or hardware performance limitation to a certain extent, and reduce the adverse effects of video lag on the user. That is, the present application can also reduce the implementation cost of cooperative display to a certain extent.

[0050] Preferably, the score interface in the embodiment refers to a display interface for simultaneously displaying score and video data.

[0051] In some embodiments, the score interface can have a default visualization rule when displayed. During the display process, the visualization rule can be dynamically updated according to the user's performance or data receiving situation.

[0052] For example, in some embodiments, the default visualization rule can be a user preset visualization rule.

[0053] Alternatively, in some embodiments, the default visualization rule can be a rule preferred according to the user's usage habit, such as the default visualization rule can be the visualization rule with the highest usage frequency in the historical period.

[0054] Alternatively, in some embodiments, the default visualization rule can be the visualization rule used when the score interface was last displayed.

[0055] For example, in some embodiments, the video data can be a video obtained by an image device (such as a camera), and the image device is connected to a display device (such as the user's mobile phone, tablet device) to synchronously display the video on the display device.

[0056] For example, in some embodiments, one or more cameras can be arranged on both sides of the piano or above the piano to collect video data, and the video data is recorded with a timestamp.

[0057] For example, in some embodiments, the hardware data refers to the user's fingering data (also known as motion capture data, or simply motion capture data), which includes: hand information (such as left hand or right hand) and finger information (such as recording which finger is pressed) for performing pressing action (i.e. pressing the key), the identification of the key (i.e. which key), and the corresponding motion data. The fingering data is recorded with a timestamp.

[0058] For example, in some embodiments, the motion data can be force and / or displacement.

[0059] Specifically, the force here can be the pressure on the key, which is used to feedback the fingering force of the user; the displacement can be the pressing depth of the key, or the rotation angle of the key in a certain period.

[0060] For example, in some embodiments, a speed sensor, a displacement sensor or an acceleration sensor is arranged below the key, which can detect the speed, displacement or acceleration of the corresponding key in real time, and then indirectly calculate the force on the key.

[0061] In some embodiments, the method further comprises the steps of: generating a progress marker according to the hardware data, the progress marker being associated with the musical note; Correspondingly, in S203, the first visualization interface and the progress marker are synchronously displayed in the score interface.

[0062] For example, in some embodiments, for the played score, the standard data of the score (such as the playing order (or time), playing force, playing tone, etc. of each musical note) can be obtained in advance, and the musical note currently played by the user can be determined through the matching relationship between the hardware data and the standard data, so as to mark the playing progress on the score in the score interface.

[0063] For example, in some embodiments, the score interface is generated according to the progress marker and the score data. The dynamic updating of the visualization rule in the application actually adjusts the dynamic display mode (such as display position) of the video and the score.

[0064] In some embodiments, the video data includes: first image data collected by a top view angle; second image data collected by a side view angle. Wherein, the top view angle is arranged to collect the performance posture of the user's fingers, and the side view angle is arranged to collect the performance posture of the user's upper limbs and / or body.

[0065] For example, in some embodiments, at least one camera can be arranged above the piano to collect the first image data, which can be used to capture the complete key image of the piano and the playing state of the user's hands. At least one camera can be arranged on the side of the piano, which can collect the user's upper limbs, body (such as the sitting posture of the upper body, the stepping posture of the legs).

[0066] In some embodiments, step S2022 comprises: S20221, selecting a recommended display interval according to the playing evaluation result; S20222, selecting recommended video data from the video data according to the recommended display interval; S20223, updating the visualization rule according to the recommended display interval when the content integrity of the recommended video data is greater than a preset integrity threshold.

[0067] In some embodiments, one fingering error or hand shape error has at least one recommended display interval.

[0068] For example, in some embodiments, the recommended display interval can be a recommended shooting angle (or in other words, a display perspective), such as a top-down perspective or a side-view perspective, etc.

[0069] For example, when the type of fingering error is a finger error or weak finger independence, the recommended display interval thereof can be a top-down perspective. When the type of fingering error is a folded finger or a thumb key, the recommended display interval thereof can be a side-view perspective, so as to facilitate observation of the folding state of the finger.

[0070] For another example, when the type of hand shape error is a collapsed palm or a high wrist / low wrist, the recommended display interval thereof can be a side-view perspective, and when the type of hand shape error is a stiff finger rotation, the recommended display interval thereof can be a top-down perspective.

[0071] For another example, in some embodiments, the recommended display interval can also be a recommended display perspective amplitude. For example, for some errors with relatively low observation difficulty, a larger perspective amplitude can be recommended for display, so that the user can make a more comprehensive observation. For some errors with relatively high observation difficulty (especially errors that beginners are prone to overlook), a smaller perspective amplitude can be recommended for targeted display.

[0072] For example, for the fingering error type of finger error, the recommended perspective amplitude can be marked as a first perspective amplitude, and for the fingering error type of weak finger independence or the hand shape error type of stiff finger rotation, the recommended perspective amplitude can be marked as a second perspective amplitude, and the first perspective amplitude is greater than the second perspective amplitude.

[0073] In some embodiments, the recommended perspective amplitude can be a perspective amplitude interval.

[0074] It can be understood that different error types can be marked with different recommended perspective amplitudes.

[0075] Further, in some embodiments, when there are multiple different error types, the error frequency of the error types can be integrated to select the recommended perspective amplitude.

[0076] For example, in some embodiments, selecting the recommended display interval according to the performance evaluation result comprises the steps of: 1. Obtain at least two recommended display intervals according to multiple error types; For example, in some embodiments, when the recommended display intervals of two error types are the same, one recommended display interval is generated.

[0077] 2. Generate a suggestion degree for the recommended display interval according to the error frequency of the error type; For example, the higher the error frequency, the higher the suggestion degree.

[0078] For example, in some embodiments, the more error types corresponding to the recommended display interval, the higher the suggestion degree. It can be understood that the specific rules of the suggestion degree can be customized or selected according to the actual application scenario (such as the user group or performance scene requirements).

[0079] 3. Select the final recommended display interval according to the suggestion procedure.

[0080] For example, the recommended display interval with the highest suggestion degree is selected as the final recommended display interval. In some embodiments, selecting the recommended interval according to the performance evaluation result includes the following steps: Obtain the score category of the score, which is used to define the priority performance characteristics of the score, and the priority performance characteristics are associated with performance posture indicators, and the performance postures include one or more of the following: performance postures of fingers (such as specific fingering, hand posture), performance postures of upper limbs, performance postures of upper body (such as whether to hunch back), performance postures of lower limbs (such as whether to step on the pedal correctly); Preferably, the above different performance posture information can be obtained by one or more cameras.

[0081] Select the shooting angle according to the performance evaluation result and the score category.

[0082] For example, in some embodiments, the score category is associated with at least one performance posture indicator, and when there are at least two performance posture indicators, different performance posture indicators can also be marked with performance priority.

[0083] For example, in some embodiments, the score category can be one or more of the following: polyphonic music (typical representatives are "Well-Tempered Clavier", "Creative Collection"), fast skill type music (which mainly focuses on skill training and is helpful for improving fingering skills, such as typical representatives are Clementi's "Path of Virtuoso", Moszkowski's Exercise, etc.), allegro movement (which often requires finger flexibility and uniformity, such as typical representatives are the first movement (allegro) of "Twentieth Piano Concerto in D minor"), etc.

[0084] For example, when the score category is polyphonic music, allegro movement or fast technique music, the recommended performance posture indicator can be the performance posture of the fingers.

[0085] For example, in some embodiments, when the score category is late Romantic music, the recommended performance posture indicator can be the performance posture of the upper limbs and / or the performance posture of the upper body.

[0086] For late Romantic music, it is often necessary to play a rich, loud and penetrating sound. Such a sound must rely on the weight and coordinated force of the entire body (from the waist, back, shoulders, arms to the fingertips).

[0087] For example, in some embodiments, when the score category is music with large span, continuous chord and octave, or impressionist music, the recommended performance posture indicator can be the performance posture of the upper limbs and / or the performance posture of the upper body.

[0088] For example, in some embodiments, when the score category is Russian-style large-scale works or late Romantic music, the recommended performance posture indicator can be the performance posture of the lower limbs.

[0089] For example, in some embodiments, when there are at least two shooting angles, the display priority of the shooting view can also be defined according to the score category, and the score with a higher display priority has a larger display interface.

[0090] For example, in some embodiments, when there are at least two recommended performance posture indicators, the display priority of the shooting angle can be generated according to the recommended priority of the performance posture indicator. For example, the higher the recommended priority of the posture indicator, the higher the display priority of the shooting angle corresponding to it.

[0091] For example, in some embodiments, one camera can capture one or more shooting angles, such as the shooting angle of the camera, which can be automatically adjusted according to the real-time demand (i.e. the recommended shooting angle).

[0092] For example, for beginners, they may make more mistakes in the early stage of practicing the score, and beginners have relatively limited understanding or adjustment ability of errors. Therefore, in this embodiment, a scheme is provided for comprehensively selecting the shooting view according to the performance evaluation result and the score category, so as to highlight the display of the preferred error correction content.

[0093] For example, in some embodiments, when the beginner has many types of problems, the score category can be used to filter and display them, so as to guide the beginner to focus on correcting the errors that have a greater impact on the performance effect, so as to relatively efficiently improve the performance level.

[0094] And this kind of concentration guiding scheme is beneficial to the user to quickly correct the core error and improve the user's practice interest, avoid the practice difficulty being too large to cause the user's practice interest or motivation to weaken, and then lose the purpose of guiding correction.

[0095] In some embodiments, further comprising steps of: (1) obtaining a second visualization interface at a previous time; (2) calculating the image difference degree between the first visualization interface and the second visualization interface; For example, in some embodiments, the image difference degree can refer to the difference in the viewing angle amplitude of the two.

[0096] (3) when the image difference degree is less than a preset difference degree, then allowing updating to the first visualization interface.

[0097] In this embodiment, the update of the visualization interface is further limited from the switching amplitude, so as to reduce the high requirement of the video and score sheet cooperative display on the data receiving ability of the user, that is, to reduce the interference of the switching on the user's performance to a certain extent, so as to assist in maintaining / guiding the user's attention on the performance itself.

[0098] In some embodiments, S2022 comprises steps of: judging whether the data progress (such as parsing progress or receiving progress) of the video data reaches a preset progress threshold; if yes, updating or keeping the lower limit boundary of the viewing angle amplitude as a first lower limit boundary; if no, updating or keeping the lower limit boundary of the viewing angle amplitude as a second lower limit boundary; wherein the first lower limit boundary is smaller than the second lower limit boundary, and the viewing angle amplitude is used to define the displayed scene range.

[0099] For example, in some embodiments, the recommended display interval can also be a recommended display viewing angle amplitude, which can be a viewing angle amplitude interval, wherein the interval size of the viewing angle amplitude interval is defined by its lower limit boundary, for example, the larger the lower limit boundary, the smaller the range of the selectable viewing angle amplitude, and the smaller the lower limit boundary, the larger the range of the viewing angle amplitude.

[0100] For example, in some embodiments, the parsing progress of the video data refers to the parsing progress (such as the parsing speed) of the received data packet, when the parsing progress is large, it may indicate that the current communication environment is relatively good, and therefore the local viewing angle is allowed to be displayed in high definition. On the contrary, when the communication environment is relatively poor, it is preferred to use a larger viewing angle amplitude for relatively complete display, so as to weaken the feeling of video lag to a certain extent.

[0101] For example, in some embodiments, the progress of receiving the video data refers to the progress of receiving the data packet, and when the complete data packet is received within a specified time, it also indicates that the current communication environment is better.

[0102] In this embodiment, the data progress of the video data selects the range of the view angle amplitude interval, which can weaken the sense of lag in poor communication environment while trying to meet the display content highlights, maintain a relatively smooth display environment, and reduce the interference to the user's playing process.

[0103] Moreover, the guidance scheme in the present application, which is conducive to the realization of error concentration and rapid correction, cooperates with the restrictive switching scheme (such as limiting the switching amplitude or limiting the switching time), on the one hand, it can guide the user to improve the playing level (i.e. the overall state of playing) relatively efficiently through limited error correction, that is, to guide the user to gradually enter a good state, and to avoid excessive disturbance to the user's playing process. The gradual improvement of the playing level cooperates with the low interference characteristics of the visualization interface, which is more conducive to guiding the user to maintain or enter an immersive piano practice state, thereby effectively improving the user's application experience.

[0104] In some embodiments, the visualization rule further comprises: visual range: display area of the score interface, display priority of different shooting angles.

[0105] For example, in some embodiments, the display area of the score interface refers to the display area of the first visualization interface on the score interface, that is, the display size.

[0106] The display area of the score interface can have a default set display area size.

[0107] Alternatively, the display area of the score interface can be selected by the user's input adjustment signal.

[0108] Further, when there are at least two display view angles, each display view angle has a display area, and the size of the display area can be determined according to the priority (such as display priority), such as the greater the priority, the larger the display area.

[0109] Further, in some embodiments, it further comprises: Obtaining the user's input adjustment signal; The visualization rule can be adjusted according to the user's input adjustment signal.

[0110] For example, the user can manually adjust the display position, such as Figure 4 , Figure 5 The first visualization interface can be displayed above or below the score page.

[0111] As shown in Figure 4 The first visualization interface allows to provide one or more perspectives (e.g. one perspective can provide a perspective display icon), and the user can manually switch the display perspective. Figure 4 For an exemplary display interface, a video can be displayed above the display interface, and a score can be displayed below the display interface.

[0112] In some embodiments, the video data can be actual performance video of the user himself, or the video data can also be demonstration video of the teacher, or performance / demonstration video of a professional performer.

[0113] It should be noted that the video and score data cooperative display scheme provided by the present application has technical advantages in the following multiple application scenarios: 1) Online teaching in live mode: For example, if the teacher and the student are both online, the teacher's performance data (such as video data, hardware data) can be transmitted to the student end through the network, such as playing the teacher's performance video on the student's tablet, or playing the performance video on the student's tablet while playing the performance sound on the student's piano.

[0114] 2, synchronously transmitting video data and score data online: During the live broadcast, one party can play the piano, and the other party can see the performance data of the other party on the staff on the local device (such as a tablet), such as displaying real-time dynamics, time value, foot pedal, etc. on the staff, which are aligned with the staff, easy to understand, and synchronized with the video, ensuring that the progress of the staff on the screen and the finger movements in the video demonstration are aligned. 3, video playback For example, for demonstration videos recorded by the teacher or practice videos recorded by the student during practice, they can be kept in the local or cloud disk for playback. For example, the teacher or the student can manually input a positioning signal (such as clicking any staff position that has been played, and then finding the video under the position to play).

[0115] For example, after the performance, the teacher or the student can quickly show the video, performance data, and playback sound to both parties to form a three-dimensional teaching. Through this form, the piano teaching can be completed, and the experience is even better than when the teacher is present.

[0116] The beneficial technical effects of the first embodiment are: Firstly, the application provides a real-time and immersive collaborative display scheme for video data, which can reduce the receiving pressure of users on complex information on the basis of timely and significant problem feedback to the users.

[0117] Specifically, the application can comprehensively generate an optimized video interface by comprehensively considering the performance level / state of the user and the quality of the video data. Moreover, the optimized video interface can prompt the user on the basis of multi-dimensional data synchronization display with the score interface, so that the user can quickly capture the key information at the current time and avoid multi-dimensional data display interfering with the immersive performance state of the user.

[0118] Embodiment two Referring to Figure 3 The application further provides a video data display system 200, which comprises: A data receiving module 201 is configured to receive video data and hardware data of a user during performance, and the video data and the hardware data are associated with a timestamp; wherein the video data comprises image data collected by the user and a performance device at at least one shooting angle, and the hardware data comprises fingering data generated by the performance device during performance; A visual interface module 202 is configured to generate a first visual interface according to the video data; wherein the visual interface module comprises: A performance evaluation unit 2021 is configured to generate or update a performance evaluation result according to the hardware data and / or the video data, wherein the performance evaluation result comprises at least one of the following evaluation indexes: a type of fingering error, a type of upper limb error, a type of body error, and a frequency of at least one type of error, and the evaluation indexes are pre-associated with at least one recommended display range of video data; A rule generating unit 2022 is configured to generate a visual rule according to the performance evaluation result and a data progress of the video data, wherein the data progress is an analysis progress and / or a receiving degree of the video data, and the visual rule comprises: A data range, and the data range comprises at least one of the following recommended display ranges: a time scale of image data to be displayed, a shooting angle, and a visual angle amplitude of image data; An interface generating unit 2023 is configured to generate a first visual interface according to the visual rule and the video data; A display module 203 is configured to display the first visual interface in a score interface.

[0119] It can be understood that the system in the embodiment can also be used to execute the method or steps in any of the above exemplary embodiments.

[0120] Embodiment three Although the prior art attempts to provide an online teaching scheme (such as patent application CN202211311948.X discloses a one-to-many piano online accompaniment method). However, the applicant noticed that, unlike traditional online teaching, piano teaching involves strong interactivity and concentration requirements, so this existing online accompaniment method is difficult to meet the special requirements under piano teaching.

[0121] For example, in the process of students practicing the piano, the teacher needs to pay close attention to the student's fingering, hand shape, posture, and final performance effect in real time, and needs to give timely feedback (such as giving adjustment suggestions) to the student's performance during the student's performance. Or, from the student's point of view, especially in the early stage of learning or when the student is not familiar with the content of the score, the student not only needs to focus on reading the score, but also needs to concentrate on his or her fingering and body movement, so the student's feedback adjustment ability or reception level may also be relatively limited.

[0122] In this case, especially for students, the distance of online teaching will further weaken the guidance efficiency or guidance effect of the teacher. Therefore, the traditional online teaching scheme is difficult to meet the special needs of interactivity and attention in the special scene of piano teaching.

[0123] To this end, the present application provides a method for cooperatively displaying video and score interface. This cooperative display method can intelligently recommend and display videos for the teaching interaction between teachers and students, or for piano teaching / exercising scenes such as students practicing the piano independently. Or, for the characteristics of strong interactivity and attention, a method is provided for teachers or students to synchronously display multi-dimensional data of video and score, while highlighting the multi-dimensional data, so that the user (such as a student) can maintain a high degree of self-attention while quickly capturing external key feedback information.

[0124] In other words, the present application provides a multi-dimensional data display scheme with strong interactivity and cooperation, which can to some extent alleviate the distance of online piano teaching and reduce the user's understanding pressure of multi-dimensional data display.

[0125] As shown in Figure 6 The present application also provides a performance data transmission method, comprising the steps of: S101, obtaining a first data set collected by a first device end of a first user, the first device end comprising a first musical instrument, a first image device (such as a camera) arranged corresponding to the first musical instrument, and the first data set comprising image data collected by the first image device at at least one shooting angle; S102, transmitting the first data set to a second device end of at least one second user, the second device end comprising a second musical instrument and / or a second display device; S103, generating a score interface according to score advancing data, the score advancing data being determined by positioning signals input by the first user or the second user, or the score advancing data being generated according to second data set, the second data set comprising second hardware data and / or second video data, the second hardware data being fingering data of the second user, and the second video data being image data collected by a second image device of the second device end at at least one shooting angle; In some embodiments, the score advancing data refers to the current performance progress of the score, which can also be referred to as score positioning data.

[0126] S104, adaptively generating a visualization rule in response to a visualization signal issued by the first user and / or the second user, wherein the visualization signal comprises the first data set or the second data set, and the visualization rule comprises: a data range, and the data range comprising at least one recommended display interval of a time scale of image data to be displayed, a shooting angle, and a visual angle amplitude of image data; wherein the time scale is determined by the score advancing data; S105, generating a first visualization interface (which is a video interface) in response to the visualization rule and the first data set; S106, displaying the score interface and the first visualization interface on the second display device.

[0127] The embodiments provided in the present embodiment provide an adjustment scheme for optimizing the display mode of the video and score display interface of the second user end according to the interaction relationship (such as the respective performance data) between the first user and the second user.

[0128] In some embodiments, the at least one device end can comprise a musical instrument (which can also be referred to as a performance device), a display device, and an image device. The image device is arranged corresponding to the musical instrument to record the action data of the user when the user performs the musical instrument.

[0129] Preferably, the at least one musical instrument can be further provided with a data acquisition module for acquiring fingering data.

[0130] Preferably, the at least one device end can further include a sound acquisition module for acquiring sound data generated during the performance.

[0131] Correspondingly, the data set acquired by the device end can include one or more of the following: video data, fingering data, sound data.

[0132] Further, the at least one device end can further include a display device for providing a data display interface, such as a first visualization interface and a fourth visualization interface (i.e., a score interface), and different visualization interfaces can be displayed by one or more devices, such as a display device that can be used to synchronously play the first visualization interface and the fourth visualization interface, or two sub-display devices that can be used to respectively display the first visualization interface and the fourth visualization interface. In some embodiments, the first user is a musical instrument teacher, and the second user is a student; or, in some embodiments, the first user is a student, and the second user is a musical instrument teacher.

[0133] In some embodiments, the first musical instrument and the second musical instrument are pianos.

[0134] It can be understood that the visualization rules in the present embodiment are mainly used to optimize the display mode of the video and the score interface according to the real-time needs of the first user / second user, and in particular, the video display can be adjusted based on the visualization rules.

[0135] The visualization rules mainly adjust the display content (such as the angle and amplitude) of the video interface, the display area, the display position on the interface, and the displayed score segment (such as the displayed time period).

[0136] Next, the generation of the visualization rules will be introduced through exemplary schemes in different scenarios: In some embodiments, the time scale determined by the score advancing data means that the performance data of at least a set time (i.e., the time scale) after the corresponding performance time node is called according to the performance time node determined by the score advancing data.

[0137] In some embodiments, the specific process of determining the score advancing data (or score positioning data) by the positioning signal input by the first user or the second user can be as follows: For example, the score interface is displayed on the display device (such as a piano roll interface), and the score advancing data is determined by the positioning signal input by the first user or the second user. Figure 4When the user clicks and selects one note (equivalent to the user triggering or inputting a positioning signal) at this time, the note (or the playing time node or playing position corresponding to the note) is taken as the score advancing data to select the video data corresponding to the score advancing data (e.g., playing the playing video from the playing time node).

[0138] For example, when the user selects the second measure, the playing video of the second measure is played.

[0139] For example, for a student, he / she can autonomously select a piece of score to be played back according to his / her practice needs. Alternatively, for a teacher, he / she can select a piece of score to be played for a student according to the teaching situation.

[0140] Alternatively, in some embodiments, the score advancing data can be predicted by a score alignment algorithm through the data sets (e.g., the first data set and the second data set).

[0141] As described, the score advancing data is generated according to the second data set means that the score alignment algorithm is used to predict the current playing time node according to the second data set.

[0142] For example, the display device can analyze the continuous playing actions (e.g., pitch, dynamics, time value, etc.) according to the received hardware data, dynamically match the current score position corresponding to the current hardware data in the local score to obtain score positioning data, wherein the score positioning data is the information of the specific position of the current playing in the preset score, which can include measure number, beat point, etc. It can also be used to drive the automatic page turning and highlighting of the score.

[0143] Further, the score positioning data can be bound with the timestamp of the current hardware data, based on which, the corresponding score positioning data can be called to be synchronously displayed on the display device when the hardware data is played on the musical instrument.

[0144] Further, the generation process of the score positioning data is completed locally (e.g., when the second display device is used to display, the data processing unit of the second display device can be used for data processing), which does not need to rely on remote data, avoiding network transmission delay, and realizing the time delay-free score display synchronized with the hardware data.

[0145] Specifically, based on a preset score alignment algorithm, the continuous hardware data received in a preset time period before and after the current time is matched and analyzed with the locally pre-stored performance score, so as to accurately calculate the score position corresponding to the current hardware data, generate score positioning data with a timestamp, and then determine the score paragraph (or score segment) that should be displayed at present according to the score positioning data, and control the display device to display the corresponding score segment in real time, that is, play the performance video.

[0146] The preset time period is a time window obtained by extending a preset time before and after the current time of the current hardware data, so as to generate accurate score positioning data according to the context information of the current hardware data. The preset performance score is a score data pre-stored locally or downloaded, which contains note sequence, rhythm and other information.

[0147] The preset score alignment algorithm refers to an algorithm for matching real-time performance data (such as pitch and time value corresponding to hardware data) with a standard score. For example, the score alignment algorithm can adopt the matching method of audio and score disclosed in CN2025102799337, or other existing alignment methods, which are not limited in the present application.

[0148] It can be understood that the video and score cooperative display method in the present application is preferably applied to a piano teaching live scene.

[0149] For example, in application scenario 1, a student practices piano at home, and a teacher teaches online in a different place. The teacher can demonstrate performance by playing a local piano, and the video corresponding to the performance demonstration can be synchronously transmitted to the student's tablet (i.e., display device). The student can practice synchronously by imitating the teacher's demonstration fingering.

[0150] Further, the teacher can also realize one-to-many teaching according to online teaching, that is, multiple students can synchronously follow and practice.

[0151] For another example, in application scenario 2, a student practices piano at home. The video of the student's practice can be synchronously transmitted to the teacher's tablet. The teacher can watch the video to check whether the student's fingering or posture is accurate, and then give timely guidance to the student. Further, the practice videos of multiple students can be transmitted to the teacher's end, so that the teacher can give synchronous guidance.

[0152] For another example, in some embodiments, the first data set or the second data set obtained can also be pre-recorded data. For example, during performance, the teacher or the student can store the performance data (such as video data or hardware data), and then the teacher or the student can play back the performance data.

[0153] In some embodiments, S104 comprises: S1041, generating or updating a performance evaluation result according to the second data set, wherein the performance evaluation result comprises at least one of the following evaluation indexes: a type of fingering error, a type of upper limb error, a type of body error, and a frequency of at least one type of error, and the evaluation indexes are previously associated with at least one recommended display interval of the video data; In some embodiments, the performance evaluation result can be obtained by analyzing the fingering data or the video data in the second data set.

[0154] S1042, generating a visualization rule according to the performance evaluation result and a data progress of the video data, wherein the data progress is an analysis progress and / or a receiving degree of the video data.

[0155] Preferably, taking the scenario of a teacher demonstrating teaching as an example, the video data used in S1042 is a performance video of the teacher (e.g., the first user), also referred to as the first video data.

[0156] That is to say, in the present embodiment, the recommended display content is generated by comprehensively considering the performance effect of the student and the video data quality of the teacher end.

[0157] In the present embodiment, the second data set is used as a guide to optimize the visualization rule, so that the video data is called according to the performance progress and the performance state of the student, so that the student can intuitively understand the external guidance (e.g., video feedback) while maintaining the coherence of the performance.

[0158] In other words, in the present embodiment, the performance evaluation result of the student is used to flexibly adjust the collaborative visualization of the video and the score, so that the key data can be emphasized to reduce the difficulty of understanding the data for the student, and the student can maintain a high degree of concentration to maintain an immersive performance state.

[0159] It can be understood that the specific generation method of the first visualization interface in the present embodiment can also refer to the method part in Embodiment One, which will not be described here.

[0160] In some embodiments, the method further comprises the following steps: storing the first data set and / or the second data set to a storage unit.

[0161] For example, in some embodiments, the storage unit can be a smart device (e.g., a display device) of the user, or can also be a network cloud disk connected to the smart device of the user to support data review function.

[0162] Therefore, in some embodiments, the first data set or the second data set can be called from the storage unit, In some embodiments, the first device end comprises a first display device; correspondingly, the method further comprises: displaying, by the first display device (such as a tablet, computer, etc. of the teacher end), at least one third visual interface, the at least one third visual interface being generated by the second video data corresponding to at least one second user; wherein the display form of the at least one third visual interface is set by the performance evaluation result, and the display form is used to define at least one of the following reinforced display attributes: display position, display brightness, display area, display mark (such as error prompt information, such as error type icon, which can be added on the video interface).

[0163] In this embodiment, the third visual interface is preferably used to define the video interface generated according to the student's performance video. Preferably, the teacher end can simultaneously receive the video interface transmitted by one or more students, and different reinforcement reminders can be provided for the teacher according to the performance state of different students.

[0164] For example, the specific steps that the display form of the at least one third visual interface is set by the performance evaluation result can be: when the performance evaluation result of the student belongs to a preset first performance state, adding or updating a first reinforced display attribute for the corresponding third visual interface; when the performance evaluation result of the student belongs to a predetermined second performance state, adding or updating a second reinforced display attribute for the corresponding third visual interface; wherein the visual emphasis effect of the first reinforced display attribute is greater than the visual emphasis effect of the second reinforced display attribute, such as the display position of the first reinforced display attribute being located before the display position of the second reinforced display attribute, or the display area / display brightness of the first reinforced display attribute being greater than the display area / display brightness of the second reinforced display attribute, or the number / size of the display mark of the first reinforced display attribute being greater than the number / size of the display mark of the second reinforced display attribute.

[0165] For example, the performance evaluation result belonging to the first performance state can mean that the type number of errors belongs to a larger first numerical interval, or the error frequency of at least one error is greater than a preset frequency threshold.

[0166] For example, the performance evaluation result belonging to the second performance state can mean that the type number of errors belongs to a smaller second numerical interval, or the error frequency of at least one error is less than or equal to a preset frequency threshold. In some embodiments, the visual signal comprises: a first visual signal input by the first user, the first visual signal comprising one or more of the following information: shooting angle, angle amplitude, display area.

[0167] For example, the first user can be a teacher. At this time, the teacher can freely select information that the student should focus on, such as freely switching the shooting angle in the visualization rule, or adjusting the display area, and the like, for the needs of teaching.

[0168] In some embodiments, the visualization signal comprises: The second visualization signal input by the second user, the second visualization signal comprising one or more of the following information: shooting angle, angle amplitude, display area.

[0169] For another example, the second user can be a student, that is, the student can autonomously select the display information to flexibly set in combination with his own needs. For example, the student can also freely set the parameters such as the angle and size of the display interface on his own tablet, to adapt to the variable learning needs in the piano learning scenario.

[0170] Further, in some embodiments, the first visualization signal or the second visualization signal can further comprise at least one enhanced display attribute of the shooting angle.

[0171] For example, in some embodiments, in the case of synchronous display of multiple shooting angles, one or more of the shooting angles can be selected for emphasis display, such as displaying the shooting angle in the front, or adding an enhanced display element to it. For example, the enhanced display element can be a brightness value, a border display, and the like. For example, when the enhanced display element is a brightness value, the display brightness of the emphasized shooting angle can be appropriately increased, or a display border can be added to the shooting angle (or the display border can be thickened, highlighted, and the like).

[0172] In some embodiments, the second device end further comprises a second broadcasting device, and correspondingly further comprises the steps of: According to the data set, the performance sound effect of the first user is reproduced in the second musical instrument or the second broadcasting device.

[0173] For example, in some embodiments, the data set can be a first data set, that is, in addition to the teacher's performance video, the teacher's performance sound effect can also be transmitted to the student end synchronously.

[0174] For example, the synchronous transmission mode of the performance sound effect can be that when the student uses a smart piano, the performance effect of the teacher can be directly reproduced in the smart piano according to the teacher's fingering data (which can be converted by hardware data).

[0175] For example, the synchronous transmission mode of the playing sound effect can be that the fingering data of the teacher (i.e. the pitch, dynamics, time value, etc. for the music attribute of each note) is converted into MIDI data, and the user's smart device (such as a mobile phone, a tablet, etc. broadcasting device) is played to realize the synchronous transmission of vision and audio. The beneficial technical effects of the embodiment are that in the piano education scene, the embodiment can provide a method for the teacher or student to display the video and score multi-dimensional data synchronously and highlight the multi-dimensional data, so that the user (such as a student) can maintain a high self-attention, and quickly capture external key feedback information.

[0176] Preferably, in the embodiment, the display mode of the video and score display interface of the second user port is optimized and adjusted according to the interaction between the first user and the second user (such as the respective playing data).

[0177] In particular, in the embodiment, the student's playing evaluation result and the data completeness of the teacher's end are used to flexibly adjust the video and score collaborative visualization, so that the key data can be highlighted, and the quality of the video display is ensured. Further, on the basis of the multi-dimensional data display, the difficulty of the student's understanding of the data is reduced, so that the student can maintain a high concentration to maintain an immersive playing state.

[0178] Embodiment four Referring to Figure 7 The present application provides a playing data transmission system, comprising: The first data acquisition module 101 is used to acquire the first data set collected by the first device end of the first user, wherein the first device end comprises a first musical instrument, a first image device arranged corresponding to the first musical instrument, and the first data set comprises image data collected by the first image device at at least one shooting angle; The first data transmission module 102 is used to transmit the first data set to the second device end of at least one second user, wherein the second device end comprises a second musical instrument and a second display device; The second interface generation module 103 is used to generate a score interface according to score advancing data, wherein the score advancing data is determined by a positioning signal input by the first user or the second user, or the score advancing data is generated according to a second data set, the second data set comprises second hardware data and / or second video data, the second hardware data is fingering data of the second user, and the second video data is image data collected by a second image device of the second device end at at least one shooting angle; The visualization signal analysis module 104 is configured to adaptively generate a visualization rule in response to a visualization signal issued by the first user and / or the second user, wherein the visualization signal comprises the first data set or the second data set, and the visualization rule comprises: a data range, and the data range comprises at least one recommended display interval of a time scale of image data to be displayed, a shooting angle, and a visual angle amplitude of the image data, wherein the time scale is determined by the score sheet advancing data; The first interface generation module 105 is configured to generate a first visualization interface in response to the visualization rule and the first data set; The collaborative display module 106 is configured to display the first visualization interface and the score sheet interface on the second display device.

[0179] It can be understood that the system in the embodiment can also be used to implement the method or steps in any of the above embodiments, which will not be described here.

[0180] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0181] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a computer terminal (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0182] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.

Claims

1. A method for transmitting performance data, characterized in that: Including steps: S101, obtaining a first data set collected by a first device of a first user, where the first device includes: a first musical instrument, a first imaging device configured for the first musical instrument, and the first data set includes: image data collected by the first imaging device at at least one shooting angle; S102, transmitting the first data set to a second device of at least one second user, where the second device includes: a second musical instrument and a second display device; S103: Generate a music score interface based on the music score advancement data; the music score advancement data is determined by a positioning signal input by the first user or the second user, or the music score advancement data is generated based on a second data set, the second data set including: second hardware data and / or second video data, wherein the second hardware data is fingering data of the second user, and the second video data is image data captured by a second imaging device on the second device end at at least one shooting angle; S104: Adaptively generate visualization rules in response to a visualization signal sent by the first user and / or the second user, wherein the visualization signal includes: the first data set or the second data set; and the visualization rules include: a data range, wherein the data range includes at least one of the following recommended display intervals: a time scale of the image data to be displayed, a shooting angle, and a viewing angle amplitude of the image data; wherein the time scale is determined by the music score advancement data; S105 , generating a first visualization interface in response to the visualization rule and the first data set; S106: Display the first visualization interface and the music score interface on the second display device.

2. The method according to claim 1, characterized in that The first user is a musical instrument teacher, and the second user is a student; or, the first user is a student, and the second user is a musical instrument teacher.

3. The method according to claim 1, characterized in that The first musical instrument and the second musical instrument are pianos.

4. The method according to claim 1, wherein S104 includes: S1041, generating or updating a performance evaluation result based on the second data set, wherein the performance evaluation result includes at least one of the following evaluation indicators: a type of fingering error, a type of upper limb error, a type of body error, and an error frequency of at least one error type, and the evaluation indicator is pre-associated with a recommended display interval of at least one video data, the video data being the first video data or the second video data; S1042: Generate visualization rules according to the performance evaluation result and the data progress of the first video data, wherein the data progress is the parsing progress and / or reception degree of the first video data.

5. The method according to claim 1, wherein Also includes the steps: The first data set and / or the second data set are stored in a storage unit.

6. The method according to claim 4, characterized in that The first device end includes: a first display device; correspondingly, the method further includes: Displaying at least one third visual interface on the first display device, where the at least one third visual interface is generated by at least one second video data; Among them, the enhanced display form of at least one of the third visual interfaces is set according to the performance evaluation result, and the enhanced display form is used to define at least one of the following enhanced display attributes: display position, display brightness, display area, and display mark.

7. The method according to claim 1, characterized in that The visual signal includes: A first visual signal input by a first user, wherein the first visual signal includes one or more of the following information: shooting angle, viewing angle, and display area; And / or, a second visual signal input by a second user, wherein the second visual signal includes one or more of the following information: shooting angle, viewing angle range, and display area.

8. The method according to claim 1, characterized in that The second device end further includes: a second broadcasting device, and correspondingly, further includes the steps of: The performance sound effect of the first user is reproduced in the second musical instrument or the second broadcasting device according to the first data set.

9. A performance data transmission system, characterized in that: include: A first data acquisition module is configured to acquire a first data set acquired by a first device of a first user, wherein the first device comprises: a first musical instrument and a first imaging device configured for the first musical instrument, and the first data set comprises: image data acquired by the first imaging device at at least one shooting angle; A first data transmission module is configured to transmit the first data set to a second device of at least one second user, wherein the second device includes: a second musical instrument and a second display device; a second interface generation module, configured to generate a score interface based on score advancement data; the score advancement data being determined by a positioning signal input by the first user or the second user, or being generated based on a second data set, the second data set comprising: second hardware data and / or second video data, wherein the second hardware data is fingering data of the second user, and the second video data is image data captured by a second imaging device on the second device end at at least one shooting angle; A visualization signal parsing module is configured to adaptively generate visualization rules in response to a visualization signal sent by the first user and / or the second user, wherein the visualization signal includes: the first data set or the second data set; and the visualization rules include: a data range, wherein the data range includes at least one of the following recommended display intervals: a time scale of the image data to be displayed, a shooting angle, and a viewing angle amplitude of the image data; wherein the time scale is determined by the music score advancement data; A first interface generating module, configured to generate a first visualization interface in response to the visualization rule and the first data set; A collaborative display module is used to display the first visualization interface and the music score interface on the second display device.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images