A data alignment method in remote performance, a remote performance system and a medium

Abstract

The application relates to the technical field of remote performance, in particular to a data alignment method in remote performance, a remote performance system and a medium. Hardware data and video data collected by a first device end are sent to a second device end; the second device end comprises a musical instrument end and a first display end and a second display end; the musical instrument end is controlled to present the hardware data; the first display end is controlled to present the video data; score positioning data of current hardware data is determined according to the hardware data in a preset time period; when the current hardware data is presented on the musical instrument end, the score positioning data or score advancing data collected by the first device end is selected to be presented on the second device end according to device computing power and position deviation; when the first time deviation of the simultaneously presented hardware data and video data is greater than or equal to a first deviation threshold, target video data is called based on a target time stamp of the hardware data to be presented on the first display end. Various data is aligned with high precision under complex computing power fluctuation, and a high-immersion remote music experience is realized.

Description

Technical Field

[0001] This application relates to the field of remote performance technology, and in particular to a data alignment method, remote performance system and medium in remote performance. Background Technology

[0002] With the rapid development of internet technology, remote performance, as an emerging art form, is gradually changing the way music is created, performed, and disseminated. Remote performance not only breaks geographical limitations, enabling artists to collaborate across geographical boundaries, but also provides new possibilities for music education, cultural exchange, and art popularization. To enhance the immersiveness and interactivity of remote performance, various types of data are introduced, such as real-time control signals from smart instruments, multi-angle high-definition video streams, and audio signals.

[0003] For example, patent application CN116939237A discloses a live teaching method based on an IoT piano, including setting up a broadcaster terminal; a first terminal connects to the broadcaster's IoT piano via a MIDI data cable to collect MIDI data from the IoT piano, and the first terminal connects to multiple video devices via multi-channel data cables to collect audio and video data from multiple directions; the video acquisition devices combine the multi-channel audio and video data into live video data of different scenes, encode and compress it, and then push it to the server along with the MIDI data of the IoT piano via the RTMP protocol; setting up a viewer terminal; a second terminal connects to the viewer's IoT piano via a MIDI data cable, pulls the live stream from the server via the RTMP protocol for parsing, and parses it into combined audio and video video data and MIDI data; the MIDI data is sent to the viewer's IoT piano via the viewer's MIDI data cable, and the combined audio and video video data and MIDI data are aligned with timestamps and played synchronously.

[0004] For example, patent application CN110392276A discloses a live streaming and recording method for synchronously transmitting MIDI based on the RTMP protocol. First, it acquires image, audio, and MIDI signals, and encodes and compresses the acquired image and audio separately. Second, it mixes the acquired MIDI signals and the encoded and compressed audio, and pushes them, along with the encoded and compressed image, to the server, while simultaneously recording and playing the encoded and compressed image, mixed audio, and MIDI signals. Then, it pulls the stream from the server based on the RTMP protocol and separates the mixed audio and MIDI data, while simultaneously decoding the audio and image data. Next, it integrates the decoded image and audio data into a playback audio-video file, and sends the separated MIDI data to the playing instrument. Finally, it achieves synchronous linkage between the playback of the audio-video file and the playing instrument.

[0005] However, the delay differences between various data at the receiving end make it difficult to achieve precise alignment, resulting in problems such as audio-visual asynchrony and misalignment of actions and sounds, which seriously affect the continuity and realism of the performance and significantly reduce the user experience. Summary of the Invention

[0006] The main objective of this application is to provide a data alignment method, a remote performance system, and a medium for remote performance. To solve the aforementioned technical problems, this application specifically adopts the following technical solution:

[0007] A first aspect of this application is to provide a data alignment method for remote performance, the method comprising:

[0008] S101, hardware data and video data collected during the performance process by the first device are sent to the second device. The hardware data and video data are associated with timestamps. The second device includes an instrument, a first display, and a second display.

[0009] S102, control the instrument terminal to parse and present the hardware data; control the first display terminal to parse and present the video data;

[0010] S103, based on a preset score alignment algorithm, determines the score location data of the current hardware data according to several hardware data received within a preset time period of the current hardware data and a preset performance score. The score location data is associated with the timestamp of the current hardware data.

[0011] Depending on the device's computing power, either proceed to step S1031 or proceed to steps S1032 to S1034:

[0012] S1031, when the current hardware data is presented on the instrument, the second display terminal is controlled to synchronously present the musical score segment pointed to by the musical score positioning data;

[0013] S1032, Based on the score progression data collected during the performance process by the first device, the data is sent to the second device, and the score progression data is associated with a timestamp;

[0014] S1033, Based on a preset period, compare the positional deviation of the musical score segment pointed to by the musical score positioning data and the musical score advancement data at the same timestamp;

[0015] S1034, when the current hardware data is presented on the instrument end, the score segment pointed to by the score positioning data or the score advancement data is selected according to the position deviation, and the second device end is controlled to present the corresponding score segment.

[0016] S1041, extract the timestamps of the hardware data and video data presented at the same time, and calculate the first time deviation between the two timestamps;

[0017] S1042, when the first time deviation is greater than or equal to the first deviation threshold, based on the target timestamp of the hardware data to be presented at the next moment, the target video data of the target timestamp is called, and the first display terminal is controlled to present the target video data at the next moment.

[0018] In some embodiments, the preset time period includes a first time period before the timestamp of the current hardware data and a second time period after it, wherein the length of the first time period is a first preset duration and the length of the second time period is a second preset duration.

[0019] In some embodiments, S1034 includes: when the position deviation is less than a preset difference, presenting a second musical score segment pointed to by the score progression data on the second device; or, when the position deviation is greater than or equal to the preset difference, presenting a first musical score segment pointed to by the score positioning data on the second device.

[0020] In some embodiments, the instrument end is used to drive the sound-producing structure to execute the hardware data; the first display end is used to play the video data; and the second display end is used to display the score fragments corresponding to the score progression data and / or score positioning data.

[0021] In some embodiments, the method further includes: when the first time deviation is greater than a second deviation threshold and less than a first deviation threshold, analyzing the presentation time difference between the hardware data and the video data at the same timestamp based on the first time deviation within a preset time period; adjusting the presentation speed of the video data based on the presentation time difference so that the first time deviation remains less than the first deviation threshold.

[0022] In some embodiments, the method further includes: in response to a user's progress adjustment operation on the first display terminal, determining video data to be displayed, and presenting hardware data of the first timestamp based on a first timestamp of the video data to be displayed; and / or, in response to a user's sheet music selection operation on the second display terminal, determining sheet music progression data or sheet music positioning data to be displayed, and presenting hardware data of the second timestamp based on a second timestamp of the sheet music progression data or sheet music positioning data to be displayed.

[0023] In some embodiments, the method further includes: obtaining first performance data, second performance data, and third performance data based on the hardware data, video data, and score progression data under the same timestamp; generating correction hardware data based on the second performance data and the third performance data when the first performance data is different from both the second performance data and the third performance data; and presenting the correction hardware data on the instrument.

[0024] In some embodiments, the method further includes: encapsulating the hardware data and the video data having the same timestamp into a data packet based on the timestamp; or, encapsulating the hardware data and the video data, and the score progression data having the same timestamp into a data packet based on the timestamp; and distributing the encapsulated data packet over a network.

[0025] A second aspect of this application is to provide a remote performance system, wherein the system is applied to the data alignment method in remote performance provided in any embodiment of this application; the system includes:

[0026] The first device is used to collect hardware data, video data, and score progression data during the performance process, and send them to the second device. The hardware data, video data, and score progression data are associated with timestamps.

[0027] The second device includes an instrument end, a first display end, and a second display end; the instrument end is used to parse and present the hardware data; the first display end is used to parse and present the video data; the second display end is used to determine the score positioning data of the current hardware data based on a preset score alignment algorithm, according to several hardware data received within a preset time period of the current hardware data and a preset performance score, wherein the score positioning data is associated with the timestamp of the current hardware data; when the current hardware data is presented on the instrument end, the score segment pointed to by the score positioning data or the score progression data is presented;

[0028] The control unit is used to extract the timestamps of hardware data and video data presented at the same time, calculate the first time deviation between the two timestamps; when the first time deviation is greater than or equal to the first deviation threshold, based on the target timestamp of the hardware data to be presented at the next time, call the target video data of the target timestamp, and control the first display terminal to present the target video data at the next time.

[0029] The control unit is also used to, when the device computing power is detected to be sufficient, control the second display terminal to synchronously display the score segment pointed to by the score positioning data; or, when the device computing power is detected to be insufficient, compare the position deviation of the score segment pointed to by the score positioning data and the score advancement data at the same timestamp based on a preset period; select the score segment pointed to by the score positioning data or the score advancement data according to the position deviation, and control the second device terminal to display the corresponding score segment.

[0030] A third aspect of this application is that a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the data alignment method in remote performance provided in any embodiment of this application.

[0031] Beneficial effects:

[0032] This application provides a data alignment method, a remote performance system, and a medium for remote performance. Based on stable and reliable hardware data, it combines video data and sheet music data for intelligent collaborative presentation, achieving high-precision alignment of heterogeneous data in environments with differences in processing links and fluctuations in device computing power. While ensuring high synchronization of audio and video, it expands the device compatibility of high-quality remote performance, realizing a low-threshold and highly immersive remote music experience. It can be widely applied to remote performance scenarios such as remote teaching and virtual ensembles.

[0033] During data parsing and processing, priority is given to ensuring real-time processing and accurate reconstruction of hardware data. Limited computing power is allocated preferentially to hardware data processing paths directly related to sound production, ensuring low latency and high responsiveness in driving the instrument's sound-producing structure, maintaining the stability of the auditory experience and the continuity of the performance rhythm. Video and sheet music serve as supplementary information, and their processing can be dynamically adjusted according to the device load.

[0034] In terms of video display, a dynamic adaptive strategy is adopted: based on the deviation threshold, the system jumps to the corresponding timestamp frame in a timely manner to complete the alignment, which not only ensures audio-visual synchronization but also avoids obvious frame skipping that affects the visual experience. At the same time, regular deviations are analyzed and the video presentation speed is pre-adjusted for smooth compensation, so that the user is not aware of the process. This enables high-quality immersive presentation even on devices with weaker computing power.

[0035] In terms of sheet music display, performance and experience are optimized through dual-source data collaboration: when computing power is sufficient, high-precision sheet music positioning is generated based on hardware data within a preset time period to achieve zero-latency progression; when computing power is insufficient, the sheet music progression signal pushed from the remote end is used as the basis, and the deviation is dynamically corrected by combining local lightweight correction algorithms to avoid sheet music lag or stuttering due to insufficient local processing capabilities, significantly reducing the terminal processing burden and enabling ordinary devices to participate in high-quality remote performances smoothly.

[0036] Furthermore, for special situations (such as user selection of video progress, sheet music position, or hardware data anomalies), an interaction priority and fault tolerance mechanism is introduced: when responding to user operations, the hardware data benchmark can be temporarily bypassed and directly aligned to the specified position; when hardware data is abnormal, reverse calibration is performed through video data or sheet music positioning data to avoid error propagation. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of this application; for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0038] Figure 1 This is a schematic flowchart illustrating a data alignment method in remote performance provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of a data transmission process provided in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram illustrating a process for dynamically adjusting the display of musical score, provided in an embodiment of this application.

[0041] Figure 4 This is a schematic diagram of a process for dynamically adjusting video display provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0043] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0044] In this document, suffixes such as “module,” “part,” or “unit” used to denote elements are used only for illustrative purposes and have no specific meaning in themselves. Therefore, “module,” “part,” or “unit” may be used interchangeably.

[0045] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0046] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," and "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0047] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0048] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0049] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0050] In this article, the remote performance scenario includes a first device and a second device. The first device, also known as the live stream initiator or data acquisition end, includes a smart musical instrument (such as a smart piano) located on the performer's side, and has built-in or external image acquisition devices and sensors. Correspondingly, the second device, also known as the live stream receiver or data receiver, includes various receiving devices of the remote user, such as tablets, mobile phones, and another smart musical instrument, etc., used to synchronously reproduce sound, play video, and display sheet music, realizing a real-time performance experience across regions.

[0051] Based on this, this application provides a data alignment method, a remote performance system, and a medium for remote performance. It uses stable and reliable hardware data as a benchmark and combines video data and sheet music data for intelligent collaborative presentation. This achieves high-precision alignment of heterogeneous data in environments with differences in processing links and fluctuations in device computing power. While ensuring high synchronization of audio and video, it expands the device compatibility of high-quality remote performance, realizes a low-threshold and highly immersive remote music experience, and can be widely applied to remote performance scenarios such as remote teaching and virtual ensemble.

[0052] The following detailed description, in conjunction with the accompanying drawings, outlines some embodiments of this application. Unless otherwise specified, the following embodiments and features described herein can be combined with each other. Please refer to... Figure 1 , Figure 1 This is a schematic flowchart illustrating a data alignment method in remote performance provided in an embodiment of this application, such as... Figure 1 As shown in the figure, this application provides a data alignment method for remote performance.

[0053] S101, hardware data and video data collected during the performance process by the first device are sent to the second device, and the hardware data and video data are associated with timestamps.

[0054] Specifically, performers can play using smart musical instruments, which collect hardware data via sensors and video data of the user's performance via image acquisition devices. The hardware and video data are then timestamped and sent to a second device. The hardware data refers to the instrument control signals collected during the performance, which the remote second device uses to reconstruct the performer's actions on the instrument, such as the ID, displacement, force, and speed of pressed or released keys, and the ID, force, and speed of pedals. The video data refers to the synchronously recorded performance footage, allowing remote users to observe the performer's hand shapes, postures, expressions, and other visual information, enhancing the immersive experience of teaching or performance.

[0055] In some embodiments, the first device can also process real-time hardware data to generate score progression data, and the score progression data is associated with the timestamp of the real-time hardware data. When the second device needs to use the score progression data, the hardware data, video data, and score progression data are sent to the second device.

[0056] In some embodiments, the first device includes a source instrument end, a third display end, and / or a fourth display end. The source instrument end is used for performance by a performer, such as a smart piano. The third display end is used to present the video data. The fourth display end is used to process real-time hardware data to generate score progression data and display a second score fragment corresponding to the score progression data.

[0057] In some embodiments, the first device collects various types of data (such as hardware data, video data, and sheet music progression data) and associates them with timestamps, then encapsulates them through a network protocol and transmits them synchronously to the second device.

[0058] In some embodiments, the second device includes a cooperating musical instrument and a first and a second display. The musical instrument receives hardware data and drives the sound-producing structure to execute the hardware data to reproduce the playing actions and generate sound. The first display receives and plays corresponding video data, presenting the performer's actions in real time. The second display shows a musical score segment generated from score progression data or score positioning data. This achieves distributed synchronous presentation of sound, images, and musical score, suitable for remote teaching and multi-screen interactive scenarios.

[0059] In some embodiments, multiple ports of the second device (such as the musical instrument end and the first and second display ends) can be connected to a specially configured remote data processing unit to process various types of received data through shared computing resources.

[0060] In another embodiment, multiple ports on the second device can independently process the received data using the data processing unit of their respective application devices. For example, hardware data can be parsed by a dedicated processor on the musical instrument side (such as a smart piano), ensuring stable computing power. The first and second display ends can be applied to user-defined devices, such as mobile phones, tablets, or PCs, with differences in corresponding hardware configurations, system loads, and decoding capabilities, resulting in uncontrollable computing power.

[0061] In some embodiments, the sound-generating structure refers to the physical or electronic device at the end of the instrument used to produce sound. In acoustic instruments, such as a smart piano, the sound-generating structure includes mechanical components such as strings, soundboard, and action mechanism, and these components can be automatically controlled through mechanical structures to achieve playing actions such as striking the strings with the keys to produce sound. In electroacoustic or digital instruments, the sound-generating structure consists of an audio processor, speaker, or headphone output module, which is responsible for converting received hardware data (such as notes and dynamics) into audible sound signals.

[0062] In some embodiments, the source instrument end of the first device and the instrument end of the second device can be the same or different, and both can be intelligent musical instruments with autonomous sound-producing capabilities. Existing intelligent pianos with automated remote control can be used. The sound-producing structure of the source instrument end responds to the performer's playing actions to produce sound, while the instrument end automatically controls the sound-producing structure to produce sound based on the hardware data corresponding to the playing actions. Furthermore, the instrument end can also be a sound player or an independent automatic sound-producing structure.

[0063] In some embodiments, the application devices of the first display terminal and the second display terminal may be the same or different, or they may be the same device. After data processing, the display functions of the first display terminal and the second display terminal are realized through different interfaces in the device, such as displaying video screen and sheet music screen in separate zones.

[0064] In some embodiments, when the second device does not need to use the score to advance data, the method includes: based on the timestamp, encapsulating the hardware data and the video data with the same timestamp into a data packet, and distributing the encapsulated data packet over the network.

[0065] In some embodiments, when the second device needs to use sheet music advancement data, the method includes: based on the timestamp, encapsulating the hardware data, the video data, and the sheet music advancement data with the same timestamp into a data packet, and distributing the encapsulated data packet over the network.

[0066] Specifically, at the live stream initiator, various data types, including hardware data, video data, and sheet music progression data, are synchronized using timestamps, and data with the same timestamp are encapsulated into a unified data packet. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a data transmission process provided in an embodiment of this application, such as... Figure 2 As shown, when it is necessary to transmit sheet music progression data, hardware data with the same timestamp, video data, and sheet music progression data are encapsulated into a unified data packet and transmitted to the second device.

[0067] In some embodiments, data is uniformly distributed over the network via the UDP protocol to ensure consistent timing at the receiving end.

[0068] S102, control the instrument terminal to parse and present the hardware data; control the first display terminal to parse and present the video data.

[0069] The first display end parses the video data based on the corresponding data processing unit. The video data is a high-bandwidth streaming media, which needs to go through multiple processing stages such as decapsulation, decoding, and rendering at the receiving end. The processing link is long and complex. On this basis, the allocation priority of the shared computing power of the video data is lower than that of the instrument end, or the computing power of the user-defined device is unstable. Therefore, the video data is easily affected by the fluctuation of device performance, resulting in delays in presentation.

[0070] Specifically, the instrument uses its dedicated or shared data processing unit to parse hardware data, which consists of lightweight control signals such as key press status, touch force, and pedal position. This data directly drives the sound-producing structure via a short processing link, resulting in minimal computational overhead. Furthermore, resource scheduling prioritizes the allocation of computing power to these tasks directly related to sound production, ensuring their highest processing priority. In addition, the instrument typically uses a dedicated processor, providing a stable operating environment and ample available computing power. Consequently, hardware data processing latency is low and highly stable, providing a stable and reliable auditory experience for remote performances.

[0071] The primary display device needs to parse the video data through its own equipment or a shared data processing unit. The video data is a high-bandwidth streaming media, requiring multiple processing stages such as decapsulation, decoding, and rendering, resulting in a long and complex process chain. Furthermore, in scenarios with shared computing resources, the computing power allocation priority for video data is lower than that for musical instruments, posing a higher risk of latency. In scenarios where user-defined devices provide computing power, performance fluctuations between devices can lead to delayed or premature image display.

[0072] It should be understood that even if the second display receives two types of data simultaneously, the final presentation time may still result in audio-visual delays due to differences in data processing links and computing power. This audio-visual asynchrony is particularly noticeable on low-performance display devices and may seriously affect the immersive experience of remote performances and the accuracy of teaching.

[0073] In some embodiments, hardware data has a higher priority than video data, sheet music location data, and sheet music progression data in the computing power allocation, to avoid excessive resource consumption from processing auxiliary information. Furthermore, depending on different user needs, the priority of sheet music location data and sheet music progression data can be higher than that of video data to prioritize sheet music tracking, or the priority of video data can be higher than that of sheet music location data and sheet music progression data to prioritize smooth video playback.

[0074] In some embodiments, the second display terminal is controlled to generate score positioning data of the current hardware data based on a plurality of received hardware data. When the current hardware data is presented on the instrument terminal, the score positioning data is presented synchronously on the second display terminal.

[0075] Specifically, the second device analyzes continuous performance actions (such as note sequences and rhythm patterns) based on the received hardware data stream. It then dynamically matches the current position of the music score corresponding to the current hardware data within the local score, obtaining score positioning data. This score positioning data contains information about the specific position of the current performance within a preset score, including bar numbers, beats, etc., and is used to drive automatic page turning and highlighting of the score. Furthermore, the score positioning data can be bound to the timestamp of the current hardware data. Based on the timestamp, when the hardware data is played on the instrument, the corresponding score positioning data can be called and displayed synchronously on the second display. Since the generation of the score positioning data is completed locally at the receiving end, it does not rely on remote data, avoiding network transmission delays and achieving latency-free score display synchronized with the hardware data.

[0076] S103, based on a preset score alignment algorithm, determine the score positioning data of the current hardware data according to several hardware data received within a preset time period of the current hardware data and the preset performance score, wherein the score positioning data is associated with the timestamp of the current hardware data.

[0077] Specifically, based on the timestamp of the current hardware data, the system obtains continuous hardware data received within a preset time period before and after the current moment, as well as locally pre-stored or pre-acquired musical scores. It then performs matching analysis based on a preset musical score alignment algorithm to accurately calculate the position of the musical score corresponding to the current hardware data, generating musical score positioning data with timestamps. Subsequently, the system can determine the musical score segment that should be displayed at the current time, i.e., the first musical score segment, based on the musical score positioning data.

[0078] In some embodiments, the preset time period includes a first time period before the timestamp of the current hardware data and a second time period after it, wherein the length of the first time period is a first preset duration, and the length of the second time period is a second preset duration. That is, the preset time period is a time window obtained by extending the current time of the current hardware data as a base time, and then extending it before and after by a first preset time or a second preset duration, to generate accurate music score positioning data based on the context information of the current hardware data. The first preset duration and the second preset duration can be the same or different, and their specific values ​​can be flexibly set and adjusted according to actual needs.

[0079] In some embodiments, the preset performance score is pre-stored locally or downloaded sheet music data, containing information such as note sequences and rhythms. The preset score alignment algorithm refers to an algorithm used to match real-time performance data (such as pitch, duration, etc. corresponding to hardware data) with standard sheet music; for details, please refer to relevant technologies.

[0080] Depending on the device's computing power, either step S1031 or steps S1032 to S1034 can be executed. The device's computing power refers to the computational capability of each component in the second device (such as the musical instrument, the first display, and the second display) in processing its received data. This can be categorized into centralized shared computing power and independent distributed computing power for each end. It should be understood that when the device's computing power is sufficient, various types of data can be decoded and presented in real-time without delay, with minimal stuttering, frame drops, or misalignment. When the device's computing power is insufficient or strained, audio and video may become out of sync (e.g., frequent occurrences of a first time deviation greater than or equal to the first deviation threshold), there may be delays in sheet music refresh (e.g., the generation rate of sheet music positioning data is lower than the preset rate), or data packet loss and response delays may occur due to processing capacity overload.

[0081] For example, pre-set criteria for determining whether computing power is sufficient or insufficient are used, and relevant technologies are employed to monitor centralized shared computing power and independent distributed computing power at each end in real time to assess the computing power of the devices. For instance, real-time monitoring of CPU utilization, memory usage, GPU load, and temperature; real-time monitoring of processing speed for critical processing tasks (such as video frame decoding); and monitoring of data packet loss rate, etc.

[0082] When the device's computing power is detected to be sufficient, step S1031 is executed; when the device's computing power is detected to be insufficient, steps S1032 to S1034 are executed.

[0083] S1031, when the current hardware data is presented on the instrument, the second display terminal is controlled to synchronously present the musical score segment pointed to by the musical score positioning data.

[0084] Specifically, when the device has sufficient computing power, the timestamp of the music score positioning data is the same as the timestamp of the current hardware data. Therefore, the second device can be controlled to display the corresponding first music score fragment when the current hardware data is presented, based on the timestamp.

[0085] S1032, based on the score progression data collected during the performance process by the first device, the score progression data is sent to the second device, and the score progression data is associated with a timestamp.

[0086] S1033, based on a preset period, compare the positional deviation of the musical score segment pointed to by the musical score positioning data and the musical score advancement data at the same timestamp.

[0087] The preset period is a pre-set time interval for comparing the two types of musical scores. The specific value can be flexibly set and adjusted according to actual needs.

[0088] S1034, when the current hardware data is presented on the instrument end, the music score segment pointed to by the music score positioning data or the music score advancement data is selected according to the position deviation, and the second device end is controlled to present the corresponding music score segment.

[0089] Specifically, in special circumstances such as limited device computing power or excessive local computing burden, the second device needs to use remote score progression data. In this case, the score progression data generated by the first device is collected and sent to the second device. At the same time, the second device generates score positioning data locally based on hardware data. During the remote performance, the positional deviation between the score positioning data generated locally and the score progression data pointed to by the transmitted score fragment is compared at the same time stamp based on a preset period.

[0090] It should be understood that score positioning data is based on hardware data within a certain period before and after the current moment. It analyzes the preceding and following performance sequences through a preset score alignment algorithm. The contextual information it relies on is complete, thus it has higher stability and accuracy. On the other hand, score progression data is dynamically generated by the first device based on real-time performance. It predicts the current score position based only on the already performed part (i.e., the information above), lacks subsequent information correction, and is easily affected by various factors such as repeated passages, accidental touches, and improvisation, which poses a risk of recognition bias.

[0091] Therefore, the accuracy of the real-time generated score progression data is verified by using score positioning data. When the accuracy is low, the score segment pointed to by the score positioning data is selected for presentation, and when the accuracy is high, the score segment pointed to by the score progression data is selected for presentation.

[0092] In some embodiments, S1034 includes: when the position deviation is less than a preset difference, presenting a second musical score segment pointed to by the score progression data on the second device; or, when the position deviation is greater than or equal to the preset difference, presenting a first musical score segment pointed to by the score positioning data on the second device.

[0093] The preset difference is the maximum threshold for the positional deviation between the two types of musical score data that is allowed to be set in advance. The specific value can be flexibly set and adjusted according to actual needs.

[0094] If the positional deviation is less than the preset difference, the accuracy of the score progression data is considered high, and the second score segment corresponding to the score progression data is maintained to save local computing power. If the positional deviation is greater than or equal to the preset difference, the score progression data is considered to have recognition errors due to relying solely on the preceding information, and the display switches back to the first score segment corresponding to the score positioning data. This ensures display accuracy under different network and device performance conditions. Furthermore, after displaying the first score segment corresponding to the score positioning data for a certain period of time, the accuracy of the score progression data is re-verified. If the score progression data has completed self-correction by this time, the display of the second score segment corresponding to the score progression data is restored.

[0095] The preset period is a pre-set time interval for comparing the two types of musical score data, and the preset difference is a pre-set maximum threshold for the positional deviation of the two types of musical score data. The specific values ​​can be flexibly set and adjusted according to actual needs.

[0096] It should be understood that score positioning data is based on hardware data within a certain period before and after the current moment. It analyzes the preceding and following performance sequences through a preset score alignment algorithm. The contextual information it relies on is complete, thus it has higher stability and accuracy. On the other hand, score progression data is dynamically generated by the first device based on real-time performance. It predicts the current score position based only on the already performed part (i.e., the information above), lacks subsequent information correction, and is easily affected by various factors such as repeated passages, accidental touches, and improvisation, which poses a risk of recognition bias.

[0097] Please see Figure 3 , Figure 3 This is a schematic diagram of a music score display process provided in an embodiment of this application. For example... Figure 3 As shown, the first device collects hardware data and score progression data during the performance and transmits them to the second device.

[0098] When computing resources are sufficient, locally generated sheet music positioning data should be prioritized to drive the sheet music presentation on the second display terminal, ensuring display accuracy. For example... Figure 3 As shown, the second device can determine the score positioning data of the current hardware data based on a preset score alignment algorithm, according to several hardware data received within a preset time period of the current hardware data and the preset performance score, and present the first score segment pointed to by the score positioning data on the second device.

[0099] Furthermore, in situations where computing power is limited or system load is too high, to alleviate the burden on local computing, switching to remotely transmitted score progression data can be used as an alternative to maintain the continuity of score progression. For example... Figure 3As shown, based on a preset period, the positional deviation between the score positioning data and the score progression data is compared; when the positional deviation is less than the preset difference, the second score segment corresponding to the score progression data is presented on the second device; or, when the positional deviation is greater than or equal to the preset difference, the corresponding first score segment is presented on the second device according to the score positioning data.

[0100] It should be understood that, to control display errors, a periodic comparison mechanism is set up to verify the consistency of the two types of data, limiting the number of times the second device generates score positioning data, significantly reducing its computational resource consumption. Once a deviation exceeds a threshold, the system immediately switches back to high-precision score positioning data for correction and maintains its use for a second preset time. As the performance progresses, the score advancement data gradually self-corrects, triggering verification again after the second preset time. If the positional deviation is less than the preset difference, the use of the score advancement data is restored. The second preset time can be flexibly determined based on the computational power of the second device. For example, when computational power is limited, the second preset time can be set shorter to actively verify whether the score advancement data has completed self-correction and to switch back to the application of the score advancement data in a timely manner. Thus, while ensuring accuracy, the system expands the device compatibility for high-quality remote performances.

[0101] S1041, extract the timestamps of the hardware data and video data presented at the same time, and calculate the first time deviation between the two timestamps.

[0102] Specifically, the timestamps associated with the hardware data and video data presented at the same moment are extracted on the second device, and the time difference between the two is calculated, i.e., the first time deviation. The first time deviation reflects the degree of synchronization between sound and video in actual output, and is used to determine whether there is audio-visual asynchrony, so as to dynamically trigger subsequent correction mechanisms and improve the consistency of sound and picture in remote performance.

[0103] S1042, when the first time deviation is greater than or equal to the first deviation threshold, based on the target timestamp of the hardware data to be presented at the next moment, the target video data of the target timestamp is called, and the first display terminal is controlled to present the target video data at the next moment.

[0104] Specifically, when a first time deviation is detected to exceed a preset first deviation threshold, the target timestamp of the hardware data to be presented in the next moment is used as a reference to search for and call target video data with the same timestamp from the received video data stream. If the target video data has been parsed and processed at this time, it can be directly called for rendering and display in the next moment. If the target video data has not been parsed and processed at this time, the processing and presentation steps of the intermediate delayed video data can be skipped, and the first display terminal can be directly instructed to process and present the target video data in the next moment.

[0105] It should be understood that achieving rapid frame-by-frame alignment of video footage through precise timestamp matching can omit the presentation of some video data. This saves computing resources while ensuring a high degree of consistency between the hardware and video data presented to the user in the next moment. Even if the processing and presentation of the target video data may introduce new delays, it also corrects, to some extent, any obvious and perceptible audio-visual delays that are about to occur. This avoids audio-visual disconnect caused by accumulated decoding or rendering delays at the display end, and is especially suitable for scenarios with insufficient device performance. It effectively restores the synchronization between hearing and vision, ensuring the continuity and realism of remote performances.

[0106] The first deviation threshold is a pre-set maximum allowable time deviation between audio and video presentation, used to trigger video frame jumps. The specific value can be set considering the human eye's sensitivity to audio-visual asynchrony and frame jumps. For example, when the first time deviation is equal to or slightly greater than the first deviation threshold, frame jumps are less noticeable or have a weaker impact, thus mitigating the effect of frame jumps on the viewing experience to some extent.

[0107] In some embodiments, the method further includes: when the first time deviation is greater than a second deviation threshold and less than a first deviation threshold, accelerating or decelerating the playback speed of the video data based on a preset playback speed.

[0108] Specifically, when the first time deviation is greater than the second deviation threshold but less than the first deviation threshold, it is determined that there is a slight asynchrony between the audio and video. Although it has not yet reached the level that is clearly perceptible to the human eye, it may continue to accumulate and eventually affect the experience if left unattended. In this case, frame skipping is not performed. Instead, the playback rate of the video data is dynamically fine-tuned based on the preset playback speed to achieve smooth synchronization. This smooth adjustment method is particularly suitable for scenarios where the performance of the first display device is good and the processing speed of video data is fast. It can support slightly speeding up or slowing down the playback rhythm of the video, allowing the video picture to gradually catch up with or delay the processing and presentation performance of the actual hardware data and its changes. This avoids visual interruptions caused by sudden changes in the picture or frame skipping, and maintains the continuity of playback.

[0109] The second deviation threshold is lower than the first deviation threshold. It is used to distinguish between slight and perceived frame skipping or audio-visual mismatch. When the deviation exceeds the second deviation threshold but does not reach the first deviation threshold, speed adjustment can be initiated instead of frame skipping. The preset playback speed refers to the unit of video speed adjustment. Acceleration or deceleration is adjusted based on this. The numerical setting should consider the naturalness of the adjustment process and the fact that it is not easily noticed by the user.

[0110] In some embodiments, the method further includes: when the first time deviation is greater than a second deviation threshold and less than a first deviation threshold, analyzing the presentation time difference between the hardware data and the video data at the same timestamp based on the first time deviation within a preset time period; adjusting the presentation speed of the video data based on the presentation time difference so that the first time deviation remains less than the first deviation threshold.

[0111] Specifically, based on the first time deviation within a preset time period, the number of frame skips within the preset time period is counted. The number of frame skips is the number of times step S105 is executed, which is used to quantify the difference in data processing and presentation capabilities between the second device and the instrument. For example, during a remote piano lesson, if the student's device triggers four frame skips within one minute due to limited computing resources or insufficient decoding performance.

[0112] This allows for the analysis of the actual presentation time difference between hardware data and video data. For example, when the initial time deviation is 80ms, each frame skip is due to the time deviation accumulating to 80ms. After each frame skip, the deviation returns to zero and begins to accumulate again, resulting in a total accumulated delay of 320ms, which is the presentation time difference. This identifies a persistent delay trend. Based on this presentation time difference, the computing resources used by the first display device can be increased or decreased, thereby increasing or decreasing its presentation speed of video data. This keeps the initial time deviation continuously within the initial deviation threshold, achieving pre-defined smooth synchronization, reducing the frequency of frame skipping, and improving playback continuity. The preset time refers to a fixed time window, such as 1 minute, used to statistically analyze the initial time deviation and evaluate the recent time difference performance of video data and hardware data in application presentation.

[0113] In some embodiments, a mapping table between presentation time difference and computing resources of the first display terminal is preset based on prior data. This allows for the rapid determination of the required adjustment of computing resources based on the presentation time difference using the mapping table. Furthermore, an upper limit is set for the computing resources occupied by the first display terminal. This upper limit is determined based on the total computing resources of the remote data processing unit, the computing resources occupied by the instrument terminal, and other necessary computing resources. This avoids allocating excessive computing resources to the first display terminal, which could affect the presentation of hardware data.

[0114] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating a process for dynamically adjusting video display according to an embodiment of this application. For example... Figure 4 As shown, the timestamps of hardware data and video data presented at the same time are extracted, the first time deviation between the two timestamps is calculated, and the degree of audio-visual mismatch is distinguished based on the first deviation threshold and the second deviation threshold.

[0115] When the first time deviation is greater than the first deviation threshold, the audio-visual delay will be perceived by the user. Synchronization is quickly achieved by skipping frames: based on the target timestamp of the hardware data, the target video data of the target timestamp is called, and the first display terminal is controlled to present the target video data.

[0116] Furthermore, when the first time deviation is greater than the second deviation threshold but less than the first deviation threshold, a slight audio-visual delay occurs, and speed adjustment can be initiated instead of frame skipping. For example... Figure 4 As shown, the playback speed of the video data can be increased or decreased based on a preset playback speed; or, after running for a period of time, the presentation time difference between hardware data and video data can be analyzed based on the first time deviation within a preset time period; the presentation speed of the video data can be increased or decreased based on the presentation time difference, so that the first time deviation is kept less than the first deviation threshold, thereby avoiding frame skipping operations.

[0117] It should be understood that humans are extremely sensitive to audio latency. A direct link between hardware data and sound control, with hardware data as the core benchmark, ensures a superior auditory experience. Latencies or errors in video and sheet music, which serve as supplementary information, are relatively tolerable. Overhead can be reduced by switching between video displays with on-screen transitions and lightweight sheet music displays, avoiding resource contention with hardware data. While ensuring high audio-visual synchronization, this significantly reduces reliance on terminal device performance, effectively improving cross-platform compatibility and providing a low-barrier, highly immersive remote music experience for scenarios such as remote teaching and virtual performances.

[0118] In some embodiments, the method further includes: in response to a user's progress adjustment operation on the first display terminal, determining video data to be displayed, and presenting hardware data of the first timestamp based on a first timestamp of the video data to be displayed; and / or, in response to a user's sheet music selection operation on the second display terminal, determining sheet music progression data or sheet music positioning data to be displayed, and presenting hardware data of the second timestamp based on a second timestamp of the sheet music progression data or sheet music positioning data to be displayed.

[0119] Specifically, when a user performs a progress adjustment operation (such as dragging a video playback progress bar) on the first display end, the video data to be displayed corresponding to the target position is determined, and its associated first timestamp is extracted. Then, the hardware data-driven sound structure with the same timestamp is searched in the hardware data stream, and the sound output at that moment is restored first, achieving synchronized audio-visual transitions. Similarly, when a user performs a music score selection operation (such as clicking on a measure of the music score) on the second display end, the music score progression data or music score positioning data at the corresponding position and its second timestamp are determined, and the hardware data-driven sound structure with the same timestamp is called, prioritizing the restoration of the sound output at that moment, ensuring that the sound matches the selected music score position. Thus, triggered by user operation, the original playback sequence is broken, achieving interactive transitions and ensuring that multiple types of data from different sources maintain time alignment even after manual adjustment.

[0120] The progress adjustment operation refers to the user's operation on the first display screen to select the video playback progress, such as manually sliding or clicking the progress bar or using voice control to jump to a specified time point. The sheet music selection operation refers to the user's operation on the second display screen to select a segment of the sheet music, such as clicking a musical phrase or measure to locate the performance position.

[0121] In some embodiments, the method further includes: obtaining first performance data, second performance data, and third performance data based on the hardware data, video data, and score progression data under the same timestamp; generating correction hardware data based on the second performance data and the third performance data when the first performance data is different from both the second performance data and the third performance data; and presenting the correction hardware data on the instrument.

[0122] Specifically, after receiving hardware data, video data, and sheet music progression data at the same timestamp, the corresponding first performance data, second performance data, and third performance data are parsed out respectively. Among them, the performance data can be specific pitch, duration, and other data.

[0123] For example, the first performance data is obtained directly from hardware data parsing, such as by looking up the ID, displacement, force, and speed of the pressed or released keys to obtain directly acquired performance data. The second performance data can be obtained through visual recognition analysis of video data, using image processing technology to detect the finger pressing position and the trajectory of the keys, and combining this with the piano keyboard layout to recognize the performance data reflected in the screen. The third performance data is determined based on the musical score segment pointed to by the score progression data to determine the performance data that should be played at that moment. The second and third performance data can be ranges of values, rather than a single, fixed data value.

[0124] When the first performance data differs from both the second and third performance data, and the second and third performance data are sufficiently similar, there may be errors in the acquisition or transmission of the hardware data. The video and sheet music data, as supplementary information, provide reference performance content. In this case, correction hardware data is generated based on the common data in the second and third performance data. If there are minor differences between the second and third performance data, the second performance data determined by the video data is prioritized, and correction hardware data for correction is generated and sent to the instrument for sound presentation.

[0125] This mechanism is activated in the event of hardware failure, and uses cross-validation of data to achieve fault-tolerant recovery, ensuring the accuracy and continuity of the performance content and improving the robustness of the system.

[0126] In some embodiments, this application provides a schematic flowchart of another data alignment method in remote performance, the method including S201 to S204: S201, collecting hardware data and video data during the performance process based on a first device, wherein the hardware data and the video data are associated with timestamps; and sending the hardware data and the video data to a second device; wherein the second device includes an instrument and a first display; S202, controlling the instrument to parse and present the hardware data; controlling the first display to parse and present the video data; S203, extracting the timestamps of the hardware data and video data presented at the same time, and calculating a first time deviation between the two timestamps; S204, when the first time deviation is greater than or equal to a first deviation threshold, based on the target timestamp of the hardware data to be presented at the next time, calling the target video data of the target timestamp, and controlling the first display to present the target video data at the next time.

[0127] This application also provides a remote performance system, which is applied to the data alignment method in remote performance provided in any embodiment of this application; the system includes:

[0128] The first device is used to collect hardware data, video data, and score progression data during the performance process, and send them to the second device. The hardware data, video data, and score progression data are associated with timestamps.

[0129] The second device includes an instrument end, a first display end, and a second display end; the instrument end is used to parse and present the hardware data; the first display end is used to parse and present the video data; the second display end is used to determine the score positioning data of the current hardware data based on a preset score alignment algorithm, according to several hardware data received within a preset time period of the current hardware data and a preset performance score, wherein the score positioning data is associated with the timestamp of the current hardware data; when the current hardware data is presented on the instrument end, the score segment pointed to by the score positioning data or the score progression data is presented;

[0130] The control unit is used to extract the timestamps of hardware data and video data presented at the same time, calculate the first time deviation between the two timestamps; when the first time deviation is greater than or equal to the first deviation threshold, based on the target timestamp of the hardware data to be presented at the next time, call the target video data of the target timestamp, and control the first display terminal to present the target video data at the next time.

[0131] The control unit is also used to, when the device computing power is detected to be sufficient, control the second display terminal to synchronously display the score segment pointed to by the score positioning data; or, when the device computing power is detected to be insufficient, compare the position deviation of the score segment pointed to by the score positioning data and the score advancement data at the same timestamp based on a preset period; select the score segment pointed to by the score positioning data or the score advancement data according to the position deviation, and control the second device terminal to display the corresponding score segment.

[0132] This application provides a remote performance system designed to achieve cross-device, high-precision audio, video, and sheet music collaborative presentation, thereby enhancing the immersion and interactivity in scenarios such as remote teaching and virtual ensemble performance.

[0133] The system comprises a first device, a second device, and a control unit. The first device collects hardware data (such as key status and dynamics) and video data (performance action images) during the performance, generates score progression data, and assigns a high-precision unified timestamp to all three. The data is then sent to the second device. The second device includes an instrument, a first display, and a second display: the instrument parses the hardware data and drives the sound-producing structure to reproduce sound; the first display plays the video data, presenting the performance; and the second display, based on the received continuous hardware data, generates score positioning data for the current performance position using a score alignment algorithm. This score positioning data synchronously uses the unified timestamp of the hardware data and displays the corresponding score fragment simultaneously with the sound output of the hardware data, achieving precise linkage between notes and the score.

[0134] The control unit continuously monitors the audio-visual synchronization status, extracts the timestamps of hardware and video data presented at the same moment, and calculates the first time deviation. When the deviation exceeds the first deviation threshold, the target timestamp of the hardware data is used as the standard, the target video frame with the corresponding timestamp is called, and the first display terminal is instructed to display it directly, achieving frame skipping alignment. This effectively addresses the audio-visual asynchrony problem caused by insufficient terminal computing power or network fluctuations, ensuring the real-time performance and audio-visual consistency of remote performance. Furthermore, the control unit also compares and selects between score progression data and score positioning data when the device's computing power is insufficient.

[0135] For example, the control unit is also used to implement the steps of the data alignment method in remote performance provided in any embodiment of this application, which will not be described again here.

[0136] The embodiments of this application provide a computer device, which may be a terminal device or a server. Exemplarily, the above-described method can be implemented as a computer program that can run on the computer device.

[0137] The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0138] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any data alignment method used in remote performance.

[0139] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0140] Internal memory provides an environment for the execution of computer programs in non-volatile storage media, which, when executed by a processor, enable the processor to perform any data alignment method in remote performance.

[0141] This network interface is used for network communication, such as sending assigned tasks.

[0142] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0143] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0144] S101, hardware data and video data collected during the performance process by the first device are sent to the second device. The hardware data and video data are associated with timestamps. The second device includes an instrument, a first display, and a second display.

[0145] S102, control the instrument terminal to parse and present the hardware data; control the first display terminal to parse and present the video data;

[0146] S103, based on a preset score alignment algorithm, determines the score location data of the current hardware data according to several hardware data received within a preset time period of the current hardware data and a preset performance score. The score location data is associated with the timestamp of the current hardware data.

[0147] Depending on the device's computing power, either proceed to step S1031 or proceed to steps S1032 to S1034:

[0148] S1031, when the current hardware data is presented on the instrument, the second display terminal is controlled to synchronously present the musical score segment pointed to by the musical score positioning data;

[0149] S1032, Based on the score progression data collected during the performance process by the first device, the data is sent to the second device, and the score progression data is associated with a timestamp;

[0150] S1033, Based on a preset period, compare the positional deviation of the musical score segment pointed to by the musical score positioning data and the musical score advancement data at the same timestamp;

[0151] S1034, when the current hardware data is presented on the instrument end, the score segment pointed to by the score positioning data or the score advancement data is selected according to the position deviation, and the second device end is controlled to present the corresponding score segment.

[0152] S1041, extract the timestamps of the hardware data and video data presented at the same time, and calculate the first time deviation between the two timestamps;

[0153] S1042, when the first time deviation is greater than or equal to the first deviation threshold, based on the target timestamp of the hardware data to be presented at the next moment, the target video data of the target timestamp is called, and the first display terminal is controlled to present the target video data at the next moment.

[0154] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0155] S201, based on the hardware data and video data collected during the performance process by the first device, the hardware data and the video data are associated with timestamps; and send the hardware data and the video data to the second device; wherein, the second device includes an instrument and a first display.

[0156] S202, control the instrument terminal to parse and present the hardware data; control the first display terminal to parse and present the video data;

[0157] S203, extract the timestamps of hardware data and video data presented at the same time, and calculate the first time deviation between the two timestamps;

[0158] S204, when the first time deviation is greater than or equal to the first deviation threshold, based on the target timestamp of the hardware data to be presented at the next moment, the target video data of the target timestamp is called, and the first display terminal is controlled to present the target video data at the next moment.

[0159] For example, the processor is used to run a computer program stored in a memory, and is also used to implement the steps of the data alignment method in remote performance provided in any embodiment of this application, which will not be described again here.

[0160] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and a processor executing the program instructions to implement the steps of the data alignment method in remote performance provided in any of the embodiments of this application. The computer-readable storage medium can be a product program.

[0161] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for aligning data in a remote performance, characterized by, The method comprises: S101, collecting hardware data and video data in a playing process based on a first device end, and sending to a second device end, the hardware data and the video data being associated with a timestamp; the second device end comprising a musical instrument end and a first display end and a second display end; S102, controlling the musical instrument end to parse and present the hardware data; controlling the first display end to parse and present the video data; S103, determining, based on a preset score alignment algorithm, score positioning data of current hardware data according to a plurality of hardware data received in a preset period in which the current hardware data is located and a preset playing score, the score positioning data being associated with a timestamp of the current hardware data; According to the device computing power condition, step S1031 is executed or steps S1032 to S1034 are executed: S1031, when the current hardware data is presented on the musical instrument end, controlling the second display end to synchronously present a score segment pointed to by the score positioning data; S1032, collecting score advancing data in a playing process based on the first device end, and sending to the second device end, the score advancing data being associated with a timestamp; S1033, comparing a position deviation of score segments pointed to by the score positioning data and the score advancing data under the same timestamp based on a preset period; S1034, when the current hardware data is presented on the musical instrument end, selecting a score segment pointed to by the score positioning data or the score advancing data according to the position deviation, and controlling the second display end to present the corresponding score segment; S1041, extracting timestamps of the hardware data and the video data presented at the same time, and calculating a first time deviation between the two timestamps; S1042, when the first time deviation is greater than or equal to a first deviation threshold, calling target video data of a target timestamp based on a target timestamp of hardware data to be presented at a next time, and controlling the first display end to present the target video data at the next time.

2. The method of claim 1, wherein, The preset period comprises a first time period before the timestamp of the current hardware data and a second time period after the timestamp of the current hardware data, wherein the length of the first time period is a first preset time length, and the length of the second time period is a second preset time length.

3. The method of claim 1 or 2, wherein, The S1034 comprises: when the position deviation is less than a preset difference, presenting a second score segment pointed to by the score advancing data on the second device end; or, when the position deviation is greater than or equal to the preset difference, presenting a first score segment pointed to by the score positioning data on the second device end.

4. The method of claim 3, wherein, The musical instrument end is used to drive a sound producing structure to execute the hardware data; the first display end is used to play the video data; and the second display end is used to display a score segment corresponding to the score advancing data and / or the score positioning data.

5. The method of claim 1, wherein, The method further comprises: when the first time deviation is greater than a second deviation threshold and less than the first deviation threshold, analyzing a presentation time difference of the hardware data and the video data under the same timestamp according to the first time deviation in a preset time; adjust a presentation speed of the video data based on the presentation time difference to keep the first time deviation less than the first deviation threshold.

6. The method of claim 1, wherein, The method further comprises: in response to a user's progress adjustment operation on the first display end, determining video data to be displayed, and calling hardware data of a first timestamp of the video data to be displayed for presentation based on the first timestamp; and / or, in response to a user's score selection operation on the second display end, determining score positioning data or score advancing data to be displayed, and calling hardware data of a second timestamp of the score positioning data or score advancing data to be displayed for presentation based on the second timestamp.

7. The method of claim 3, wherein, The method further comprises: obtaining first performance data, second performance data, and third performance data from the hardware data, the video data, and the score advancing data respectively according to the same timestamp; generating corrected hardware data based on the second performance data and the third performance data when the first performance data is different from the second performance data and the third performance data; presenting the corrected hardware data on the musical instrument end.

8. The method of claim 1, wherein, The method further comprises: based on the timestamp, packaging the hardware data and the video data with the same timestamp into one data packet, or based on the timestamp, packaging the hardware data, the video data, and the score advancing data with the same timestamp into one data packet; distributing the packaged data packet over a network.

9. A remote playing system characterized by comprising: The system is applied to the data alignment method in the remote performance as claimed in any one of claims 1 to 8; the system comprises: a first device end for collecting hardware data, video data, and score advancing data during a performance process and sending the data to a second device end, wherein the hardware data, the video data, and the score advancing data are associated with timestamps; the second device end comprises a musical instrument end and a first display end and a second display end; the musical instrument end is configured to analyze and present the hardware data; the first display end is configured to analyze and present the video data; the second display end is configured to determine score positioning data of current hardware data based on a preset score alignment algorithm according to a plurality of hardware data received within a preset time period in which the current hardware data is located and a preset performance score, the score positioning data being associated with a timestamp of the current hardware data; when the current hardware data is presented on the musical instrument end, a score segment pointed by the score positioning data or score advancing data is presented; a control unit is configured to extract timestamps of the hardware data and the video data presented at the same time, calculate a first time deviation between the two timestamps, and when the first time deviation is greater than or equal to a first deviation threshold, call target video data of a target timestamp of hardware data to be presented at a next time based on the target timestamp, and control the first display end to present the target video data at the next time. The control unit is further configured to, when the device computing power is sufficient, control the second display terminal to synchronously present the score segment pointed to by the score positioning data; or, when the device computing power is insufficient, compare the position deviation of the score segments pointed to by the score positioning data and the score advancing data under the same timestamp based on a preset period; select the score segment pointed to by the score positioning data or the score advancing data according to the position deviation, and control the second device terminal to present the corresponding score segment.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to enable the processor to implement the data alignment method in remote performance according to any one of claims 1 to 8.

Images