Performance assistance system and method of performance assistance thereof

Abstract

The present application provides a kind of performance auxiliary system and performance auxiliary method.Performance auxiliary system includes data processing end and performance interaction end.Data processing end is used to obtain score data and parse and generate multi-path score data, and built-in timing synchronization unit and filter distribution unit.Timing synchronization unit is used to generate global timestamp, and periodically issues clock calibration instruction, unified clock.Filter distribution unit is used to perform hierarchical filtering operation, and score data is filtered according to instrument or performance role.Performance interaction end is used to receive score data, collect the performance data generated by performer and return to data processing end.Performance interaction end also contains performance prompt unit, synchronously outputs the real-time performance prompt of current note and the pre-judgment prompt of subsequent note.The present application can adapt to the diversified needs in different application scenarios, and can realize visual performance guidance containing pre-judgment prompt and accurate filtering and distribution of score data.

Description

Technical Field

[0001] This invention relates to the field of music performance, and in particular to a performance assistance system and method. Background Technology

[0002] Existing ensemble training architectures often distribute ensemble control commands by broadcasting from a central node to all terminals or through simple addressing. For example, they use part information from the score to push light guidance commands to the corresponding instrument terminals to control the on / off state and color of the corresponding LED lights, thereby providing instructions such as part switching previews. However, this command distribution method cannot achieve precise instrument-level control and channel isolation at the "one person, one score" level, and the lights can only indicate part switching previews, failing to provide comprehensive teaching guidance.

[0003] Furthermore, existing ensemble training frameworks are mostly tied to fixed scenarios, lacking comprehensive teaching guidance and feedback mechanisms such as rhythm guidance and multi-part collaborative performance. They are difficult to adapt to the refined division of labor needs in diverse derivative scenarios such as multi-part and multi-terminal, and also difficult to meet the real-time requirements of collective teaching and collaborative performance. In particular, they are difficult to support high-requirement collaborative scenarios such as teaching of hundreds of people or rehearsals of professional orchestras.

[0004] Furthermore, existing performance terminals only possess basic performance functions, lacking visual instruction and rhythmic feedback features, thus failing to meet the diverse needs of modern teaching, such as real-time guidance, collaborative ensemble playing, and data feedback. This is particularly problematic for beginners, who not only face a high learning curve in reading music but also frequently make mistakes, omissions, and struggle to accurately grasp the rhythm (a disconnect from rhythmic concepts, preventing beginners from developing rhythmic habits), leading to low teaching efficiency. Moreover, the lack of visual instruction and rhythmic feedback features hinders performers' expressive abilities and prevents the use of dynamics and other techniques to achieve nuanced and personalized artistic expression.

[0005] In addition, existing performance terminals lack predictive performance prompts and cannot predict the entire melody, which can cause serious discontinuities and breaks in the performance, thus affecting teaching. Summary of the Invention

[0006] The purpose of this invention is to provide a performance assistance system and a performance assistance method, which can effectively solve at least one defect of the prior art.

[0007] To achieve the above objectives, this invention provides a performance assistance system, comprising a data processing terminal and a performance interaction terminal. The data processing terminal acquires full score data and parses it to generate multiple individual score data, and includes a built-in timing synchronization unit and a filtering and distribution unit. The timing synchronization unit generates a global timestamp and periodically issues clock calibration commands to unify the clocks of all terminal hardware in the performance assistance system. The filtering and distribution unit performs layered filtering operations, filtering the full score data according to the instrument or performance role, and sending the filtered individual score data, carrying timing tags, to the performance interaction terminal. The performance interaction terminal establishes a bidirectional data channel with the data processing terminal to receive the individual score data. The performance interaction terminal also collects performance data generated by the performer and transmits it back to the data processing terminal. The performance interaction terminal further includes a performance prompt unit, which is bound to the note timing of the corresponding individual score data and synchronously outputs real-time performance prompts for the current note, as well as predictive prompts for subsequent notes based on the performance timeline.

[0008] In some embodiments of the present invention, the hierarchical filtering includes at least one or any combination of instrument group filtering, single instrument filtering, and chord single-note filtering; the data processing terminal further includes a mapping table generation unit for performing a mapping table generation operation to generate a mapping table reflecting the four-dimensional relationship between "terminal-seat-instrument-data channel", wherein the filtering distribution unit distributes the score data according to the mapping table; wherein the data processing terminal supports independent masking of single instruments, and the filtered score data is distributed through the isolated data channel.

[0009] In some embodiments of the present invention, the performance prompt unit includes a light array unit, which contains at least one row of lights, and distinguishes the real-time performance prompt from the predictive prompt by brightness, number of rows, or color; wherein, the visual intensity of the predictive prompt is negatively correlated with the time elapsed between the corresponding note and the current performance moment, and the predictive prompt is implemented by using a gradient brightness method of multiple rows of lights or a single row of lights.

[0010] In some embodiments of the present invention, the performance interaction terminal is further configured to attach a local timing stamp to the performance data before sending it back to the data processing terminal; the timing synchronization unit is further configured to perform timing deviation verification on the returned performance data.

[0011] In some embodiments of the present invention, the performance prompting unit further includes a vibration unit and / or a visual metronome unit; the vibration unit is used to perform a vibration indication operation, providing performance beat guidance prompts in a first vibration manner according to the strong beat information in the score data; the visual metronome unit is used to perform a beat prompting operation, providing beat guidance prompts in a first visual manner according to the strong beat information in the score data; and / or, the light array unit is further used to perform a time value indication operation, illuminating a corresponding number of indicator lights in the light array unit according to the note time value information in the score data.

[0012] In some embodiments of the present invention, the performance interaction terminal further includes a performance input unit for performing performance input operations, wherein the performer generates first performance data when performing through the performance input unit; and / or, the performance interaction terminal further includes a multimodal sensing unit for performing multimodal control operations, wherein the performer generates second performance data when performing multimodal control through the multimodal sensing unit; and / or, the performance signal generated when the performer performs through an acoustic instrument is converted by a pickup to generate third performance data, wherein the third performance data is uploaded to the data processing terminal via the performance interaction terminal.

[0013] In some embodiments of the present invention, the data processing terminal further includes one or more of the following units:

[0014] The rendering unit is used to perform the spectrum rendering operation, rendering the corresponding spectrum data into visualized spectrum data, wherein the visualized spectrum data is a visualized staff notation or a visualized simplified notation.

[0015] A feedback analysis unit is used to perform feedback analysis operations; wherein, the feedback analysis unit includes a misplay feedback unit and / or a rhythm feedback unit; the misplay feedback unit is used to perform a misplay indication operation, and performs a performance judgment based on the first performance data or the third performance data and the corresponding score data, and when a misplay is determined to occur, it provides a misplay indication through the light array unit and / or the vibration unit; the rhythm feedback unit is used to perform a rhythm feedback indication operation, and provides a strong beat indication through the vibration unit;

[0016] The role configuration unit is used to perform role configuration operations, select one or more roles from melody player, harmony player, orchestrator, rhythm player, and sound effects player, and configure one or more role terminals accordingly.

[0017] The relay unit is used to perform relay operations, forwarding the performance data from the first terminal to the second terminal for relay.

[0018] An audio output unit is used to perform audio output operations, and to achieve audio output by calling a preset VST sound source according to the performance data;

[0019] The page-turning unit is used to perform the physical music stand page-turning operation. It determines the current performance progress based on the first performance data or the third performance data and the corresponding part score data, and turns the physical music score page when the current performance progress reaches the last measure of the current page.

[0020] In some embodiments of the present invention, the data processing terminal is configured to select from multiple working modes, including a symphony rehearsal mode, a multi-person teaching mode, a multi-person ensemble mode, an improvisation co-creation mode, and a fun practice mode. In the symphony rehearsal mode, the filtering and distribution unit of the data processing terminal is configured to perform single-layer filtering including instrument group filtering. In the multi-person teaching mode or the multi-person ensemble mode, the filtering and distribution unit of the data processing terminal is configured to perform double-layer filtering including instrument group filtering and single-instrument filtering, or triple-layer filtering including instrument group filtering, single-instrument filtering, and chord single-note filtering; and in the multi-person teaching mode, the multi-person ensemble mode, or the improvisation co-creation mode, the data processing terminal and multiple performance interaction terminals are networked in parallel. In the improvisational co-creation mode, the improvisational performance data generated by the melody player through the corresponding melody player terminal, the harmony data generated by the harmony player through the corresponding harmony player terminal, the orchestration instructions generated by the orchestration player through the corresponding orchestration player terminal, the rhythm instructions generated by the rhythm player through the corresponding rhythm player terminal, and the sound effect instructions generated by the sound effect player through the corresponding sound effect player terminal are uploaded to the data processing terminal and output through the audio output unit. In the fun practice mode, the performance data from the first terminal can be forwarded to the second terminal for broadcast via the data processing terminal.

[0021] In some embodiments of the present invention, the performance assistance system further includes: a cloud platform, communicatively connected to the data processing terminal. The cloud platform includes at least: a training unit for performing performance data training operations, training a model based on the performance data to generate a virtual musician model corresponding to the performer; and a cloud database for storing data. The data processing terminal further includes a virtual musician loading unit for performing a virtual musician loading operation, loading the virtual musician model corresponding to the performer onto the data processing terminal to replace the performer in playing.

[0022] To achieve the above objectives, the present invention also provides a performance assistance method, which includes the following steps:

[0023] S1. The data processing end acquires the total spectrum data and parses it to generate multi-channel spectrum data;

[0024] S2. The data processing terminal performs a hierarchical filtering operation on the full score data, filtering the full score data hierarchically according to the instrument or performance role, and the filtered score data is sent down through the data channel.

[0025] S3. The performance interaction terminal receives the score data and outputs the real-time performance prompt of the current note and the prediction prompt of the subsequent notes based on the performance timeline through the performance prompt unit.

[0026] S4. The performance interaction terminal collects the performer's performance data and sends it back to the data processing terminal.

[0027] This invention performs layered filtering and distribution through the filtering and distribution unit at the data processing end, which can adapt to the diverse needs of different application scenarios. In particular, through double-layer and triple-layer filtering, it can achieve accurate filtering and distribution of "instrument-level" and "chord-level" score data, and can eliminate signal interference from channels other than the instrument group.

[0028] This invention, through the performance prompt unit (such as the light array unit) on the performance interaction terminal, can simultaneously realize real-time performance prompts and visual guidance (i.e., rational cognition) of predictive performance prompts, which can help performers predict notes in advance, reduce wrong playing and omissions, and is especially suitable for teaching beginners, which is conducive to improving teaching efficiency.

[0029] This invention can also achieve precise acquisition of performance expression parameters (such as, but not limited to, CC1 vibrato, CC2 breathing, dynamics, CC7 volume, CC11 expression, etc.) through the multimodal sensing unit of the performance interaction terminal. Furthermore, by deeply coordinating the light array unit, visual metronome unit, multimodal sensing unit, and vibration unit, it can form a dual rhythmic guidance of "visual + tactile" and "rational + emotional" approaches. This enriches the expressive forms of symphonic performance, fun practice, and improvisation, further enhancing performance expressiveness and teaching guidance effectiveness. Simultaneously, the performance expression parameters (such as dynamics envelopes) acquired through the multimodal sensing unit can provide nuanced and personalized emotional teaching support for artistic processing.

[0030] This invention, adaptable to various application scenarios, can solve the problem of monotonous timbre in traditional acoustic instruments by connecting external mobile terminals (such as mobile phones / tablets / laptops), software sound sources, and microphones. It can convert the performance signals of acoustic instruments into MIDI / XML signals and transmit them to a data processing terminal for pitch and rhythm determination. This not only facilitates acoustic instrument teaching but also adapts to diverse scenario requirements. Furthermore, by setting different working modes, it can adapt to various scenarios such as classroom teaching, outdoor performances, individual practice, symphony rehearsals, and VST recording in a recording studio. It can also be further extended into various systems such as symphony rehearsal teaching, multi-person teaching, group music performances, multi-person collaborative creation, and multi-person fun practice. These extended systems do not require modification of the underlying control logic of the performance assistance system of this invention; they can accurately cover different teaching, performance, practice, and collaborative creation scenarios simply through scenario-based parameter adaptation.

[0031] This invention utilizes a cloud-based training unit to train a corresponding virtual musician model (sometimes referred to as a "shadow model") based on the performer's performance data. By loading the virtual musician model locally, it can highly replicate the playing habits of a real musician, thus replacing the performer in the performance. Furthermore, through the virtual musician model, it enables collaborative ensemble performances between virtual and real musicians, as well as extended applications such as remote ensemble performances, multi-part ensemble performances by one person, ensemble performances with one's past self, and cross-terminal shadow broadcasting for remote teaching.

[0032] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] The above and other features and advantages of the present invention will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic diagram of the performance assistance system of the present invention;

[0035] Figure 2 yes Figure 1 The diagram shows the specific structure of the performance assistance system.

[0036] Figure 3 This is a schematic diagram illustrating the specific process by which the data processing end of the present invention performs hierarchical filtering and distribution operations through the filtering and distribution unit;

[0037] Figure 4 This is a schematic diagram illustrating the specific process by which the performance assistance system of the present invention selects different layered filtering methods according to different working modes;

[0038] Figure 5This is a schematic diagram of the total spectrum data FD of the present invention, which also shows different sub-spectral data FD1 generated after different levels of hierarchical filtering;

[0039] Figure 6 The diagram illustrates the specific structure of the playing terminal of the present invention, using a keyboard instrument as an example.

[0040] Figure 7 yes Figure 2 A schematic diagram illustrating the specific process by which the data processing end executes the misplay instruction operation through the misplay feedback unit.

[0041] Figure 8A This is a flowchart illustrating the performance assistance method of the present invention;

[0042] Figure 8B This is a schematic diagram illustrating the specific process of the performance assistance method of the present invention in a multi-person teaching mode. Detailed Implementation

[0043] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0044] In describing the elements / components / etc. described and / or illustrated herein, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of one or more elements / components / etc. The terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that additional elements / components / etc. may exist in addition to those listed. Furthermore, the terms “first,” “second,” etc., in the claims are used only as designations and are not intended to limit the number of objects to which they pertain.

[0045] It should be understood that the wording or terminology used herein is for descriptive purposes and not for limitation, so that those skilled in the art can interpret the terms or wording of this specification based on the teachings herein.

[0046] Different embodiments or examples are provided below for implementing different features of the subject matter provided by the present invention. Of course, these are merely examples and are not intended to be limiting. For example, the following description of "a first feature forming on or above a second feature" may, in embodiments, include direct contact between the first and second features, and may also include the formation of an additional feature between the first and second features such that the first and second features do not have direct contact. Furthermore, element symbols and / or letters may be repeated in various embodiments or examples. This repetition is for simplicity and clarity and does not in itself limit the relationship between the various embodiments and / or configurations discussed.

[0047] like Figure 1 and Figure 2 As shown, the performance assistance system 100 of the present invention may include a data processing terminal 100D and a performance interaction terminal 100P.

[0048] The data processing terminal 100D can acquire the full score data FD and parse it to generate multiple partial score data FD1. It also has a built-in timing synchronization unit 12 and a filtering and distribution unit 13. The timing synchronization unit 12 can generate a global timestamp and periodically issue clock calibration commands to unify the clocks of all terminal hardware in the performance assistance system 100. The filtering and distribution unit 13 can perform hierarchical filtering operations, filtering the full score data FD according to the instrument or performance role (sometimes also called the "performance subject"). The filtered partial score data FD2, carrying a timing tag, is sent to the performance interaction terminal 100P.

[0049] The performance interaction terminal 100P establishes a bidirectional data channel with the data processing terminal 100D to receive the score data FD2. The performance interaction terminal 100P also collects the performance data PD generated by the performer and transmits it back to the data processing terminal 100D. The performance interaction terminal 100P may also include a performance prompt unit 34, which can be bound to the note timing of the corresponding score data FD2 and synchronously output real-time performance prompts for the current note, as well as predictive prompts for subsequent notes based on the performance timeline.

[0050] In some embodiments, layered filtering may include at least one or any combination of instrument group filtering, single instrument filtering, and chord monotone filtering.

[0051] In some embodiments, the data processing terminal 100D may further include a mapping table generation unit 15, used to perform a mapping table generation operation to generate a mapping table reflecting the four-dimensional relationship between "terminal-seat-instrument-data channel". The filtering and distribution unit 13 may distribute the spectral data FD1 according to the mapping table.

[0052] In some embodiments, the bidirectional data channel established between the performance interaction terminal 100P and the data processing terminal 100D may include, for example, mutually independent and isolated downlink MIDI data channels (e.g., used for transmitting data and / or instructions) and uplink MIDI data channels (e.g., used for transmitting data and / or instructions), but the present invention is not limited thereto. Preferably, in the present invention, the data processing terminal 100D can support independent masking of a single instrument. The filtered part score data FD1 can be sent down through the isolated data channel.

[0053] In some embodiments, the performance prompt unit 34 of the performance interaction terminal 100P may include a light array unit 341. The light array unit 341 may include at least one row of lights, and distinguish between real-time performance prompts and predictive prompts by brightness, number of rows, or color. Preferably, the visual intensity of the predictive prompt may be negatively correlated with the time elapsed between the corresponding note and the current performance moment; that is, the closer the note, the stronger its visual intensity (e.g., the brighter its brightness). In some embodiments, the predictive prompt may be implemented using a gradient brightness method with multiple rows of lights or a single row of lights. For example, the light array unit 341 can be a multi-row light array, which may include a row of real-time performance light arrays for real-time performance prompts and at least one row of predictive light arrays for predictive prompts; alternatively, the light array unit 341 can also be a single-row light array. By illuminating multiple indicator lights in this single-row light array with different brightness levels in the horizontal direction (i.e., having gradient brightness in the horizontal direction), real-time performance prompts and predictive prompts can be achieved. For example, the indicator light for "real-time performance prompts" has the maximum brightness, while the several indicator lights for "predictive prompts" have a gradually decreasing brightness as the time away from the current performance moment increases. However, the present invention is not limited to this, and more specific structures and implementation processes will be described in detail later. In some other embodiments, indicator lights of different colors can also be used to distinguish between real-time performance prompts and predictive prompts.

[0054] In some embodiments, the performance interaction terminal 100P can also be used to attach a local timing stamp to the performance data PD before sending it back to the data processing terminal 100D. The timing synchronization unit 12 of the data processing terminal 100D can also be used to perform timing deviation verification on the sent-back performance data PD.

[0055] In some embodiments, the performance prompting unit 34 of the performance interaction terminal 100P may further include a vibration unit 342 and / or a visual metronome unit 343. The vibration unit 342 can be used to perform vibration indication operations, providing performance beat guidance prompts in a first vibration manner based on the strong beat information in the score data FD2. The visual metronome unit 343 can be used to perform beat prompting operations, providing beat guidance prompts in a first visual manner based on the strong beat information in the score data FD2.

[0056] In some embodiments, the light array unit 341 of the performance interaction terminal 100P can also be used to perform a timing indication operation, lighting up a corresponding number of indicator lights in the light array unit 341 according to the note timing information in the score data FD2.

[0057] In some embodiments, the performance interaction terminal 100P may further include a performance input unit 32 for performing performance input operations. When the performer performs through the performance input unit 32, first performance data PD1 may be generated.

[0058] In some embodiments, the performance interaction terminal 100P may further include a multimodal sensing unit 35 for performing multimodal control operations. When the performer performs multimodal control through the multimodal sensing unit 35, second performance data PD2 may be generated.

[0059] In some embodiments, the performance interaction terminal 100P may include an acoustic instrument 50. When the performer plays using the acoustic instrument 50, the resulting performance signal is converted by a pickup to generate third performance data PD3. The third performance data PD3 can be uploaded to the data processing terminal 100D via the performance interaction terminal 100P.

[0060] In some embodiments, the data processing terminal 100D may further include one or more of the following units: rendering unit 23, feedback analysis unit 16, role configuration unit 17, broadcasting unit 18, audio output unit (including first audio output unit 22 and / or second audio output unit 19), page turning unit 10A, etc.

[0061] The rendering unit 23 can be used to perform spectrum rendering operations, rendering the corresponding spectrum data FD1 into visualized spectrum data, such as visualized staff notation or visualized simplified notation.

[0062] The feedback analysis unit 16 can be used to perform feedback analysis operations. The feedback analysis unit 16 may include a misplay feedback unit 161 and / or a rhythm feedback unit 162. The misplay feedback unit 161 can be used to perform misplay indication operations, and can make performance judgments based on the first performance data PD1 or the third performance data PD3 and the corresponding part score data. When a misplay is determined, a misplay indication is given through the light array unit 341 and / or the vibration unit 342. The rhythm feedback unit 162 can be used to perform rhythm feedback indication operations, and can indicate strong beats through the vibration unit 342.

[0063] The role configuration unit 17 can be used to perform role configuration operations, and can select one or more roles from melody player, harmony player, orchestrator, rhythm player and sound effects player, and configure one or more role terminals accordingly.

[0064] The relay unit 18 can be used to perform relay operations, which can forward the performance data of the first terminal to the second terminal for relay.

[0065] The audio output unit (including the first audio output unit 22 and / or the second audio output unit 19) can be used to perform audio output operations. According to the performance data PD, audio output can be achieved by calling the preset VST sound source.

[0066] The page-turning unit 10A can be used to perform physical music stand page-turning operations. The current performance progress can be determined based on the first performance data PD1 or the third performance data PD3 and the corresponding part score data. The physical music score can be turned when the current performance progress reaches the last measure of the current page.

[0067] In some embodiments of the present invention, such as Figure 1 and Figure 2 As shown, the performance assistance system 100 of the present invention can be constructed, for example, as a three-layer architecture including a main control terminal 10 (sometimes referred to as a "teacher's machine"), a voice part terminal 20 (sometimes referred to as a "voice part machine"), and a performance terminal 30. The main control terminal 10 and the voice part terminal 20 can serve as different running roles of the same software (e.g., server and client), respectively used to implement some functions or operations of the data processing terminal 100D. The performance terminal 30 can be used to implement the functions or operations of the performance interaction terminal 100P. More specifically, the main control terminal 10 (sometimes referred to as a "teacher's machine") can, for example, serve as a global timestamp reference node for the entire system, responsible for timing control, score parsing, score voice part coarse screening (i.e., "instrument group filtering"), and instruction issuance, i.e., undertaking some non-real-time tasks (e.g., score import, parameter configuration, interface display, teaching report generation, etc.) and some real-time teaching timing control (e.g., synchronization instructions, score distribution, performance data upload, feedback instruction issuance, etc.). The section terminal 20 (sometimes referred to as the "section machine") can handle tasks such as relaying, filtering, and preliminary processing of teaching instructions / performance data, achieving section-level control. The performance terminal 30 can handle tasks such as performance data acquisition, teaching guidance display, and real-time feedback. Through the linkage of the main control terminal 10, section terminal 20, and performance terminal 30, diverse scenario requirements such as simultaneous multi-instrument teaching, interactive practice, and improvisation can be efficiently met.

[0068] In some other embodiments of the present invention, such as Figure 1 and Figure 2 As shown, the performance assistance system 100 of the present invention can also be constructed as a "two-layer architecture" including a master control terminal 10 (sometimes referred to as a "teacher machine") that integrates the functions of the voice terminal 20 and the performance terminal 30.

[0069] In this invention, the performance assistance system 100 can adopt either a "three-level architecture" or a "two-level architecture" depending on the application scenario. For example, the "three-level architecture" performance assistance system 100 is more suitable for scenarios with a large number of terminals and high data processing pressure, such as large-scale group teaching and orchestra rehearsals. On the other hand, the "two-level architecture" performance assistance system 100 is more suitable for scenarios with a smaller number of terminals and lower data processing pressure, such as extracurricular activity groups, home teaching, and leisure and entertainment.

[0070] For more specific details, please refer to [link / reference]. Figure 1 and Figure 2 As shown, the "three-layer architecture" performance assistance system 100 of the present invention may include, for example, a master control terminal 10 located in the first layer L1, at least one voice terminal 20 located in the second layer L2, and at least one performance terminal 30 located in the third layer. The master control terminal 10 can be used to parse the acquired score data into multiple channel score data, and to perform hierarchical filtering and distribution of the parsed score data, and synchronously output synchronization control signals. The voice terminal 20 is communicatively connected to the master control terminal 10 and can be used to receive corresponding score data and output first audio and / or visual score data. The performance terminal 30 is communicatively connected to the voice terminal and can be used to generate performance data and / or output at least one prompt. In this invention, the performance assistance system 100 can support bidirectional transmission of downlink score data and uplink performance data.

[0071] In some embodiments, a voice terminal 20 corresponds to an instrument group, an instrument group may correspond to a voice, and an instrument group may contain one or more instruments. For example, in Figure 1 and Figure 2 In the illustrated embodiment, the N voice terminals 20-1, 20-2, 20-3, ..., 20-N correspond to instrument group 1, instrument group 2, instrument group 3, ..., instrument group N, respectively, and each voice terminal 20-1, 20-2, 20-3, ..., 20-N can communicate with the main control terminal 10 via a dedicated instrument group channel CH_G1, CH_G2, CH_G3, ..., CH_GN (e.g., a MIDI data channel).

[0072] In some embodiments, a performance terminal 30 corresponds to a musical instrument. Each performance terminal 30 can communicate with its corresponding instrument terminal 20 via a dedicated instrument channel (e.g., a MIDI data channel) for data transmission. For example, in... Figure 1 and Figure 2In the illustrated embodiment, performance terminals 30-1-1 to 30-1-8 and 30-1-9 to 30-1-16 can communicate with the voice terminal 20-1 corresponding to instrument group 1 via dedicated instrument channels CH_S1-1 to CH_S1-8 and CH_S1-9 to CH_S1-16 respectively through corresponding convergent 31 (e.g., 8-in-1 convergent 31-1-1, 31-1-2); performance terminals 30-2-1 to 30-2-8 can communicate with the voice terminal 20-2 corresponding to instrument group 2 via dedicated instrument channels CH_S2-1 to CH_S2-8 through corresponding convergent 31 (e.g., 8-in-1 convergent 31-2-1); performance terminals 30-3-1 to 30-2-8 can communicate with the voice terminal 20-2 corresponding to instrument group 2 via dedicated instrument channels CH_S2-1 to CH_S2-8 through corresponding convergent 31 (e.g., 8-in-1 convergent 31-2-1); performance terminals 30-3-1 to 30-2-8 can communicate with the voice terminal 20-2 corresponding to instrument group 2; performance terminals 30-3-1 to 30-2-8 can communicate with the voice terminal 20-2 corresponding to instrument group 2. Instruments 30-3-8 can communicate with the corresponding instrument terminal 20-3 of instrument group 3 via dedicated instrument channels CH_S3-1 to CH_S3-8 through the corresponding convergent 31 (e.g., an 8-in-1 convergent 31-3-1). The communication connections between the corresponding instrument terminals 20 and the corresponding performance terminals 30 of other instrument groups are similar to the above-described communication connections and will not be repeated here.

[0073] In some embodiments, with Figure 1 and Figure 2 Taking the performance assistance system 100 shown as an example applied to a symphony rehearsal scenario, these instrument groups 1 to N correspond to different parts. For example, with N equal to 6, it can be divided into 6 instrument groups, including strings, woodwinds, brass, and percussion. Instrument group 1 may correspond to instruments such as violins; instrument group 2 may correspond to instruments such as cellos, woodwinds, and brass; instrument group 3 may correspond to percussion, solo, and color instruments; and so on. Instrument group N may correspond to other orchestral instruments, and so on. However, it is understood that the division of instrument groups, the number of part terminals, the number of performance terminals connected to each convergent unit, dedicated instrument channels, and dedicated instrument group channels can be designed and configured according to actual application scenarios and needs, and these are not considered limitations of the present invention.

[0074] Preferably, these instrument group channels (e.g., including CH_G1, CH_G2, CH_G3, ..., CH_GN, etc.) and these instrument channels (e.g., including CH_S1-1 to CH_S1-16, CH_S2-1 to CH_S2-8, CH_S3-1 to CH_S3-8, etc.) can be configured, for example, to enable bidirectional communication with electrical isolation.

[0075] Preferably, the performance terminal 30, the instrument terminal 20, and the main control terminal 10 can all be configured to support dual-link backup connections of "wired link + wireless link," wherein the wireless link may include, for example, dual wireless links of "Bluetooth 5.0 / WIDI X8" and Wi-Fi. More preferably, the performance terminal 30 can also be configured to perform redundant link switching operations in the event of a communication failure. When the primary wired link fails (e.g., the transmission delay of the wired link is greater than a delay threshold (e.g., 15ms, but this invention is not limited thereto), the performance terminal 30 can perform redundant link switching operations through the link switching unit to seamlessly switch to the dual wireless links (e.g., Bluetooth 5.0 / WIDI X8 and Wi-Fi) as backup links to continue transmission, thereby ensuring uninterrupted transmission of commands and data. When the wired link returns to normal, the performance terminal 30 can also automatically switch back to the wired link, thereby ensuring the stability and low latency of command and data transmission.

[0076] like Figure 2 As shown, the main control terminal 10 may include at least a first user interface 11, a timing synchronization unit 12, a spectrum analysis unit 13, and a filtering and distribution unit 14.

[0077] The first user interface 11 can be used by the first user for operation. For example, it can be operated by the teacher in teaching mode, or by the conductor in symphony rehearsal mode.

[0078] The timing synchronization unit 12 can be used to perform synchronization control operations and output synchronization control signals synchronously. For example, the master control terminal 10 can issue a "synchronization control command" through the timing synchronization unit 12 to drive all terminals in the entire system (including the master control terminal 10, the voice terminal 20, and the performance terminal 30) to establish a unified timing reference, ensuring that the clocks of all terminals are aligned. For example, after the master control terminal 10 starts up, it can generate a unified global timestamp (e.g., in SMPTE format, but this invention is not limited thereto) through its built-in high-precision hardware clock chip. This global timestamp can serve as the sole timing reference for all control operations and data transmission in the system, ensuring the accuracy of multi-device collaboration. In some preferred embodiments, the master control terminal 10 can preferably also issue a "synchronization calibration command" to the voice terminal 20 and the performance terminal 30 every 1 second via UDP multicast for timing calibration. After calibration, the timing deviation of the system can be strictly controlled within the range of ≤15ms, thereby providing high-precision timing assurance for different application scenarios and avoiding rhythmic errors.

[0079] The score parsing unit 13 can be used to perform score parsing operations, parsing the score data into multiple individual score data. For example, the main control terminal 10 can use the score parsing unit 13 to parse the acquired score data FD into multiple individual score data FD1 according to different instrument types; that is, each individual score data corresponds to one instrument, for example. Preferably, the score parsing unit 13 can also extract information such as notes, note values, performance sequence (e.g., through "sequence tags"), and paragraph markings from each individual score data FD1. Preferably, the score data FD can be, for example, a Musescore format file, which can be parsed to generate a MIDI or Music XML format file, but this is not intended to limit the invention.

[0080] The filtering and distribution unit 14 can be used to perform a hierarchical filtering and distribution operation, distributing the hierarchically filtered spectral data downlink to the corresponding acoustic terminal. In some preferred embodiments, in conjunction with reference to... Figure 3 Layered filtering may include, for example, one or more of the following: instrument group filtering based on different instrument groups (i.e., "first layer filtering"), single instrument filtering based on different instrument types (i.e., "second layer filtering"), and chord single-note filtering based on chord single notes (i.e., "third layer filtering"). For ease of description, the score data after the first layer filtering is labeled "FD1-1", the score data after the second layer filtering is labeled "FD1-2", and the score data after the third layer filtering is labeled "FD2-3".

[0081] Reference Figure 4 As shown, in some preferred embodiments, different levels of layered filtering can be selected according to different working modes (or different application scenarios). For example, in the symphony rehearsal mode (i.e., mode 1), the filtering distribution unit 14 of the main control terminal 10 can be configured to perform single-layer filtering including instrument group filtering (i.e., "first layer filtering"). In the multi-person teaching mode (i.e., mode 2) or multi-person ensemble mode (i.e., mode 3), the filtering distribution unit 14 of the main control terminal 10 can be configured to perform two-layer filtering including instrument group filtering (i.e., "first layer filtering") and single instrument filtering (i.e., "second layer filtering"), or three-layer filtering including instrument group filtering (i.e., "first layer filtering"), single instrument filtering (i.e., "second layer filtering") and chord single-note filtering (i.e., "third layer filtering"). However, it is understood that two-layer filtering can also be selected in the symphony rehearsal mode (i.e., mode 1), and these are not intended to limit the present invention.

[0082] Reference Figure 5 As shown, the total spectrum data FD of the present invention can be split into multiple spectrum data FD1 after parsing. These spectrum data FD1 are displayed, for example, in a line-by-line manner (i.e., one line corresponds to one spectrum data FD1). Figure 5The document also shows different spectral data FD1 generated after different levels of layered filtering, such as FD1-1 after the first layer of filtering, FD1-2 after the second layer of filtering, and FD1-3 after the third layer of filtering. Figure 5 The paragraph mark PM is also schematically shown in the text.

[0083] like Figure 2 As shown, each audio terminal 20 may include at least a second user interface 21, a first audio output unit 22, and a rendering unit 23.

[0084] The second user interface 21 can be used by a second user for operation. For example, it can be used by musicians in each instrument group.

[0085] The first audio output unit 22 can be used to perform a first audio output operation, generate a first audio based on the performance data uploaded by one or more performance terminals, and output the first audio by calling a preset first VST sound source.

[0086] The rendering unit 23 can be used to perform a spectrum rendering operation, rendering the corresponding spectrum data FD1 into visualized spectrum data FD2. Preferably, the visualized spectrum data FD2 can be, for example, a visualized staff notation or a visualized simplified musical notation.

[0087] like Figure 2 As shown, each performance terminal 30 may include at least a third user interface 31, a performance input unit 32, a data acquisition unit 33, and a performance prompt unit 34.

[0088] The third user interface 31 can be used by a third user for operation. For example, it can be used by musicians playing various instruments.

[0089] The performance input unit 32 can be used to perform performance input operations. When the performer performs through the performance input unit 32, the first performance data PD1 is generated.

[0090] The acquisition unit 33 can be used to perform acquisition operations to acquire the performer's performance data. The performance data includes at least the first performance data.

[0091] The performance prompt unit 34 can be used to perform a prompt output operation and output at least one prompt.

[0092] In some preferred embodiments, the performance prompting unit 34 may preferably include one or more of a plurality of units such as a light array unit 341, a vibration unit 342, and a visual metronome unit 343.

[0093] The light array unit 341 can be used to perform performance prompts. The light array unit 341 may include multiple indicator lights arranged in an array. By illuminating the real-time performance light among the multiple indicator lights, a real-time performance prompt for the currently played note can be provided. By illuminating the prediction light among the multiple indicator lights, a prediction performance prompt for at least one predicted note following the currently played note can be provided.

[0094] Preferably, the light array unit 341 is arranged in M ​​rows of lights, where 2 ≤ M ≤ 8. The M rows of lights may include one row of real-time performance lights for providing real-time performance cues for the currently played note, and (M-1) rows of pre-judgment lights for providing pre-judgment performance cues for (M-1) predicted notes. The light array unit 341 can perform performance cues by simultaneously illuminating the real-time performance lights in the real-time performance light row and the pre-judgment lights in the pre-judgment light row. For example, the timing of illuminating the real-time performance lights, the number of rows of pre-judgment lights illuminated, and their duration can be directly driven based on note events (e.g., pitch, start time, duration, etc.) in the score file (e.g., a MIDI score file). More specifically, in conjunction with reference... Figure 6 Taking a light array unit 341 with four rows of lights as an example, by controlling the real-time playing lights in the real-time playing light row 3411 corresponding to the current playing note and the three prediction lights in the three prediction light rows corresponding to the three prediction notes to be lit at the same time, the overall synchronous falling of multiple notes can be achieved, thus guiding the entire melody including the current playing note and the multiple prediction notes after it.

[0095] Preferably, the light array unit 341 can also be arranged in a single row of lights, for example. The light array unit 341 performs performance prompting by simultaneously illuminating multiple indicator lights in the single row of lights in a multi-level brightness decrease manner in the horizontal direction. The brightness of each indicator light is negatively correlated with the time elapsed since the corresponding note was played. That is, the real-time performance light corresponding to the currently played note can have the maximum brightness, and the brightness of the prediction light corresponding to the predicted note decreases sequentially as it gets further away from the current performance time of the currently played note (i.e., the closer the note is to the current performance time, the brighter the corresponding indicator light). Furthermore, as the performance progresses, after the currently played note is triggered (i.e., the performer presses the corresponding key), the brightness of the indicator lights for the remaining notes to be played sequentially increases, supplementing the indicator lights for subsequent notes, thus ensuring that these indicator lights used for performance prompting always maintain a continuously decreasing brightness display state. More specifically, for example, one indicator light with 100% brightness can serve as a real-time performance light to provide real-time performance prompts, while three indicator lights with brightness levels of 75%, 50%, and 25% can serve as prediction lights to provide prediction prompts for the first, second, and third predicted notes, respectively. Similarly, it is understood that the number of these indicator lights used for performance prompts can be adjusted according to actual needs and is not limited to the aforementioned number.

[0096] More preferably, in some embodiments, with Figure 6 Taking the light array unit 341, which includes four rows of lights, as an example, the real-time performance lights in the real-time performance light row 3411 used for real-time performance prompts may have a first-level brightness (i.e., maximum brightness, such as 100%). The three prediction lights in the three prediction light rows 3412 used for prediction prompts of the first, second, and third prediction notes may have a second-level brightness (such as 75%), a third-level brightness (such as 50%), and a fourth-level brightness (such as 25%), respectively. That is, the brightness of one real-time performance light and the three prediction lights decreases in multiple levels in the horizontal direction. It is understood that the multi-level brightness of these indicator lights used for performing performance prompt operations is not limited to the above values, and these are not intended to limit the invention.

[0097] Preferably, in some embodiments, the lamp array unit 341 can also be used to perform timing indication operations, corresponding to the sound terminal 20 (see reference). Figure 1 Based on the note value information obtained from the corresponding musical score data, it can issue time value indication commands to control the illumination of a corresponding number of indicator lights in the light array unit. For example, using... Figure 6 Taking the illustrated embodiment of the 4x37 light array unit 341 as an example, for instance, illuminating one indicator light in these light arrays can indicate an eighth note, illuminating two indicator lights in these light arrays can indicate a quarter note, and so on. It is understood that in other embodiments, Figure 6 The 4x37 light array unit can also be replaced by an 8x37 light array unit. In this case, for example, lighting one indicator light in these light arrays can indicate a sixteenth note, lighting two indicator lights can indicate an eighth note, lighting four indicator lights can indicate a quarter note, and so on. In other words, different note values ​​can be indicated by different numbers of indicator lights, and the 4x4 light array unit and the 8x8 light array unit are equivalent substitutes for each other.

[0098] The vibration unit 342 can be used to perform vibration indication operations, and the corresponding sound terminal 20 (see reference) Figure 1Based on the strong beat information in the corresponding score data FD1, a first vibration command can be issued to control the vibration unit 342 to provide performance beat guidance in a first vibration mode (e.g., "slight vibration," but this invention is not limited to this). This enables tactile guidance of rhythm, which is beneficial for strengthening rhythm guidance, helping beginners accurately grasp the performance beat, and improving the teaching guidance effect. Preferably, the vibration unit 342 can be, for example, a miniature drive motor. In other embodiments, the vibration unit 342 can also adjust the vibration intensity according to different working modes (e.g., reduced in classroom mode, increased in outdoor mode) to adapt to different scenario requirements.

[0099] The visual metronome unit 343 can be used to perform beat cues, and the corresponding sound terminal 20 (see reference) Figure 1 Based on the strong beat information in the corresponding score data FD1, a first beat command can be issued to control the visual metronome unit 343 to provide beat guidance in a first visual cue manner (e.g., flashing). Preferably, the visual metronome unit 343 can be, for example, a pad-type metronome light, which can prompt the musician to keep the beat by flashing the pad-type metronome light while the vibration unit 342 performs vibration indication operation, thereby realizing visual guidance of rhythm.

[0100] Preferably, the prompt display unit 34 may further include a status indicator light 344, which can be used to provide feedback on different connection states of the performance terminal 30, such as turning off when disconnected and flashing rapidly when a redundant circuit failure occurs. However, it is understood that these are not intended to limit the invention.

[0101] In this invention, the light array unit 341, the visual metronome unit 343, and the status indicator light 344 can work together to achieve visual guidance.

[0102] like Figure 2 As shown, in some embodiments, the main control terminal 10 may further include a mapping table generation unit 15 for performing mapping table generation operations. Preferably, for example, a mapping table reflecting the four-dimensional correspondence between "terminal (i.e., a vocal part terminal) - seat - instrument (i.e., instrument type) - data channel" can be generated based on the seat information SID input by the third user via the third user interface 31. The mapping table is then distributed to the corresponding vocal part terminal 20 (see reference 20). Figure 1 (and forwarded to the corresponding performance terminal 30 via the corresponding voice terminal 20).

[0103] Preferably, the filtering and distribution unit 14 of the main control terminal 10 can perform hierarchical filtering and distribution operations according to the mapping table.

[0104] Preferably, the second user interface 21 of the instrument group's part terminal 20 can display only the visual score data FD2 corresponding to this instrument group according to the mapping table.

[0105] Preferably, the third user interface 31 of the performance terminal 30 corresponding to this instrument can display only the visual score data FD2 corresponding to this instrument according to the mapping table.

[0106] like Figure 2 As shown, in some embodiments, each performance terminal 30 may further include a multimodal sensing unit 35 for performing multimodal control operations. The performance data PD collected by the acquisition unit 33 may also include second performance data PD2 generated by the performer through the multimodal sensing unit 35 for multimodal control.

[0107] Ideally, in conjunction with references Figure 6 The multimodal sensing unit 35 may include, for example, one or more of the following: a breathing controller 351 for generating a breathing control signal (e.g., a resistive thin-film pressure sensor); a vibrato controller 352 for generating a vibrato control signal (e.g., a resistive thin-film pressure sensor); a volume controller 353 for generating a volume control signal (e.g., a first capacitive touch slider); an expression controller 354 for generating an expression control signal (e.g., a second capacitive touch slider); and a force controller 355 for generating a force control signal (e.g., a third capacitive touch slider). Through these sensing components in the multimodal sensing unit 50, control of MIDI parameters such as vibrato (corresponding to the CC1 parameter), breathing (corresponding to the CC2 parameter), volume (corresponding to the CC7 parameter), expression (corresponding to the CC11 parameter), and force can be achieved.

[0108] Taking the performance terminal 30 as an electric melodica as an example, the performer can match the melodica's blowpipe through the breathing controller 351. The greater the performer's blowing pressure, the stronger the breathing force. The breathing control signal generated by the breathing controller 351 can control the envelope of the CC2 parameter. The control methods or processes of other sensing components are similar and will not be described in detail here.

[0109] In this invention, the performance expression parameters (such as the envelope of dynamics) collected by the multimodal sensing unit can provide sensitive teaching support for delicate and personalized artistic processing.

[0110] like Figure 2As shown, in some embodiments, each performance terminal 30 can also be connected to the acoustic instrument 50 via a MIDI pickup (not shown). The performance data PD collected by the acquisition unit 33 may also include third performance data PD3 generated by the performance signal produced by the performer playing the acoustic instrument 50, after conversion by the MIDI pickup. More specifically, the performance signal (e.g., waveform signal) generated by the performer playing the acoustic instrument 50 can be converted into MIDI data (i.e., third performance data PD3) via the MIDI pickup. Based on the third performance data PD3 collected by the acquisition unit 33, it can be uploaded to the corresponding vocal terminal 20 and the master control terminal 10 to perform one or more operations.

[0111] like Figure 2 As shown, in some embodiments, the main control terminal 10 may further include one or more of the following units: a feedback analysis unit 16, a role configuration unit 17, a broadcasting unit 18, a second audio output unit 19, and a page-turning unit 10A.

[0112] The feedback analysis unit 16 can be used to perform feedback analysis operations. Preferably, the feedback analysis unit 16 may include a misplay feedback unit 161 and / or a rhythm feedback unit 162. The misplay feedback unit 161 can be used to perform a misplay indication operation, performing a performance judgment based on the upward first performance data PD1 or third performance data PD3 and the corresponding part data FD1. When a misplay is determined, a misplay indication command is issued to control the light array unit 341 and / or vibration unit 342 in the corresponding performance terminal 30 to indicate the misplay. The rhythm feedback unit 162 can be used to perform a rhythm feedback indication operation, indicating strong beats through the vibration unit 342.

[0113] In some preferred embodiments, the misplay feedback unit 161 preferably determines whether a misplay has occurred using a dual misplay determination method. (Referring to the reference...) Figure 7 Among them, teacher's machine (e.g.) Figure 2After receiving performance data (e.g., first performance data PD1 or third performance data PD3), the "main control terminal 10" can read the corresponding notes in the score data and perform a first-level judgment based on whether the pitch of the played notes in the first performance data PD1 or third performance data PD3 matches the pitch of the corresponding notes in the score data FD1. If the first-level judgment fails, it is determined that a wrong note has occurred. If the first-level judgment passes, a second-level judgment is performed based on whether the timing deviation between the timing of the played notes in the first performance data PD1 or third performance data PD3 and the timing of the corresponding notes in the score data FD1 exceeds a deviation threshold (e.g., timing deviation > ±0.5s?). If the second-level judgment fails, it is determined that a rhythmic misplay has occurred. If the second-level judgment passes, it is determined that the performance is normal. If either the first-level or second-level judgment fails, it is determined that a wrong note has occurred.

[0114] In some preferred embodiments, after a misplay is determined to have occurred, the teacher's computer can generate a misplay instruction and send it to the student's computer. The student's computer can then indicate the misplay based on the instruction, for example, by providing feedback through a "long vibration" of the vibration unit. In other embodiments, the student's computer can also control the real-time performance lights of the light array unit to flash in a way that indicates the misplay, thereby accurately locating the misplay position.

[0115] In some preferred embodiments, after a misplay is determined, the teacher's computer can also perform misplay statistics based on the performance data uploaded to the teacher's computer, allowing the teacher to make real-time teaching adjustments. More specifically, in teaching mode, the feedback analysis unit 16 can make real-time teaching adjustments based on the received teaching feedback data. For example, the teacher can explain some common misplays (such as a chord playing error) and issue correction instructions. The student's computer can view the correction prompts and perform correction exercises.

[0116] The role configuration unit 17 can be used to perform role configuration operations, select one or more roles from melody player, harmony player, orchestrator, rhythm player, and sound effects player, and configure one or more role terminals accordingly.

[0117] In some preferred embodiments, in conjunction with reference Figure 2 and Figure 4In the improvisational co-creation mode (sometimes also called "Remix co-creation"), the improvisational performance data generated by the melody player through the corresponding melody player terminal, the harmony data generated by the harmony player through the corresponding harmony player terminal, the orchestration instructions generated by the orchestration player through the corresponding orchestration player terminal based on MIDI control signals, the rhythm instructions generated by the rhythm player through the corresponding rhythm player terminal, and the sound effects instructions generated by the sound effects player through the corresponding sound effects player terminal are uploaded to the main control terminal 10. The second audio output unit 19 of the main control terminal 10 generates a second audio based on the improvisational performance data and harmony data, and outputs the second audio by calling the corresponding second VST sound source according to the orchestration instructions, rhythm instructions, and sound effects instructions based on MIDI control signals. Among them, "orchestration instructions" may include, but are not limited to, timbre switching instructions based on MIDI control signals (e.g., for switching between octaves or instrumentation) and timbre overlay instructions (e.g., for overlaying timbre of parallel octaves / parallel fifths). "Rhythm commands" may include, but are not limited to, rhythm material recall commands based on MIDI control signals. For example, a rhythm material recall command can be used to select and recall preset rhythm materials (such as drum loops, bass loops, etc.) within the teacher's machine. "Sound effect commands" may include, but are not limited to, sound effect material recall commands based on MIDI control signals.

[0118] Preferably, in multi-person teaching mode, multi-person ensemble mode, or improvisational co-creation mode, the main control terminal and multiple performance terminals can be networked in parallel.

[0119] The relay unit 18 can be used to perform relay operations, forwarding the performance data of the first terminal to the second terminal for relay.

[0120] In some preferred embodiments, the performance data generated by the first terminal (e.g., including first performance data or third performance data) can be forwarded to the third user interface of the second terminal for display via the corresponding voice terminal, and the performance data generated by the second terminal (e.g., including first performance data or third performance data) can be uploaded to the main control terminal via the corresponding voice terminal.

[0121] More specifically, in conjunction with references Figure 2 and Figure 4In a fun teaching mode, for example, demonstration performance data generated when a first terminal connected to the teacher's computer performs a demonstration can be forwarded to a second terminal (sometimes called a "student's computer") for broadcast, such as live streaming of the performance on the student's computer. More specifically, the teacher's computer can collect the teacher's demonstration performance data and push it to the student's computer's light array unit for live streaming of the performance. Students follow the performance instructions on the student's computer's light array unit, and the performance data can be sent back to the teacher's computer, which loads the VST audio source through the second audio output unit 19 and outputs it.

[0122] The second audio output unit 19 can be used to perform a second audio output operation, generate a second audio based on the performance data uploaded by one or more performance terminals, and output the second audio by calling a preset second VST sound source.

[0123] The page-turning unit 10A is used to perform the physical music stand page-turning operation. It determines the current performance progress based on the first performance data PD1 or the third performance data PD3 and the corresponding part score data, and turns the physical music score when the current performance progress reaches the last measure of the current page.

[0124] For example, in symphony rehearsal mode (not limited to this invention), the performance signals generated by live musicians playing acoustic instruments can be converted into MIDI signals by a MIDI pickup and enter the performance assistance system 100 of this invention to participate in determinations such as pitch and rhythm, for example, comparing them with the pitch and rhythm of corresponding notes in the score data, thereby determining the current performance progress. When the performance reaches the last measure of the current page, automatic page turning can be performed (e.g., driving an electronic or physical sheet music page turner to replace manual page turning), thereby assisting symphony orchestra rehearsals.

[0125] like Figure 1 and Figure 2 As shown, in some embodiments, the performance assistance system 100 of the present invention further includes a cloud 40, which is communicatively connected to the main control terminal 10, and includes at least a training unit 41 and a cloud database 42. Each vocal part terminal 30 may also include a virtual musician loading unit 24.

[0126] Among them, the training unit 41 in the cloud 40 can be used to perform performance data training operations, train the model based on the performance data PD, and generate a virtual musician model VD corresponding to the performer.

[0127] More specifically, during the performance data training process, training unit 41 first collects the performer's performance data (including but not limited to notes, dynamics, tempo, rhythmic fluctuations, ADSR envelope, and playing habits) based on the performer's (sometimes referred to as a musician) unique ID. Next, training unit 41 performs feature extraction, such as extracting dynamic curves, attack / attenuation / suspension / release envelopes, phrasing pauses, and breathing rhythms, forming the performer's personal performance feature vector. Finally, based on these personal performance feature vectors, training unit 41 generates a virtual musician model (VD) for the performer (sometimes referred to as a "shadow model") through a learning model.

[0128] Among them, the cloud database 42 of the cloud 40 can be used to store data, such as performance data PD, score FD, virtual musician model VD and other data.

[0129] The virtual musician loading unit 24 of the voice terminal 20 can be used to perform virtual musician loading operations. For example, a virtual musician model VD corresponding to the performer can be loaded into the corresponding voice terminal 20 to replace the performer in playing.

[0130] In some application implementations, for example, in a multi-person ensemble mode, a shadow model can be invoked at any time as a virtual ensemble teammate to perform in real time in sync with the real musicians according to a global timestamp.

[0131] In some other extended application embodiments, high latency in remote ensemble performances has always been a challenge due to physical reasons such as geographical distance. However, this invention can learn the playing habits of real people through the training unit 41, and perform as an ensemble agent by loading a virtual musician model VD locally. After the performance, the audio tracks can be merged through the second audio output unit of the main control terminal, thereby realizing "real-time ensemble performance" in place of real people.

[0132] In some other extended application embodiments, by loading virtual musician models (VD) on different performance terminals, applications such as one person playing multiple parts in ensemble, playing with one's past self, and remote teaching through cross-terminal shadow broadcasting can be realized.

[0133] In this invention, the performance terminal 30 is preferably, for example, a keyboard instrument, as described in the reference. Figure 2 and Figure 6 Its structure may include, for example, a performance input unit 32 containing multiple piano keys 321, a processor (not shown) containing a data acquisition unit 33, a light array unit 341, a vibration unit 342, a visual metronome unit 343, and a multimodal sensing unit 35, etc. Furthermore, when the performance terminal 30 is as follows... Figure 6When playing a keyboard instrument as shown, the playing terminal 30 may use a mobile terminal (such as a mobile phone / tablet / laptop, etc., but the present invention is not limited thereto) as a third user interface 31 that is connected to it for communication.

[0134] In this invention, during operations such as issuing / uploading instructions and data, a standardized data structure can unify the data format of all instructions and data to avoid transmission deviations. For example, the standardized data structure may include components such as a frame header, terminal ID, timestamp, data type, data length, data content, checksum, and frame trailer. More specifically, the "frame header" may be 2 bytes, fixed at "0xAA55", used for hardware to quickly identify valid control frames and filter interference signals. The "terminal ID" may be 2 bytes, consistent with the mapping table, used for precise addressing; it can be filled with 0xFFFF for broadcast synchronization. The "timestamp" may be 4 bytes, carrying a global timestamp to ensure that all control and performance actions are strictly aligned with the timing reference. The "Data Type" field can be 1 byte, used to identify the hardware control purpose of the current frame. For example, "0x01" can be used to identify "synchronization command," "0x02" can be used to identify "score data," "0x03" can be used to identify "control command," "0x04" can be used to identify "performance data," "0x05" can be used to identify "feedback command," and "0x06" can be used to identify "fault alarm," etc., but this invention is not limited to these. The "Data Length" field can be 2 bytes, used to indicate the length of subsequent data segments for easy hardware parsing. The "Data Content" field can be N bytes, carrying synchronization parameters, score notes, performance keys, sensor values, feedback intensity, etc., according to type. The "Check Digit" field can be 1 byte, used for verification to ensure error-free transmission. The "Frame End" field can be 2 bytes, fixed as "0x55AA," used to mark the end of a control frame. This invention, through a standardized data structure, ensures accurate command transmission, reliable timing alignment, and interference resistance between the teacher's computer, the instrument part computer, and the performance terminal.

[0135] In this invention, each device can perform various control commands (CS) including but not limited to, such as synchronization commands, synchronization control commands, segment positioning commands, performance speed adjustment commands, start / pause / continue / stop commands, performance start commands, teaching guidance commands, vibration control commands, and global synchronization calibration commands. Figure 2This includes the transmission / uploading of control commands (CS1) from the main control terminal 10 to the voice terminal 20, and control commands (CS2) from the voice terminal 20 to the performance terminal 30, etc., as well as the transmission / uploading of various data including full score data FD, part score data FD1, visualized part score data FD2, and performance data (e.g., PD, PD1, PD2, PD3). Furthermore, it is understood that the transmitted commands and data may differ in different application scenarios, and this is not intended to limit the invention.

[0136] like Figure 8A As shown, the present invention also provides a performance assistance method 200A, which may include, for example, the following steps:

[0137] Step S1: The data processing terminal acquires the total spectrum data and parses it to generate multi-channel spectrum data;

[0138] Step S2: The data processing terminal performs a hierarchical filtering operation on the full score data, filtering the full score data according to the instrument or performance role, and the filtered score data is sent down through the data channel.

[0139] Step S3: The performance interaction terminal receives the score data and outputs the real-time performance prompts for the current note and the prediction prompts for subsequent notes based on the performance timeline through the performance prompt unit according to the note sequence.

[0140] Step S4: The performance interaction terminal collects the performer's performance data and sends it back to the data processing terminal.

[0141] In the performance assistance method 200A of the present invention, the data processing end and the performance interaction end of the performance assistance system are configured to perform at least one of the above-described operations. These operations may include, for example, one or more of the following: synchronization control operation, score parsing operation, score rendering operation, hierarchical filtering and distribution operation, mapping table generation operation, performance operation, multimodal control operation, acquisition operation, performance prompt operation, vibration indication operation, beat visualization prompt operation, feedback analysis operation, audio output operation, role configuration operation, broadcasting operation, page turning operation, etc.

[0142] However, it is understood that, depending on the application scenario, the data processing end and the performance interaction end of the performance assistance system may also be configured to perform some other operations, which is not intended to limit the present invention.

[0143] like Figure 8B As shown, taking the performance assistance method 200B in the teaching mode as an example, the performance assistance method 200 of the present invention may more specifically include, for example:

[0144] Step S201, Synchronization Control. This step can be achieved, for example, by issuing synchronization control commands to drive all terminals in the entire system to establish a unified timing reference, ensuring that the clocks of all terminals are aligned.

[0145] Step S202, Spectrum Analysis. This step, for example, completes the spectrum analysis and terminal binding, laying the foundation for accurate distribution of individual spectra later.

[0146] Step S203, layered filtering and distribution. This step can employ single-layer filtering, double-layer filtering, or multi-layer filtering to achieve accurate distribution of scores based on instrument groups or individual instruments, adapting to the needs of multi-part and multi-terminal collaborative ensemble performances.

[0147] Step S204: Issuance of teaching instructions. This step, for example, enables the precise issuance of teaching instructions from the teacher's computer, covering the control and guidance needs of all scenarios such as teaching and practicing.

[0148] Step S205: Upload performance data. This step, for example, enables the accurate collection and uploading of performance data, providing data support for teachers' real-time monitoring and error judgment.

[0149] Preferably, the performance assistance method 200B of the present invention may further include, for example:

[0150] Step S206, Teaching Feedback. This step, for example, can enable error identification and real-time feedback, forming a closed loop of "collection-judgment-feedback-optimization," thereby improving the relevance of teaching and practice.

[0151] Step S207, Redundant Link Switching. This step, for example, enables automatic switching of redundant links, ensuring the reliability of data transmission links and preventing link interruptions from affecting teaching and practice processes.

[0152] Step S208, temporary data clearing. This step, for example, can automatically clear the data cache of the main control terminal / audio terminal after power-off, ensuring that no data remains.

[0153] In this invention, depending on the application scenario, some or all of the above steps can be selected for execution (that is, some of the above eight steps can be reused in different scenarios), which is not intended to limit the invention.

[0154] In some embodiments, the performance assistance method of the present invention can be implemented through a specific application (APP). For example, a specific application can be loaded in a corresponding terminal device, and various operations can be performed through this specific application.

[0155] The following will describe in detail the performance assistance method of the present invention by combining specific application embodiments of the performance assistance system 100 of the present invention in multiple scenarios.

[0156] Application Example 1: Symphony Rehearsal

[0157] In this application embodiment 1, the reusable process is as follows: Step S201 (Global Timing Synchronization) → Step S202 (Mapping Table Generation) → Step S203 (First-Level Filtering) → Step S204 (Issuance of Command Instructions) → Step S207 (Redundancy Switching) → Step S208 (State Reset). In this application embodiment 1, partial score distribution (first-level filtering) and command instruction transmission can be achieved, accompanied by rendered and visualized staff notation guidance. There is no multimodal acquisition, no multimodal feedback, and no keyboard performance acquisition function; only basic redundancy switching and state reset are retained, meeting the core needs of professional rehearsals.

[0158] More specifically, in this application embodiment 1, for example, one master terminal can be used as the teacher's machine for command and operation, and six instrument terminals can be used as instrument machines (slave machines, for example, corresponding to six instrument groups such as strings / woodwinds / brass / percussion) for operation by the musicians of each instrument group. External displays / tablet computers, sound systems, etc., can also be connected.

[0159] The following is a description of several stages.

[0160] Section 1: Score Analysis

[0161] 1. The instructor operates on the teacher's computer (e.g., uploading the full score, confirming the parsing rules for the 6 major instrument groups, etc.). The teacher's computer screen can display the progress of the full score parsing in real time, as well as a preview of the score data corresponding to the 6 instrument groups (corresponding to 6 voice parts).

[0162] 2. After the total score is analyzed, the commander checks the completeness and accuracy of each sub-score data (for example, by clicking to preview the corresponding sub-score data). After confirming that there are no errors, click "Confirm and Send".

[0163] 3. Each instrument group member checks the external monitor / tablet connected to the instrument group unit. The monitor displays the real-time progress of receiving the score data of their instrument group (i.e., their own instrument part). After confirming that the score data of their instrument group matches the scores of each instrument in their instrument group, they click "Confirm Instrument Part Ready".

[0164] 4. The instructor checks the readiness status of all voice parts on the teacher's computer (all 6 voice parts are shown as ready), completing the preparation work for this step.

[0165] Step 2: Global Timing Calibration and Spectral Synchronization Rendering

[0166] 1. The instructor clicks "Global Timing Calibration" on the teacher's computer. The teacher's computer screen displays the calibration progress and timing deviation values ​​of each voice part in real time. After all voice parts are calibrated, a message "Global Timing Alignment Successful" is displayed.

[0167] 2. The commander issues a score rendering command, and the external monitors / tablets of each section simultaneously generate visual score data (such as staff notation or numbered musical notation) for that section, allowing musicians to test score scaling, etc.

[0168] 3. After the musicians confirm that the score rendering is complete and the initial cursor position is correct, the conductor checks the readiness status of all parts on the teacher's computer;

[0169] 4. After the conductor confirms that the entire system is ready, he signals to all musicians that rehearsal preparation is complete and the system is ready to start.

[0170] Step 3: Rehearsal begins with synchronized playback of full instrumental scores.

[0171] 1. When the instructor clicks "Start Playback" on the teacher's computer, the external monitors / tablets of the 6 voice part machines start scrolling the score at the same time, the dynamic cursor moves synchronously with the global timing, and the sound system plays the beat prompts synchronously.

[0172] 2. Musicians follow the dynamic cursor and score markings on the external monitor / tablet to play, and the score scrolling progress of all parts is completely consistent throughout the entire performance, with no fast forward or lag;

[0173] 3. When the score reaches the end of a section, all external displays / tablets connected to the instrument panel will turn the page simultaneously, with the cursor moving seamlessly and the musicians playing without interruption;

[0174] 4. The instructor can view the full score playback progress and the synchronization status of all parts in real time on the teacher's computer screen, and monitor the consistency of the score scrolling throughout the process.

[0175] Step 4: The conductor controls all vocal parts to execute synchronously according to different control instructions.

[0176] Control command 1: Pause / Continue playback

[0177] 1. When the conductor clicks "Pause," the external monitors / tablets of all voice parts can simultaneously freeze the score and cursor (feedback), allowing the conductor to provide explanations and annotations for the current section;

[0178] 2. When the conductor clicks "Continue Playing", the external monitors / tablets of all instrument parts can resume scrolling synchronously from the paused position, and the musicians can continue to play along.

[0179] Control Instruction 2: Practice Memory Point Jump

[0180] 1. The conductor marks key practice passages on the full score and sets them as memory points. All voice parts can receive the memory point parameters synchronously.

[0181] 2. When the conductor clicks the corresponding memory point, all external monitors / tablets connected to all parts of the instrument can simultaneously jump to the score of that section, and the cursor is reset to the beginning of the section, without the musicians needing to manually turn the page.

[0182] Control Instruction 3: Repeated Paragraph Practice

[0183] 1. Select the section that needs to be practiced repeatedly, set the number of loops, and click "Start Repeat". All external monitors / tablets connected to the voice parts can synchronously scroll the score within the target section.

[0184] 2. Musicians practice in a targeted manner following the continuously scrolling score. After the cycle ends, the score on all parts stops synchronously at the end of the section.

[0185] Control command 4: Playback speed adjustment

[0186] 1. The conductor adjusts the BPM (e.g., stepless adjustment from 0.5x to 1x) via the speed slider on the teacher's machine, and the scrolling speed of the score on the external monitors / tablets of all voice parts changes synchronously to adapt to different needs such as slow practice and playing at the original speed;

[0187] 2. After the speed adjustment is completed, the conductor can click "Restore Original Speed" to synchronize the scoreboard on all external monitors / tablets of all voice parts back to the initial scrolling speed.

[0188] Step 5: Rehearsal Ends and System Status Reset

[0189] 1. After the conductor finishes the rehearsal, click "End Rehearsal" on the teacher's computer. The external monitors / tablets of all voice parts will stop scrolling on the score simultaneously, and the cursor will be reset to the starting position of the score.

[0190] 2. Musicians should check the standby indicator on the instrument panel to confirm that temporary data has been cleared, and turn off the external monitor / tablet or keep the device in standby mode.

[0191] 3. The instructor can view the progress record and parameter settings of this rehearsal on the teacher's computer, and can set the starting memory point and parameters for the next rehearsal;

[0192] 4. The conductor confirms that all vocal parts have completed their status reset, shuts down the system, and concludes the rehearsal.

[0193] Application Example 2: Multi-person teaching

[0194] In this application embodiment 2, the reusable process is as follows: Step S201 (Global Timing Synchronization) → Step S202 (Mapping Table Generation) → Step S203 (Two-Level Filtering and Score Division) → Step S204 (Teacher Machine Command Issuance) → Step S205 (Performance Data Upload) → Step S207 (Redundancy Switching) → Step S208 (State Reset). The difference between this application embodiment 2 and application embodiment 1 is the addition of a second layer of filtering, along with functions such as issuing dedicated teaching commands, keyboard performance acquisition, multimodal acquisition, and visual + haptic feedback from light arrays. Steps S201 to S208 are fully reused, achieving a complete closed loop of synchronization, score division (two-level filtering), guidance, performance data upload, multimodal acquisition, multimodal feedback, redundancy switching, and state reset, adapting to the standardized and large-scale needs of group teaching.

[0195] More specifically, in this application embodiment 2, for example, one master terminal can be used as a teacher's machine for operation, and six instrument terminals can be used as instrument slaves (e.g., corresponding to violin groups 1 / 2, cello groups, woodwind groups, brass groups, percussion groups, and color groups), and 30 keyboard instruments (including light array units, vibration units, multimodal sensing units, and external displays / tablets, etc.) can be used for operation by 30 students. Furthermore, it can be connected to a high-definition projector, a gigabit switch, a four-channel power amplifier, desktop monitor speakers, etc.

[0196] The following is a description of several stages.

[0197] Step 1: Pre-class preparation and full system initialization calibration

[0198] 1. The teacher operates the teacher's computer (e.g., starting the system, importing the teaching score, confirming the four-dimensional mapping table, etc.). The teacher's computer screen displays the self-test status, synchronization progress, and score preview. At the same time, the projector can display the global teaching interface synchronously.

[0199] 2. Students operate the corresponding keyboard instruments (e.g., inputting the corresponding seat information) to test the light array unit, vibration unit, and multimodal sensing unit (e.g., touch sensor, pressing piano keys, etc.). An external monitor / tablet can display the test results, synchronization status, and a dedicated score preview.

[0200] 3. Teachers can initiate global synchronization calibration with one click. All external monitors / tablets connected to keyboard instruments can display the calibration countdown simultaneously. After students confirm that the synchronization is normal, they can click to confirm, and the system will provide a synchronization ready prompt.

[0201] 4. Teachers can randomly check the display of musical scores and performance prompts on the terminals through the teacher's computer (e.g., by selecting a terminal to view). The teacher's computer screen can display the status of the corresponding terminal. After confirming that everything is correct, click to start the teaching preparation. The system will then enter the teaching ready state (e.g., the global ready indicator light will turn on).

[0202] Section 2: Classroom Teaching and Dual-Guided Performance

[0203] 1. Teachers operate the teacher's computer (e.g., start / stop performance, adjust BPM, practice phrases, and mark key teaching points). The teacher's computer screen and projector can simultaneously display operation feedback and the whole class's performance progress.

[0204] 2. Students follow the performance prompts from the light array unit and the dynamic cursor on the external monitor / tablet to guide their playing (e.g., pressing the keys), and receive rhythmic feedback (e.g., "slight vibration" to match the beat) through the vibration unit.

[0205] 3. Students can adjust parameters such as CC1 vibrato, CC7 volume, and CC11 expression by touching / sliding the multimodal sensing unit. The second synthesis unit of the voice terminal can synthesize it into a VST sound source output, and the speaker can output expressive performance audio synchronously (for example, it can provide feedback on the timbre intensity and vibrato effect changes).

[0206] 4. Students view their own playing progress and fingering prompts on the external monitor / tablet, and adjust their playing actions in a timely manner to ensure that their playing is synchronized with the guidance;

[0207] 5. Teachers can view the entire class's performance status through a projector and issue unified prompts for common problems. All external monitors / tablets connected to keyboard instruments can display the prompt information simultaneously.

[0208] Step 3: Feedback on Mistakes and Real-time Teaching Adjustment

[0209] 1. When a student plays a wrong note or deviates from the rhythm, the keyboard instrument will trigger a long vibration to indicate the wrong note. The student can then adjust their playing actions according to the indication (e.g., correct the key press, adjust the rhythm, etc.).

[0210] 2. Teachers can view the error statistics displayed on the teacher's computer and projector, and explain common errors (such as playing a certain chord incorrectly). Students can view the correction prompts on the external monitor / tablet and practice accordingly.

[0211] Step 4: End of Teaching and Status Reset

[0212] 1. After completing the lesson, the teacher should click "End Lesson" on their computer.

[0213] 2. The student stops playing, checks the standby prompt displayed on the external monitor / tablet, and turns off the keyboard instrument or puts it into standby mode;

[0214] 3. Teachers can view the classroom practice reports generated by their computers (e.g., by clicking to view or save the report). The computer screen can display report details, error statistics charts, etc., for subsequent teaching review.

[0215] 4. The teacher turns off the projector and sound system, confirms that all keyboard instruments have been reset and are in standby mode (the teacher's computer displays that all terminals are in normal status), and concludes the lesson.

[0216] Application Example 3: Multi-person ensemble (or "100-person ensemble")

[0217] In this application embodiment 3, the reusable process is as follows: Step S201 (multiple voice machine parallel operation + global timing synchronization) → Step S202 (mapping table generation) → Step S203 (optional 1 / 2 level filtering score distribution) → Step S204 (basic command issuance) → Step S205 (performance data upload) → Step S207 (redundancy switching) → Step S208 (state reset). In this application embodiment 3, the parallel operation of multiple voice machines focuses on ensuring multi-terminal timing synchronization and dedicated score distribution, enabling large-scale, synchronized playing by a large group of people.

[0218] More specifically, in this application embodiment 3, for example, one master terminal can be used as a teacher's machine for command and operation, and 6×N multi-parallel-connectable instrument terminals can be used as instrument machines (slave machines, for example, corresponding to 6 instrument groups such as violin group 1 / 2, cello group, woodwind group, brass group, percussion group, and color group), and 100 keyboard instruments (including light array units, vibration units, multimodal sensing units, and external displays / tablets, etc.) can be used for operation by 100 performers. Furthermore, it can also connect to a high-definition large screen, multiple gigabit switches, 6×N sets of professional sound systems (including main amplifier speakers, monitor speakers, and power amplifiers), timing test instruments, backup power supplies, etc.

[0219] The following is a description of several stages.

[0220] Step 1: Group performance preparation and system initialization calibration

[0221] 1. Instruct the teacher's computer to operate (e.g., start the system, import the ensemble score, confirm the four-dimensional mapping table, check the VST3 sound source adaptation status, etc.). The teacher's computer screen and the high-definition large screen will simultaneously display the self-test status, synchronization progress and score preview.

[0222] 2. The performer operates the corresponding keyboard instrument (e.g., confirms personal part and terminal binding information), tests the light array unit, vibration unit, multimodal sensing unit, etc. (e.g., touch sensor, pressing piano keys, etc.), and can display test results, synchronization status and exclusive score preview;

[0223] 3. The conductor can initiate a global synchronization calibration with one click. All keyboard instruments can display the calibration countdown simultaneously. After the performer confirms that the synchronization is normal, they can click to confirm, and the system will provide a synchronization ready prompt.

[0224] 4. The instructor checks the score display and light array guidance functions of the terminals through the teacher's computer (e.g., by selecting a terminal to view). The teacher's computer screen can display the status of the corresponding terminal. After confirming that everything is correct, click to start the group performance preparation. The system then enters the group performance ready state (e.g., the global ready indicator lights up and the sound system is in standby mode).

[0225] Step 2: Group Performance Execution and Guidance

[0226] 1. The command is issued via the teacher's computer to start the group performance, and the teacher's computer broadcasts a synchronization control signal every 10ms to ensure synchronization of the entire system;

[0227] 2. After receiving the synchronization control signal, each instrument part forwards the synchronization signal and its dedicated instrument score (including dynamic cursor, timing marks, chord analysis, individual notes, etc.) to the corresponding keyboard instrument;

[0228] 3. The performer operates the corresponding keyboard instrument to play, and the performer's performance data can be uploaded to the corresponding part machine. The corresponding part machine can output the performance data through the first audio output unit and transmit it to the corresponding sound system to realize the independent sound production of each part.

[0229] 4. Each instrument can synchronously transmit its performance data back to the teacher's computer. The teacher's computer then summarizes the performance data of all keyboard instruments in real time, generates a visual chart of the performance status of 100 people and time-series synchronous monitoring data, and displays it on a high-definition large screen.

[0230] Step 3: Real-time Control and Detail Optimization

[0231] 1. The conductor can view the performance status of each section, timing synchronization data and sound output effect through a high-definition screen. For common deviations (such as a section being too slow), the conductor can issue annotations and correction instructions. The performers of that section can view the highlighted prompts on the terminal and adjust their performance rhythm.

[0232] 2. Based on the auditory feedback from the scene, the conductor adjusts the volume of a certain section of the sound system and the timbre of the VST sound source through the teacher's computer, and listens to the changes in the sound output in real time until the ideal group performance effect is achieved.

[0233] Step 4: End of Ensemble Performance, Recording Results and Status Reset

[0234] 1. After the ensemble performance ends, the conductor clicks "Stop Ensemble Performance" on the teacher's computer. The teacher's computer screen will then display the command issuance status and the terminal reset progress.

[0235] 2. The conductor clicks "Save Performance Data". The system displays the data saving progress. Once the saving is complete, a success message will be displayed. The conductor can preview the performance data and timing statistics (e.g., by clicking "Preview") and view the performance record and synchronization effect.

[0236] 3. The performer stops playing, checks the standby prompt displayed on the terminal screen, and turns off the power to the keyboard instrument or puts it into standby mode;

[0237] 4. Instruct the teacher to view the group performance timing statistics generated by the teacher's computer (e.g., click to view), and conduct a review based on the performance data (e.g., generate data charts, timing replay) for subsequent optimization and adjustment;

[0238] 5. Instruct the staff to shut down the high-definition screen, sound system, and timing test equipment, and confirm that all terminals have been reset and are in standby mode (e.g., the teacher's computer screen shows that all terminals are in normal status), and complete the group performance.

[0239] Application Example 4: Improvisational Collaboration

[0240] In this application embodiment 4, the reusable process is as follows: Step S201 (Global Timing Synchronization) → Step S202 (Mapping Table Generation) → Step S203 (Two-Level Filtering and Score Distribution, Achieving Second-Level Distribution) → Step S204 (Command Distribution) → Step S205 (Performance / Preset Data Upload) → Step S207 (Redundancy Switching) → Step S208 (State Reset). In this application embodiment 4, bidirectional data interaction can be simultaneously achieved between the teacher's computer's command distribution and the upload of preset data by each terminal role, adapting to the needs of multi-person collaborative creation scenarios.

[0241] In this application embodiment 4, for example, a main control terminal integrating the functions of the vocal part terminals can be used as a teacher's machine for operation by the teacher, and 4 to 6 keyboard instruments (including light array units, vibration units, multimodal sensing units, and external displays / tablets, etc.) can be used as role terminals for 4 to 6 students to take on roles such as melody player, harmony player, orchestrator, rhythm player, and sound effects player. Furthermore, it can also be connected to a main amplifier system, monitor speakers, a high-definition touch screen, a gigabit switch, a dedicated teaching sound card, a backup power supply, audio recording equipment, etc.

[0242] The following is a description of several stages.

[0243] Step 1: Co-creation preparation and full system initialization calibration

[0244] 1. Teachers operate the teacher's computer (e.g., start the system, set BPM, assign five roles, import the baseline melody, and send out strong beats, etc.). The teacher's computer screen can synchronously display role assignment, timing calibration status, and melody preview with the large screen.

[0245] 2. The melody player checks the baseline melody and guidance prompts displayed on the corresponding character's terminal to confirm that the melody information has been received successfully;

[0246] 3. The harmonist views the melody strong beat markers and chord mode selection interface displayed on the corresponding character's terminal, and selects block / arpeggiated chord mode;

[0247] 4. The orchestrator checks the tone library and overlay function panel displayed on the corresponding role's terminal to confirm that the tone control permissions are in effect;

[0248] 5. The rhythmist and sound effects designer should check the loop material library and sound effect material list displayed on their respective character's terminal to confirm that the material access permissions are in effect;

[0249] 6. The teacher checks the ready status of all role terminals. After confirming that everything is correct, the co-creation preparation is complete, and the system enters the co-creation ready state.

[0250] Part 2: Remix Improvisational Co-creation Execution and Real-time Collaboration

[0251] 1. Teachers can monitor the overall co-creation progress, track status, and synchronization quality on the teacher's computer, and can calibrate the timing, adjust the BPM, or issue prompts at any time;

[0252] 2. The melody player follows the guidance of the light array unit of the corresponding character terminal to play the baseline melody, and can freely improvise, and upload the improvisational performance data in real time and display the melody track status synchronously;

[0253] 3. The harmonist arranges thirds harmony based on the strong beats of the melody, can switch between block / arpeggiated chord modes, and uploads harmony data in real time;

[0254] 4. The orchestrator selects the melody / harmony track timbre on the corresponding role terminal, and can overlay parallel octave / parallel fifth thickened timbre. The teacher's machine and the speakers update the timbre effects synchronously.

[0255] 5. The rhythmist selects and triggers the drum beat and bass loop on the corresponding character's terminal, and the teacher's terminal plays the corresponding rhythm synchronously in sequence;

[0256] 6. The sound effects operator selects preset sound effects materials on the corresponding role terminal. The teacher's computer can synthesize the improvisational performance data, harmony data, etc. through the second audio output unit to generate a second audio and output it in real time. The teacher's computer produces sound by loading the preset sound effects materials.

[0257] 7. All characters can view the overall co-creation status through the external monitor / tablet connected to their respective character's terminal. If they find that they are out of sync, they can initiate a synchronization request, and the system will automatically complete the timing calibration.

[0258] Step 3: Real-time Adjustment and Improvisation

[0259] 1. Each character performs real-time control operations on their corresponding character terminal, including melody improvisation, harmony optimization, instrument timbre switching, rhythm pattern replacement, and real-time sound effect triggering.

[0260] 2. The corresponding role terminal receives operation instructions from each role in real time, uploads them to the teacher's computer immediately, and completes instruction parsing and global synchronization within 0.5 seconds.

[0261] 3. The teacher's computer returns a synchronization confirmation signal to each role's terminal in real time. The teacher's computer screen displays the optimized status in real time, including the current timbre, voice part status, chord sequence, rhythm mode, sound effect trigger status, etc., forming a complete closed loop of "operation-upload-synchronization-feedback".

[0262] 4. Each role can view the real-time collaboration status at any time. The system does not generate any recordings or save any audio; it only displays the real-time performance and improvisation optimization status.

[0263] Step 4: Collaborative Creation Ends, Results Recording, and Status Reset

[0264] 1. The teacher's computer sends a co-creation end command, and all role terminals simultaneously display a co-creation end prompt.

[0265] 2. Each role performs the result recording confirmation operation on the corresponding role terminal, and the system provides real-time feedback: Recording in progress → Recording completed → Result saved.

[0266] 3. Each role can view the collaborative creation results display interface, which may show only: performance trajectory, parameter configuration, sequence flow, timeline, and role division, without playing or presenting any audio.

[0267] 4. Each character can trigger a status reset operation, and the corresponding character's terminal can provide real-time feedback: Resetting → Cleared → Globally ready → A new round of co-creation can be started.

[0268] Application Example 5: Fun Teaching

[0269] In this application embodiment 5, the reusable process is as follows: Step S201 (Global Timing Synchronization) → Step S202 (Mapping Table Generation) → Step S203 (Two-Level Filtering and Score Division) → Step S204 (Issuance of Fun Practice Commands) → Step S205 (Upload of Performance Data) → Step S206 (Real-Time Feedback) → Step S207 (Redundancy Switching) → Step S208 (State Reset). This application embodiment 5 achieves a closed loop of synchronization, score division, guidance, data acquisition, and feedback, enhancing the fun interactive mode (melody-harmony interchange, etc.) and practice report generation function, adapting to fun practice scenarios for 2-4 people, and balancing fun and pedagogy.

[0270] In this application embodiment 5, for example, a main control terminal integrating the functions of the vocal part terminal can be used as a teacher's machine for operation by the teacher, and several keyboard instruments (including light array units, vibration units, multimodal sensing units, and external displays / tablets, etc.) can be used as performance terminals for operation by several students. Furthermore, it can also be connected to a high-definition touchscreen, gigabit switch, backup power supply, audio recording equipment, etc.

[0271] The following is a description of several stages.

[0272] Step 1: System Startup and Initialization Calibration

[0273] 1. When the teacher operates the teacher's computer (e.g., starting the system, importing teaching scores, etc.), the self-test status is simultaneously displayed on the teacher's computer screen and the high-definition large screen;

[0274] 2. Students start the corresponding keyboard instrument, check the terminal self-test prompts, and wait for registration and calibration after confirming that the terminal is normal. The terminal will display the registration progress in real time.

[0275] 3. The teacher checks the student terminal registration status displayed on the teacher's computer. After confirming that all terminals have been successfully registered, the teacher issues a calibration command, and the student terminals simultaneously display a calibration countdown.

[0276] 4. After calibration, the student terminal will indicate that it is ready. The student confirms that the terminal is in normal condition and the light array can be lit normally (e.g., test the light array unit), and then sends feedback confirmation to the teacher's terminal.

[0277] 5. The instructor conducts a brief demonstration and test (e.g., playing a simple musical phrase) to verify the live broadcast effect of the light array unit (e.g., the light array unit corresponding to the student's keyboard instrument lights up synchronously). After confirming that there are no errors, the instructor announces that the teaching mode has been entered.

[0278] Part 2: Live Broadcast and Demonstration Teaching with Performance Lights

[0279] 1. The instructor demonstrates playing the piece (e.g., playing a complete musical phrase), explains the key points of fingering and rhythmic techniques, and displays the playing data and synchronization status on the teacher's computer screen simultaneously;

[0280] 2. Students can view the live broadcast effect of the light array unit of the corresponding keyboard instrument. Through the display interface of the external monitor / tablet, students can intuitively see the teacher's key operation, touch timing and left and right hand division, etc. Students only need to follow the playing lights to familiarize themselves and play along, and can signal the teacher to pause at any time.

[0281] 3. After the demonstration is completed, the teacher clicks the "Save Demonstration" command, and the teacher's computer will prompt that the demonstration file has been saved successfully (feedback), which is convenient for subsequent playback.

[0282] Part 3: Teachers and students perform and practice together synchronously.

[0283] 1. Full Ensemble Mode: The teacher selects "Full Ensemble Mode" and issues synchronous performance instructions through the teacher's computer; students receive synchronous performance instructions through the corresponding keyboard instruments, complete the performance according to the corresponding mode, and collect performance data (notes, rhythm, etc.) in real time.

[0284] 2. Left and Right Hand Division Mode: When the teacher selects the "Left and Right Hand Division Mode", the teacher issues division instructions through the teacher's computer. Two students can take on the left and right hand playing tasks respectively through the corresponding keyboard instruments (or the teacher plays the harmony and the students play the melody). The performance data can be sent back to the teacher's computer, which also monitors the synchronization status in real time to ensure that the rhythm of both hands is unified and the connection is smooth. The teacher's computer can also switch the division mode.

[0285] 3. New Four-Hand Piano Playing Mode: When the teacher selects the "New Four-Hand Piano Playing Mode", the teacher's computer issues pairing instructions and the timing of the two terminals is aligned through synchronous control signals. Two students can take on different parts of the performance using their corresponding keyboard instruments, and the performance data can be sent back to the teacher's computer. The teacher's computer can also monitor the coordination of the parts, support mode switching, and ensure that the two terminals play synchronously without deviation.

[0286] 4. Melody-Harmony Swap Mode: The teacher selects the "Melody-Harmony Swap Mode" and sends a mode switching command through the teacher's computer. Students receive the command through the corresponding keyboard instrument and switch their playing role (e.g., melody / harmony swap) to perform. The performance data can be sent back to the teacher's computer. The teacher's computer can also monitor the synchronization status to ensure that the voices are aligned and the rhythm is consistent. The teacher's computer can also freely switch modes.

[0287] Step 4: Demonstration and saving, post-lesson practice setup, and lesson conclusion.

[0288] 1. When the teacher operates on the teacher's computer (e.g., clicking "Save Demonstration"), the teacher's computer screen will simultaneously display the demonstration saving progress, follow-up parameter configuration, and terminal readiness status;

[0289] 2. Teachers can view the demonstration save completion message to confirm that the data has been successfully archived;

[0290] 3. After the teacher has checked and confirmed that everything is correct, the teacher can shut down the system on the teacher's computer and put it into standby mode.

[0291] This invention performs layered filtering and distribution through the filtering and distribution unit at the data processing end, which can adapt to the diverse needs of different application scenarios. In particular, through double-layer and triple-layer filtering, it can achieve accurate filtering and distribution of "instrument-level" and "chord-level" score data, and can eliminate signal interference from channels other than the instrument group.

[0292] This invention employs a dual spectrum filtering mechanism of "coarse screening of the whole score (i.e., instrument group filtering) - fine screening of the seated instrument channel (i.e., single instrument filtering)" to achieve accurate matching and distribution of spectrum data to the target terminal, while also achieving channel isolation between terminals to avoid data crosstalk.

[0293] This invention, by adopting a standardized data frame structure (i.e., a fixed structure including frame header, terminal ID, global timestamp, data type, data content, check bit, frame tail, etc.), can realize the full-process transmission of synchronization instructions, score data, performance data, feedback instructions, etc.

[0294] This invention employs a redundant data transmission link architecture of "wired + dual wireless," enabling seamless link switching and ensuring uninterrupted and latency-free synchronous teaching / performance across multiple devices. Furthermore, based on global timestamps, this invention achieves a closed-loop interaction encompassing real-time acquisition of terminal performance data, accurate error detection, and multimodal feedback command issuance. The core components of this closed-loop interaction can be implemented directly by hardware logic circuits, with software handling only non-real-time parameter configuration and data aggregation.

[0295] This invention enables simultaneous real-time performance prompts and visual guidance (i.e., rational cognition) through the light array unit of the performance terminal. It can help performers anticipate notes in advance, reduce wrong playing and omissions, and is especially suitable for teaching beginners, thus improving teaching efficiency.

[0296] This invention can also achieve precise acquisition of performance expression parameters (such as, but not limited to, CC1 vibrato, CC2 breathing, dynamics, CC7 volume, CC11 expression, etc.) through a multimodal sensing unit. Furthermore, by deeply coordinating performance cues such as light array units and visual metronome units with the multimodal sensing unit and vibration unit, it can form a dual rhythmic guidance of "visual + tactile" and "rational + emotional" approaches. This enriches the expressive forms of symphonic performance, fun practice, and improvisation, further enhancing performance expressiveness and teaching guidance effectiveness. Simultaneously, through dynamic envelopes, it can provide nuanced and personalized emotional teaching support for artistic processing.

[0297] This invention, adaptable to various application scenarios, can solve the problem of monotonous timbre in traditional acoustic instruments by connecting external mobile terminals (such as mobile phones / tablets / laptops), software sound sources, and microphones. It can convert the performance signals of acoustic instruments into MIDI / XML signals and transmit them to the main control terminal for pitch and rhythm determination. This not only facilitates acoustic instrument teaching but also adapts to diverse scenario requirements. Furthermore, by setting different working modes, it can adapt to various scenarios such as classroom teaching, outdoor performances, individual practice, symphony rehearsals, and VST recording in a recording studio. It can also be further extended into various systems such as symphony rehearsal teaching, multi-person teaching, group music performances, multi-person collaborative creation, and multi-person fun practice. These extended systems do not require modification of the underlying control logic of the performance assistance system of this invention; they can accurately cover different teaching, performance, practice, and collaborative creation scenarios simply through scenario-based parameter adaptation.

[0298] This invention utilizes a cloud-based training unit to train a corresponding virtual musician model (sometimes referred to as a "shadow model") based on the performer's performance data. By loading the virtual musician model locally, it can highly replicate the playing habits of a real musician, thus replacing the performer in the performance. Furthermore, through the virtual musician model, it enables collaborative ensemble performances between virtual and real musicians, as well as extended applications such as remote ensemble performances, multi-part ensemble performances by one person, ensemble performances with one's past self, and cross-terminal shadow broadcasting for remote teaching.

[0299] Exemplary embodiments of the present invention have been specifically illustrated and described above. It should be understood that the present invention is not limited to the disclosed embodiments; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A performance assistance system, characterized in that, This includes both the data processing end and the performance interaction end; The data processing terminal is used to acquire total spectrum data and parse it to generate multi-channel sub-spectral data, and has a built-in timing synchronization unit and filtering and distribution unit. The timing synchronization unit is used to generate a global timestamp and periodically issue clock calibration commands to unify the clocks of all terminal hardware in the performance assistance system. The filtering and distribution unit is used to perform a hierarchical filtering operation, which filters the score data hierarchically according to the instrument or performance role, and the filtered score data is sent to the performance interaction terminal with a time sequence tag. The performance interaction terminal establishes a bidirectional data channel with the data processing terminal to receive the musical score data; The performance interaction terminal is also used to collect performance data generated by the performer and send it back to the data processing terminal. The performance interaction terminal also includes a performance prompt unit, which is bound to the note timing of the corresponding score data and synchronously outputs real-time performance prompts for the current note, as well as predictive prompts for subsequent notes based on the performance timeline.

2. The performance assistance system according to claim 1, characterized in that, The layered filtering includes at least one or any combination of instrument group filtering, single instrument filtering, and chord single note filtering. The data processing terminal also includes a mapping table generation unit, which is used to perform mapping table generation operations to generate a mapping table that reflects the four-dimensional relationship between "terminal-seat-instrument-data channel". The filtering and distribution unit distributes the spectral data according to the mapping table. The data processing terminal supports independent masking of individual musical instruments, and the filtered spectral data is sent through the isolated data channel.

3. The performance assistance system according to claim 1, characterized in that, The performance prompt unit includes a light array unit, which contains at least one row of lights, and distinguishes the real-time performance prompt from the predictive prompt by brightness, number of rows, or color. The visual intensity of the predictive prompt is negatively correlated with the time elapsed between the corresponding note and the current performance moment, and the predictive prompt is implemented by using a gradient brightness method with multiple rows of lights or a single row of lights.

4. The performance assistance system according to claim 1, characterized in that, The performance interaction terminal is also used to attach a local time stamp to the performance data before sending it back to the data processing terminal. The timing synchronization unit is also used to perform timing deviation verification on the returned performance data.

5. The performance assistance system according to claim 3, characterized in that, The performance prompting unit further includes a vibration unit and / or a visual metronome unit; the vibration unit is used to perform vibration indication operation, and to provide performance beat guidance prompts in a first vibration mode according to the strong beat information in the score data. The visual metronome unit is used to perform beat cues and provide beat guidance instructions in a first visual cues manner based on the strong beat information in the spectral data. And / or, the light array unit is also used to perform a timing indication operation, illuminating a corresponding number of indicator lights in the light array unit according to the note timing information in the musical score data.

6. The performance assistance system according to claim 5, characterized in that, The performance interaction terminal also includes a performance input unit for performing performance input operations, and the performer generates first performance data when performing through the performance input unit. And / or, the performance interaction terminal also includes a multimodal sensing unit for performing multimodal control operations, and the performer generates second performance data when performing multimodal control through the multimodal sensing unit; And / or, the performance signal generated by the performer when playing an acoustic instrument is converted by a pickup to generate third performance data, which is then uploaded to the data processing terminal via the performance interaction terminal.

7. The performance assistance system according to claim 6, characterized in that, The data processing terminal may also include one or more of the following units: The rendering unit is used to perform the spectrum rendering operation, rendering the corresponding spectrum data into visualized spectrum data, wherein the visualized spectrum data is a visualized staff notation or a visualized simplified notation. A feedback analysis unit is used to perform feedback analysis operations; wherein, the feedback analysis unit includes a misplay feedback unit and / or a rhythm feedback unit; the misplay feedback unit is used to perform a misplay indication operation, and performs a performance judgment based on the first performance data or the third performance data and the corresponding score data, and when a misplay is determined to occur, it provides a misplay indication through the light array unit and / or the vibration unit; the rhythm feedback unit is used to perform a rhythm feedback indication operation, and provides a strong beat indication through the vibration unit; The role configuration unit is used to perform role configuration operations, select one or more roles from melody player, harmony player, orchestrator, rhythm player, and sound effects player, and configure one or more role terminals accordingly. The relay unit is used to perform relay operations, forwarding the performance data from the first terminal to the second terminal for relay. An audio output unit is used to perform audio output operations, and to achieve audio output by calling a preset VST sound source according to the performance data; The page-turning unit is used to perform the physical music stand page-turning operation. It determines the current performance progress based on the first performance data or the third performance data and the corresponding part score data, and turns the physical music score page when the current performance progress reaches the last measure of the current page.

8. The performance assistance system according to claim 7, characterized in that, The data processing terminal is configured to select one of multiple working modes, including symphony rehearsal mode, multi-person teaching mode, multi-person ensemble mode, improvisation co-creation mode, and fun practice mode. In the symphony rehearsal mode, the filtering and distribution unit of the data processing terminal is configured to perform single-layer filtering that includes the filtering of the instrument group. In the multi-person teaching mode or the multi-person ensemble mode, the filtering and distribution unit of the data processing terminal is configured to perform two-layer filtering including instrument group filtering and single instrument filtering, or three-layer filtering including instrument group filtering, single instrument filtering, and chord single note filtering; and in the multi-person teaching mode, the multi-person ensemble mode, or the improvisation co-creation mode, the data processing terminal and multiple performance interaction terminals are connected in parallel. In the improvisational co-creation mode, the improvisational performance data generated by the melody player through the corresponding melody player terminal, the harmony data generated by the harmony player through the corresponding harmony player terminal, the orchestration instructions generated by the orchestration player through the corresponding orchestration player terminal, the rhythm instructions generated by the rhythm player through the corresponding rhythm player terminal, and the sound effect instructions generated by the sound effect player through the corresponding sound effect player terminal are uploaded to the data processing terminal and audio output is achieved through the audio output unit. In the fun practice mode, the performance data of the first terminal can be forwarded to the second terminal for broadcast via the data processing terminal.

9. The performance assistance system according to any one of claims 1 to 8, characterized in that, Also includes: The cloud, which is communicatively connected to the data processing terminal, includes at least: The training unit is used to perform performance data training operations, train the model based on the performance data, and generate a virtual musician model corresponding to the performer. A cloud-based database used to store data; The data processing terminal further includes a virtual musician loading unit, which performs a virtual musician loading operation to load the virtual musician model corresponding to the performer onto the data processing terminal to replace the performer in playing the role.

10. A performance assistance method, characterized in that, Includes the following steps: S1. The data processing end acquires the total spectrum data and parses it to generate multi-channel spectrum data; S2. The data processing terminal performs a hierarchical filtering operation on the full score data, filtering the full score data hierarchically according to the instrument or performance role, and the filtered score data is sent down through the data channel. S3. The performance interaction terminal receives the score data and outputs the real-time performance prompt of the current note and the prediction prompt of the subsequent notes based on the performance timeline through the performance prompt unit. S4. The performance interaction terminal collects the performer's performance data and sends it back to the data processing terminal.

Images