Musical instrument playing interaction system, audio signal processing method and electronic equipment

Abstract

The invention relates to the technical field of audio signal processing, and provides a musical instrument playing interaction system, an audio signal processing method and electronic equipment. The system comprises a sound mixing table, a musical instrument and a monitoring device, the sound mixing table comprises a first wireless transceiver and a first sound mixer, and the musical instrument comprises a second wireless transceiver and an audio collector; the audio collector of each musical instrument is used for collecting a first audio signal of the musical instrument and sending the first audio signal to the first wireless transceiver through the second wireless transceiver; the first wireless transceiver sends the first audio signals to the first audio mixer; the first audio mixer receives the first audio signals, carries out audio mixing processing on the first audio signals based on the personalized configuration information corresponding to the monitoring devices to obtain at least one second audio signal, and sends the second audio signals to the corresponding monitoring devices or the second wireless transceivers through the first wireless transceivers; and the stage arrangement flexibility of a music playing scene is improved.

Description

Technical Field

[0001] The present application relates to the technical field of audio signal processing, and in particular to a musical instrument performance interaction system, an audio signal processing method and an electronic device. Background Art

[0002] In traditional music performances, the connection between multiple musical instruments and mixing equipment usually relies on wired transmission. Although this connection method is stable and reliable, it increases the complexity of stage layout, reduces the efficiency of venue construction, and limits the musicians' freedom of movement. Especially in large-scale performances or frequent changes of venues, wiring and disassembly are time-consuming and laborious, and are easily affected by on-site conditions, resulting in connection failures, which seriously affect the quality of performance. It can be seen that wired transmission methods have limitations in terms of stage layout flexibility, venue adaptability, and large-scale band management. Summary of the invention

[0003] In order to improve the flexibility of stage layout for live music performances, improve the venue adaptability of musical instruments and mixing equipment, and reduce the difficulty of managing large bands, an embodiment of the present application provides a musical instrument performance interaction system, the system includes a mixing console, at least one musical instrument and at least one monitoring device, the mixing console includes a first wireless transceiver and a first mixer, and the musical instrument includes a second wireless transceiver and an audio collector; wherein the audio collector of each musical instrument is used to collect a first audio signal of the musical instrument, and send the first audio signal to the first wireless transceiver through the second wireless transceiver; the first wireless transceiver sends each first audio signal to the first mixer; the first mixer receives each first audio signal, and mixes each first audio signal based on the personalized configuration information corresponding to each monitoring device to obtain at least one second audio signal, and sends each second audio signal to the corresponding monitoring device or the second wireless transceiver through the first wireless transceiver.

[0004] Based on the above technical solution, the audio data can be exchanged between each musical instrument and the mixing console in a wireless manner, thus avoiding the wiring problems and equipment line management problems in wired connections, thereby improving the flexibility of stage layout and the venue adaptability of related equipment, while reducing the difficulty of band equipment management and improving the freedom of performance movement. Furthermore, the mixing console can perform personalized mixing processing on the audio data based on the personalized configuration information of each monitoring device, thereby meeting the monitoring needs of different listeners, especially suitable for application scenarios where different performers have different monitoring needs for audio in large-scale performances.

[0005] In one implementation, the musical instrument further includes a second mixer, and the second mixer is used to perform secondary mixing processing on the second audio signal to obtain a third audio signal, and send the third audio signal to the monitoring device.

[0006] Based on the above technical solution, the second audio signal is remixed by the musical instrument, which can reduce the data processing pressure of the mixing console on the one hand, and can also be adjusted by the performer according to the controller by setting a controller on the musical instrument or monitoring device end, so that the performer can flexibly adjust the monitoring needs according to the on-site situation.

[0007] In one implementation, the monitoring device includes a wireless receiving interface, and the wireless receiving interface is used to receive the second audio signal sent by the first wireless transceiver or the third audio signal sent by the second wireless transceiver.

[0008] In one implementation, the system further includes a mixing console, which is used to generate the personalized configuration information and send it to the mixing console.

[0009] In one embodiment, the system also includes at least one broadcasting device; the mixing console is also used to mix each of the first audio signals based on the personalized configuration information corresponding to each of the broadcasting devices, obtain the corresponding second audio signal, and send it to the broadcasting device via the first wireless transceiver.

[0010] In one implementation, the system also includes a remote access subsystem, which is used to receive a fourth audio signal from other performance sites and send it to the mixing console, and the mixing console is used to mix the first audio signal and the fourth audio signal based on the personalized configuration information corresponding to the monitoring device of the other performance site to obtain the fifth audio signal, and send the fifth audio signal to the monitoring device of the other performance site through the remote access subsystem.

[0011] Based on the above technical solution, it is possible to access remote audio signals, thereby realizing ensemble linkage at different performance sites through mixing processing by a mixing console.

[0012] In one implementation, the monitoring device includes an ultrasonic transmitter, which is used to transmit an ultrasonic signal while the monitoring device plays the second audio signal; the system also includes an ultrasonic receiver, which is used to receive the ultrasonic signals transmitted by each of the monitoring devices, and determine whether the second audio signals played by each of the monitoring devices within the target monitoring range are listened to synchronously based on whether the reception time of each of the ultrasonic signals is consistent; if not, the reception time is sent to the mixing console, so that the mixing console adjusts the order of sending the second audio signals to each of the monitoring devices based on the reception time.

[0013] Based on the above technical solution, it is possible to monitor in real time whether the listener can listen synchronously during the performance, and promptly feedback abnormalities to the mixing console, so that the mixing console can adjust the signal sending order according to each receiving time to ensure the listening experience.

[0014] The embodiment of the present application also provides an audio signal processing method, which applies the above-mentioned musical instrument performance interaction system, wherein the first mixer performs mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the monitoring devices to obtain at least one second audio signal, including: obtaining the personalized configuration information corresponding to the monitoring device; extracting a target mixing mode and its corresponding mixing parameters from the personalized configuration information respectively; when the target mixing mode is mono mixing, calculating the mixing parameters corresponding to each of the first audio signals and each of the first audio signals based on a first preset algorithm to obtain the second audio signal; or, when the target mixing mode is stereo mixing, determining the target mixing parameters based on the time information and the mixing parameters; calculating the target mixing parameters corresponding to each of the first audio signals and each of the first audio signals based on a second preset algorithm to obtain a left channel mixing signal and a right channel mixing signal respectively, and determining them as the second audio signal; wherein determining the target mixing parameters based on the time information and the mixing parameters includes: The time information selects a mixing parameter corresponding to the time information from the mixing parameters as the target mixing parameter or calculates the mixing parameter based on the time information and the parameter change rule to obtain the target mixing parameter; or, when the target mixing mode is panoramic sound mixing, determines a virtual position origin and a target HRTF function from the mixing parameters, and determines a left ear HRTF coefficient and a right ear HRTF coefficient corresponding to each of the first audio signals according to the target HRTF function, determines a value of a channel mixing parameter based on the virtual position origin, calculates the left ear HRTF coefficient, the right ear HRTF coefficient, the channel mixing parameters and each of the first audio signals based on a third preset algorithm, and obtains a corresponding left ear panoramic sound audio signal and a right ear panoramic sound audio signal as the second audio signal; or, when the target mixing mode is surround sound mixing, determines a surround sound type from the mixing parameters, processes each of the first audio signals according to a surround processing method corresponding to the surround type, and obtains the second audio signal.

[0015] Based on the above technical solution, the mixing console can implement corresponding mixing processing according to the mixing modes and mixing parameters selected by the user for different monitoring devices.

[0016] In one implementation, the method further includes performing sound effect processing on the second audio signal based on the personalized configuration information to obtain a sixth audio signal and send it to a corresponding monitoring device and a broadcasting device.

[0017] Based on the above technical solution, the mixing console can further realize personalized sound effect processing during the mixing process, thereby enhancing the richness of the performance content and enhancing the performance atmosphere.

[0018] In addition, an embodiment of the present application also provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the program or instruction implements the above method when executed by the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of the present application are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1A A schematic diagram of the structure of a musical instrument performance interaction system provided in one embodiment of the present application is shown.

[0022] Figure 1B A schematic diagram of the structure of a musical instrument performance interaction system provided in another embodiment of the present application is shown.

[0023] Figure 2 A flowchart of an audio signal processing method provided by an embodiment of the present application is shown.

[0024] Figure 3 A flow chart of a mixing processing method in an embodiment of the present application is shown.

[0025] Figure 4 A flow chart of a panoramic sound mixing processing method in an embodiment of the present application is shown.

[0026] Figure 5 A flowchart of an audio signal processing method provided by another embodiment of the present application is shown.

[0027] Figure 6 A flow chart of a secondary mixing processing method in an embodiment of the present application is shown.

[0028] Figure 7 A schematic diagram of the structure of a musical instrument performance interaction system provided in yet another embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, and "first", "second" and various digital numbers are only distinguished for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0031] The features, structures or characteristics in this application may be combined in one or more embodiments in any suitable manner. In various embodiments of this application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0032] Some optional features in the embodiments of the present application may be implemented independently in some scenarios without relying on other features to solve corresponding technical problems and achieve corresponding effects. They may also be combined with other features in some scenarios as needed.

[0033] In this application, unless otherwise specified, the same or similar parts between the various embodiments can refer to each other. In the various embodiments of this application, if there is no special description and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0034] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0035] The embodiment of the present application provides an interactive musical instrument performance system that can be applied to professional concerts, outdoor music festivals, remote music recording collaboration and other performances or recording rehearsal sites. The data interaction between the musical instrument and the mixing console is achieved wirelessly, thereby avoiding the related problems caused by the wired connection method, improving the flexibility of stage layout and venue adaptability, and also facilitating the management of large bands. In addition, for the application scenario of multi-instrument ensemble, it can also realize differentiated real-time mixing processing of audio signals to adapt to the personalized monitoring needs of each performer during the performance.

[0036] Please refer to Figure 1AThe musical instrument performance interaction system provided in the embodiment of the present application includes a mixing console 11, at least one musical instrument 12, at least one monitoring device 13 and a mixing console 14, wherein each musical instrument 12, each monitoring device 13 and the mixing console 14 can be connected to the mixing console 11 in a wireless manner.

[0037] Specifically, the mixing console 11 includes a first wireless transceiver and a first mixer, wherein the first wireless transceiver is used to realize the reception and transmission of wireless signals; and the first mixer is used to mix the audio signals of various musical instruments.

[0038] The musical instrument 12 includes a musical instrument body, a second wireless transceiver and an audio collector, wherein the musical instrument body is a structural body for realizing performance and has an electrical interface, the second wireless transceiver is used to realize the reception and transmission of wireless signals; the audio collector is used to collect the performance audio signal of the musical instrument. In a specific example, the musical instrument body can be a piano, a guitar, a bass, or an electronic keyboard, and the second wireless transceiver and the audio collector can be externally connected or built into the musical instrument body. In implementation, the second wireless transceiver and the audio collector can be implemented based on the same hardware structure, or can be set as independent and interconnected hardware structures.

[0039] In some embodiments of the present application, the musical instrument 12 further includes a second mixer, which is mounted on the main body of the musical instrument and is communicatively connected to the audio collector and the second wireless transceiver, respectively, for performing secondary mixing processing on the audio signal.

[0040] The monitoring device 13 includes a wireless receiving interface or a wired receiving interface, wherein the monitoring device 13 can receive wireless audio signals sent from the first wireless transceiver and the second wireless transceiver through the wireless receiving interface, and connect to the signal line of the musical instrument through the wired receiving interface to receive the wired audio signal. In one example, the monitoring device 13 is a monitoring headset.

[0041] The mixer 14 can be a software program running on a terminal device such as a computer, a mobile phone, or a PAD. The mixer 11 is connected to a device equipped with the mixer program via a wired or wireless method. The mixer 14 is used to control the mixing method of the mixer 11 for multiple signals.

[0042] Specifically, the mixer 14 can generate personalized configuration information for each monitoring device 13 according to the user operation and send it to the mixer 11, so that the mixer 11 can perform personalized processing on the audio signal sent to each monitoring device 13 based on the personalized configuration information. The personalized configuration information of each monitoring device 13 can be the same or different, and can be configured according to the actual monitoring needs of the performer.

[0043] In one example, the personalized configuration information may include a mixing mode and mixing parameters; in another example, the personalized configuration information may further include a pre-mixing processing method or a post-mixing processing method.

[0044] In some implementations of the present application, the personalized configuration information may also include sound effect control parameters, and the mixing console may further perform personalized sound effect processing on the audio signal obtained after the mixing process based on the sound effect control parameters, wherein the sound effect control parameters in different personalized configuration information may be the same or different, and are specifically set according to monitoring requirements. In one example, personalized sound effect processing includes but is not limited to adding other virtual performers or background sounds.

[0045] Please refer to Figure 1B In some embodiments of the present application, the musical instrument performance interactive system may further include at least one broadcasting device 15, wherein the broadcasting device 15 may be connected to the mixing console 15 in a wired or wireless manner to play the received audio signal. Similarly, the user may also set personalized configuration information for each broadcasting device 15 based on the mixing console to meet different playback requirements.

[0046] Please refer to Figure 2 In a specific application example, when the musical instrument performance interaction system processes audio signals, the interaction of various components specifically includes the following steps.

[0047] S201, the audio console sends personalized configuration information to the mixing console.

[0048] In implementation, the user can set personalized configuration information for each monitoring device and each broadcasting device through the mixer before the performance begins, and send it to the mixing console. It is understandable that the user can configure different personalized configuration information for each monitoring device and each broadcasting device, or configure the same personalized configuration information for some or all monitoring devices and broadcasting devices, that is, multiple monitoring devices or broadcasting devices can share the same personalized configuration information.

[0049] The mixing console receives and stores each personalized configuration information, and establishes a mapping relationship between each personalized configuration information and a corresponding monitoring device or broadcasting device.

[0050] S202: The second wireless transceiver on each musical instrument sends a first audio signal to the mixing console.

[0051] In implementation, when the performance begins, the audio collectors on musical instrument 1, musical instrument 2 and musical instrument 3 collect the first audio signal generated by the performance of the musical instrument in real time, and send it to the mixing console through the second wireless transceiver.

[0052] In a specific example, the first audio signal x1(n) played by musical instrument 1, the first audio signal x2(n) played by musical instrument 2, and the first audio signal x3(n) played by musical instrument 3 are acquired in real time by corresponding audio collectors and encoded to obtain corresponding encoded audio signals: y1(n), y2(n), y3(n). The audio signals y1(n), y2(n), y3(n) are sent to the mixing console via the second wireless transceivers on musical instrument 1, musical instrument 2, and musical instrument 3, respectively.

[0053] S203: A first mixer on the mixing console performs mixing processing on each first audio signal based on each personalized configuration information to obtain each second audio signal.

[0054] In implementation, the first mixer performs mixing processing on each first audio signal based on different personalized configuration information to obtain different second audio signals.

[0055] In a specific example, the first mixer receives the encoded audio signals y1'(n), y2'(n), y3'(n) through the first wireless transceiver, and decodes the received audio signals to obtain decoded audio signals: z1(n), z2(n), z3(n), and then performs different mixing processes on z1(n), z2(n), z3(n) based on the personalized configuration information to obtain corresponding second audio signals v1(n), v2(n), v3(n). Each monitoring device (taking monitoring device 1 and monitoring device 2 as examples in the figure) and the broadcasting device corresponds to a second audio signal, and each second audio signal may be the same or different.

[0056] Please refer to Figure 3 In the embodiment provided in the present application, the method for obtaining the second audio signal by mixing each first audio signal based on the personalized configuration information by the mixing console includes the following steps.

[0057] S301, obtaining personalized configuration information corresponding to a monitoring device.

[0058] In an implementation, the first mixer may read the personalized configuration information from a local storage.

[0059] S302: extract target mixing modes and their corresponding mixing parameters from the personalized configuration information.

[0060] S303: Perform mixing processing on each first audio signal according to a target mixing mode and corresponding mixing parameters to obtain a second audio signal.

[0061] Among them, the calling console can provide different mixing modes for users to choose from, and can provide a parameter editing interface according to the mixing mode selected by the user, so that the user can further personalize the specific parameters used in the mixing mode to meet the different needs of monitoring or broadcasting.

[0062] In one implementation, when the target mixing mode is mono mixing, the first mixer may calculate the mixing parameters corresponding to each first audio signal and each first audio signal based on the first preset algorithm to obtain the second audio signal. In a specific example, the first mixer performs mono mixing on the first audio signals z1(n), z2(n), and z3(n) to obtain the second audio signal A(n), and the calculation formula is as follows: A(n)= a1z1(n)+ a2z2(n)+a3z3(n) Among them, a1, a2, and a3 are mixing coefficients corresponding to the signals of each channel.

[0063] In one implementation, when the target mixing mode is stereo mixing, the first mixer first determines the target mixing parameters based on time information and mixing parameters, wherein the time information may be the time of the current mixing console system or the acquisition time of the first audio signal. In one example, the first mixer may select mixing parameters corresponding to the time information from the mixing parameters based on the time information as the target mixing parameters. In another example, the first mixer may calculate the mixing parameters based on the time information and the parameter change rule to obtain the target mixing parameters, wherein the parameter change rule includes but is not limited to determining the current performance stage based on the current time information, and then determining the parameter adjustment coefficient according to the current performance stage, and calculating part or all of the mixing parameters based on the parameter adjustment coefficient to obtain the target mixing parameters. Based on this, the mixing parameters can be changed with the performance stage to meet the monitoring requirements of different performance stages. For example, different parameter adjustment coefficients are set in the solo, ensemble, concerto and duet stages of the symphony performance to obtain different target mixing parameters, thereby meeting the monitoring or broadcasting requirements in the corresponding stages.

[0064] Then, the first mixer may calculate the target mixing parameters corresponding to each first audio signal and each first audio signal based on the second preset algorithm to obtain a left channel mixing signal AL(n) and a right channel mixing signal AR(n) respectively, and determine them as the second audio signals.

[0065] In one example, the target mixing parameters include a left channel mixing coefficient and a right channel mixing coefficient, and the calculation formula is as follows: AL(n)= al1z1(n)+al2z2(n)+al3z3(n) AR(n)= ar1z1(n)+ar2z2(n)+ar3z3(n) Among them, al1, al2, and al3 are the left channel mixing coefficients corresponding to the first audio signals of each channel; ar1, ar2, and ar3 are the right channel mixing coefficients corresponding to the first audio signals of each channel.

[0066] When the target mixing mode is panoramic sound mixing, please refer to Figure 4 The method for the first mixer to perform panoramic sound mixing processing on each first audio signal includes the following steps.

[0067] S401, determining a virtual position origin and a target HRTF function from mixing parameters.

[0068] The virtual position origin is the calculation origin of each monitoring virtual position. In implementation, the center of the stage can be used as the virtual position origin or a certain monitoring virtual position can be used as the virtual position origin. The monitoring virtual position is the virtual position of the listener, and the listener can be a musical instrument player, a field staff, etc. The virtual position of the listener is the position of the listener at the scene, and the virtual position of the musical instrument player can be the position of the instrument being played, or the position of the monitoring device being worn.

[0069] The target HRTF function is used to calculate the left and right ear HRTF coefficients. Different listeners have different corresponding HRTF functions. During the mixing process, a unified HRTF function can be configured as the target HRTF function, or the HRTF function corresponding to a certain listener can be selected as the target HRTF function.

[0070] It is understandable that the user can set this information through the mixing console and record it in the mixing parameters of the personalized configuration information, and the first mixer can directly read this information from the mixing parameters.

[0071] S402: Determine corresponding left-ear HRTF coefficients and right-ear HRTF coefficients according to the target HRTF function.

[0072] In one example, when a unified HRTF function is selected as the target HRTF function, the left ear HRTF coefficient of the listener of instrument 1 is HRTF1_L1 and the right ear HRTF coefficient is HRTF1_R1, that is, the HRTF coefficients corresponding to the first audio signal z1(n) are HRTF1_L1 and HRTF1_R1.

[0073] In another example, when the target HRTF function is the HRTF function corresponding to a certain listener 2, the HRTF coefficients corresponding to the first audio signal z1(n) are HRTF2_L1 and HRTF2_R1.

[0074] S403: Determine the value of the channel mixing parameter based on the virtual position origin.

[0075] The channel mixing parameters include left channel mixing parameters and right channel mixing parameters. Each first audio signal corresponds to a left channel mixing parameter and a right channel mixing parameter.

[0076] In one example, when the virtual position origin is the center of the stage, the left and right channel mixing parameter values ​​may be set to 1.

[0077] In another example, when the virtual position origin is a certain monitoring virtual position, left and right channel mixing parameters corresponding to each first audio signal are obtained.

[0078] S404: Calculate the left ear HRTF coefficient, the right ear HRTF coefficient, the channel mixing parameter, and each first audio signal based on a third preset algorithm to obtain a corresponding left ear panoramic sound audio signal and a right ear panoramic sound audio signal as a second audio signal.

[0079] In one example, the center of the stage is used as the virtual position origin and a unified HRTF function is used as the target function. The left ear HRTF coefficient corresponding to the virtual listening position of the listener of instrument 1 is HRTF_L1(m), and the right ear HRTF coefficient is HRTF_R1(m). The left ear HRTF coefficient corresponding to the virtual listening position of the listener of instrument 2 is HRTF_L2(m), and the right ear HRTF coefficient is HRTF_R2(m). The left ear HRTF coefficient corresponding to the virtual listening position of the listener of instrument 3 is HRTF_L3(m), and the right ear HRTF coefficient is HRTF_R3(m). The first mixer calculates the above parameters based on the third preset algorithm to obtain the left ear panoramic sound audio signal AL3D(n) and the left ear panoramic sound audio signal AR3D(n). The specific calculation formula example is as follows: AL3D(n)=HRTF_L1(m)z1(n)+HRTF_L2(m)z2(n)+HRTF_L3(m)z3(n) AR3D(n)=HRTF_R1(m)z1(n)+HRTF_R2(m)z2(n)+HRTF_R3(m)z3(n) In another example, each listener may also use his own monitoring virtual position as the virtual position origin and his own HRTF function as the target function to perform mixing processing.

[0080] For example, the second audio signal received by the monitoring headphones worn by the listener of the instrument 1 is recorded as A1L3D(n) and A1R3D(n), and the calculation formulas of A1L3D(n) and A1R3D(n) are as follows: A1L3D(n)=al1HRTF1_L1(m)z1(n)+al2HRTF1_L2(m)z2(n)+al3HRTF1_L3(m)z3(n) A1R3D(n)=ar1HRTF1_R1(m)z1(n)+ar2HRTF1_R2(m)z2(n)+ar3HRTF1_R3(m)z3(n) Among them, HRTF1_L1(m) is the left ear HRTF coefficient corresponding to the first audio signal z1(n), HRTF1_L1(m) is the right ear HRTF coefficient corresponding to the first audio signal z1(n); HRTF1_L2(m) is the left ear HRTF coefficient corresponding to the first audio signal z2(n), HRTF1_L2(m) is the right ear HRTF coefficient corresponding to the first audio signal z2(n); HRTF1_L3(m) is the left ear HRTF coefficient corresponding to the first audio signal z3(n), HRTF1_L3(m) is the right ear HRTF coefficient corresponding to the first audio signal z3(n); al1(n), al2(n), al3(n) are the left channel mixing coefficients corresponding to each audio signal; ar1(n), ar2(n), ar3(n) are the right channel mixing coefficients corresponding to each audio signal.

[0081] Correspondingly, the second audio signal received by the monitoring headphones worn by the listener of the instrument 2 is recorded as A2L3D(n) and A2R3D(n). The calculation formulas of A2L3D(n) and A2R3D(n) are as follows: A2L3D(n)=al1HRTF2_L1(m)z1(n)+al2HRTF2_L2(m)z2(n) +al3HRTF2_L3(m)z3(n) A2R3D(n)=ar1HRTF2_R1(m)z1(n)+ar2HRTF2_R2(m)z2(n) +ar3HRTF2_R3(m)z3(n) Among them, HRTF2_L1(m) is the left ear HRTF coefficient corresponding to the first audio signal z1(n), HRTF2_L1(m) is the right ear HRTF coefficient corresponding to the first audio signal z1(n); HRTF2_L2(m) is the left ear HRTF coefficient corresponding to the first audio signal z2(n), HRTF2_L2(m) is the right ear HRTF coefficient corresponding to the first audio signal z2(n); HRTF2_L3(m) is the left ear HRTF coefficient corresponding to the first audio signal z3(n), HRTF2_L3(m) is the right ear HRTF coefficient corresponding to the first audio signal z3(n); al1(n), al2(n), al3(n) are the left channel mixing coefficients corresponding to each audio signal; ar1(n), ar2(n), ar3(n) are the right channel mixing coefficients corresponding to each audio signal.

[0082] The mixed monitoring signals received by the monitoring headphones worn by the monitor of the instrument 3 are recorded as A3L3D(n) and A3R3D(n). The calculation formulas of A3L3D(n) and A3R3D(n) are as follows: A3L3D(n)=al1HRTF3_L1(m)z1(n)+al2HRTF3_L2(m)z2(n) +al3HRTF3_L3(m)z3(n) A3R3D(n)=ar1HRTF3_R1(m)z1(n)+ar2HRTF3_R2(m)z2(n) +ar3HRTF3_R3(m)z3(n) Among them, HRTF3_L1(m) is the left ear HRTF coefficient corresponding to the first audio signal z1(n), HRTF3_L1(m) is the right ear HRTF coefficient corresponding to the first audio signal z1(n); HRTF3_L2(m) is the left ear HRTF coefficient corresponding to the first audio signal z2(n), HRTF3_L2(m) is the right ear HRTF coefficient corresponding to the first audio signal z2(n); HRTF3_L3(m) is the left ear HRTF coefficient corresponding to the first audio signal z3(n), HRTF3_L3(m) is the right ear HRTF coefficient corresponding to the first audio signal z3(n); al1(n), al2(n), al3(n) are the left channel mixing coefficients corresponding to each audio signal; ar1(n), ar2(n), ar3(n) are the right channel mixing coefficients corresponding to each audio signal.

[0083] It can be seen from this that users can set the virtual position origin and target HRTF function of different listeners through the mixing console, which can provide customized monitoring experience for different listeners, thereby meeting different monitoring needs and providing different monitoring experiences.

[0084] In one example, when setting mixing parameters through the mixing console, the user can also configure the proportions of different instruments in different mixed channels at different times, and the first mixer can introduce corresponding proportion coefficients for calculation during the calculation process.

[0085] When the target mixing mode is surround sound mixing, the surround sound type is determined from the mixing parameters, and each first audio signal is processed according to the surround processing method corresponding to the surround sound type to obtain a second audio signal. The surround sound type includes but is not limited to 5.1 channels, 7.1.4 channels, and 22.2 channels.

[0086] It is worth noting that in some embodiments of the present application, the first mixer may further perform sound effect processing on the second audio signal based on the sound effect control parameters in the personalized configuration information to obtain a sixth audio signal. The sound effect processing includes but is not limited to restorative sound effect processing based on Dolby noise reduction or BBE technology, and modified sound effect processing based on SRS technology.

[0087] In one example, when the first mixer determines that other virtual performers need to be added according to the personalized configuration information, the specific position of the virtual performers in the live performance can be obtained from the sound effect control parameters, and the audio signals of the virtual performers are mixed with the audio signals of the live performers (i.e., the first audio signals) through mixing processing. The specific mixing processing method can refer to the above mixing processing method and will not be repeated here.

[0088] In one example, when the first mixer determines that background sound needs to be added based on personalized configuration information, it can obtain the locally stored recorded background sound and mix the background sound with each first audio signal to obtain a second audio signal. The mixing method can refer to the above method and will not be repeated here.

[0089] In this way, by further processing the audio signal with sound effects, the monitoring experience can be improved. At the same time, by adding virtual performers or background sounds for mixing, the performance content can be enriched and a diversified display of the performance content can be achieved.

[0090] S204: The first wireless transceiver on the mixing console sends a corresponding second audio signal to each monitoring device and each broadcasting device respectively.

[0091] In implementation, the mixing console can determine that the receiving devices of each second audio signal v1(n), v2(n), v3(n) are monitoring device 1, monitoring device 2 and broadcasting device respectively according to the mapping relationship between personalized configuration information and devices, and send each second audio signal v1(n), v2(n), v3(n) to the corresponding monitoring device and broadcasting device through the first wireless transceiver.

[0092] The monitoring device and the broadcasting device receive the second audio signal and play it.

[0093] Please refer to Figure 5 In another specific application scenario, the musical instrument can perform secondary mixing processing on the second audio signal and then send it to the corresponding monitoring device. Accordingly, the interaction of each component specifically includes the following steps.

[0094] S501, the audio console sends personalized configuration information to the mixing console.

[0095] S502: The second wireless transceiver on each musical instrument sends a first audio signal to the mixing console.

[0096] S503: A first mixer on the mixing console performs mixing processing on each first audio signal based on each personalized configuration information to obtain each second audio signal.

[0097] Among them, steps S501 to S503 may correspond to the description of steps S201 to S203 and will not be repeated here.

[0098] S504, the mixing console determines an associated musical instrument of the monitoring device or the broadcasting device, and sends a corresponding second audio signal to a second wireless transceiver of the associated musical instrument.

[0099] In implementation, both the monitoring device and the broadcasting device can be connected to the musical instrument by wire or wirelessly, so as to establish an association relationship and send it to the mixing console. In this embodiment, the monitoring device 1 is associated with the musical instrument 3, and the broadcasting device is associated with the musical instrument 2. Based on this, when the mixing console determines that the second audio signals v1(n) and v3(n) need to be remixed according to the personalized configuration information, the musical instrument that receives the second audio signal can be determined according to the association relationship, so as to send the second audio signals v1(n) and v3(n) to the musical instrument 3 and the musical instrument 2 respectively.

[0100] It can be understood that this step also includes sending the second audio signal v2 (n) directly to the monitoring device 2 .

[0101] S505, the second mixer of the associated musical instrument performs secondary mixing processing on the received second audio signal to obtain a third audio signal, and sends it to the corresponding monitoring device and broadcasting device.

[0102] Please refer to Figure 6 In implementation, the method of performing secondary mixing processing on the second audio signal by the second mixer on the associated musical instrument includes the following steps.

[0103] S601: Acquire a fifth audio signal from local storage.

[0104] The fifth audio signal is the first audio signal stored locally by the audio collector or an audio signal with the same collection time that is extracted from a locally recorded audio signal according to the collection time of the first audio signal.

[0105] Accordingly, in one example, the method for obtaining the fifth audio signal includes reading the latest first audio signal from the local storage area as the fifth audio signal or, based on the signal identifier carried in the second audio signal, querying the first audio signal with the same signal identifier from the local storage area as the fifth audio signal. In another example, the method for obtaining the fifth audio signal includes, based on the acquisition time of the first audio signal used when generating the second audio signal, extracting a segment with the same acquisition time from the locally recorded complete performance audio signal as the fifth audio signal.

[0106] S602: Perform alignment processing on the fifth audio signal based on the second audio signal.

[0107] In implementation, the fifth audio signal may be time-aligned according to the playback time corresponding to the second audio signal, so that the processed fifth audio signal can be time-synchronized with the second audio signal.

[0108] S603: Calculate the processed fifth audio signal, the second audio signal, and the secondary mixing coefficient based on a sixth preset algorithm to obtain a third audio signal.

[0109] In implementation, the secondary mixing can play a role in highlighting the solo audio signal of the associated instrument, and the second mixer can mix the solo audio signal of the instrument (ie, the fifth audio signal) with the second audio signal.

[0110] Taking musical instrument 3 as an example, the third audio signal received by the monitoring headphones (i.e., monitoring device 1) worn by the listener of musical instrument 3 is recorded as A1L3D(n) and A1R3D(n), the solo audio signal of musical instrument 3 is recorded as solo3(n), and the signals after secondary mixing are recorded as A3L3D'(n) and A3R3D'(n). Among them, A3L3D'(n) and A3R3D'(n) can be calculated based on the following formula: A3L3D'(n)=A3L3D(n)+WLsolo3(n) A3R3D'(n)=A3R3D(n)+WRsolo3(n) Among them, WL is the left ear secondary mixing ratio coefficient, WR is the right ear secondary mixing ratio coefficient, WL<=1, WR<=1.

[0111] In this embodiment, the third audio signals obtained by mixing the musical instrument 2 and the musical instrument 3 are respectively recorded as v2'(n) and v3'(n), as shown in the figure.

[0112] Based on the above method, the secondary mixing of the second audio signal is achieved through the musical instrument, which not only allows the listener to monitor the solo audio of the musical instrument more clearly, but also allows the listener to operate the controller by himself by setting a controller on the musical instrument or the monitoring device, and choose to turn on or off the enhanced monitoring of the solo audio signal.

[0113] In other embodiments of the present application, audio signals from other performance sites may be accessed through the network and mixed together to achieve cross-site rehearsal or ensemble.

[0114] Specifically, the musical instrument performance interaction system also includes a remote access subsystem 16, which can be deployed on a cloud server to achieve remote access to a remote performance site.

[0115] Please refer to Figure 7 In one example, the remote access subsystem 16 is used to receive a fourth audio signal from another performance mixing console 71 or a musical instrument 72 and send it to the mixing console 11, wherein the musical instrument 72 can be a musical instrument at another performance site or a separately connected musical instrument.

[0116] The mixing console 11 can mix the first audio signal and the fourth audio signal or mix the second audio signal and the fourth audio signal based on the personalized configuration information corresponding to the monitoring device 73 and the broadcasting device 74 in the local performance scene and other performance scenes to obtain each fifth audio signal, and send the corresponding fifth audio signal to the monitoring device 73, the broadcasting device 74 or the musical instrument 72 in other performance scenes through the remote access subsystem 16. The fourth audio signal can be generated by mixing the audio signals of each musical instrument by the mixing console 71 in other performance scenes, or can be the audio signal of each musical instrument.

[0117] Based on this, it can meet the needs of cross-site ensembles, and at the same time meet the different monitoring needs of each on-site monitor.

[0118] In other embodiments of the present application, the musical instrument playing interaction system also includes an ultrasonic receiver, and multiple monitoring devices or broadcasting devices are installed with ultrasonic transmitters, wherein the system assigns a different ultrasonic frequency range to each monitoring device, so that the ultrasonic receiver can uniquely determine the sending device based on the frequency range of the received ultrasonic wave.

[0119] During the performance, when each monitoring device receives the audio signal sent by the musical instrument or the mixing console and plays it, it synchronously emits an ultrasonic signal in the corresponding frequency range; the ultrasonic receiver receives the ultrasonic signal sent by each monitoring device in real time, and determines whether the audio signal played by each monitoring device can be synchronously listened to by each listener within the target monitoring range according to the receiving time and frequency range of each ultrasonic signal; if it cannot be listened to synchronously, the receiving time is sent to the mixing console, so that the mixing console can adjust the order of sending the second audio signal to each monitoring device based on the receiving time.

[0120] In one implementation, the ultrasonic monitoring device can determine whether each listener within the target monitoring range can listen to the audio signal synchronously based on whether the reception time of each ultrasonic signal is the same. If the reception time is the same, it is determined that synchronous listening is possible, otherwise, synchronous listening is not possible.

[0121] In another implementation, the installation position of the ultrasonic receiver can be determined first, and then the distance from the ultrasonic receiver to the origin of the target monitoring range can be calculated, and then the reception time difference between each ultrasonic signal can be determined. Then, when determining whether synchronous reception is possible, it can be determined based on whether the reception time difference requirements are met.

[0122] When the ultrasonic receiver determines that it is impossible to listen synchronously, the reception time can be sent to the mixing console.

[0123] The mixing console can adjust the order of sending the second audio signal to each monitoring device based on each receiving time. In one example, the mixing console can first determine the receiving time with the largest number of the same receiving times as the reference value, and then calculate the difference between the other receiving times and the reference value, and then determine the order adjustment plan for sending the second audio signal to each monitoring device. The adjustment plan includes but is not limited to advancing the signal sending order of the monitoring device corresponding to the receiving time with a positive difference, and delaying the signal sending order of the monitoring device corresponding to the receiving time with a negative difference, and determining the sending interval according to the size of the difference. Based on this, the reception synchronization optimization of the target monitoring area can be achieved, thereby ensuring that the listeners in the target monitoring area can listen synchronously, thereby avoiding the situation where the listeners cannot listen synchronously due to problems with network transmission or monitoring equipment, thereby affecting the performance.

[0124] Among them, the target monitoring range can be selected and determined according to the performance site, and then the installation location of the ultrasonic monitor can be determined. In one example, assuming that the minimum time interval between two identical sounds that the human ear can distinguish is 0.1 seconds, the ultrasonic receiver can be used as the origin and 34 meters as the diameter, and the determined range is the target monitoring range. It is understandable that due to interference from on-site environmental factors, the target monitoring range may change accordingly. In addition, in a scenario where an ultrasonic receiver is not specifically installed within a target monitoring range, the ultrasonic receiver can be adjusted to adapt to the actual scenario by determining the synchronous listening method.

[0125] It is understandable that the synchronous reception processing of the third audio signal and the second audio signal can be implemented based on the above method. In addition, the above synchronization method can also be implemented based on a broadcasting device, thereby ensuring that listeners within the target monitoring range can obtain the best experience.

[0126] The ultrasonic receiver can be installed in the main monitoring area of ​​the performance site, such as the VIP audience area, the center of the stage, or the virtual position origin, or any other convenient location for installation.

[0127] Based on the above technical solution, the ultrasonic receiver determines whether the listeners or listeners within the target monitoring range can synchronously hear the audio played by each monitoring device and broadcasting device based on the reception time of each ultrasonic signal, thereby optimizing the way the audio signal is sent for the mixing console to avoid listening asynchrony caused by network or device differences, thereby ensuring the listening experience.

[0128] In addition, an embodiment of the present application also provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the embodiments of the present application; wherein the processor may adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, for executing relevant programs to implement the method in any one of the implementations in the embodiments of the present application.

[0129] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of the embodiments of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.

[0130] The above-mentioned processor can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined and executed.

[0131] The software module may be located in a random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines its hardware to complete the functions required to be performed by the units included in the data processing device of the embodiment of the present application, or executes the method in any one of the implementation modes in the embodiment of the present application.

[0132] Those skilled in the art can understand that all or part of the steps in the above-mentioned implementation methods can be completed by instructing related hardware through a program, and the program is stored in a storage medium, including several instructions for making a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) perform all or part of the steps of each implementation method of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0133] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A musical instrument performance interactive system, characterized in that: The system includes a mixing console, at least one musical instrument and at least one monitoring device, wherein the mixing console includes a first wireless transceiver and a first mixer, and the musical instrument includes a second wireless transceiver and an audio collector; wherein, The audio collector of each musical instrument is used to collect a first audio signal of the musical instrument, and send the first audio signal to the first wireless transceiver via the second wireless transceiver; The first wireless transceiver sends each of the first audio signals to the first mixer; The first mixer receives each of the first audio signals, and mixes each of the first audio signals based on the personalized configuration information corresponding to each of the monitoring devices to obtain at least one second audio signal, and sends each of the second audio signals to the corresponding monitoring device or the second wireless transceiver through the first wireless transceiver.

2. The system according to claim 1, characterized in that The musical instrument also includes a second mixer, which is used to perform secondary mixing processing on the second audio signal to obtain a third audio signal and send it to the monitoring device.

3. The system according to claim 2, characterized in that The monitoring device includes a wireless receiving interface, and the wireless receiving interface is used to receive the second audio signal sent by the first wireless transceiver or the third audio signal sent by the second wireless transceiver.

4. The system according to claim 1, characterized in that The system further comprises a sound mixer, which is used to generate the personalized configuration information and send it to the mixing console.

5. The system according to claim 1, characterized in that The system also includes at least one broadcasting device; the mixing console is also used to mix each of the first audio signals based on the personalized configuration information corresponding to each of the broadcasting devices to obtain a corresponding second audio signal, and send it to the broadcasting device via the first wireless transceiver.

6. The system according to claim 1, characterized in that The system also includes a remote access subsystem, which is used to receive a fourth audio signal from other performance sites and send it to the mixing console. The mixing console is used to mix the first audio signal and the fourth audio signal based on the personalized configuration information corresponding to the monitoring device of the other performance site to obtain the fifth audio signal, and send the fifth audio signal to the monitoring device of the other performance site through the remote access subsystem.

7. The system according to claim 1, characterized in that The monitoring device includes an ultrasonic transmitter, which is used to transmit an ultrasonic signal while the monitoring device plays the second audio signal; the system also includes an ultrasonic receiver, which is used to receive the ultrasonic signals transmitted by each of the monitoring devices, and determine whether the second audio signals played by each of the monitoring devices within the target monitoring range are listened to synchronously according to whether the reception time of each of the ultrasonic signals is consistent; if not, the reception time is sent to the mixing console, so that the mixing console adjusts the order of sending the second audio signals to each of the monitoring devices based on the reception time.

8. An audio signal processing method, characterized in that: The method is applied to the musical instrument performance interactive system according to any one of claims 1 to 6, wherein the first mixer performs mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the monitoring devices to obtain at least one second audio signal including: Obtaining personalized configuration information corresponding to the monitoring device; Extracting target mixing modes and their corresponding mixing parameters respectively from the personalized configuration information; When the target mixing mode is mono mixing, the mixing parameters corresponding to the first audio signals and the first audio signals are calculated based on a first preset algorithm to obtain the second audio signals; or When the target mixing mode is stereo mixing, the target mixing parameters are determined based on the time information and the mixing parameters; the target mixing parameters corresponding to each of the first audio signals and each of the first audio signals are calculated based on a second preset algorithm to obtain a left channel mixing signal and a right channel mixing signal, respectively, and determined as the second audio signal; wherein, determining the target mixing parameters based on the time information and the mixing parameters includes selecting a mixing parameter corresponding to the time information from the mixing parameters based on the time information as the target mixing parameter or calculating the mixing parameters based on the time information and the parameter change rule to obtain the target mixing parameter; or, When the target mixing mode is panoramic sound mixing, a virtual position origin and a target HRTF function are determined from the mixing parameters, and a left ear HRTF coefficient and a right ear HRTF coefficient corresponding to each of the first audio signals are determined according to the target HRTF function, and a value of a channel mixing parameter is determined based on the virtual position origin, and the left ear HRTF coefficient, the right ear HRTF coefficient, the channel mixing parameters and each of the first audio signals are calculated based on a third preset algorithm to obtain a corresponding left ear panoramic sound audio signal and a right ear panoramic sound audio signal as the second audio signal; or When the target mixing mode is surround sound mixing, the surround sound type is determined from the mixing parameters, and each of the first audio signals is processed according to a surround processing method corresponding to the surround type to obtain the second audio signal.

9. The method according to claim 8, characterized in that The method further includes performing sound effect processing on the second audio signal based on the personalized configuration information to obtain a sixth audio signal and send the sixth audio signal to a corresponding monitoring device and a broadcasting device.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the method according to claim 8 or 9 when executed by the processor.

Images