Musical instrument playing interaction system, audio signal processing method and electronic device

Abstract

The application relates to the technical field of audio signal processing, and provides a musical instrument performance interaction system, an audio signal processing method and electronic equipment. The system comprises a mixing console, musical instruments and listening devices. The mixing console comprises a first wireless transceiver and a first mixer, and the musical instruments comprise second wireless transceivers and audio collectors. The audio collector of each musical instrument is used for collecting a first audio signal of the musical instrument, and sending the first audio signal to the first wireless transceiver through the second wireless transceiver. The first wireless transceiver sends each first audio signal to the first mixer. The first mixer receives each first audio signal, and performs mixing processing on each first audio signal based on individualized configuration information corresponding to each listening device, so as to obtain at least one second audio signal. Each second audio signal is sent to the corresponding listening device or the second wireless transceiver through the first wireless transceiver, and the flexibility of stage layout of a music performance site is improved.

Description

Technical Field

[0001] This application relates to the field of audio signal processing technology, and in particular to an interactive system for playing musical instruments, an audio signal processing method, and an electronic device. Background Technology

[0002] In traditional music performances, the connection between multiple instruments and mixing equipment usually relies on wired transmission. While this method is stable and reliable, it increases the complexity of stage setup, reduces venue construction efficiency, and restricts the freedom of movement for musicians. Especially in large-scale performances or frequent venue changes, wiring and dismantling are time-consuming and labor-intensive, and are easily affected by on-site conditions, leading to connection failures that seriously affect the performance quality. It is evident that wired transmission has limitations in terms of stage setup flexibility, venue adaptability, and the management of large orchestras. Summary of the Invention

[0003] To enhance the flexibility of stage setup for live music performances, improve the adaptability of instruments and mixing equipment to different venues, and reduce the difficulty of managing large bands, this application provides an interactive instrument performance system. The system includes a mixing console, at least one instrument, and at least one monitoring device. The mixing console includes a first wireless transceiver and a first mixer. The instrument includes a second wireless transceiver and an audio acquisition device. The audio acquisition device of each instrument acquires a first audio signal for that instrument and transmits the first audio signal to the first wireless transceiver via the second wireless transceiver. The first wireless transceiver transmits each of the first audio signals to the first mixer. The first mixer receives each of the first audio signals and performs mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each monitoring device, obtaining at least one second audio signal. The second audio signal is then transmitted to the corresponding monitoring device or the second wireless transceiver via the first wireless transceiver.

[0004] Based on the above technical solution, the instruments and the mixing console can exchange audio data wirelessly, thus avoiding the wiring and equipment management issues associated with wired connections. This improves the flexibility of stage setup and the site adaptability of related equipment, while also reducing the difficulty of band equipment management and increasing the freedom of movement during performances. Furthermore, the mixing console can perform personalized mixing processing on the audio data based on the individual configuration information of each monitoring device, thereby meeting the monitoring needs of different listeners. This is particularly suitable for application scenarios in large-scale performances where different performers have different audio monitoring requirements.

[0005] In one embodiment, the musical instrument further includes a second mixer for performing secondary mixing processing on the second audio signal to obtain a third audio signal, which is then sent to the monitoring device.

[0006] Based on the above technical solution, using musical instruments to perform secondary mixing of the second audio signal can, on the one hand, reduce the data processing burden on the mixing console, and on the other hand, allow performers to make personalized mixing adjustments by setting up controllers at the instrument or monitoring device end. This enables performers to flexibly adjust their monitoring needs according to the situation.

[0007] In one implementation, the monitoring device includes a wireless receiving interface for receiving the second audio signal transmitted by the first wireless transceiver or the third audio signal transmitted by the second wireless transceiver.

[0008] In one implementation, the system further includes a mixing console for generating the personalized configuration information and sending it to the mixing console.

[0009] In one implementation, the system further includes at least one broadcasting device; the mixer is also used to perform mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the broadcasting devices to obtain a corresponding second audio signal, and to send it to the broadcasting device through the first wireless transceiver.

[0010] In one implementation, the system further includes a remote access subsystem, which is used to receive a fourth audio signal from other performance venues and send it to the mixing console. The mixing console is used to mix each of the first audio signal and the fourth audio signal based on the personalized configuration information corresponding to the monitoring equipment at the other performance venues to obtain the fifth audio signal, and then send the fifth audio signal to the monitoring equipment at the other performance venues through the remote access subsystem.

[0011] Based on the above technical solution, remote audio signal access can be achieved, thereby enabling ensemble collaboration between different performance venues through mixing processing by a mixing console.

[0012] In one implementation, the monitoring device includes an ultrasonic transmitter for emitting ultrasonic signals while the monitoring device is playing the second audio signal; the system also includes an ultrasonic receiver for receiving ultrasonic signals emitted by each of the monitoring devices, and determining whether the second audio signals played by each of the monitoring devices within the target monitoring range are synchronized based on whether the reception times of each ultrasonic signal are consistent. If they are not synchronized, the reception times are sent to the mixing console, so that the mixing console adjusts the order in which the second audio signals are sent to each of the monitoring devices based on the reception times.

[0013] Based on the above technical solution, it is possible to monitor whether the listener can hear synchronously in real time during the performance and promptly report any abnormalities to the mixing console, so that the mixing console can adjust the signal transmission order according to each reception time to ensure the listening experience.

[0014] This application embodiment also provides an audio signal processing method. The method applies the above-mentioned musical instrument performance interaction system. The method involves the first mixer performing mixing processing on each first audio signal based on personalized configuration information corresponding to each of the monitoring devices to obtain at least one second audio signal. This includes: acquiring personalized configuration information corresponding to the monitoring devices; extracting target mixing modes and their corresponding mixing parameters from the personalized configuration information; when the target mixing mode is mono mixing, calculating the mixing parameters corresponding to each of the first audio signals and each of the first audio signals based on a first preset algorithm to obtain the second audio signal; or, when the target mixing mode is stereo mixing, determining target mixing parameters based on the time information and the mixing parameters; calculating the target mixing parameters corresponding to each of the first audio signals and each of the first audio signals based on a second preset algorithm to obtain a left channel mixing signal and a right channel mixing signal, which are then determined as the second audio signal; wherein, determining the target mixing parameters based on the time information and the mixing parameters includes... The timing information is used to select the mixing parameters corresponding to the timing information from the mixing parameters, which are then used as the target mixing parameters. Alternatively, the mixing parameters are calculated based on the timing information and parameter change rules to obtain the target mixing parameters. Or, when the target mixing mode is immersive sound mixing, the virtual position origin and target HRTF function are determined from the mixing parameters, and the left ear HRTF coefficient and right ear HRTF coefficient corresponding to each of the first audio signals are determined according to the target HRTF function. The channel mixing parameters are determined based on the virtual position origin, and the left ear HRTF coefficient, the right ear HRTF coefficient, the channel mixing parameters, and each of the first audio signals are calculated based on a third preset algorithm to obtain the corresponding left ear immersive sound audio signal and right ear immersive sound audio signal, which are then used as the second audio signal. Or, when the target mixing mode is surround sound mixing, the surround sound type is determined from the mixing parameters, and each of the first audio signals is processed according to the surround sound processing method corresponding to the surround sound type to obtain the second audio signal.

[0015] Based on the above technical solution, the mixing console can perform corresponding mixing processing according to the mixing mode and mixing parameters selected by the user for different monitoring devices.

[0016] In one implementation, the method further includes performing sound effect processing on the second audio signal based on the personalized configuration information to obtain a sixth audio signal and sending it to the corresponding monitoring device and broadcasting device.

[0017] Based on the above technical solutions, the mixing console can further realize personalized sound effects processing during the mixing process, thereby enhancing the richness of the performance content and creating a better atmosphere for the performance.

[0018] Furthermore, this application also provides an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the above-described method. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A A schematic diagram of the structure of a musical instrument playing interactive system provided in an embodiment of this application is shown.

[0022] Figure 1B A schematic diagram of the structure of a musical instrument playing interactive system provided in another embodiment of this application is shown.

[0023] Figure 2 A flowchart illustrating an embodiment of the audio signal processing method provided in this application is shown.

[0024] Figure 3 A flowchart illustrating the audio mixing method in the embodiments of this application is shown.

[0025] Figure 4 A flowchart illustrating the panoramic sound mixing processing method in the embodiments of this application is shown.

[0026] Figure 5 A flowchart illustrating an audio signal processing method provided in another embodiment of this application is shown.

[0027] Figure 6 A flowchart illustrating the secondary mixing processing method in the embodiments of this application is shown.

[0028] Figure 7 A schematic diagram of the structure of a musical instrument playing interactive system provided in another embodiment of this application is shown. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, and "first", "second" and various numerical designations are only for the convenience of description and are not used to limit the scope of the embodiments of this application.

[0031] The features, structures, or characteristics in this application can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence number of each process does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0032] Some optional features in the embodiments of this application can be implemented independently without relying on other features in certain scenarios to solve the corresponding technical problems and achieve the corresponding effects. They can also be combined with other features according to needs in certain scenarios.

[0033] In this application, unless otherwise specified, the same or similar parts between the various embodiments can be referred to each other. In the various embodiments of this application, unless otherwise specified or there is a logical conflict, the terminology and / or descriptions between different embodiments are consistent and can be mutually referenced. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0034] The embodiments of this application will be described in detail below with reference to the figures.

[0035] This application provides an interactive instrument performance system applicable to professional concerts, outdoor music festivals, remote music recording collaborations, and other performance or recording rehearsal settings. By employing a wireless method to achieve data interaction between the instrument and the mixing console, it avoids the problems associated with wired connections, improves stage setup flexibility and venue adaptability, and facilitates the management of large orchestras. Furthermore, for multi-instrument ensemble applications, it can also achieve differentiated real-time mixing processing of audio signals, adapting to the personalized monitoring needs of each performer during their performance.

[0036] Please refer to Figure 1AThe musical instrument performance interactive system provided in this application includes a mixer 11, at least one musical instrument 12, at least one monitoring device 13, and a mixing console 14. Each musical instrument 12, each monitoring device 13, and the mixing console 14 can be wirelessly connected to the mixer 11.

[0037] Specifically, the mixing console 11 includes a first wireless transceiver and a first mixer, wherein the first wireless transceiver is used to receive and transmit wireless signals; and the first mixer is used to mix the audio signals of each instrument.

[0038] The musical instrument 12 includes an instrument body, a second wireless transceiver, and an audio acquisition unit. The instrument body is the structural component for performance and has an electrical interface. The second wireless transceiver is used for receiving and transmitting wireless signals. The audio acquisition unit is used to acquire the instrument's performance audio signals. In a specific example, the instrument body can be a piano, guitar, bass, or electronic keyboard. The second wireless transceiver and audio acquisition unit can be externally connected to or internally integrated into the instrument body. In implementation, the second wireless transceiver and audio acquisition unit can be implemented based on the same hardware structure, or they can be configured as independent but interconnected hardware structures.

[0039] In some embodiments of this application, the instrument 12 further includes a second mixer, which is mounted on the instrument body and is communicatively connected to an audio acquisition unit and a second wireless transceiver, respectively, for secondary mixing of audio signals.

[0040] The monitoring device 13 includes a wireless receiving interface or a wired receiving interface. The monitoring device 13 can receive wireless audio signals transmitted from a first wireless transceiver and a second wireless transceiver via the wireless receiving interface, and can connect to the signal cable of an instrument via the wired receiving interface to receive wired audio signals. In one example, the monitoring device 13 is a monitoring headset.

[0041] The mixing console 14 can be a software program running on terminal devices such as computers, mobile phones, and tablets. The mixing console 11 is connected to the device equipped with the mixing console program via wired or wireless means. The mixing console 14 is used to control the mixing mode of the mixing console 11 for multiple signals.

[0042] Specifically, the mixing console 14 can generate personalized configuration information for each monitoring device 13 based on user operations and send it to the mixing console 11, enabling the mixing console 11 to perform personalized processing on the audio signals sent to each monitoring device 13 based on the personalized configuration information. The personalized configuration information for each monitoring device 13 can be the same or different, depending on the performer's actual monitoring needs.

[0043] In one example, personalized configuration information may include mixing mode and mixing parameters; in another example, personalized configuration information may further include pre-mixing processing method or post-mixing processing method.

[0044] In some embodiments of this application, the personalized configuration information may also include sound effect control parameters. The mixer can further perform personalized sound effect processing on the audio signal obtained after mixing based on the sound effect control parameters. The sound effect control parameters in different personalized configuration information can be the same or different, depending on the monitoring requirements. In one example, personalized sound effect processing includes, but is not limited to, adding other virtual performers or background sounds.

[0045] Please refer to Figure 1B In some embodiments of this application, the musical instrument performance interactive system may further include at least one broadcasting device 15, which can be connected to the mixing console 15 via wired or wireless means to play received audio signals. Similarly, users can also set personalized configuration information for each broadcasting device 15 based on the mixing console to meet different playback needs.

[0046] Please refer to Figure 2 In a specific application example, the interaction between the components of the musical instrument playing interactive system during the audio signal processing process includes the following steps.

[0047] S201, the mixing console sends personalized configuration information to the mixer.

[0048] In practice, users can set personalized configuration information for each monitoring device and each broadcasting device via the mixing console before the performance begins, and then send it to the mixing console. It is understood that users can configure different personalized configuration information for each monitoring device and each broadcasting device separately, or they can configure the same personalized configuration information for some or all monitoring devices and broadcasting devices; that is, multiple monitoring devices or broadcasting devices can share the same personalized configuration information.

[0049] The mixer receives and saves each personalized configuration information and establishes a mapping relationship between each personalized configuration information and the corresponding monitoring or broadcasting devices.

[0050] S202, the second wireless transceiver on each instrument sends the first audio signal to the mixer.

[0051] In practice, when the performance begins, the audio acquisition devices on instruments 1, 2 and 3 acquire the first audio signal generated by the instrument in real time and send it to the mixing console through the second wireless transceiver.

[0052] In a specific example, the first audio signal x1(n) played by instrument 1, the first audio signal x2(n) played by instrument 2, and the first audio signal x3(n) played by instrument 3 are acquired in real time by the corresponding audio acquisition devices and encoded to obtain the corresponding encoded audio signals y1(n), y2(n), and y3(n). The audio signals y1(n), y2(n), and y3(n) are sent to the mixing console through the second wireless transceivers on instrument 1, instrument 2, and instrument 3, respectively.

[0053] S203, the first mixer on the mixing console performs mixing processing on each first audio signal based on each personalized configuration information to obtain each second audio signal.

[0054] In practice, the first mixer mixes each first audio signal based on different personalized configuration information to obtain different second audio signals.

[0055] In a specific example, the first mixer receives encoded audio signals y1'(n), y2'(n), and y3'(n) via a first wireless transceiver, and decodes these signals to obtain decoded audio signals z1(n), z2(n), and z3(n). Based on personalized configuration information, z1(n), z2(n), and z3(n) undergo different mixing processes to obtain corresponding second audio signals v1(n), v2(n), and v3(n). Each monitoring device (using monitoring device 1 and monitoring device 2 as examples in the diagram) and broadcasting device corresponds to a second audio signal, and these second audio signals can be the same or different.

[0056] Please refer to Figure 3 In the embodiments provided in this application, the method of mixing each first audio signal based on personalized configuration information to obtain a second audio signal by a mixing console includes the following steps.

[0057] S301, obtain the personalized configuration information corresponding to the monitoring device.

[0058] In practice, the first mixer can read personalized configuration information from local storage.

[0059] S302 extracts the target mixing mode and its corresponding mixing parameters from the personalized configuration information.

[0060] S303, perform mixing processing on each first audio signal according to the target mixing mode and the corresponding mixing parameters to obtain the second audio signal.

[0061] The call console offers different mixing modes for users to choose from, and provides a parameter editing interface based on the selected mixing mode, allowing users to further personalize the specific parameters used in the mixing mode to meet different monitoring or broadcasting needs.

[0062] In one implementation, when the target mixing mode is mono mixing, the first mixer can calculate the mixing parameters corresponding to each first audio signal and each first audio signal based on a first preset algorithm to obtain the second audio signal. In a specific example, after the first mixer performs mono mixing on the first audio signals z1(n), z2(n), and z3(n), it obtains the second audio signal A(n), and the calculation formula is as follows: A(n)= a1z1(n)+ a2z2(n)+a3z3(n) Where a1, a2, and a3 are the mixing coefficients corresponding to each signal.

[0063] In one implementation, when the target mixing mode is stereo mixing, the first mixer first determines the target mixing parameters based on time information and mixing parameters. The time information can be the current time of the mixing console system or the acquisition time of the first audio signal. In one example, the first mixer can select the mixing parameters corresponding to the time information from the mixing parameters as the target mixing parameters. In another example, the first mixer can calculate the mixing parameters based on time information and parameter change rules to obtain the target mixing parameters. The parameter change rules include, but are not limited to, determining the current performance stage based on the current time information, determining parameter adjustment coefficients based on the current performance stage, and calculating some or all mixing parameters based on the parameter adjustment coefficients to obtain the target mixing parameters. Based on this, the mixing parameters can change with the performance stage to meet the monitoring needs of different performance stages. For example, different parameter adjustment coefficients can be set for solo, ensemble, concerto, and chamber music stages in a symphony performance to obtain different target mixing parameters, thereby meeting the monitoring or broadcasting needs of the corresponding stages.

[0064] Then, the first mixer can calculate the target mixing parameters corresponding to each first audio signal and each first audio signal based on the second preset algorithm, and obtain the left channel mixing signal AL(n) and the right channel mixing signal AR(n) respectively, which are determined as the second audio signal.

[0065] In one example, the target mixing parameters include the left channel mixing coefficient and the right channel mixing coefficient, calculated as follows: AL(n)= al1z1(n)+al2z2(n)+al3z3(n) AR(n)= ar1z1(n)+ar2z2(n)+ar3z3(n) Where al1, al2, and al3 are the left channel mixing coefficients corresponding to each first audio signal; ar1, ar2, and ar3 are the right channel mixing coefficients corresponding to each first audio signal.

[0066] When the target mix mode is Dolby Atmos mix, please refer to... Figure 4 The method for the first mixer to perform panoramic mixing processing on each first audio signal includes the following steps.

[0067] S401 determines the virtual position origin and target HRTF function from the mixing parameters.

[0068] The virtual position origin is the calculation origin for each virtual monitoring position. In implementation, the stage center can be used as the virtual position origin, or a specific virtual monitoring position can be used as the virtual position origin. The virtual monitoring position refers to the virtual position of the monitor, which can be a musician, on-site staff, etc. The virtual position of the monitor is the monitor's location on-site; for a musician, this could be the position where they are playing their instrument or the position of their monitoring equipment.

[0069] The target HRTF function is used to calculate the HRTF coefficients for the left and right ears. Different listeners have different HRTF functions. During the mixing process, a uniform HRTF function can be configured as the target HRTF function, or the HRTF function corresponding to a specific listener can be selected as the target HRTF function.

[0070] Understandably, users can set this information through the mixing console and record it in the mixing parameters of the personalized configuration information. The first mixer can directly read this information from the mixing parameters.

[0071] S402, determine the corresponding left ear HRTF coefficient and right ear HRTF coefficient based on the target HRTF function.

[0072] In one example, when a uniform HRTF function is selected as the target HRTF function, the HRTF coefficients of the left ear of the listener of instrument 1 are HRTF1_L1 and HRTF1_R1, that is, the HRTF coefficients corresponding to the first audio signal z1(n) are HRTF1_L1 and HRTF1_R1.

[0073] In another example, when the target HRTF function is the HRTF function corresponding to a certain listener 2, the HRTF coefficients corresponding to the first audio signal z1(n) are HRTF2_L1 and HRTF2_R1.

[0074] S403 determines the values ​​of channel mixing parameters based on the virtual position origin.

[0075] The channel mixing parameters include left channel mixing parameters and right channel mixing parameters. Each first audio signal corresponds to left channel mixing parameters and right channel mixing parameters.

[0076] In one example, when the virtual position origin is the center of the stage, the left and right channel mixing parameters can be set to 1.

[0077] In another example, when the virtual location origin is a certain virtual listening location, the left and right channel mixing parameters corresponding to each first audio signal are obtained.

[0078] S404, based on the third preset algorithm, calculates the left ear HRTF coefficient, right ear HRTF coefficient, channel mixing parameters and each first audio signal to obtain the corresponding left ear panoramic audio signal and right ear panoramic audio signal, which are used as the second audio signal.

[0079] In one example, using the stage center as the virtual position origin and a unified HRTF function as the objective function, the HRTF coefficients for the left ear and right ear of the listener at the virtual position of instrument 1 are HRTF_L1(m) and HRTF_R1(m), respectively; for instrument 2, the HRTF coefficients for the left ear and right ear are HRTF_L2(m) and HRTF_R2(m), respectively; and for instrument 3, the HRTF coefficients for the left ear and right ear are HRTF_L3(m) and HRTF_R3(m), respectively. The first mixer calculates the above parameters based on the third preset algorithm to obtain the left ear panoramic audio signal AL3D(n) and the left ear panoramic audio signal AR3D(n). A specific calculation formula example is shown below: AL3D(n)=HRTF_L1(m)z1(n)+HRTF_L2(m)z2(n)+HRTF_L3(m)z3(n) AR3D(n)=HRTF_R1(m)z1(n)+HRTF_R2(m)z2(n)+HRTF_R3(m)z3(n) In another example, each listener can also use its own virtual listening position as the origin of the virtual position and its own HRTF function as the target function for mixing.

[0080] For example, the second audio signal received by the monitoring headphones worn by the listener of instrument 1 is denoted as A1L3D(n) and A1R3D(n). The formulas for calculating A1L3D(n) and A1R3D(n) are as follows: A1L3D(n)=al1HRTF1_L1(m)z1(n)+al2HRTF1_L2(m)z2(n)+al3HRTF1_L3(m)z3(n) A1R3D(n)=ar1HRTF1_R1(m)z1(n)+ar2HRTF1_R2(m)z2(n)+ar3HRTF1_R3(m)z3(n) Wherein, HRTF1_L1(m) is the left ear HRTF coefficient corresponding to the first audio signal z1(n), and HRTF1_L1(m) is the right ear HRTF coefficient corresponding to the first audio signal z1(n); HRTF1_L2(m) is the left ear HRTF coefficient corresponding to the first audio signal z2(n), and HRTF1_L2(m) is the right ear HRTF coefficient corresponding to the first audio signal z2(n); HRTF1_L3(m) is the left ear HRTF coefficient corresponding to the first audio signal z3(n), and HRTF1_L3(m) is the right ear HRTF coefficient corresponding to the first audio signal z3(n); al1(n), al2(n), and al3(n) are the left channel mixing coefficients corresponding to each audio signal; ar1(n), ar2(n), and ar3(n) are the right channel mixing coefficients corresponding to each audio signal.

[0081] Correspondingly, the second audio signal received by the monitoring headphones worn by the listener of instrument 2 is denoted as A2L3D(n) and A2R3D(n). The formulas for calculating A2L3D(n) and A2R3D(n) are as follows: A2L3D(n)=al1HRTF2_L1(m)z1(n)+al2HRTF2_L2(m)z2(n) +al3HRTF2_L3(m)z3(n) A2R3D(n)=ar1HRTF2_R1(m)z1(n)+ar2HRTF2_R2(m)z2(n) +ar3HRTF2_R3(m)z3(n) Wherein, HRTF2_L1(m) is the left ear HRTF coefficient corresponding to the first audio signal z1(n), and HRTF2_L1(m) is the right ear HRTF coefficient corresponding to the first audio signal z1(n); HRTF2_L2(m) is the left ear HRTF coefficient corresponding to the first audio signal z2(n), and HRTF2_L2(m) is the right ear HRTF coefficient corresponding to the first audio signal z2(n); HRTF2_L3(m) is the left ear HRTF coefficient corresponding to the first audio signal z3(n), and HRTF2_L3(m) is the right ear HRTF coefficient corresponding to the first audio signal z3(n); al1(n), al2(n), and al3(n) are the left channel mixing coefficients corresponding to each audio signal; ar1(n), ar2(n), and ar3(n) are the right channel mixing coefficients corresponding to each audio signal.

[0082] The mixed monitoring signals received by the monitoring headphones worn by the monitor of instrument 3 are denoted as A3L3D(n) and A3R3D(n). The formulas for calculating A3L3D(n) and A3R3D(n) are as follows: A3L3D(n)=al1HRTF3_L1(m)z1(n)+al2HRTF3_L2(m)z2(n) +al3HRTF3_L3(m)z3(n) A3R3D(n)=ar1HRTF3_R1(m)z1(n)+ar2HRTF3_R2(m)z2(n) +ar3HRTF3_R3(m)z3(n) Wherein, HRTF3_L1(m) is the left ear HRTF coefficient corresponding to the first audio signal z1(n), and HRTF3_L1(m) is the right ear HRTF coefficient corresponding to the first audio signal z1(n); HRTF3_L2(m) is the left ear HRTF coefficient corresponding to the first audio signal z2(n), and HRTF3_L2(m) is the right ear HRTF coefficient corresponding to the first audio signal z2(n); HRTF3_L3(m) is the left ear HRTF coefficient corresponding to the first audio signal z3(n), and HRTF3_L3(m) is the right ear HRTF coefficient corresponding to the first audio signal z3(n); al1(n), al2(n), and al3(n) are the left channel mixing coefficients corresponding to each audio signal; ar1(n), ar2(n), and ar3(n) are the right channel mixing coefficients corresponding to each audio signal.

[0083] Therefore, users can set the virtual position origin and target HRTF function for different listeners through the mixing console, which can provide a customized listening experience for different listeners, thereby meeting different listening needs and providing different listening experiences.

[0084] In one example, when setting mixing parameters through the mixing console, the user can also configure the proportion of different instruments in different mixed channels at different times, and the first mixer can introduce the corresponding scaling factor for calculation during the calculation process.

[0085] When the target mixing mode is surround sound mixing, the surround sound type is determined from the mixing parameters, and each first audio signal is processed according to the surround sound processing method corresponding to the surround sound type to obtain the second audio signal. The surround sound type includes, but is not limited to, 5.1 channel, 7.1.4 channel, and 22.2 channel.

[0086] It is worth noting that, in some embodiments of this application, the first mixer may also perform audio effect processing on the second audio signal based on the audio effect control parameters in the personalized configuration information to obtain a sixth audio signal. The audio effect processing includes, but is not limited to, restorative audio effect processing based on Dolby noise reduction or BBE technology, and decorative audio effect processing based on SRS technology.

[0087] In one example, when the first mixer determines that other virtual performers need to be added based on personalized configuration information, it can obtain the specific location of the virtual performers in the live performance from the sound effect control parameters, and mix the audio signals of the virtual performers with the audio signals of the live performers (i.e., each first audio signal) through mixing processing. The specific mixing processing method is the same as the mixing processing method described above, and will not be repeated here.

[0088] In one example, when the first mixer determines that background sound needs to be added based on the personalized configuration information, it can obtain the locally stored background sound that has been recorded and mix the background sound with each of the first audio signals to obtain the second audio signal. The mixing method can refer to the above method and will not be repeated here.

[0089] Thus, by further processing the audio signal with sound effects, the listening experience can be improved. At the same time, by adding virtual performers or background sounds for mixing, the performance content can be enriched, and a more diverse presentation of the performance content can be achieved.

[0090] S204, the first wireless transceiver on the mixer sends the corresponding second audio signal to each monitoring device and each broadcasting device.

[0091] In implementation, the mixing console can determine the receiving devices for each second audio signal v1(n), v2(n), and v3(n) according to the mapping relationship between personalized configuration information and devices, namely monitoring device 1, monitoring device 2, and broadcast device, and send each second audio signal v1(n), v2(n), and v3(n) to the corresponding monitoring device and broadcast device through the first wireless transceiver.

[0092] The monitoring and broadcasting equipment receives the second audio signal and plays it.

[0093] Please refer to Figure 5 In another specific application scenario, the musical instrument can perform secondary mixing processing on the second audio signal and then send it to the corresponding monitoring device. Accordingly, the interaction of each component includes the following steps.

[0094] S501: The mixing console sends personalized configuration information to the mixer.

[0095] S502, the second wireless transceiver on each instrument sends the first audio signal to the mixer.

[0096] S503, the first mixer on the mixing console performs mixing processing on each first audio signal based on each personalized configuration information to obtain each second audio signal.

[0097] The steps S501 to S503 can be referred to in the description of steps S201 to S203, and will not be repeated here.

[0098] S504, the mixer identifies the associated instrument of the monitoring or broadcasting equipment and sends the corresponding second audio signal to the second wireless transceiver of the associated instrument.

[0099] In implementation, both the monitoring device and the broadcasting device can be connected to the musical instruments via wired or wireless means to establish an association and send the signals to the mixing console. In this embodiment, the monitoring device 1 is associated with musical instrument 3, and the broadcasting device is associated with musical instrument 2. Based on this, when the mixing console determines, according to the personalized configuration information, that secondary mixing processing of the second audio signals v1(n) and v3(n) is required, it can determine the musical instrument receiving the second audio signals based on this association, and thus send the second audio signals v1(n) and v3(n) to musical instrument 3 and musical instrument 2 respectively.

[0100] Understandably, this step also includes sending the second audio signal v2(n) directly to the monitoring device 2.

[0101] S505, the second mixer of the associated instrument performs secondary mixing processing on the received second audio signal to obtain a third audio signal, and sends it to the corresponding monitoring and broadcasting equipment.

[0102] Please refer to Figure 6 In practice, the method for secondary mixing of the second audio signal by the second mixer on the associated instrument includes the following steps.

[0103] S601 retrieves the fifth audio signal from local storage.

[0104] The fifth audio signal is either the first audio signal stored locally by the audio acquisition device or an audio signal extracted from the locally recorded audio signal based on the acquisition time of the first audio signal, which is consistent with the acquisition time.

[0105] Accordingly, in one example, obtaining the fifth audio signal includes reading the latest first audio signal from the local storage area and using it as the fifth audio signal, or retrieving a first audio signal with the same signal identifier from the local storage area based on the signal identifier carried in the second audio signal and using it as the fifth audio signal. In another example, obtaining the fifth audio signal includes extracting a segment from the locally recorded complete performance audio signal that matches the acquisition time of the first audio signal used when generating the second audio signal and using it as the fifth audio signal.

[0106] S602, the fifth audio signal is aligned based on the second audio signal.

[0107] In practice, the fifth audio signal can be time-aligned according to the playback time corresponding to the second audio signal, so that the processed fifth audio signal can be time-synchronized with the second audio signal.

[0108] S603 calculates the processed fifth audio signal, second audio signal, and secondary mixing coefficients based on the sixth preset algorithm to obtain the third audio signal.

[0109] In practice, secondary mixing can highlight the solo audio signal of the associated instrument. The second mixer can mix the solo audio signal (i.e., the fifth audio signal) of this instrument with the second audio signal.

[0110] Taking instrument 3 as an example, the third audio signal received by the monitoring headphones (i.e., monitoring device 1) worn by the listener of instrument 3 is denoted as A1L3D(n) and A1R3D(n), the solo audio signal of instrument 3 is denoted as solo3(n), and the signal after secondary mixing is denoted as A3L3D'(n) and A3R3D'(n). A3L3D'(n) and A3R3D'(n) can be calculated based on the following formulas: A3L3D'(n) = A3L3D(n) + WLsolo3(n) A3R3D'(n) = A3R3D(n) + WRsolo3(n) Where WL is the left ear secondary mixing ratio coefficient, WR is the right ear secondary mixing ratio coefficient, WL<=1, WR<=1.

[0111] In this embodiment, the third audio signal obtained after mixing instrument 2 and instrument 3 is denoted as v2'(n) and v3'(n), respectively, as shown in the figure.

[0112] Based on the above method, the secondary mixing of the second audio signal is achieved through an instrument. This not only allows the listener to listen more clearly to the instrument's solo audio, but also allows the listener to operate the controller on the instrument or monitoring device to choose to turn the enhanced monitoring of the solo audio signal on or off.

[0113] In other embodiments of this application, audio signals from other performance venues can be accessed via the network and mixed together to achieve cross-venue rehearsals or ensemble performances.

[0114] Specifically, the instrument performance interaction system also includes a remote access subsystem 16, which can be deployed on a cloud server to enable remote access to performance venues in different locations.

[0115] Please refer to Figure 7 In one example, the remote access subsystem 16 is used to receive a fourth audio signal from a mixer 71 or an instrument 72 at another performance venue and send it to the mixer 11. The instrument 72 can be a performance instrument at another performance venue or a separately accessed instrument.

[0116] The mixing console 11 can mix the first and fourth audio signals, or the second and fourth audio signals, based on the personalized configuration information corresponding to the monitoring equipment 73 and broadcasting equipment 74 in this performance venue and other performance venues, to obtain the fifth audio signals. The corresponding fifth audio signals are then sent to the monitoring equipment 73, broadcasting equipment 74, or musical instruments 72 in other performance venues via the remote access subsystem 16. The fourth audio signals can be generated by mixing the audio signals of each instrument using the mixing console 71 in other performance venues, or they can be the audio signals of each instrument themselves.

[0117] Based on this, it can meet the needs of cross-site ensemble performances, and at the same time meet the different listening needs of listeners at each site.

[0118] In other embodiments of this application, the musical instrument playing interactive system further includes an ultrasonic receiver, and multiple monitoring devices or broadcasting devices are equipped with ultrasonic transmitters, wherein the system assigns different ultrasonic frequency ranges to each monitoring device, so that the ultrasonic receiver can uniquely determine the transmitting device based on the frequency range of the received ultrasonic waves.

[0119] During the performance, each monitoring device simultaneously emits ultrasonic signals within the corresponding frequency range when it receives and plays the audio signals sent by the instrument or mixer. The ultrasonic receiver receives the ultrasonic signals sent by each monitoring device in real time and determines whether the audio signals played by each monitoring device can be heard synchronously by each listener within the target monitoring range based on the reception time and frequency range of each ultrasonic signal. If they cannot be heard synchronously, the reception time is sent to the mixer so that the mixer can adjust the order in which the second audio signals are sent to each monitoring device based on the reception time.

[0120] In one implementation, the ultrasonic monitoring device can determine whether each listener within the target monitoring range can synchronously hear the audio signal based on whether the reception time of each ultrasonic signal is the same. If the reception time is the same, it is determined that they can listen synchronously; otherwise, they cannot listen synchronously.

[0121] In another implementation, the installation location of the ultrasonic receiver can be determined first, and then the distance from the ultrasonic receiver to the origin of the target listening range can be calculated. The receiving time difference between each ultrasonic signal can then be determined. When determining whether synchronous reception is possible, the receiving time difference requirement can be used as a basis for determination.

[0122] When the ultrasonic receiver determines that it cannot receive the audio synchronously, it can send each reception time to the mixer.

[0123] The mixer can adjust the order in which the second audio signal is sent to each monitoring device based on the received time. In one example, the mixer can first determine the received time with the most identical frequencies as a baseline value, then calculate the difference between the other received times and the baseline value, and thus determine the adjustment scheme for the order of sending the second audio signal to each monitoring device. The adjustment scheme includes, but is not limited to, advancing the signal transmission order of monitoring devices corresponding to received times with positive differences and delaying the signal transmission order of monitoring devices corresponding to received times with negative differences, and determining the transmission interval based on the magnitude of the differences. Based on this, the reception synchronization of the target monitoring area can be optimized, thereby ensuring that the listeners in the target monitoring area can listen synchronously, thus avoiding situations where the listeners cannot listen synchronously due to network transmission or monitoring equipment problems, which would affect the performance.

[0124] The target monitoring range can be selected and determined based on the performance venue, thereby determining the installation location of the ultrasonic receiver. In one example, assuming the minimum time interval between two identical sounds that the human ear can distinguish is 0.1 seconds, the target monitoring range can be defined as the ultrasonic receiver as the origin and 34 meters as the diameter. It is understandable that the target monitoring range may change due to environmental factors. Furthermore, in scenarios where no specific ultrasonic receiver is installed within the target monitoring range, the method by which the ultrasonic receiver determines synchronous listening can be adjusted to adapt to the actual scenario.

[0125] It is understandable that the synchronous reception and processing of both the third and second audio signals can be implemented based on the above method. Furthermore, the above synchronization method can also be implemented using broadcast equipment, thereby ensuring that listeners within the target monitoring range receive the best possible experience.

[0126] The ultrasonic receiver can be installed in the main monitoring area of ​​the performance venue, such as the VIP audience area, the center of the stage, or the virtual location origin, or any location that is convenient for installation.

[0127] Based on the above technical solution, the ultrasonic receiver determines whether the listeners or listeners within the target monitoring range can synchronously hear the audio played by each monitoring device and broadcasting device based on the reception time of each ultrasonic signal. This optimizes the way the mixing console sends audio signals, avoiding asynchronous listening due to network or device differences, and thus ensuring a good listening experience.

[0128] Furthermore, this application also provides an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the method in any of the implementations in this application. The processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the method in any of the implementations in this application.

[0129] The processor can also be an integrated circuit electronic device with signal processing capabilities. In implementation, each step of the method in any of the embodiments of this application can be completed by the integrated logic circuitry in the processor's hardware or by software instructions.

[0130] The aforementioned processor can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.

[0131] The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory. The processor reads information from the memory and, in conjunction with its hardware, performs the functions required by the units included in the data processing apparatus of this application embodiment, or executes the methods in any implementation of this application embodiment.

[0132] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0133] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A musical instrument playing interactive system, characterized in that, The system includes a mixing console, at least one musical instrument, and at least one monitoring device. The mixing console includes a first wireless transceiver and a first mixer, and the musical instrument includes a second wireless transceiver and an audio acquisition device. The audio acquisition unit of each instrument is used to acquire the first audio signal of the instrument and transmit the first audio signal to the first wireless transceiver through the second wireless transceiver; The first wireless transceiver sends each of the first audio signals to the first mixer; The first mixer receives each of the first audio signals and performs mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the monitoring devices to obtain at least one second audio signal. The first wireless transceiver then transmits each of the second audio signals to the corresponding monitoring device or the second wireless transceiver. The system also includes a mixing console, which is used to generate the personalized configuration information and send it to the mixing console; The first mixer performs mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the monitoring devices, to obtain at least one second audio signal, including: Obtain the personalized configuration information corresponding to the monitoring device; The target mixing mode and its corresponding mixing parameters are extracted from the personalized configuration information. When the target mixing mode is mono mixing, the mixing parameters corresponding to each first audio signal and each first audio signal are calculated based on the first preset algorithm to obtain the second audio signal; When the target mixing mode is stereo mixing, the target mixing parameters are determined based on time information and the mixing parameters. The time information is the current mixing console system time or the acquisition time of the first audio signal. The target mixing parameters corresponding to each of the first audio signals and each of the first audio signals are calculated based on a second preset algorithm to obtain the left channel mixing signal and the right channel mixing signal, which are then determined as the second audio signal. Determining the target mixing parameters based on time information and the mixing parameters includes selecting the mixing parameter corresponding to the time information from the mixing parameters based on the time information, and using it as the target mixing parameter; or calculating the mixing parameters based on the time information and parameter change rules to obtain the target mixing parameter. The parameter change rules include determining the current performance stage based on the current time information, determining parameter adjustment coefficients based on the current performance stage, and calculating the mixing parameters based on the parameter adjustment coefficients. The monitoring device includes an ultrasonic transmitter, which is used to emit ultrasonic signals while the monitoring device is playing the second audio signal; the system also includes an ultrasonic receiver, which is used to receive the ultrasonic signals emitted by each of the monitoring devices, and to determine whether the second audio signals played by each of the monitoring devices within the target monitoring range are listened to synchronously based on whether the reception times of each ultrasonic signal are consistent. If they are not synchronized, the reception times are sent to the mixing console, so that the mixing console adjusts the order in which the second audio signals are sent to each of the monitoring devices based on the reception times.

2. The system according to claim 1, characterized in that, The instrument also includes a second mixer, which is used to perform secondary mixing processing on the second audio signal to obtain a third audio signal, and send it to the monitoring device.

3. The system according to claim 2, characterized in that, The monitoring device includes a wireless receiving interface, which is used to receive the second audio signal sent by the first wireless transceiver or the third audio signal sent by the second wireless transceiver.

4. The system according to claim 1, characterized in that, The system further includes at least one broadcasting device; the mixer is also used to perform mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the broadcasting devices to obtain the corresponding second audio signal, and send it to the broadcasting device through the first wireless transceiver.

5. The system according to claim 1, characterized in that, The system further includes a remote access subsystem, which is used to receive a fourth audio signal from other performance venues and send it to the mixing console. The mixing console is used to mix each of the first audio signal and the fourth audio signal based on the personalized configuration information corresponding to the monitoring equipment at the other performance venues to obtain a fifth audio signal, and then send the fifth audio signal to the monitoring equipment at the other performance venues through the remote access subsystem.

6. An audio signal processing method, characterized in that, The method is applied to the musical instrument performance interactive system according to any one of claims 1 to 5, wherein the first mixer performs mixing processing on each of the first audio signals based on the personalized configuration information corresponding to each of the monitoring devices to obtain at least one second audio signal, including: Obtain the personalized configuration information corresponding to the monitoring device; The target mixing mode and its corresponding mixing parameters are extracted from the personalized configuration information. When the target mixing mode is mono mixing, the mixing parameters corresponding to each first audio signal and each first audio signal are calculated based on the first preset algorithm to obtain the second audio signal; When the target mixing mode is stereo mixing, the target mixing parameters are determined based on time information and the mixing parameters. The time information is the current mixing console system time or the acquisition time of the first audio signal. The target mixing parameters corresponding to each of the first audio signals and each of the first audio signals are calculated based on a second preset algorithm to obtain the left channel mixing signal and the right channel mixing signal, which are then determined as the second audio signal. Determining the target mixing parameters based on time information and the mixing parameters includes selecting the mixing parameter corresponding to the time information from the mixing parameters based on the time information, and using it as the target mixing parameter; or calculating the mixing parameters based on the time information and parameter change rules to obtain the target mixing parameter. The parameter change rules include determining the current performance stage based on the current time information, determining parameter adjustment coefficients based on the current performance stage, and calculating the mixing parameters based on the parameter adjustment coefficients. When the target mixing mode is immersive mixing, the virtual position origin and the target HRTF function are determined from the mixing parameters, and the left ear HRTF coefficient and right ear HRTF coefficient corresponding to each of the first audio signals are determined according to the target HRTF function. The values ​​of the channel mixing parameters are determined based on the virtual position origin. The left ear HRTF coefficient, the right ear HRTF coefficient, the channel mixing parameters and each of the first audio signals are calculated based on the third preset algorithm to obtain the corresponding left ear immersive audio signal and right ear immersive audio signal, which are used as the second audio signal. When the target mixing mode is surround sound mixing, the surround sound type is determined from the mixing parameters, and each of the first audio signals is processed according to the surround sound processing method corresponding to the surround sound type to obtain the second audio signal.

7. The method according to claim 6, characterized in that, The method further includes performing sound effect processing on the second audio signal based on the personalized configuration information to obtain a sixth audio signal and sending it to the corresponding monitoring device and broadcasting device.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as described in claim 6 or 7.

Images