Sound mixing method and device, computer device and sound mixing processing system

Abstract

The application relates to a mixing method and device, computer equipment, a storage medium and a computer program product, and a mixing processing system. The method comprises the following steps: acquiring a plurality of audio data streams in a multi-audio convergence scene; according to scene identifiers carried by the audio data streams, audio data streams with the same scene identifier are introduced into the same buffer queue; determining the audio energy of each audio data stream in each buffer queue, wherein the audio energy is used to represent the total amplitude of pulse code modulation of the audio data stream; from each buffer queue, a target audio data stream with audio energy satisfying an energy screening condition is screened out; and mixing processing is performed on the target audio data streams with the same scene identifier, so that the output audio of the multi-audio convergence scene is obtained, thereby effectively reducing the participation amount of audio data in the mixing process, and the media data transmission quality is ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a mixing method, apparatus, computer equipment, and mixing processing system. Background Technology

[0002] With the development of network technology, network-based audio and video technologies have also continuously evolved. For example, remote conferencing products rely on audio and video technologies to enable meetings across different regions. During a meeting, multiple audio streams are typically transmitted over the network via software applications or hardware terminals to a media server for processing.

[0003] Currently, for network-based audio mixing, a long-lived connection is typically maintained between remote terminals and a media server. All remote terminals upload audio data streams to the media server in real time via this connection. The media server calculates the energy of each terminal's audio data stream in real time, then selects the few with higher energy streams for mixing processing. The final mixing result is then synchronized to all conference terminals. However, this mixing method requires all audio data to be transmitted directly to the media server over the network. This results in high bandwidth consumption on the media server's network card, impacting the quality of other media data transmissions on the media server. Summary of the Invention

[0004] Therefore, it is necessary to provide a mixing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can guarantee the quality of media data transmission in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a mixing method. The method includes:

[0006] Acquire multiple audio data streams in multi-audio convergence scenarios;

[0007] According to the scene identifier carried by the audio data stream, audio data streams with the same scene identifier are imported into the same buffer queue;

[0008] Determine the audio energy of each audio data stream in each buffer queue, the audio energy being used to characterize the total amplitude of the pulse code modulation of the audio data stream;

[0009] From each of the buffer queues, select the target audio data stream whose audio energy meets the energy filtering conditions;

[0010] The target audio data streams with the same scene identifier are mixed to obtain the output audio of the multi-audio merging scene.

[0011] Secondly, this application also provides a mixing device. The device includes:

[0012] The audio data acquisition module is used to acquire multiple audio data streams in multi-audio convergence scenarios;

[0013] The audio data import module is used to import audio data streams with the same scene identifier into the same buffer queue according to the scene identifier carried by the audio data stream;

[0014] An audio energy identification module is used to determine the audio energy of each audio data stream in each buffer queue, wherein the audio energy is used to characterize the total amplitude of the pulse code modulation of the audio data stream;

[0015] The target audio data identification module is used to filter out target audio data streams whose audio energy meets the energy filtering conditions from each of the buffer queues;

[0016] The audio mixing module is used to mix the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

[0017] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0018] Acquire multiple audio data streams in multi-audio convergence scenarios;

[0019] According to the scene identifier carried by the audio data stream, audio data streams with the same scene identifier are imported into the same buffer queue;

[0020] Determine the audio energy of each audio data stream in each buffer queue, the audio energy being used to characterize the total amplitude of the pulse code modulation of the audio data stream;

[0021] From each of the buffer queues, select the target audio data stream whose audio energy meets the energy filtering conditions;

[0022] The target audio data streams with the same scene identifier are mixed to obtain the output audio of the multi-audio merging scene.

[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0024] Acquire multiple audio data streams in multi-audio convergence scenarios;

[0025] According to the scene identifier carried by the audio data stream, audio data streams with the same scene identifier are imported into the same buffer queue;

[0026] Determine the audio energy of each audio data stream in each buffer queue, the audio energy being used to characterize the total amplitude of the pulse code modulation of the audio data stream;

[0027] From each of the buffer queues, select the target audio data stream whose audio energy meets the energy filtering conditions;

[0028] The target audio data streams with the same scene identifier are mixed to obtain the output audio of the multi-audio merging scene.

[0029] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0030] Acquire multiple audio data streams in multi-audio convergence scenarios;

[0031] According to the scene identifier carried by the audio data stream, audio data streams with the same scene identifier are imported into the same buffer queue;

[0032] Determine the audio energy of each audio data stream in each buffer queue, the audio energy being used to characterize the total amplitude of the pulse code modulation of the audio data stream;

[0033] From each of the buffer queues, select the target audio data stream whose audio energy meets the energy filtering conditions;

[0034] The target audio data streams with the same scene identifier are mixed to obtain the output audio of the multi-audio merging scene.

[0035] The aforementioned mixing method, apparatus, computer equipment, storage medium, and computer program products, after acquiring multiple audio data streams in a multi-audio convergence scenario, first import audio data streams with the same scene identifier into the same buffer queue according to the scene identifier carried by the audio data streams, thereby classifying them according to the scene to which the audio data streams belong; then determine the audio energy corresponding to each audio data stream in each buffer queue; then, based on the audio energy, target audio data streams that meet the energy screening conditions and need to be mixed can be extracted from multiple audio data streams in each buffer queue, ensuring the effectiveness of the mixing process; and finally, by mixing the target audio data streams with the same scene identifier, the output audio of the multi-audio convergence scenario is obtained, thereby effectively reducing the amount of audio data involved in the mixing process and ensuring the quality of media data transmission.

[0036] Sixthly, this application also provides a mixing processing system, the system including multiple audio collectors, multiple audio filters and a mixing processor;

[0037] The audio collector is used to collect multiple audio data streams in multi-audio convergence scenarios and send the multiple audio data streams to the audio filter;

[0038] The audio filter is used to import audio data streams with the same scene identifier into the same buffer queue according to the scene identifier carried by the audio data stream, determine the audio energy of each audio data stream in each buffer queue, the audio energy is used to characterize the total amplitude of pulse code modulation of the audio data stream, and then select the target audio data stream whose audio energy meets the energy filtering conditions from each buffer queue, and send the obtained target audio data stream to the mixing processor.

[0039] The mixing processor is used to mix target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

[0040] The aforementioned audio mixing system first acquires multiple audio data streams from a multi-audio convergence scenario using an audio acquisition device. Then, it uses an audio filter to import audio data streams with the same scene identifier into the same buffer queue, thereby classifying them according to the scene to which they belong. Next, it determines the audio energy corresponding to each audio data stream in each buffer queue. Then, based on the audio energy, it extracts the target audio data streams that meet the energy filtering conditions from multiple audio data streams in each buffer queue, ensuring the effectiveness of the mixing process. Finally, the mixing processor performs mixing processing on the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio convergence scenario. This effectively reduces the amount of audio data involved in the mixing process, thereby ensuring the quality of media data transmission. Attached Figure Description

[0041] Figure 1 This is a diagram illustrating the application environment of the mixing method in one embodiment;

[0042] Figure 2 This is a flowchart illustrating a mixing method in one embodiment;

[0043] Figure 3 This is a flowchart illustrating the data alignment process steps in one embodiment;

[0044] Figure 4 This is a flowchart illustrating the mixing method in another embodiment;

[0045] Figure 5This is a diagram illustrating the conceptual idea behind audio mixing in one embodiment.

[0046] Figure 6 This is a diagram of the mixing framework for implementing mixing processing in one embodiment;

[0047] Figure 7 This is a block diagram of the audio mixing system in one embodiment;

[0048] Figure 8 This is a flowchart of the audio mixing process related to a web conferencing in one embodiment;

[0049] Figure 9 This is a structural block diagram of a mixing device in one embodiment;

[0050] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The mixing method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, multiple terminals, including terminal 102 and terminal 104, communicate with server 108 via a network and intermediate device 106. Terminal 102, terminal 104, intermediate device 106, and server 108 constitute a mixing system, which can be built based on MapReduce. A data storage system can store the data that server 108 needs to process. The data storage system can be integrated on server 108 or located in the cloud or on other servers. Terminals 102 and 104 can be hardware terminals for acquiring audio or smart terminals integrated with audio applications. When a user needs to mix audio data from multiple terminals through server 108, they can obtain the target audio data stream to be mixed through intermediate device 106, and then perform the mixing processing based on server 108. First, intermediate device 106 acquires multiple audio data streams from terminals 102 and 104 for the multi-audio merging scenario. Based on the scene identifier carried by the audio data streams, audio data streams with the same scene identifier are imported into the same buffer queue. The audio energy of each audio data stream in each buffer queue is determined; the audio energy characterizes the total amplitude of the pulse-code modulation of the audio data stream. From each buffer queue, target audio data streams whose audio energy meets the energy selection criteria are selected. Then, server 108 performs mixing processing on the target audio data streams with the same scene identifier to obtain the output audio for the multi-audio merging scenario. Terminals 102 and 104 can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Intermediate device 106 can be implemented using a smart router. Server 108 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0053] In one embodiment, such as Figure 2 As shown, a mixing method is provided, which can be applied to... Figure 1 Taking intermediate device 106 as an example, the explanation includes the following steps:

[0054] Step 201: Obtain multiple audio data streams in the multi-audio convergence scenario.

[0055] Multi-audio convergence scenarios refer to situations where multiple audio data from different sources need to be processed within a single context. Examples include web conferencing, where multiple audio streams from different sources are typically transmitted via application software or hardware terminals to a server for processing. Similarly, music production may require processing audio data from different instruments or vocals. In this case, mixing is usually performed to ensure the audio data maintains its order. Mixing is a step in audio processing that integrates multiple audio data streams into a single stereo or mono track. Audio data streams refer to the data streams obtained from a terminal via a network connection, specifically network data in the Transmission Control Protocol (TCP) format. Each audio data stream consists of multiple frames of audio data.

[0056] Specifically, when a user needs to mix multiple audio data streams in a multi-audio convergence scenario through server 108, they can first collect audio data streams from different scenarios through terminals 102 and 104. These audio data streams are then sent to intermediate device 106 for filtering. The filtered target audio data is then fed back to server 108 for mixing. Intermediate device 106 directly obtains multiple audio data streams in the multi-audio convergence scenario from terminals 102 and 104. These audio data streams may belong to the same multi-audio convergence scenario or to different scenarios. The purpose of intermediate device 106 is to filter multiple audio data streams within the same multi-audio convergence scenario. In one embodiment, the multi-audio convergence scenario in this application specifically refers to a web conferencing scenario. In a web conferencing scenario, since there are multiple different participants, and these participants all need to speak during the meeting, mixing is required during the meeting. At the start of a web conference, participants can log in to the web conferencing application using smartphones or desktop computers. During the conference, different participants can speak through their logged-in terminals, generating audio data streams corresponding to their speeches. These audio data streams are first transmitted over the network to intermediate device 106, where intermediate device 106 initiates the mixing process for these audio data streams.

[0057] Step 203: According to the scene identifier carried by the audio data stream, import the audio data streams with the same scene identifier into the same buffer queue.

[0058] The scene identifier carried by the audio data stream is used to represent the specific scene to which the audio data stream belongs. For example, for an audio data stream in a web conferencing scenario, the scene identifier can be the conference number of the web conferencing session, while for a music production scenario, the scene identifier can be the concert code. The buffer queue is a special type of linear list, characterized by allowing only deletion operations at the front of the list and insertion operations at the back. Like a stack, a queue is a linear list with restricted operations. The end where insertion operations are performed is called the tail, and the end where deletion operations are performed is called the head. In this application, a buffer queue is specifically used to store the audio data stream. Each multi-audio merging scenario corresponds to one buffer queue, which can be pre-built or generated in real time.

[0059] Specifically, audio mixing involves mixing multiple audio data streams within the same multi-audio convergence scenario. Therefore, at the intermediate device 106, after acquiring multiple audio data streams within the multi-audio convergence scenario, these audio data streams can be grouped. The grouping is based on the scenario identifier carried by the audio data streams. When audio data streams carry the same scenario identifier, it indicates that they belong to the same multi-audio convergence scenario. At this time, audio data streams belonging to the same multi-audio convergence scenario can be imported into the same buffer queue for further processing. In one embodiment, the multi-audio convergence scenario of this application is specifically a web conferencing scenario. In this case, the scenario identifier carried by the audio data stream is specifically the conference number of the current web conferencing session. When the participant's terminal records the participant's speech, it generates an audio data stream under the transmission control protocol based on the conference number of the web conferencing session. In the generated audio data stream, the header of each frame of audio data carries the conference number of the current web conferencing session. This allows the intermediate device to identify the conference number carried in the audio data stream by parsing the audio data stream, and then place each audio data stream into a buffer queue based on the conference number. When each frame of audio data is placed into the buffer queue, the intermediate device 106 first checks the existing buffer queues to determine if a buffer queue corresponding to the conference number exists. If it does not exist, a new buffer queue needs to be created, and the audio frame of data is placed into the newly created buffer queue. If it exists, the audio frame of data can be directly placed into the buffer queue corresponding to the conference number.

[0060] Step 205: Determine the audio energy of each audio data stream in each buffer queue. The audio energy is used to characterize the total amplitude of the pulse code modulation of the audio data stream.

[0061] Step 207: Select the target audio data stream from each buffer queue whose audio energy meets the energy filtering criteria.

[0062] In this application, audio energy specifically refers to the total energy carried by an audio data stream. The calculation of audio energy carried by an audio data stream is generally determined by the total amplitude of the pulse code modulation (PCM) corresponding to the audio data stream. Pulse code modulation (PCM) transforms a time-continuous, continuously valued analog signal into a time-discrete, discrete-valued digital signal for transmission in a channel. PCM involves sampling the analog signal, quantizing the sample amplitude, and encoding the signal. In this application, the intermediate device 106 specifically converts the audio data stream transmitted over the network into a PCM format, and then determines the audio energy of each audio data stream by the total amplitude of the audio data stream under the PCM format. Meeting the energy screening criteria specifically means that the intensity of the audio energy meets the energy screening conditions, such as selecting the audio data stream with the highest audio energy in the buffer queue as the target audio data stream, selecting the top three audio data streams with the highest audio energy in the buffer queue as the target audio data stream, or selecting an audio data stream with audio energy greater than a preset energy threshold as the target audio data stream.

[0063] Specifically, during audio mixing, to ensure the mixing effect, multiple audio data streams with higher energy (usually six) are typically selected for mixing. This process usually occurs in the mixing processor. Therefore, during mixing, all audio data streams collected by the terminal need to be submitted to the mixing processor for processing, resulting in the audio data uploaded to the mixing processor occupying a significant amount of bandwidth and affecting data transmission efficiency. The solution in this application, however, directly determines the audio energy of each audio data stream in each buffer queue in the intermediate device 106 by using the total amplitude of the pulse code modulation of the audio data streams. This performs a screening process, selecting only audio data streams whose audio energy meets the energy screening criteria within the same multi-audio merging scenario as the target audio data streams for mixing. In one embodiment, the multi-audio merging scenario of this application is specifically a web conferencing scenario. At this point, the multiple audio data streams specifically represent the audio data corresponding to the speeches of participants in the online meeting. The same buffer queue contains audio data streams corresponding to different participants in the same meeting. If someone speaks, the audio energy of that stream is higher and needs to be mixed. If no one speaks, the audio energy of that stream is lower, and it can be removed from the mixing process. Therefore, when mixing, for the same meeting, audio data streams with lower audio energy from participants can be removed from the buffer queue first. The basis for removing audio data streams is their audio energy level. After determining the audio energy, the target audio data stream needs to be further filtered based on the identified audio energy level. For example, if the audio energies of five audio energy streams in a buffer queue are determined to be 2dB, 3dB, 10dB, 60dB, and 5dB respectively, and the energy filtering condition is to select the audio data stream with the highest audio energy, then the target audio data stream in these five buffer queues can be determined to be the 60dB audio data stream.

[0064] Step 209: Mix the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

[0065] Specifically, target audio data streams with the same scene identifier refer to target audio data streams from multiple different intermediate devices.

[0066] Specifically, after obtaining the target audio data stream, the intermediate device 106 can directly upload the obtained target audio data stream to the server 108 for mixing processing. The server 108 can obtain multiple target audio data streams from multiple different intermediate devices. Then, it filters out the target audio data streams with the same scene identifier and mixes these target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene. In one embodiment, the multi-audio merging scene of this application is specifically a web conference scene. At this time, the filtered target audio data stream is specifically the speaking content of the conference between terminal 102 and terminal 104. The intermediate device 106 can specifically feed back the filtered speaking content to the server 108, which mixes the speaking content. The server 108 can obtain different speaking content from multiple different intermediate devices and then mix them to obtain the output audio of the multi-audio merging scene, that is, the final web conference mixing result. The audio mixing results of these web conferences will also be fed back from server 108 to terminals 102 and 104 via the network. Participants on terminals 102 and 104 can obtain the latest conference speaking content by receiving the audio mixing results.

[0067] The above-described mixing method, after acquiring multiple audio data streams in a multi-audio convergence scenario, first imports audio data streams with the same scene identifier into the same buffer queue according to the scene identifier carried by the audio data stream, thereby classifying them according to the scene to which the audio data streams belong; then determines the audio energy corresponding to each audio data stream in each buffer queue; then, based on the audio energy, the target audio data streams that meet the energy screening conditions and need to be mixed can be extracted from the multiple audio data streams in each buffer queue, ensuring the effectiveness of the mixing process; and finally, by mixing the target audio data streams with the same scene identifier, the output audio of the multi-audio convergence scenario is obtained, thereby effectively reducing the amount of audio data involved in the mixing process and ensuring the quality of media data transmission.

[0068] In one embodiment, step 205 includes: for each buffer queue, performing data alignment processing on each audio data stream in the buffer queue to obtain an audio data alignment result; and based on the audio data alignment result, calculating the pulse code modulation amplitude of the audio data stream to obtain the audio energy of the audio data stream.

[0069] Data alignment is used to compare different audio data streams using the same standard. By aligning the data formats and standards of different audio data streams, the accuracy of audio energy calculation and comparison is ensured. Pulse code modulation amplitude calculation aims to obtain the specific numerical value of the audio energy.

[0070] Specifically, when calculating audio energy, firstly, each audio data stream in the buffer queue undergoes data alignment preprocessing to ensure the accuracy of the audio energy calculation. Data alignment processing may include sequential steps such as sampling rate conversion, frame length fixing, sampling bit count fixing, and channel alignment. After data alignment, the pulse code modulation amplitude can be directly calculated from the audio data alignment result to obtain the audio energy of the audio data stream. In one embodiment, a monotonic rate scheduling algorithm can be used to calculate the pulse code modulation amplitude to obtain the audio energy of the audio data stream. In another embodiment, the method of this application is specifically used for audio mixing in a web conferencing scenario. When determining the audio energy of each audio data stream in each buffer queue, to ensure the accuracy of the comparison process between the audio data streams corresponding to the spoken content, data alignment can be directly performed on the audio data streams corresponding to these spoken content. After data alignment, a monotonic rate scheduling algorithm is then used to calculate the audio energy corresponding to each spoken content. In this embodiment, data alignment is performed first, followed by pulse code modulation amplitude calculation to obtain the audio energy of the audio data stream. This effectively ensures the accuracy of the calculated audio energy, thereby guaranteeing the accuracy of the target audio data stream participating in the mixing process.

[0071] In one embodiment, the method further includes: saving the audio energy data of each audio data stream to the energy data storage queue corresponding to the buffer queue; traversing the energy data storage queue and filtering out target audio energy that meets the energy filtering conditions.

[0072] Step 207 includes: selecting target audio data streams whose audio energy meets the energy screening conditions from the buffer queue corresponding to the energy data storage queue.

[0073] The energy data storage queue and the buffer queue have a one-to-one correspondence. The buffer queue is specifically used to store the energy data in the corresponding buffer queue. When the audio energy of an audio data stream in the buffer queue is calculated, the identification information (or index information) of the audio data stream and the audio energy value can be saved together into the energy data storage queue.

[0074] Specifically, the solution in this application uses an energy data storage queue to store audio energy data. After calculating the audio energy of an audio data stream, the identification information and audio energy value of the audio data stream are saved together in the energy data storage queue. Thus, after the audio energy calculation is completed, the energy data storage queue will store the specific audio energy value information of all audio data streams in a given scene. When it is necessary to filter out the target audio data streams whose audio energy meets the energy filtering criteria, the energy data storage queue can be directly traversed to find the target audio data streams among all audio data streams whose audio energy meets the energy filtering criteria. For example, the audio data stream with the highest audio energy can be determined as the target audio data stream by directly comparing audio energy. In one embodiment, the method of this application is specifically used for audio mixing in a network conferencing scenario. In this case, the target audio data stream is filtered through an intermediate device. The intermediate device is specifically used to implement edge computing, which can identify and parse multiple audio data streams sent within the local area network. The intermediate device specifically includes a packet engine, which can depack the received audio data streams to identify the scene identifier of the audio data streams. After identifying the scene and importing the audio data streams into various buffer queues, each data stream in the buffer queue undergoes data alignment preprocessing and audio energy calculation. During the calculation process, the grouping engine saves the calculated audio energy result in another queue corresponding to the current buffer queue, namely the energy data storage queue. After all audio energies in the buffer queues have been calculated, the energy data storage queue can be directly traversed to complete the filtering of target audio energies. In this embodiment, the audio energy data is stored through the energy data storage queue, which effectively identifies target audio data streams whose audio energy meets the energy filtering conditions from the buffer queues, ensuring the accuracy of data identification.

[0075] In one embodiment, selecting a target audio data stream whose audio energy meets the energy screening criteria from a buffer queue corresponding to the energy data storage queue includes: determining the audio data stream with the largest audio energy in the buffer queue corresponding to the energy data storage queue, and using the audio data stream with the largest audio energy as the target audio data stream matched with the target audio energy.

[0076] Specifically, when identifying the target audio data stream, the magnitude of the audio energy can be used as the identification criterion. This is because during mixing, the N audio data streams with the highest energy from multiple audio data streams are typically selected for mixing calculations. When the target audio data stream is fed back to server 108 for mixing, it also needs to be compared with other audio data streams in its corresponding multi-audio merging scenario before mixing. Therefore, the audio data stream with the highest audio energy in the buffer queue corresponding to the energy data storage queue can be used as the target audio data stream. In another embodiment, depending on the specific application scenario, multiple groups of audio data streams with relatively high audio energy can be selected as the target audio data stream, or the target audio data stream can be filtered directly by setting an audio energy threshold. In this implementation, by selecting the audio data stream with the highest audio energy as the target audio data stream, it can be effectively ensured that the identified target audio data stream can effectively participate in the mixing, thereby ensuring the effectiveness of the mixing process.

[0077] In one embodiment, such as Figure 3 As shown, each audio data stream in the buffer queue undergoes data alignment processing to obtain the audio data alignment results, including:

[0078] Step 302: Parse each audio data stream in the buffer queue into audio data in pulse modulation coding format.

[0079] Step 304: Perform sampling rate conversion processing on the audio data in pulse modulation coding format to obtain sampling rate converted data.

[0080] Step 306: Perform frame length fixing processing on the sampling rate conversion data to obtain fixed frame length data.

[0081] Step 308: Perform data sampling processing on the fixed frame length data to obtain audio sampling data.

[0082] Step 310: Perform channel number alignment processing on the audio sampling data to obtain the audio data alignment result.

[0083] Pulse code modulation (PCM) transforms a continuous-time, continuously-valued analog signal into a discrete-time, discrete-valued digital signal for transmission in a channel. PCM involves sampling the analog signal, quantizing the sample amplitude, and encoding the signal. Specifically, the scheme in this application converts continuous audio data segments into discrete digital audio signals. Sampling rate conversion is a method in digital audio processing. It can reduce noise or simplify signal processing. Specifically, sampling rate conversion can be achieved by converting a digital-to-analog signal to an analog signal, then re-sampling the analog signal using analog-to-digital conversion, or by interpolation or decimation of the input signal to achieve pure digital domain sampling rate conversion. Since different acquisition devices may capture audio data at different sampling rates, this application requires a consistent sampling rate conversion for the audio data. Fixed frame length processing compares different audio data streams under the same frame length reference, ensuring comparison accuracy. Data is transmitted over the network in frames, each composed of several parts that perform different functions. Frames are typically generated by the network driver and then sent via the network card to their destination machine, where the reverse process is performed. The receiving machine's Ethernet card captures these frames, informs the operating system of their arrival, and then stores them. In this application's scheme, a fixed frame length is required to prevent interference with audio energy calculations due to inconsistent frame lengths in the audio data stream. When fixing the frame length, the frame length setting can be referenced to pulse code modulation, standardizing the frame length of each frame of audio data in the audio data stream. The number of channels refers to the number of speakers capable of producing different sounds; channel alignment refers to standardizing the number of channels corresponding to different audio data streams. In this application's scheme, to simplify processing, multi-channel audio data can generally be directly converted to mono audio data. Furthermore, this application's scheme requires sampling the audio data stream, where the number of bits representing each sampling point is fixed to standardize the sampling data.

[0084] Specifically, before comparing audio energy, the audio data needs to be preprocessed. This preprocessing includes data parsing, data format alignment, and data standardization. Preprocessing facilitates unified comparison in subsequent processes. Data parsing is required to parse each audio data stream in the buffer queue into Pulse Coded (PCC) audio data. Specifically, data parsing refers to decompressing each frame of audio data in the received audio data stream to obtain PCC audio data. Then, sampling rate conversion is performed to improve the processing efficiency of the audio data processing. This PCC audio data is converted to the same sampling rate, which can be set to 8 kHz. Sampling rate conversion can be achieved by converting the audio data into an analog signal using digital-to-analog conversion (DAC), and then re-sampling the analog signal using DAC to obtain audio data at a uniform sampling rate. Afterwards, frame length fixing, data sampling, and channel alignment are performed to obtain the audio data alignment result. During alignment, the same frame length and the number of bits represented by each sampling point must be fixed for all audio data streams. In the fixed frame length processing, the frame length setting standard of pulse code modulation can be followed, setting the audio data of each frame to 125 microseconds (µs). Then, sampling is performed according to the fixed number of bits represented by the sampling points; specifically, the number of bits represented by the sampling points can be fixed at 8 bits. Next, channel alignment is required to convert audio data from different channels to the same number of channels. Specifically, all audio data can be converted to mono data for processing. In a specific embodiment, the mixing method of this application is applicable to mixing audio data streams related to web conferencing. At this time, the audio data stream acquired by the intermediate device 106 is the audio data stream recorded by participants in the web conferencing through their local hardware terminals, and the audio data streams of the same web conferencing are placed in the same buffer queue. To compare the audio energy of the audio data streams in the same buffer queue, the audio energy information of each received audio data stream needs to be calculated first. To ensure the accuracy of the audio energy information comparison, data alignment processing is required for multiple audio data streams in the same buffer queue. At this point, the intermediate device 106 can first convert the audio data stream in the transmission control protocol format in the buffer queue into audio data in pulse code modulation format, thereby starting the data alignment process and comparing the audio data in the same buffer queue under the same standard. The first step is to convert the sampling rate of these pulse code modulation format audio data to the same sampling rate, such as setting it to 8 kHz.Simultaneously, it may be necessary to align other data of these pulse code modulation format audio data by setting them to be the same. Specifically, this includes: fixing the frame length of the audio data after setting the sampling rate, setting the number of samples for the audio data with the fixed frame length to fix the number of bits representing each audio data sampling point, sampling the audio data according to the set number of samples, and setting channel alignment to obtain the audio data alignment result. In this embodiment, data alignment effectively places the audio data streams under the same benchmark for comparison, thereby ensuring the recognition accuracy during the target audio data recognition process.

[0085] In one embodiment, the scene identifier includes a network conference identifier, and step 203 includes: parsing the audio data stream to obtain the data header corresponding to the audio data stream; identifying the network conference identifier corresponding to the audio data stream based on the data header; and importing audio data streams containing the same network conference identifier into the same buffer queue.

[0086] The data header, also known as the data packet header, is a special reserved field of defined length appended to the beginning of data packets used for carrying and transmitting control information in network protocol communication. The data header typically carries identification information. In the scheme of this application, the data header carries a network conference identifier to indicate which network conference the current audio data stream belongs to.

[0087] Specifically, this application is used to achieve audio mixing in a multi-audio convergence scenario such as a web conference. In this case, the scene identifier carried by the audio data stream can be used to identify which web conference the current audio data stream belongs to. After acquiring the audio data stream, the intermediate device 106 can parse each frame of audio data to obtain the data header, and then search for an existing buffer queue based on the data header, or create a new buffer queue for a new web conference. This allows audio data streams containing the same web conference identifier to be imported into the same buffer queue for mixing. In this embodiment, using the data header of the audio data stream to identify the audio data stream scene can effectively ensure the accuracy of scene identification, thereby guaranteeing the mixing effect of the audio data.

[0088] In one embodiment, step 201 specifically includes: acquiring a portion of the audio data stream from multiple audio data streams in a multi-audio merging scenario based on multiple data processing processes. Simultaneously, step 209 includes: performing mixing processing on the target audio data stream selected by each data processing process to obtain the output audio of the multi-audio merging scenario. Specifically, as... Figure 4 As shown, in this embodiment, the mixing method of this application specifically includes:

[0089] Step 401: Based on multiple data processing processes, acquire partial audio data streams from multiple audio data streams in the multi-audio convergence scenario.

[0090] Step 403: According to the scene identifier carried by the audio data stream, import the audio data streams with the same scene identifier into the same buffer queue.

[0091] Step 405: Determine the audio energy of each audio data stream in each buffer queue. The audio energy is used to characterize the total amplitude of the pulse code modulation of the audio data stream.

[0092] Step 407: Select the target audio data stream from each buffer queue whose audio energy meets the energy filtering conditions.

[0093] Step 409: Mix the target audio data streams selected by each data processing process to obtain the output audio of the multi-audio merging scene.

[0094] In another specific embodiment, step 409 specifically includes: sending the target audio data stream selected by each data processing process to the mixing processor, so that the mixing processor can perform mixing processing on the received target audio data stream.

[0095] Specifically, multiple data processing processes refer to acquiring audio data streams through multiple intermediate devices, with each data processing process corresponding to one intermediate device. The mixing of the target audio data streams selected by each data processing process to obtain the output audio of the multi-audio merging scenario specifically refers to the merging of audio data from the same multi-audio merging scenario collected by multiple intermediate devices on the server 108 side, thereby obtaining the output audio of the multi-audio merging scenario. The mixing processor is a device used for mixing, which can merge target audio data streams from different intermediate devices and mix these audio data streams.

[0096] Specifically, in a multi-audio convergence scenario, it is generally necessary to collect audio data from multiple sources, such as through terminals 102, 104, and many other terminals to collect audio data streams from different locations. These audio data streams are then transmitted to server 108 via intermediate devices for mixing processing. Multiple terminals may transmit audio data streams to the same intermediate device or different intermediate devices, and each intermediate device has its own data processing process to obtain a portion of the audio data streams from the multiple audio data streams in the multi-audio convergence scenario. During the final mixing processing, the target audio data streams selected from each intermediate device can be sent to server 108. Server 108 then directly performs mixing processing on the target audio data streams selected by each data processing process to obtain the output audio of the multi-audio convergence scenario. In a specific embodiment, the mixing method of this application is specifically used for the mixing processing of web conferences in a large company. This large company includes several branches that need to participate in web conferences, and each branch also has multiple terminals responsible for participating in web conferences. At this point, multiple terminals in the branch office can upload audio data streams to server 108 through intermediate devices within the branch office, thereby mixing the audio data. During mixing, each intermediate device in the branch office obtains its portion of the audio data stream from the web conference through an independent data processing process, processes this portion of the audio data stream, and identifies the target audio data stream. Finally, during the mixing process, multiple intermediate devices send their selected target audio data streams to server 108 for mixing, resulting in the output audio for the multi-audio convergence scenario. In this embodiment, by using different intermediate devices to obtain portions of the audio data streams from multiple audio data streams in a multi-audio convergence scenario, the audio data streams in large multi-audio convergence scenarios can be effectively filtered, ensuring the accuracy of audio data filtering.

[0097] This application also provides an application scenario in which the above-described mixing method is applied. Specifically, the mixing method is applied in this scenario as follows:

[0098] When users hold web conferencing applications, due to the large number of participants, multiple audio streams are typically transmitted over the network from the application software or hardware terminals to a media server for processing. The media server's processing of multiple audio streams generally has three characteristics: First, real-time processing of multiple audio data is a computationally intensive service, consuming significant Central Processing Unit (CPU) resources. Second, mixing typically only requires a maximum of six of the highest-capacity audio streams. Third, the media server has limited bandwidth, but the number of upstream audio streams is proportional to the number of participants. This leads to two consequences: First, the media server needs to sort the multiple audio streams before mixing the six highest-capacity streams, resulting in high CPU overhead. Second, the upstream transmission of multiple audio streams consumes bandwidth, creating data transmission pressure. Therefore, the mixing method described in this application can reduce the amount of data uploaded during the mixing process, thereby improving the media server's mixing efficiency and freeing up CPU computing resources. At this point, the mixing method in this application is specifically implemented based on the edge computing mixing framework of MapReduce. MapReduce is a programming model used for parallel computation of large-scale datasets (greater than 1 terabyte). The concepts of "Map" and "Reduce" are their main ideas, both borrowed from functional programming languages, as well as features borrowed from vector programming languages. It greatly facilitates programmers who do not know distributed parallel programming to run their programs on distributed systems. The current software implementation specifies a Map function to map a set of key-value pairs into a new set of key-value pairs, and specifies concurrent Reduce functions to ensure that each key-value pair in all mapped pairs shares the same key group. For a detailed explanation of the concept of MapReduce-based mixing processing in this application, please refer to the following diagram. Figure 5 As shown, the process of calculating 1+5+7+3+4+9+3+5+6 can be broken down into three parts (1+5+7), (+3+4+9), and (+3+5+6) through mapping. Then, the result 13+16+14 is calculated in the reduction process. In the scheme of this application, the terminal devices that need to participate in the mixing are mapped to different intermediate devices. Each intermediate device processes a portion of the audio data stream sent by the terminal, and filters these audio data streams to obtain the target audio data stream. Then, the target audio data stream obtained by the intermediate devices is reduced on the server to output the final mixed data. In the scheme of this application, an edge computing mixing framework based on MapReduce is used, such as... Figure 6As shown, the interface is divided into three main sections from left to right: the application software or hardware access terminal, the smart router, and the media server. The mixing method in this application primarily involves audio data acquisition via the application software or hardware access terminal, audio data filtering via the smart router, and mixing via the media server. The core of the framework is to utilize the computing power of the smart router to identify, parse, and depacketize multiple audio data streams sent within the same local area network. Then, the router's own Field Programmable Gate Array (FPGA) edge computing capabilities are used to perform energy calculation and sorting of the audio data. The highest-capacity audio data stream is then grouped and finally sent to the media server for mixing. First, the smart router obtains multiple audio data streams from the application software or hardware access terminal for the network conferencing scenario; these audio data streams may belong to different network conferencing sessions. Therefore, classification is necessary. First, the header of each frame in the audio data stream is identified. Based on the header, the corresponding network conference identifier is determined. Then, audio data streams with the same network conference identifier are imported into the same buffer queue for processing. Next, the audio energy of each audio data stream in each buffer queue needs to be determined. The audio energy is used to characterize the total amplitude of the pulse-code modulation (PCM) of the audio data stream. When calculating the audio energy, data alignment processing is first performed on each audio data stream in the buffer queue to obtain the audio data alignment result. During data alignment, each audio data stream in the buffer queue is parsed into Pulse-Coded Modulation (PCM) format audio data; the PCM format audio data undergoes sampling rate conversion processing to obtain sample-rate converted data; the sample-rate converted data undergoes frame length fixing processing to obtain fixed-length frame data; the fixed-length frame data undergoes data sampling processing to obtain audio sample data; and the audio sample data undergoes channel number alignment processing to obtain the audio data alignment result. Then, based on the audio data alignment results, pulse code modulation amplitude is calculated on the audio data stream to obtain its audio energy. This audio energy is then saved to the energy data storage queue corresponding to the buffer queue. Once all audio energy has been identified, subsequent processing can begin. The intelligent router will filter out target audio data streams whose audio energy meets the energy filtering criteria from the buffer queue corresponding to the energy data storage queue. The identified target audio data streams are then sent to the media server. The media server receives target audio data streams from multiple intelligent routers. For each web conference, the target audio data streams from that conference are mixed to obtain the output audio for the multi-audio convergence scenario. Specifically, on the media server side, the audio data streams forwarded by each intelligent router can be sorted within a small range before mixing calculations are performed.After receiving the output audio, the media server can also feed it back to the corresponding application software or hardware access terminal, thereby effectively maintaining the online conference.

[0099] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0100] Based on the same inventive concept, this application also provides a mixing processing system for implementing the mixing method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more mixing processing system embodiments provided below can be found in the limitations of the mixing method described above, and will not be repeated here.

[0101] In one embodiment, such as Figure 7 As shown, a mixing processing system is provided, which includes multiple audio collectors 702, multiple audio filters 704 and a mixing processor 706.

[0102] The audio acquisition unit 702 is used to acquire multiple audio data streams in multi-audio convergence scenarios and send the multiple audio data streams to the audio filter.

[0103] The audio filter 704 is used to import audio data streams with the same scene identifier into the same buffer queue according to the scene identifier carried by the audio data stream, determine the audio energy of each audio data stream in each buffer queue, the audio energy is used to characterize the total amplitude of the pulse code modulation of the audio data stream, and then select the target audio data stream that meets the energy selection condition from each buffer queue, and send the obtained target audio data stream to the mixing processor.

[0104] The mixing processor 706 is used to mix target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

[0105] Specifically, this application also provides a mixing system based on the above-mentioned mixing method. The mixing system includes multiple audio acquisition units, multiple audio filters, and a mixing processor. The audio acquisition unit 702 consists of a terminal capable of acquiring audio data; the audio filter 704 is an intermediate device in the mixing process and can be implemented by a smart router; and the mixing processor 706 is used for mixing and can be implemented by a media server. Specifically, the mixing system of this application can be explained using an example of mixing for network conferencing. Figure 8 As shown, the audio acquisition unit 702 can be implemented through application software or hardware terminals. Multiple audio data streams sent by the audio acquisition unit 702 within the same local area network can be sent to the audio filter 704 using the audio transmission control protocol data stream. The audio filter 704 is specifically implemented through intelligent routing. Based on intelligent routing, the edge computing business logic that utilizes the field-programmable gate array (FPGA) module of the router to sort the audio data streams is first implemented. This code logic runs within the FPGA module of the intelligent router. In the overall edge computing architecture of the intelligent router, the most important module is the audio data stream sorting logic operation module based on the FPGA. This module receives network audio transmission control protocol data through the Ethernet interface (Ethernet MAC). The received data is first unpacked by the group engine, and then each frame of data is grouped. The grouping logic is performed by conference. Audio data streams whose headers contain information about the same conference are written into the corresponding buffer queue by the group engine. When the number of audio data streams in the queue does not increase within a certain time, the group engine passes the memory address of the queue to the preprocessing sorting engine. After receiving the queued data, the preprocessing sorting engine begins preprocessing each data stream in the queue. Preprocessing mainly includes: decompression into pulse code modulation format, sample rate conversion, fixing the frame length, fixing the number of bits per sample point, and aligning the number of channels. After preprocessing, the audio energy of each audio data stream is calculated, and the results are stored in another queue. Finally, the entire energy queue is traversed to find the index of the queue with the highest audio energy, and the corresponding audio data stream is retrieved from the audio data stream queue input to the preprocessing sorting engine based on this index. This data stream is then encapsulated and forwarded to the mixing processor (implemented here by the media server). The mixing processor performs a small-scale sorting of the audio data streams forwarded by each intelligent router before performing mixing calculations.

[0106] The aforementioned audio mixing system first acquires multiple audio data streams from a multi-audio convergence scenario using an audio acquisition device. Then, it uses an audio filter to import audio data streams with the same scene identifier into the same buffer queue, thereby classifying them according to the scene to which they belong. Next, it determines the audio energy corresponding to each audio data stream in each buffer queue. Then, based on the audio energy, it extracts the target audio data streams that meet the energy filtering conditions from multiple audio data streams in each buffer queue, ensuring the effectiveness of the mixing process. Finally, the mixing processor performs mixing processing on the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio convergence scenario. This effectively reduces the amount of audio data involved in the mixing process, thereby ensuring the quality of media data transmission.

[0107] Based on the same inventive concept, this application also provides a mixing apparatus for implementing the mixing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more mixing apparatus embodiments provided below can be found in the limitations of the mixing method described above, and will not be repeated here.

[0108] In one embodiment, such as Figure 9 As shown, a mixing device is provided, comprising:

[0109] The audio data acquisition module 902 is used to acquire multiple audio data streams in multi-audio convergence scenarios.

[0110] The audio data import module 904 is used to import audio data streams with the same scene identifier into the same buffer queue according to the scene identifier carried by the audio data stream.

[0111] The audio energy identification module 906 is used to determine the audio energy of each audio data stream in each buffer queue. The audio energy is used to characterize the total amplitude of the pulse code modulation of the audio data stream.

[0112] The target audio data recognition module 908 is used to filter out target audio data streams whose audio energy meets the energy filtering conditions from each buffer queue.

[0113] The mixing module 910 is used to mix target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

[0114] In one embodiment, the audio energy identification module 906 is specifically used to: perform data alignment processing on each audio data stream in each buffer queue to obtain an audio data alignment result; and calculate the pulse code modulation amplitude of the audio data stream based on the audio data alignment result to obtain the audio energy of the audio data stream.

[0115] In one embodiment, the audio energy identification module 906 is further configured to: save the audio energy data of each audio data stream to the energy data storage queue corresponding to the buffer queue; traverse the energy data storage queue and filter out target audio energy that meets the energy filtering conditions; and filter out target audio data streams that meet the energy filtering conditions from the buffer queue corresponding to the energy data storage queue.

[0116] In one embodiment, the target audio data recognition module 908 is specifically used to: determine the audio data stream with the largest audio energy in the buffer queue corresponding to the energy data storage queue, and use the audio data stream with the largest audio energy as the target audio data stream that matches the target audio energy.

[0117] In one embodiment, the audio energy identification module 906 is further configured to: parse each audio data stream in the buffer queue into audio data in pulse modulation coding format; perform sampling rate conversion processing on the audio data in pulse modulation coding format to obtain sampling rate converted data; perform frame length fixing processing on the sampling rate converted data to obtain fixed frame length data; perform data sampling processing on the fixed frame length data to obtain audio sample data; and perform channel number alignment processing on the audio sample data to obtain audio data alignment results.

[0118] In one embodiment, the scene identifier includes a web conference identifier, and the audio data import module 904 is specifically used to: parse the audio data stream and obtain the data header corresponding to the audio data stream; identify the web conference identifier corresponding to the audio data stream based on the data header; and import audio data streams containing the same web conference identifier into the same buffer queue.

[0119] In one embodiment, the audio data acquisition module 902 is specifically used to: acquire a portion of the audio data stream from multiple audio data streams in a multi-audio convergence scenario based on multiple data processing processes; the mixing processing module 910 is specifically used to: perform mixing processing on the target audio data stream selected by each data processing process to obtain the output audio of the multi-audio convergence scenario.

[0120] In one embodiment, the mixing processing module 910 is specifically used to: send the target audio data stream selected by each data processing process to the mixing processor, so that the mixing processor can perform mixing processing on the received target audio data stream.

[0121] Each module in the aforementioned mixing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0122] In one embodiment, a computer device is provided, which may be a router terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a mixing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0123] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0124] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0125] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0126] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0129] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A mixing method, characterized in that, Implemented via an intermediate device, the method includes: Acquire multiple audio data streams in a multi-audio convergence scenario, wherein the multi-audio convergence scenario is a scenario in which multiple audio data from different sources are processed in one scenario, and in the case that the multi-audio convergence scenario is a web conferencing scenario, the scenario identifier carried by the audio data stream is a web conferencing identifier; Parse the audio data stream to obtain the data header corresponding to the audio data stream; identify the network conference identifier corresponding to the audio data stream based on the data header; import audio data streams containing the same network conference identifier into the same buffer queue; Determine the audio energy of each audio data stream in each buffer queue, the audio energy being used to characterize the total amplitude of the pulse code modulation of the audio data stream; From each of the buffer queues, select the target audio data stream whose audio energy meets the energy filtering conditions; The target audio data stream is uploaded to the server, and the server performs mixing processing on the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

2. The method according to claim 1, characterized in that, The determination of the audio energy of each audio data stream in each buffer queue includes: For each buffer queue, data alignment processing is performed on each audio data stream in the buffer queue to obtain the audio data alignment result; Based on the audio data alignment results, the pulse code modulation amplitude of the audio data stream is calculated to obtain the audio energy of the audio data stream.

3. The method according to claim 2, characterized in that, The method further includes: Save the audio energy data of each audio data stream to the energy data storage queue corresponding to the buffer queue; Traverse the energy data storage queue and filter out target audio energy that meets the energy filtering conditions; The step of filtering out the target audio data stream whose audio energy meets the energy filtering conditions from each of the buffer queues includes: From the buffer queue corresponding to the energy data storage queue, the target audio data stream whose audio energy meets the energy filtering conditions is selected.

4. The method according to claim 3, characterized in that, The step of filtering out the target audio data stream whose audio energy meets the energy filtering conditions from the buffer queue corresponding to the energy data storage queue includes: The audio data stream with the highest audio energy in the buffer queue corresponding to the energy data storage queue is determined, and the audio data stream with the highest audio energy is taken as the target audio data stream that matches the target audio energy.

5. The method according to claim 3, characterized in that, The step of performing data alignment processing on each audio data stream in the buffer queue to obtain the audio data alignment result includes: Each audio data stream in the buffer queue is parsed into audio data in pulse modulation coding format; The audio data of the pulse modulation coding format is subjected to sampling rate conversion processing to obtain sampling rate converted data; The sample rate conversion data is subjected to frame length fixing processing to obtain fixed frame length data; The fixed-frame-length data is subjected to data sampling processing to obtain audio sampling data; The audio sampling data is subjected to channel number alignment processing to obtain the audio data alignment result.

6. The method according to any one of claims 1 to 5, characterized in that, The acquisition of multiple audio data streams in a multi-audio merging scenario includes: Based on multiple data processing processes, partial audio data streams from multiple audio data streams in multi-audio convergence scenarios are obtained respectively; The mixing process performed on the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene includes: The target audio data streams selected by each of the data processing processes are mixed to obtain the output audio of the multi-audio merging scenario.

7. The method according to claim 6, characterized in that, The mixing process performed on the target audio data streams selected by each of the data processing processes to obtain the output audio of the multi-audio merging scenario includes: The target audio data stream selected by each of the data processing processes is sent to the mixing processor so that the mixing processor can perform mixing processing on the received target audio data stream.

8. A mixing processing system, characterized in that, The system includes multiple audio acquisition units, multiple audio filters, and a mixing processor; The audio collector is used to collect multiple audio data streams in a multi-audio convergence scenario and send the multiple audio data streams to an audio filter. The multi-audio convergence scenario is a scenario in which multiple audio data from different sources are processed in one scenario. When the multi-audio convergence scenario is a web conferencing scenario, the scenario identifier carried by the audio data stream is a web conferencing identifier. The audio filter is used to parse the audio data stream and obtain the data header corresponding to the audio data stream; identify the network conference identifier corresponding to the audio data stream based on the data header; import audio data streams containing the same network conference identifier into the same buffer queue; determine the audio energy of each audio data stream in each buffer queue, the audio energy is used to characterize the total amplitude of pulse code modulation of the audio data stream; then, from each buffer queue, select the target audio data stream whose audio energy meets the energy selection criteria, and send the obtained target audio data stream to the mixing processor; The mixing processor is used to mix target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

9. A mixing device, characterized in that, The device, located in an intermediate device, includes: The audio data acquisition module is used to acquire multiple audio data streams in a multi-audio convergence scenario. The multi-audio convergence scenario is a scenario in which multiple audio data from different sources are processed in one scenario. When the multi-audio convergence scenario is a web conferencing scenario, the scenario identifier carried by the audio data stream is a web conferencing identifier. The audio data import module is used to parse the audio data stream, obtain the data header corresponding to the audio data stream; identify the network conference identifier corresponding to the audio data stream based on the data header; and import audio data streams containing the same network conference identifier into the same buffer queue. An audio energy identification module is used to determine the audio energy of each audio data stream in each buffer queue, wherein the audio energy is used to characterize the total amplitude of the pulse code modulation of the audio data stream; The target audio data identification module is used to filter out target audio data streams whose audio energy meets the energy filtering conditions from each of the buffer queues; The audio mixing module is used to upload the target audio data stream to the server, and the server performs audio mixing on the target audio data streams with the same scene identifier to obtain the output audio of the multi-audio merging scene.

10. The apparatus according to claim 9, characterized in that, The audio energy identification module is specifically used for: performing data alignment processing on each audio data stream in each buffer queue to obtain an audio data alignment result; and calculating the pulse code modulation amplitude of the audio data stream based on the audio data alignment result to obtain the audio energy of the audio data stream.

11. The apparatus according to claim 10, characterized in that, The audio energy identification module is further configured to: save the audio energy data of each audio data stream to the energy data storage queue corresponding to the buffer queue; traverse the energy data storage queue and filter out target audio energy that meets the energy filtering conditions; and filter out target audio data streams whose audio energy meets the energy filtering conditions from the buffer queue corresponding to the energy data storage queue.

12. The apparatus according to claim 11, characterized in that, The target audio data recognition module is specifically used to: determine the audio data stream with the largest audio energy in the buffer queue corresponding to the energy data storage queue, and use the audio data stream with the largest audio energy as the target audio data stream matched with the target audio energy.

13. The apparatus according to claim 11, characterized in that, The audio energy identification module is further configured to: parse each audio data stream in the buffer queue into audio data in pulse modulation coding format; perform sampling rate conversion processing on the audio data in pulse modulation coding format to obtain sampling rate converted data; and perform frame length fixing processing on the sampling rate converted data to obtain fixed frame length data. The fixed frame length data is sampled to obtain audio sample data; the audio sample data is then aligned to obtain audio data alignment results.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The audio data acquisition module is specifically used to: acquire a portion of the audio data stream from multiple audio data streams in a multi-audio merging scenario based on multiple data processing processes; the mixing processing module is specifically used to: perform mixing processing on the target audio data stream selected by each of the data processing processes to obtain the output audio of the multi-audio merging scenario.

15. The apparatus according to claim 14, characterized in that, The mixing module is specifically used to: send the target audio data stream selected by each of the data processing processes to the mixing processor, so that the mixing processor can perform mixing processing on the received target audio data stream.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images