Audio mixing method and device, electronic device, computer readable medium

Abstract

The present disclosure provides a mixing method and device, which relates to technical fields such as audio processing. The specific implementation scheme is as follows: receiving multiple initial audio streams at different times; assigning each initial audio stream to one of multiple period windows, where the multiple period windows are obtained based on the mixing delay; based on the timestamps of each initial audio stream, detecting whether each period window has a first audio stream that does not correspond to the period window, where the initial audio stream includes the first audio stream; in response to detecting the first audio stream, adjusting the first audio stream to the corresponding period window; based on the multiple period windows and the mixing delay, mixing the initial audio streams belonging to all period windows to obtain mixed audio. This implementation improves the quality of mixed audio.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of audio processing, and more particularly to a mixing method and device, electronic device, computer readable medium and computer program product. BACKGROUND

[0002] When a mixing device receives real-time audio streams sent by different audio devices, due to different hardware conditions of the audio devices, time delay may occur in sending the real-time audio streams. Therefore, when the mixing device mixes the audio streams with time delay, the multiple audio streams may not be synchronized, and the mixing effect is poor. SUMMARY

[0003] A mixing method and device, electronic device, computer readable medium and computer program product are provided.

[0004] According to a first aspect, a mixing method is provided, the method comprising: receiving multiple initial audio streams of different time instants; attributing each initial audio stream to a cycle window of multiple cycle windows, the multiple cycle windows being obtained based on a mixing delay; detecting, based on a timestamp of each initial audio stream, whether there is a first audio stream that does not correspond to the cycle window in each cycle window, the initial audio streams including the first audio stream; in response to detecting the first audio stream, adjusting the first audio stream to the corresponding cycle window; and mixing the initial audio streams attributed to all the cycle windows based on the multiple cycle windows and the mixing delay to obtain mixed audio.

[0005] According to a second aspect, a mixing device is provided, the device comprising: a receiving unit configured to receive multiple initial audio streams of different time instants; an attribution unit configured to attribute each initial audio stream to a cycle window of multiple cycle windows, the multiple cycle windows being obtained based on a mixing delay; a detection unit configured to detect, based on a timestamp of each initial audio stream, whether there is a first audio stream that does not correspond to the cycle window in each cycle window, the initial audio streams including the first audio stream; an adjustment unit configured to, in response to detecting the first audio stream, adjust the first audio stream to the corresponding cycle window; and a mixing unit configured to mix the initial audio streams attributed to all the cycle windows based on the multiple cycle windows and the mixing delay to obtain mixed audio.

[0006] According to a third aspect, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being configured to cause a computer to perform the method according to any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program configured to implement the method according to any implementation of the first aspect when executed by a processor.

[0009] The mixing method and device provided by the embodiments of the present disclosure first receive multiple initial audio streams at different time points; secondly, each initial audio stream is attributed to one of multiple period windows, and the multiple period windows are obtained based on a mixing delay; thirdly, based on a timestamp of each initial audio stream, it is detected whether there is a first audio stream that does not correspond to a period window in each period window, and the initial audio stream includes the first audio stream; fourthly, in response to detecting the first audio stream, the first audio stream is adjusted to the corresponding period window; and finally, based on the multiple period windows and the mixing delay, the initial audio streams attributed to all the period windows are mixed to obtain a mixed audio. Thus, after receiving the multiple initial audio streams, based on the mixing delay, the delay is divided into multiple period windows, the received audio streams are attributed to each period window, and the audio streams that do not correspond to each period window are adjusted based on the timestamp of the audio stream, so that the audio streams with a delay in sending time are adjusted to the appropriate period window, thereby suppressing the occurrence of the phenomenon that the multiple audio streams are out of synchronization due to the delay of the audio streams, and improving the quality of the mixed audio.

[0010] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0012] Figure 1 is a flowchart of one embodiment of the mixing method according to the present disclosure;

[0013] Figure 2 is a structural schematic diagram of the initial audio stream being attributed to multiple period windows in the present disclosure;

[0014] Figure 3 is a structural schematic diagram of the first audio stream being adjusted to the corresponding period window in the present disclosure;

[0015] Figure 4 is a flowchart of another embodiment of the mixing method according to the present disclosure;

[0016] Figure 5 is a structural schematic diagram of one embodiment of a mixing device according to the present disclosure;

[0017] Figure 6 is a block diagram of an electronic device for implementing a mixing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are presented for the purpose of illustration and description. It is to be understood that the embodiments described herein are exemplary only, and that various changes and modifications can be made thereto without departing from the scope and spirit of the present disclosure. As such, the exemplary embodiments described herein are to be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for combination with features or aspects of the other embodiments, unless the context explicitly dictates otherwise.

[0019] Figure 1 A flow 100 of one embodiment of a mixing method according to the present disclosure is shown, the mixing method comprising the following steps:

[0020] Step 101, receiving multiple initial audio streams at different times.

[0021] In this embodiment, the audio stream is a carrier of audio data, and the audio stream is also one of the main media forms in the field of information and signal processing. The multiple initial audio streams at different times are audio streams from different sources at different times, which can be from different musical instruments, human voices, or orchestras, recorded from live performances or in recording studios.

[0022] In this embodiment, the multiple initial audio streams at different times can also be audio streams emitted in real time at different times by multiple audio devices (such as loudspeakers) or personnel, each initial audio stream having a respective emitter and emission time. By collecting multiple initial audio streams at different times, a mixing device can be used to mix the initial audio.

[0023] In this embodiment, mixing is a step in music production, in which multiple sounds from different sources are integrated into a stereo track or a mono track. Mixing multiple initial audio streams from different sources can produce mixed audio with a hierarchical effect.

[0024] Step 102, attributing each initial audio stream to a cycle window in multiple cycle windows.

[0025] In this embodiment, when mixing multiple different initial audio streams using a mixing device or a subject running on the mixing method of the present disclosure, due to the mixing device or the subject itself (such as the hardware delay of the mixing device, the program running delay of the subject, etc.), there is a blank time before mixing, and the actual mixing of multiple initial audio streams will not be performed until the blank time is passed. Therefore, the multiple initial audio streams and the time period before mixing the multiple initial audio streams are the mixing delay.

[0026] In this embodiment, the mixing delay is a time point, such as the mixing delay a1 on the time axis a. The mixing delay is determined by the subject that mixes multiple initial audio streams. The mixing delay of different types of subjects is different, for example, the mixing delay of the mixing device is 10s, and the mixing delay of the mixing using software is 3s. Figure 2

[0027] In this embodiment, multiple periodic windows are obtained based on the mixing delay. The mixing delay is mapped on the time axis, the time period from the starting point of the time axis to the mixing delay is divided into time regions, and multiple different time regions are obtained. The time of each time region can increase to the mixing delay as time advances, and the time period from the starting point of the time axis to the mixing delay is divided and sequentially reaches the time region of the mixing delay as time advances, which is the periodic window.

[0028] In this embodiment, the above-mentioned time region can be a time period, which is obtained by dividing the time axis from the starting point of the time axis to the mixing delay, and each time period can eventually increase to the mixing delay as time advances, as shown in Figure 2 In this embodiment, the time axis a is divided into periodic window one and periodic window two. Alternatively, the time region can also be a time point, which is each time point from the starting point of the time axis to the mixing delay. Each time point increases until it eventually increases to the mixing delay as time advances.

[0029] In this embodiment, the purpose of dividing multiple periodic windows is to associate each periodic window with each initial audio stream, so as to process the initial audio stream and obtain a better initial audio in time sequence. Each initial audio stream is attributed to each periodic window, which means that each initial audio stream is sequentially placed in the region corresponding to each periodic window based on the time when each different initial audio stream is received. At this time, the obtained initial audio stream is sequentially attributed to different periodic windows, as shown in Figure 2 As shown, the audio stream of A object 00:00 time and the audio stream of B object 00:00 time are attributed to periodic window one, and the audio stream of A object 00:01 time, the audio stream of B object 00:01 time, the audio stream of C object 00:00 time, and the audio stream of C object 00:01 time are attributed to periodic window two.​

[0030] Further, as shown in Figure 2 Fig. 2, due to the delay of the sending time of the C object, in the second cycle window, there are audio streams corresponding to the C object 00:00 time and the C object 00:01 time at the same time, and if the second cycle window reaches the mixing delay a1, the initial audio stream belonging to the second cycle window is mixed, and the mixing effect is poor.

[0031] In step 103, whether each cycle window has a first audio stream not corresponding to the cycle window is detected based on the time stamp of each initial audio stream.

[0032] The time stamp is usually a character sequence, which uniquely identifies a moment in time. In this embodiment, the time stamp of the initial audio stream refers to the time stamp of the sending time of the initial audio stream, and the time stamp of the initial audio stream is used to indicate the time when the initial audio stream is sent by the sender. The execution subject on which the mixing method runs can determine the time stamp of the initial audio stream through the information carried by the initial audio stream itself, and the execution subject can also determine the time stamp of the initial audio stream by communicating with the audio device that sends the initial audio stream. Further, the specific time when the initial audio stream is sent by the audio device that sends the initial audio stream can be determined through the time stamp of the initial audio stream.

[0033] In this embodiment, the cycle windows are arranged in the order of receiving the initial audio streams, but due to the influence of the hardware of the audio device that sends the initial audio stream or the transmission medium, the time stamp of the initial audio stream may not correspond to the cycle window, as shown in Figure 2 Fig. 2, due to the delay of the sending time of the C object, in the second cycle window, there are audio streams corresponding to the C object 00:00 time and the C object 00:01 time at the same time, and if the second cycle window reaches the mixing delay a1, the initial audio stream belonging to the second cycle window is mixed, and the mixing effect is poor.

[0034] In this embodiment, the initial audio stream includes a first audio stream, and the first audio stream is an initial audio stream that does not correspond to the cycle window. The time stamp of the initial audio stream is not within the time range of the cycle window corresponding to it. It should be noted that when the first audio stream is adjusted to the cycle window corresponding to it, the first audio stream becomes an initial audio stream.

[0035] The above-mentioned detection of whether each period window has a first audio stream that does not correspond to the period window based on the timestamp of each initial audio stream includes: determining, for each period window, the timestamp of the initial audio stream corresponding to the period window; for each initial audio stream of the period window, detecting whether the time corresponding to the timestamp of the initial audio stream is within the time period corresponding to the period window, and in response to being within the time period corresponding to the period window, determining that the initial audio stream corresponds to the period window; when all the initial audio streams corresponding to the period window correspond to the period window, determining that the period window has no first audio stream that does not correspond to the period window.

[0036] Optionally, the above-mentioned detection of whether each period window has a first audio stream that does not correspond to the period window based on the timestamp of each initial audio stream also includes: determining, for each period window, the timestamp of the initial audio stream corresponding to the period window; for each initial audio stream of the period window, detecting whether the time corresponding to the timestamp of the initial audio stream is within the time period corresponding to the period window, and in response to not being within the time period corresponding to the period window, determining that the initial audio stream does not correspond to the period window; determining that the period window has a first audio stream that does not correspond to the period window, and using the initial audio stream that does not correspond to the period window as the first audio stream.

[0037] Step 104: In response to detecting the first audio stream, adjust the first audio stream to a corresponding period window.

[0038] In this embodiment, the first audio stream is an initial audio stream that incorrectly corresponds to a periodic window, and the timestamp of the initial audio stream is not within the time period of the corresponding periodic window. In order to facilitate mixing of all initial audio streams, the first audio stream can be adjusted to a corresponding periodic window, wherein the corresponding periodic window refers to a time period in which the timestamp of the first audio stream belongs to the corresponding periodic window. Adjusting the first audio stream to the corresponding periodic window includes: revoking the affiliation of the first audio stream with the original periodic window, affiliating the first audio stream to the corresponding periodic window, and mixing the first audio stream with other initial audio streams after the corresponding periodic window reaches the mixing delay.

[0039] Specifically, if Figure 3 As shown, through detection, it can be known that the audio stream of the C object at 00:00 time corresponding to the cycle window two is not in the time period (00:01) of the cycle window two, and the audio stream of the C object at 00:00 time is adjusted to the corresponding cycle window, which is the cycle window one, and the audio stream of the C object at 00:00 time corresponds to the time period (00:00) of the cycle window one, and belongs to the time period of the cycle window one.

[0040] Optionally, when the first audio stream is not detected in the initial audio stream corresponding to the cycle window, no processing is performed.

[0041] In step 105, based on the multiple cycle windows and the mixing delay, mixing the initial audio streams belonging to all the cycle windows to obtain mixed audio.

[0042] In this embodiment, mixing is a step of music production, and through mixing, multiple different source sounds are integrated into a stereo track or a mono track. In the process of mixing, the mixing device can adjust the frequency, dynamics, tone, positioning, reverberation and sound field of each initial audio stream separately, so that each track is optimized, and then superimposed on the final product. This processing method can produce a perfect effect with clear levels that listeners cannot hear in live recording.

[0043] In this embodiment, the multiple cycle windows are time regions between the start point (for example, zero point) of the time axis and the mixing delay, and can advance with time. Once the time of any one of the multiple cycle windows grows to the mixing delay, all the initial audio streams in the cycle window are mixed. It should be noted that all the initial audio streams in the cycle window can be obtained by adjusting the initial audio stream of the first audio stream. As shown in FIG. 1, when the time of cycle window one grows to mixing delay a1, all the initial audio streams in cycle window one are mixed, and the initial audio streams in cycle window one include the audio stream of 00:00 time of object C adjusted from cycle window two. Figure 3

[0044] The mixing method provided by the embodiment of the present disclosure first receives multiple initial audio streams at different times; second, attributes each initial audio stream to a cycle window in multiple cycle windows, and the multiple cycle windows are obtained based on a mixing delay; third, based on the time stamp of each initial audio stream, detects whether each cycle window has a first audio stream that does not correspond to the cycle window, and the initial audio stream includes the first audio stream; fourth, in response to detecting the first audio stream, adjusts the first audio stream to the corresponding cycle window; and finally, based on the multiple cycle windows and the mixing delay, mixes the initial audio streams belonging to all the cycle windows to obtain mixed audio. Thus, after receiving multiple initial audio streams, based on the mixing delay, the delay is divided into multiple cycle windows, so that the received audio streams are attributed to each cycle window, and based on the time stamp of the audio stream, the audio stream that does not correspond to each cycle window is adjusted, so that the audio stream with a transmission time delay is adjusted to a suitable cycle window, thereby suppressing the occurrence of the phenomenon that multiple audio streams are out of synchronization due to the delay of the audio stream, and improving the quality of the mixed audio.

[0045] ​Optionally, after obtaining the multiple initial audio streams at different time, the mixing method can further comprise: performing echo cancellation processing and noise suppression processing on each initial audio stream. The echo cancellation and noise suppression belong to the pre-processing of the voice. The echo cancellation processing estimates the size of the echo signal by an adaptive method, and removes the estimated value in the initial audio stream to offset the echo. The noise suppression processing removes the noise in the initial audio stream based on the nature of the noise itself.

[0046] In some optional implementations of the embodiment, the above-mentioned attributing each initial audio stream to one of the multiple period windows comprises: determining the period window corresponding to each initial audio stream based on the order of the receiving time of each initial audio stream; associating the timestamp of each initial audio stream with the period window corresponding to the initial audio stream to obtain the association relationship between the timestamp and the period window; and storing the association relationship in a hash table, so as to attribute each initial audio stream to the corresponding period window.

[0047] In the embodiment, the receiving time of each initial audio stream is the time when the execution subject on which the mixing method runs receives the initial audio stream. Based on the order of the time when the initial audio stream is received, each initial audio stream is sequentially associated with each period window, so that each initial audio stream is attributed to a different period window.

[0048] In the embodiment, the timestamp of the initial audio stream is a time identifier of the sending time corresponding to the initial audio stream, and the timestamp of the initial audio stream is used to indicate the time when the initial audio stream is sent by the sender.

[0049] In the optional implementation, the receiving time of each initial audio stream means that each initial audio stream is received in time sequence, and each initial audio stream is associated with each period window, for example, the initial audio stream received at 00:00 is associated with the period window at 00:00. It should be noted that each initial audio stream carries a timestamp corresponding to the time when each initial audio stream is sent. Due to the influence of the audio device or the transmission medium, the timestamp can be different from the time when the audio stream is received.

[0050] Optionally, each initial audio stream can also not carry a timestamp. In the embodiment, the execution subject on which the mixing method runs can also communicate with the audio device of each initial audio stream according to the identifier of each initial audio stream after obtaining each initial audio stream, determine the sending time of each initial audio stream, and thus obtain the timestamp of each initial audio stream.

[0051] In the optional implementation, the earlier the initial audio stream is received, the more priority the initial audio stream has to be allocated to the period window close to the mixing time.

[0052] In the embodiment, the initial audio stream is associated with the period window, and the time corresponding to the period window is increased to the mixing delay, and the initial audio stream corresponding to the period window is mixed to obtain the mixed audio which is synchronous in time sequence.

[0053] In the embodiment, the hash table is used to store the association between the timestamp of each initial audio stream and the period window. The hash table, also known as a hash table, is a data structure for direct access according to a key value. The hash table maps the key value to a position in the table through a mapping function to access the record array to speed up the search. The mapping function is also called a hash function, and the record array is called a hash table.

[0054] Alternatively, a database or a message queue can also be used to store the association between the timestamp of each initial audio stream and the period window.

[0055] In the optional implementation, the association between the initial audio stream and the period window is established based on the sending time of the initial audio stream, so that the initial audio stream advances with the time advancement of the period window. When the time of the period window corresponding to the initial audio stream reaches the mixing time, the initial audio stream is mixed. By associating the timestamp of the initial audio stream with the period window corresponding to the initial audio stream and storing it in the hash table, the reliability of determining the association between the initial audio stream and the period window is ensured.

[0056] In some optional implementations of the embodiment, the above adjusting the first audio stream to the corresponding period window in response to detecting the first audio stream comprises: determining the period window corresponding to the timestamp of the first audio stream based on the timestamp of the first audio stream; changing the association based on the first audio stream and the period window corresponding to the timestamp; and adjusting the first audio stream to the period window corresponding to the timestamp.

[0057] In the optional implementation, the timestamp of the initial audio stream is a time identifier of the sending time of the corresponding initial audio stream. The timestamp of the initial audio stream is used to indicate the time when the initial audio stream is sent. Generally, the sending time of the initial audio stream can correspond to the receiving time of the initial audio stream, but due to external conditions, the sending time of the initial audio stream can not correspond to the receiving time of the initial audio stream, and the initial audio stream that does not correspond forms the first audio stream.

[0058] In the optional implementation, the period window corresponding to the timestamp of the first audio stream is determined based on the timestamp of the first audio stream, including: searching for a time of a period window including the timestamp of the first audio stream based on the timestamp of the first audio stream, and taking the period window including the timestamp of the first audio stream as the period window corresponding to the first audio stream. The above changing the association relationship includes: canceling the association relationship between the first audio stream and the original period window, and establishing the association relationship between the first audio stream and the period window corresponding to the timestamp. The above adjusting the first audio stream into the period window corresponding to the timestamp includes: based on the association relationship between the first audio stream and the period window corresponding to the timestamp, mixing the first audio stream with other initial audio streams in the period window corresponding to the timestamp after the period window corresponding to the timestamp grows to the mixing delay, to obtain the mixed audio of the period window corresponding to the timestamp.

[0059] In the optional implementation, the timestamp of the first audio stream is the timestamp of the audio device sending the first audio stream, and the most relevant period window of the first audio stream can be determined through the timestamp of the first audio stream, thereby providing a reliable basis for the first audio stream belonging to a suitable period window.

[0060] In some optional implementations of the embodiment, the above mixing the initial audio streams belonging to all period windows based on the multiple period windows and the mixing delay to obtain the mixed audio includes:

[0061] After the time of any one of the multiple period windows grows to the mixing delay, the initial audio streams belonging to the period window are mixed to obtain the mixed audio corresponding to the period window.

[0062] In the optional implementation, each of the multiple period windows is advanced with time, and after the period window grows to the mixing delay with time, it is determined that the initial audio streams belonging to the period window can be mixed, and at this time, the initial audio streams belonging to the period window are also audio streams whose sending times have been aligned.

[0063] In the optional implementation, after the time of each period window grows to the mixing delay, the initial audio streams belonging to the period window are mixed, which can ensure that the initial audio corresponding to the period window that grows with time in the mixing delay can be effectively mixed, and the synchronization effect of each initial audio during mixing is ensured.

[0064] The calculation amount of deep learning is large, and the traditional Figure 4 Flow 400 of another embodiment of the mixing method according to the present disclosure is shown, and the mixing method includes the following steps:

[0065] In step 401, multiple initial audio streams at different times are received.

[0066] Step 402, attribute each initial audio stream to one of the plurality of cycle windows.

[0067] Step 403, filter the initial audio streams in the plurality of cycle windows to remove the initial audio streams whose timestamps do not satisfy the time of all cycle windows.

[0068] In the embodiment, the filtering of the initial audio streams in the plurality of cycle windows includes: determining the initial audio streams whose timestamps do not belong to the plurality of cycle windows based on the association between the initial audio streams and each cycle window, to obtain the initial audio streams whose timestamps do not satisfy the time of all cycle windows.

[0069] In the embodiment, the initial audio streams whose timestamps do not satisfy the time of all cycle windows refer to the timestamps of the initial audio streams do not all fall within the time range of each cycle window of the plurality of cycle windows.

[0070] Step 404, detect whether there is a first audio stream that does not correspond to a cycle window in each cycle window based on the timestamps of each initial audio stream.

[0071] In the embodiment, the initial audio streams include the first audio streams.

[0072] Step 405, in response to detecting the first audio stream, adjust the first audio stream to the corresponding cycle window.

[0073] Step 406, mix the initial audio streams belonging to all cycle windows based on the plurality of cycle windows and the mixing delay to obtain the mixed audio.

[0074] It should be understood that the operations and features of the above steps 401, 402, 404-406 correspond to the operations and features of steps 101-105 respectively, and therefore the description of the operations and features in steps 101-105 also applies to steps 401, 402, 404-406, which will not be repeated here.

[0075] The mixing method provided in the embodiment first filters the initial audio streams in the plurality of cycle windows to remove the initial audio streams whose timestamps do not satisfy the time of all cycle windows before detecting whether there is a first audio stream that does not correspond to a cycle window in each cycle window, which can effectively screen the initial audio streams and ensure the reliability of the mixed audio stream and improve the mixing effect of the mixed audio.

[0076] Further reference Figure 5 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of a mixing device, and the device embodiment corresponds toFigure 1 The method embodiment shown corresponds.

[0077] As Figure 5 shown, the mixing device 500 provided by the embodiment includes a receiving unit 501, an attribution unit 502, a detection unit 503, an adjustment unit 504, and a mixing unit 505. The receiving unit 501 can be configured to receive multiple initial audio streams at different time points. The attribution unit 502 can be configured to attribute each initial audio stream to a period window in a plurality of period windows, the plurality of period windows being based on a mixing delay. The detection unit 503 can be configured to detect, based on a timestamp of each initial audio stream, whether there is a first audio stream in each period window that does not correspond to the period window, the initial audio stream including the first audio stream. The adjustment unit 504 can be configured to adjust the first audio stream to the corresponding period window in response to detecting the first audio stream. The mixing unit 505 can be configured to mix the initial audio streams attributed to all the period windows based on the plurality of period windows and the mixing delay to obtain mixed audio.

[0078] In the embodiment, the specific processing of the receiving unit 501, the attribution unit 502, the detection unit 503, the adjustment unit 504, and the mixing unit 505 in the mixing device 500 and the technical effects brought by the specific processing can be respectively referred to the specific processing of the receiving unit 501, the attribution unit 502, the detection unit 503, the adjustment unit 504, and the mixing unit 505 and the technical effects brought by the specific processing in the corresponding embodiments. Figure 1 The corresponding embodiments of the steps 101, 102, 103, 104, and 105 are not described herein again.

[0079] In some optional implementations of the embodiment, the device 500 further includes a filtering unit (not shown in the figure). The filtering unit can be configured to perform filtering processing on the initial audio streams in the plurality of period windows to remove the initial audio streams whose timestamps do not satisfy the time of all the period windows.

[0080] In some optional implementations of the embodiment, the attribution unit 502 includes an initial determination module (not shown in the figure), an association module (not shown in the figure), and a storage module (not shown in the figure). The initial determination module can be configured to determine the period window corresponding to each initial audio stream based on the order of the receiving time of each initial audio stream. The association module can be configured to associate, for each initial audio stream, the timestamp of the initial audio stream with the period window corresponding to the initial audio stream to obtain an association relationship between the timestamp and the period window. The storage unit can be configured to store the association relationship in a hash table so as to attribute each initial audio stream to the corresponding period window.

[0081] In some optional implementation forms of the embodiment, the adjusting unit 504 comprises a first determining module (not shown in the figure), a changing module (not shown in the figure) and an adjusting module (not shown in the figure). The first determining module can be configured to determine, based on the timestamp of the first audio stream, a period window corresponding to the timestamp. The changing module can be configured to change the association relationship based on the first audio stream and the period window corresponding to the timestamp. The adjusting module can be configured to adjust the first audio stream into the period window corresponding to the timestamp.

[0082] In some optional implementation forms of the embodiment, the mixing unit 505 can be further configured to mix the initial audio streams belonging to any one of the period windows to obtain the mixed audio corresponding to the period window after the time of the period window increases by the mixing delay.

[0083] The mixing device provided by the embodiment of the disclosure first receives the initial audio streams of multiple different time points by the receiving unit 501. Then, the attributing unit 502 attributes each initial audio stream to one of the period windows, and the period windows are obtained based on the mixing delay. Next, the detecting unit 503 detects whether there is a first audio stream not corresponding to the period window based on the timestamp of each initial audio stream. The initial audio stream comprises the first audio stream. Then, the adjusting unit 504 adjusts the first audio stream to the corresponding period window in response to the detection of the first audio stream. Finally, the mixing unit 505 mixes the initial audio streams belonging to all the period windows based on the period windows and the mixing delay to obtain the mixed audio. Thus, after receiving the multiple initial audio streams, the mixing delay is divided into multiple period windows based on the mixing delay, the received audio streams are attributed to each period window, and the audio streams not corresponding to each period window are adjusted based on the timestamp of the audio stream. The audio stream with a transmission time delay can be adjusted to the appropriate period window, thereby suppressing the occurrence of the phenomenon that the multiple audio streams are out of synchronization due to the delay of the audio stream, and improving the quality of the mixed audio.

[0084] In the technical solution of the disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the user's personal information comply with the relevant laws and regulations and do not violate public order and good customs.

[0085] According to the embodiments of the disclosure, the disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0086] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0087] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0088] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0089] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the sound mixing method. For example, in some embodiments, the sound mixing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the sound mixing method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the sound mixing method by any other appropriate means, such as by means of firmware.

[0090] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0091] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable computing apparatus to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / operations specified in the flowcharts and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0092] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0094] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0095] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0096] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0097] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A mixing method, comprising: Receive multiple initial audio streams at different times, and perform echo cancellation and noise suppression on each initial audio stream; assigning each initial audio stream to one of a plurality of periodic windows, wherein the plurality of periodic windows are obtained based on a mixing delay; Based on the timestamp of each initial audio stream, detecting whether each periodic window includes a first audio stream that does not correspond to the periodic window, the initial audio stream including the first audio stream, wherein the timestamp is carried by the initial audio stream itself or determined by communicating with an audio device that sends the initial audio stream; In response to detecting the first audio stream, adjusting the first audio stream to a corresponding periodic window includes: canceling the affiliation between the first audio stream and the original periodic window and affiliating the first audio stream to the corresponding periodic window; Based on the multiple periodic windows and the mixing delay, mixing the initial audio streams belonging to all periodic windows to obtain mixed audio; The step of assigning each initial audio stream to one of the multiple periodic windows includes: Determining a periodic window corresponding to each initial audio stream based on a sequence of reception times of the initial audio streams; For each initial audio stream, associating the timestamp of the initial audio stream with the periodic window corresponding to the initial audio stream to obtain an association relationship between the timestamp and the periodic window; The association relationship is stored in a hash table so that each initial audio stream is assigned to a corresponding period window.

2. The method according to claim 1, further comprising: The initial audio streams in the multiple periodic windows are filtered to remove the initial audio streams whose timestamps do not satisfy the times of all the periodic windows.

3. The method according to claim 1, wherein In response to detecting the first audio stream, adjusting the first audio stream to a corresponding period window includes: Determining, based on a timestamp of the first audio stream, a periodic window corresponding to the timestamp; modifying the association relationship based on the first audio stream and a periodic window corresponding to the timestamp; The first audio stream is adjusted to a period window corresponding to the timestamp.

4. The method according to claim 1, wherein The mixing of the initial audio streams belonging to all periodic windows based on the multiple periodic windows and the mixing delay to obtain mixed audio includes: After the time of any one of the multiple period windows increases to the mixing delay, the initial audio stream belonging to the period window is mixed to obtain a mixed audio corresponding to the period window.

5. A sound mixing device, comprising: The receiving unit is configured to receive multiple initial audio streams at different times and perform echo cancellation and noise suppression on each initial audio stream; A classifying unit configured to classify each initial audio stream into one of a plurality of periodic windows, wherein the plurality of periodic windows are obtained based on a mixing delay; a detection unit configured to detect, based on a timestamp of each initial audio stream, whether each periodic window includes a first audio stream that does not correspond to the periodic window, the initial audio stream including the first audio stream, wherein the timestamp is carried by the initial audio stream itself or determined by communicating with an audio device that sends the initial audio stream; an adjusting unit configured to, in response to detecting the first audio stream, adjust the first audio stream to a corresponding periodic window, comprising: canceling an affiliation between the first audio stream and an original periodic window, and affiliating the first audio stream to the corresponding periodic window; a mixing unit configured to mix the initial audio streams belonging to all periodic windows based on the multiple periodic windows and the mixing delay to obtain mixed audio; The attribution unit includes: an initial determination module configured to determine a periodic window corresponding to each initial audio stream based on a sequence of reception times of the initial audio streams; an associating module configured to associate, for each initial audio stream, a timestamp of the initial audio stream with a periodic window corresponding to the initial audio stream, to obtain an association relationship between the timestamp and the periodic window; The storage unit is configured to store the association relationship in a hash table so that each initial audio stream belongs to a corresponding period window.

6. The apparatus according to claim 5, further comprising: The filtering unit is configured to filter the initial audio streams in the multiple period windows to remove the initial audio streams whose timestamps do not meet the times of all period windows.

7. The device according to claim 5, wherein The adjustment unit includes: A first determining module is configured to determine, based on a timestamp of the first audio stream, a periodic window corresponding to the timestamp; a changing module configured to change the association relationship based on the first audio stream and a period window corresponding to the timestamp; The adjustment module is configured to adjust the first audio stream to a period window corresponding to the timestamp.

8. The device according to claim 5, wherein The mixing unit is further configured to mix the initial audio stream belonging to any periodic window among the multiple periodic windows after the time of the periodic window increases to the mixing delay to obtain the mixed audio corresponding to the periodic window.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.

11. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 4.

Images