Multi-channel digital audio mixing method and system for online video conference

Abstract

The invention discloses a multi-channel digital audio mixing method and system for an online video conference, and the method comprises the steps: calculating the audio energy of a current frame in each channel of audio stream, and storing the audio energy in an audio energy sequence; acquiring position coordinates of face key points from a video picture corresponding to a current frame of each path of audio stream, calculating a pronunciation action value of the current frame, and storing the pronunciation action value in a pronunciation action value sequence; calculating a speech evaluation value of the current frame in each path of audio stream according to the audio energy sequence and the pronunciation action value sequence; calculating the conference participation degree of the current frame in each path of audio stream according to the speaking evaluation value of the current frame; and calculating an audio attenuation coefficient of the current frame in each path of audio stream according to the conference participation degree, the speaking evaluation value and the time domain sampling amplitude of the current frame, and outputting audio mixing data according to the audio attenuation coefficient. The invention relates to the technical field of video conferences and audio synthesis, and can effectively eliminate audio overflow, reduce volume attenuation and reduce environmental sound interference without influencing the audio of a main spokesman when multi-channel digital audio signals of a conference are mixed.

Description

Technical Field

[0001] The present invention relates to a multi-channel digital audio mixing method and system for an online video conference, and relates to the technical field of video conference and audio synthesis. Background Art

[0002] In a multi-person online video conference, a common scenario is that when dozens of audio channels are simultaneously using microphones, the sound heard is very chaotic. So, when multiple audio channels exist at the same time, how to mix the audio signals of multiple audio streams? There are several technical solutions:

[0003] Technical solution 1: directly add multiple audio streams, but this technical solution will cause overflow problems, which will significantly reduce the audio quality;

[0004] Technical Solution 2: Average the audio streams of multiple channels. This technical solution can eliminate the overflow problem, but it will significantly reduce the volume of each audio channel.

[0005] Technical Solution 3: Adopting the adaptive attenuation coefficient solution, that is, when multiple audio streams are directly added together and overflow occurs, an attenuation coefficient is introduced to attenuate the mixed audio stream, and then the attenuation factor is gradually eliminated. This technical solution can improve the problem of reduced volume when only one audio stream is speaking, but when multiple audio streams exist at the same time, the problem of reduced volume will still occur;

[0006] Technical solution 4: Use the newlc algorithm for mixing. This mixing solution uses an empirical formula for mixing, which has a good effect when mixing two channels, but the quality of multi-channel mixing is significantly reduced.

[0007] The above technical solutions will also consider the ambient sound of the participants as valid audio and mix it into the conference audio. Therefore, when mixing multiple digital audio signals of online video conferences, how to effectively solve the problem of audio overflow, volume attenuation, background environmental sound interference, etc. that cause the audio quality to deteriorate after mixing has become a technical issue that technical personnel focus on. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a multi-channel digital audio mixing method and system for online video conferencing, which can effectively eliminate audio overflow, reduce volume attenuation, and reduce interference from ambient sound when mixing multi-channel digital audio signals for online video conferencing, and ensure that the audio of the main speaker is not affected as much as possible in the conference scenario.

[0009] In order to achieve the above object, the present invention provides a multi-channel digital audio mixing method for an online video conference, when multiple audio streams collected and sent by all participating terminals in real time are received during the online video conference, including:

[0010] Step 1: Calculate the audio energy of the current frame audio signal in each audio stream and save it in the audio energy sequence of each audio stream. The audio energy sequence is used to save the audio energy of all frame audio signals in the audio stream according to the time sequence of each frame audio signal.

[0011] Step 2: Obtain the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in each audio stream, and then calculate the pronunciation action value of the current frame audio signal in each audio stream according to the position coordinates of the key points of the face, and save them in the pronunciation action value sequence of each audio stream, wherein the pronunciation action value is used to represent the action value of the person in the screen when speaking, and the pronunciation action value sequence is used to save the pronunciation action values ​​of all the audio frames in the audio stream according to the time sequence of each frame audio signal;

[0012] Step 3: Calculate the speech evaluation value of the current frame audio signal in each audio stream according to the audio energy sequence and the pronunciation action value sequence of each audio stream, wherein the speech evaluation value is used to characterize the speaking probability of the speaker in the audio stream within a period of time;

[0013] Step 4: Calculate the conference participation degree of the current frame audio signal in each audio stream according to the speech evaluation value of the current frame audio signal in each audio stream, where the conference participation degree is used to characterize the conference participation degree of the participant in the conference up to the current time;

[0014] Step 5: Calculate the audio attenuation coefficient of the current frame audio signal in each audio stream based on the conference participation, speech evaluation value and time domain sampling amplitude of the current frame audio signal in each audio stream, and calculate and output the mixing data of the current frame audio signals of all audio streams accordingly.

[0015] In order to achieve the above object, the present invention also provides a multi-channel digital audio mixing system for online video conferencing, comprising:

[0016] An audio stream receiving device receives multiple audio streams collected and sent in real time by all participating terminals during an online video conference;

[0017] An audio energy calculation device calculates the audio energy of the current frame audio signal in each audio stream and saves it in the audio energy sequence of each audio stream. The audio energy sequence is used to save the audio energy of all frame audio signals in the audio stream according to the time sequence of each frame audio signal;

[0018] A pronunciation action value calculation device obtains the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in each audio stream, and then calculates the pronunciation action value of the current frame audio signal in each audio stream according to the position coordinates of the key points of the face, and saves it in the pronunciation action value sequence of each audio stream, wherein the pronunciation action value is used to represent the action value of the person in the screen when speaking, and the pronunciation action value sequence is used to save the pronunciation action values ​​of all the audio frames in the audio stream according to the time sequence of each audio frame;

[0019] A speech evaluation value calculation device calculates a speech evaluation value of a current frame audio signal in each audio stream according to an audio energy sequence and a pronunciation action value sequence of each audio stream, wherein the speech evaluation value is used to characterize a speaking probability of a speaker in the audio stream within a period of time;

[0020] A conference participation calculation device, which calculates the conference participation of the current frame audio signal in each audio stream according to the speech evaluation value of the current frame audio signal in each audio stream, wherein the conference participation is used to characterize the conference participation degree of the participant in the conference up to the current time;

[0021] The mixing calculation device calculates the audio attenuation coefficient of the current frame audio signal in each audio stream according to the conference participation, speech evaluation value and time domain sampling amplitude of the current frame audio signal in each audio stream, and calculates and outputs the mixing data of the current frame audio signals of all audio streams accordingly.

[0022] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0023] Memory and processor;

[0024] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the multi-channel digital audio mixing method for online video conferencing are implemented.

[0025] In order to achieve the above-mentioned object, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the multi-channel digital audio mixing method for online video conferencing are implemented.

[0026] Compared with the prior art, the beneficial effect of the present invention is: the present invention proposes an audio mixing technology solution that fully meets the online conference scenario, which can effectively eliminate the audio overflow problem while minimizing the volume attenuation problem, greatly reduce the interference of ambient sound, and ensure that the audio of the main speaker is not affected as much as possible in the conference scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1The present invention is a flowchart of a method for mixing multi-channel digital audio in an online video conference, shown as an exemplary embodiment of the present invention.

[0028] Figure 2 An exemplary embodiment of the present invention is shown Figure 1 Detailed process flow chart of step five.

[0029] Figure 3 It is a structural schematic diagram of a multi-channel digital audio mixing system for online video conferencing shown in an exemplary embodiment of the present invention.

[0030] Figure 4 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, the present invention provides a multi-channel digital audio mixing method for an online video conference, when receiving multi-channel audio streams collected and sent by all participating terminals in real time during the online video conference, including:

[0033] Step 1: Calculate the audio energy of the current frame audio signal in each audio stream and save it in the audio energy sequence of each audio stream. The audio energy sequence is used to save the audio energy of all frame audio signals in the audio stream according to the time sequence of each frame audio signal.

[0034] Step 2: Obtain the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in each audio stream, and then calculate the pronunciation action value of the current frame audio signal in each audio stream according to the position coordinates of the key points of the face, and save them in the pronunciation action value sequence of each audio stream, wherein the pronunciation action value is used to represent the action value of the person in the screen when speaking, and the pronunciation action value sequence is used to save the pronunciation action values ​​of all the audio frames in the audio stream according to the time sequence of each frame audio signal;

[0035] Step 3: Calculate the speech evaluation value of the current frame audio signal in each audio stream according to the audio energy sequence and the pronunciation action value sequence of each audio stream, wherein the speech evaluation value is used to characterize the speaking probability of the speaker in the audio stream within a period of time;

[0036] Step 4: Calculate the conference participation degree of the current frame audio signal in each audio stream according to the speech evaluation value of the current frame audio signal in each audio stream, where the conference participation degree is used to characterize the conference participation degree of the participant in the conference up to the current time;

[0037] Step 5: Calculate the audio attenuation coefficient of the current frame audio signal in each audio stream based on the conference participation, speech evaluation value and time domain sampling amplitude of the current frame audio signal in each audio stream, and calculate and output the mixing data of the current frame audio signals of all audio streams accordingly.

[0038] In step 1, calculating the audio energy of the current frame audio signal in each audio stream may further include:

[0039] The audio PCM time domain sampling data of the current frame audio signal in each audio stream is obtained, and the square sum of all components in the audio PCM time domain sampling data is calculated, where the square sum is the audio energy of the current frame audio signal in each audio stream.

[0040] In step 2, taking the k-th audio stream as an example, obtaining the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in the k-th audio stream, and then calculating the pronunciation action value of the current frame audio signal in the k-th audio stream according to the position coordinates of the key points of the face, which may further include:

[0041] Step 21, identifying whether there is a face in the video picture corresponding to the current frame audio signal in the k-th audio stream, if yes, extracting and saving the position coordinates of the key point of the center of the face and the position coordinates of multiple key points of the lips from the video picture, and then continuing to step 22, if no, setting the pronunciation action value of the current frame audio signal in the k-th audio stream to 0, and ending this process;

[0042] Step 22: extract the position coordinates of the key point of the center of the face and the position coordinates of multiple key points of the lips in the video image corresponding to the previous frame audio signal of the current frame of the k-th audio stream, and then calculate the lip motion value Vc of the current frame audio signal in the k-th audio stream. k (i) and the overall action value Vz k (i)

[0043] Thus, the pronunciation action value V of the current frame audio signal in the k-th audio stream is calculated. k (i):V k (i) = Vc k (i)*(1+Vz k (i)), where i represents the sequence number of the current frame in the k-th audio stream, i-1 represents the sequence number of the previous frame of the current frame, and s k (n,i),s k (n, i-1) are the distances from the nth lip key point to the face center key point in the video image corresponding to the current frame and the previous frame of the audio signal of the kth audio stream, respectively. N is the number of lip key points, z k _x(i),z k_y(i) are the horizontal and vertical coordinates of the key point of the face center corresponding to the audio signal of the current frame of the k-th audio stream, z k _x(i-1),z k _y(i-1) are the horizontal and vertical coordinates of the key point of the center of the face corresponding to the previous frame audio signal of the current frame of the k-th audio stream. If there is no face in the video picture corresponding to the previous frame audio signal of the k-th audio stream, the pronunciation action value of the current frame audio signal in the k-th audio stream is directly set to 0.

[0044] In step 3, the calculation formula of the speech evaluation value of the current frame audio signal in each audio stream can be as follows: Where i represents the sequence number of the current frame in each audio stream, Sp k (i) is the speech evaluation value of the current frame audio signal in the k-th audio stream, Pv k (ia), V k (ia) are the audio energy and pronunciation action value of the audio signal of the a frames before the current frame (i.e., the iath frame) in the kth audio stream, respectively. A is the total number of all frames of each audio stream before the current frame and within a certain preset historical time range. For example, the preset historical time range is T seconds, and the time interval of the audio signal frame is t, then A is The rounded value of .

[0045] Step 4 may further include:

[0046] Step 41: Count the sum of speech evaluation values ​​of the current frame audio signal in all audio streams: K is the total number of audio streams, and then the participation gain of the current frame audio signal in each audio stream is calculated: C k is the participation gain of the current frame audio signal in the k-th audio stream;

[0047] Step 42: extract the conference participation degree of the previous frame and the previous two frames of audio signals of the current frame in each audio stream, calculate and save the conference participation degree of the current frame audio signal in each audio stream: M k (i) M k (i-1), M k (i-2) are the conference participation degrees of the current frame, previous frame, and previous two frames of audio signals in the kth audio stream. If the current frame is the 0th or 1st frame of the corresponding audio stream, the conference participation degree of the current frame of audio signals in the audio stream is set to 0.

[0048] like Figure 2 As shown, step five may further include:

[0049] Step 51: Count the sum of the conference participation Ms(i) and the sum of the speech evaluation values ​​Sps(i) of the current frame audio signal in all audio streams:

[0050] Step 52, initializing the iteration coefficient d to 0;

[0051] Step 53: Calculate the audio attenuation coefficient of the current frame audio signal in each audio stream: f k (i) f k (i-1) are the audio attenuation coefficients of the current frame and the previous frame of the audio signal in the k-th audio stream, and then the mixing results of the current frame audio signals of all audio streams are calculated: x k (i) is the time domain sampling amplitude of the current frame audio signal in the k-th audio stream;

[0052] Step 54, determine whether mix is ​​greater than MAX, MAX is the maximum positive amplitude value, and its value can be set according to actual business needs. If yes, proceed to step 55; if not, proceed to step 56;

[0053] Step 55, update d to d+1, then modify the audio attenuation coefficient of the current frame audio signal in each audio stream according to the updated d, then update the mixing result mix of the current frame audio signals of all audio streams, and finally determine whether the updated mix is ​​greater than MAX, if yes, proceed to step 55, if not, proceed to step 59;

[0054] By gradually increasing d by 1, we can find the smallest positive integer that does not cause overflow in the mixing.

[0055] Step 56: determine whether mix is ​​less than MIN, where MIN is the minimum negative value of the amplitude, and its value can be set according to actual business needs. If yes, proceed to step 57; if not, proceed to step 58;

[0056] Step 57, update d to d+1, then modify the audio attenuation coefficient of the current frame audio signal in each audio stream according to the updated d, then update the mixing result mix of the current frame audio signal of all audio streams, and finally determine whether the updated mix is ​​less than MIN, if yes, proceed to step 57, if not, proceed to step 59;

[0057] Step 58: Set d to 0, then modify the audio attenuation coefficient of the current frame audio signal in each audio stream according to the updated d, and then update the mixing results of the current frame audio signals of all audio streams;

[0058] Step 59: output the updated mixing results of the current frame audio signals of all audio streams. At the same time, according to d and the audio attenuation coefficient of the previous frame audio signal in the audio stream, modify and save the audio attenuation coefficient of the current frame audio signal in each audio stream.

[0059] In the above steps 55, 57, and 58, the calculation formulas for modifying the audio attenuation coefficient of the current frame audio signal in each audio stream and updating the mixing results of the current frame audio signals of all audio streams are the same as those in step 53, so they are not repeated here.

[0060] In step 59, modifying the audio attenuation coefficient of the current frame audio signal in each audio stream may further include:

[0061] Determine whether d is 0. If so, determine whether the audio attenuation coefficient of the previous audio signal of the current frame in each audio stream is less than 1. If it is less than 1, modify the audio attenuation coefficient of the current audio signal in the audio stream: f k ‘ (i) = f k (i-1)+(1-f k (i-1)) / 7, f k ‘ (i) is the modified audio attenuation coefficient of the current frame audio signal in the k-th audio stream.

[0062] like Figure 3 As shown, the present invention provides a multi-channel digital audio mixing system for online video conferencing, comprising:

[0063] An audio stream receiving device receives multiple audio streams collected and sent in real time by all participating terminals during an online video conference;

[0064] An audio energy calculation device calculates the audio energy of the current frame audio signal in each audio stream and saves it in the audio energy sequence of each audio stream. The audio energy sequence is used to save the audio energy of all frame audio signals in the audio stream according to the time sequence of each frame audio signal;

[0065] A pronunciation action value calculation device obtains the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in each audio stream, and then calculates the pronunciation action value of the current frame audio signal in each audio stream according to the position coordinates of the key points of the face, and saves it in the pronunciation action value sequence of each audio stream, wherein the pronunciation action value is used to represent the action value of the person in the screen when speaking, and the pronunciation action value sequence is used to save the pronunciation action values ​​of all the audio frames in the audio stream according to the time sequence of each audio frame;

[0066] A speech evaluation value calculation device calculates a speech evaluation value of a current frame audio signal in each audio stream according to an audio energy sequence and a pronunciation action value sequence of each audio stream, wherein the speech evaluation value is used to characterize a speaking probability of a speaker in the audio stream within a period of time;

[0067] A conference participation calculation device, which calculates the conference participation of the current frame audio signal in each audio stream according to the speech evaluation value of the current frame audio signal in each audio stream, wherein the conference participation is used to characterize the conference participation degree of the participant in the conference up to the current time;

[0068] The mixing calculation device calculates the audio attenuation coefficient of the current frame audio signal in each audio stream according to the conference participation, speech evaluation value and time domain sampling amplitude of the current frame audio signal in each audio stream, and calculates and outputs the mixing data of the current frame audio signals of all audio streams accordingly.

[0069] See also Figure 4 , Figure 4 4 is a block diagram of a computing device 400 shown in an exemplary embodiment of this specification. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0070] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0071] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0072] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 400 may also be a mobile or stationary server or a cloud server, etc.

[0073] The processor 420 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the multi-channel digital audio mixing method for online video conferencing.

[0074] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multi-channel digital audio mixing method of the online video conference are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the multi-channel digital audio mixing method of the online video conference.

[0075] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multi-channel digital audio mixing method for online video conferencing.

[0076] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the multi-channel digital audio mixing method of the online video conference are of the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the multi-channel digital audio mixing method or system of the online video conference.

[0077] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the multi-channel digital audio mixing method for online video conferencing.

[0078] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the multi-channel digital audio mixing method of the online video conference are of the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical scheme of the multi-channel digital audio mixing method or system of the online video conference.

[0079] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0081] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-channel digital audio mixing method for online video conferencing, characterized in that: When multiple audio streams are received from all participating terminals in real time during an online video conference, including: Step 1: Calculate the audio energy of the current frame audio signal in each audio stream and save it in the audio energy sequence of each audio stream. The audio energy sequence is used to save the audio energy of all frame audio signals in the audio stream according to the time sequence of each frame audio signal. Step 2: Obtain the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in each audio stream, and then calculate the pronunciation action value of the current frame audio signal in each audio stream according to the position coordinates of the key points of the face, and save them in the pronunciation action value sequence of each audio stream, wherein the pronunciation action value is used to represent the action value of the person in the screen when speaking, and the pronunciation action value sequence is used to save the pronunciation action values ​​of all the audio frames in the audio stream according to the time sequence of each frame audio signal; Step 3: Calculate the speech evaluation value of the current frame audio signal in each audio stream according to the audio energy sequence and the pronunciation action value sequence of each audio stream, wherein the speech evaluation value is used to characterize the speaking probability of the speaker in the audio stream within a period of time; Step 4: Calculate the conference participation degree of the current frame audio signal in each audio stream according to the speech evaluation value of the current frame audio signal in each audio stream, where the conference participation degree is used to characterize the conference participation degree of the participant in the conference up to the current time; Step 5: Calculate the audio attenuation coefficient of the current frame audio signal in each audio stream based on the conference participation, speech evaluation value and time domain sampling amplitude of the current frame audio signal in each audio stream, and calculate and output the mixing data of the current frame audio signals of all audio streams accordingly.

2. The method according to claim 1, characterized in that In step 1, the audio energy of the current frame audio signal in each audio stream is calculated, which further includes: The audio PCM time domain sampling data of the current frame audio signal in each audio stream is obtained, and the square sum of all components in the audio PCM time domain sampling data is calculated, where the square sum is the audio energy of the current frame audio signal in each audio stream.

3. The method according to claim 1, characterized in that In step 2, the position coordinates of the key points of the face are obtained from the video picture corresponding to the current frame audio signal in the k-th audio stream, and then the pronunciation action value of the current frame audio signal in the k-th audio stream is calculated according to the position coordinates of the key points of the face, which further includes: Step 21, identifying whether there is a face in the video picture corresponding to the current frame audio signal in the k-th audio stream, if yes, extracting and saving the position coordinates of the key point of the center of the face and the position coordinates of multiple key points of the lips from the video picture, and then continuing to step 22, if no, setting the pronunciation action value of the current frame audio signal in the k-th audio stream to 0, and ending this process; Step 22: extract the position coordinates of the face center key point and the position coordinates of multiple lip key points in the video image corresponding to the previous audio signal of the current frame of the k-th audio stream, and then calculate the lip motion value Vc of the current audio signal of the k-th audio stream. k (i) and the overall action value Vz k (i) Thus, the pronunciation action value V of the current frame audio signal in the k-th audio stream is calculated. k (i):V k (i) = Vc k (i)*(1+Vz k (i)), where i represents the sequence number of the current frame in the k-th audio stream, i-1 represents the sequence number of the previous frame of the current frame, and s k (n,i),s k (n, i-1) are the distances from the nth lip key point to the face center key point in the video image corresponding to the current frame and the previous frame of the audio signal of the kth audio stream, respectively. N is the number of lip key points, z k _x(i),z k _y(i) are the horizontal and vertical coordinates of the key point of the face center corresponding to the audio signal of the current frame of the k-th audio stream, z k _x(i-1),z k _y(i-1) are the horizontal and vertical coordinates of the key point of the center of the face corresponding to the previous frame audio signal of the current frame of the k-th audio stream. If there is no face in the video picture corresponding to the previous frame audio signal of the k-th audio stream, the pronunciation action value of the current frame audio signal in the k-th audio stream is directly set to 0.

4. The method according to claim 1, characterized in that: In step 3, the calculation formula of the speech evaluation value of the current frame audio signal in each audio stream is as follows: Where i represents the sequence number of the current frame in each audio stream, Sp k (i) is the speech evaluation value of the current frame audio signal in the k-th audio stream, Pv k (ia), V k (ia) are respectively the audio energy and pronunciation action value of the audio signal of the previous a frames of the current frame in the k-th audio stream, and A is the total number of all frames of each audio stream before the current frame and within a certain preset historical time range.

5. The method according to claim 1, characterized in that Step 4 further includes: Step 41: Count the sum of speech evaluation values ​​of the current frame audio signal in all audio streams: i represents the sequence number of the current frame in each audio stream, Sp k (i) is the speech evaluation value of the current frame audio signal in the k-th audio stream, K is the total number of audio streams, and then the participation gain of the current frame audio signal in each audio stream is calculated: C k is the participation gain of the current frame audio signal in the k-th audio stream; Step 42: extract the conference participation degree of the previous frame and the previous two frames of audio signals of the current frame in each audio stream, calculate and save the conference participation degree of the current frame audio signal in each audio stream: M k (i) M k (i-1), M k (i-2) are the conference participation degrees of the current frame, previous frame, and previous two frames of audio signals in the kth audio stream. If the current frame is the 0th or 1st frame of the corresponding audio stream, the conference participation degree of the current frame of audio signals in the audio stream is set to 0.

6. The method according to claim 1, characterized in that Step five further includes: Step 51: Count the sum of the conference participation Ms(i) and the sum of the speech evaluation values ​​Sps(i) of the current frame audio signal in all audio streams: i represents the sequence number of the current frame in each audio stream, M k (i) is the conference participation degree of the current frame audio signal in the k-th audio stream, Sp k (i) is the speech evaluation value of the current frame of the audio signal in the k-th audio stream, and K is the total number of audio streams; Step 52, initializing the iteration coefficient d to 0; Step 53: Calculate the audio attenuation coefficient of the current frame audio signal in each audio stream: f k (i) f k (i-1) are the audio attenuation coefficients of the current frame and the previous frame of the audio signal in the k-th audio stream, and then the mixing results of the current frame audio signals of all audio streams are calculated: x k (i) is the time domain sampling amplitude of the current frame audio signal in the k-th audio stream; Step 54, determine whether mix is ​​greater than MAX, MAX is the maximum positive amplitude value, if yes, proceed to step 55; if no, proceed to step 56; Step 55, update d to d+1, then modify the audio attenuation coefficient of the current frame audio signal in each audio stream according to the updated d, then update the mixing result mix of the current frame audio signals of all audio streams, and finally determine whether the updated mix is ​​greater than MAX, if yes, proceed to step 55, if not, proceed to step 59; Step 56: determine whether mix is ​​less than MIN, where MIN is the minimum negative value of the amplitude. If yes, proceed to step 57; if not, proceed to step 58; Step 57, update d to d+1, then modify the audio attenuation coefficient of the current frame audio signal in each audio stream according to the updated d, then update the mixing result mix of the current frame audio signal of all audio streams, and finally determine whether the updated mix is ​​less than MIN, if yes, proceed to step 57, if not, proceed to step 59; Step 58: Set d to 0, then modify the audio attenuation coefficient of the current frame audio signal in each audio stream according to the updated d, and then update the mixing results of the current frame audio signals of all audio streams; Step 59: output the updated mixing results of the current frame audio signals of all audio streams. At the same time, according to d and the audio attenuation coefficient of the previous frame audio signal in the audio stream, modify and save the audio attenuation coefficient of the current frame audio signal in each audio stream.

7. The method according to claim 6, characterized in that In step 59, the audio attenuation coefficient of the current frame audio signal in each audio stream is modified, further comprising: Determine whether d is 0. If so, determine whether the audio attenuation coefficient of the previous audio signal of the current frame in each audio stream is less than 1. If it is less than 1, modify the audio attenuation coefficient of the current audio signal in the audio stream: f k ‘ (i) = f k (i-1)+(1-f k (i-1)) / 7, f k ‘ (i) is the modified audio attenuation coefficient of the current frame audio signal in the k-th audio stream.

8. A multi-channel digital audio mixing system for online video conferencing, characterized in that: Included are: An audio stream receiving device receives multiple audio streams collected and sent in real time by all participating terminals during an online video conference; An audio energy calculation device calculates the audio energy of the current frame audio signal in each audio stream and saves it in the audio energy sequence of each audio stream. The audio energy sequence is used to save the audio energy of all frame audio signals in the audio stream according to the time sequence of each frame audio signal; A pronunciation action value calculation device obtains the position coordinates of the key points of the face from the video screen corresponding to the current frame audio signal in each audio stream, and then calculates the pronunciation action value of the current frame audio signal in each audio stream according to the position coordinates of the key points of the face, and saves it in the pronunciation action value sequence of each audio stream, wherein the pronunciation action value is used to represent the action value of the person in the screen when speaking, and the pronunciation action value sequence is used to save the pronunciation action values ​​of all the audio frames in the audio stream according to the time sequence of each audio frame; A speech evaluation value calculation device calculates a speech evaluation value of a current frame audio signal in each audio stream according to an audio energy sequence and a pronunciation action value sequence of each audio stream, wherein the speech evaluation value is used to characterize a speaking probability of a speaker in the audio stream within a period of time; A conference participation calculation device, which calculates the conference participation of the current frame audio signal in each audio stream according to the speech evaluation value of the current frame audio signal in each audio stream, wherein the conference participation is used to characterize the conference participation degree of the participant in the conference up to the current time; The mixing calculation device calculates the audio attenuation coefficient of the current frame audio signal in each audio stream according to the conference participation, speech evaluation value and time domain sampling amplitude of the current frame audio signal in each audio stream, and calculates and outputs the mixing data of the current frame audio signals of all audio streams accordingly.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multi-channel digital audio mixing method for online video conferencing described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the multi-channel digital audio mixing method for online video conferencing as described in any one of claims 1-7.

Images