Multi-channel audio and video signal processing method and device
By calculating the audio signal energy and voice activity detection information, selecting and enhancing the target channel, solving the problem of noisy sounds in multi-person audio and video calls, and improving the user's hearing and communication efficiency.
Patent Information
- Application Number
- CN202411161729.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-23
AI Technical Summary
In multi-person audio and video calls or conference scenarios, multiple audio signals cause noisy sound, making it difficult to highlight the target audio being spoken, affecting the quality of communication.
By calculating the audio signal energy and voice activity detection information of all input channels, selecting and enhancing the target channel, and attenuating other channels, the reasonable allocation of audio signals and mixing output of audio signals are achieved.
Significantly improve users' hearing feelings, simple and smooth online communication, and save users' time and cost.
Smart Images

Figure CN118692498B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio and video processing, and in particular to a multi-channel-based audio and video signal processing method and device. Background Art
[0002] In multi-person audio and video calls, audio and video teaching, or audio and video conferences, a common scenario is multi-person online interaction. However, when there are dozens of audio channels with microphones turned on at the same time, the sound heard is very messy, and the various types of equipment and different speaking environments further aggravate the noise. In actual application scenarios, at the same time, you often only need to pay attention to the speech content of a few people, and the information of most other users can be ignored, or users of multiple channels take turns to speak on stage.
[0003] When multiple audios exist simultaneously, how to select and enhance the multi-channel audio signals so as to highlight the target audio being spoken, weaken the interference of other audios, and ensure the reception quality of the target audio has become an urgent problem to be solved by technical personnel in this field. Summary of the invention
[0004] The present invention provides a multi-channel-based audio and video signal processing method and device, which can significantly improve the user's auditory experience by selecting a target audio channel and enhancing the audio, making the user's online communication more concise and smooth, and saving the user's communication cost.
[0005] The present invention provides a multi-channel audio and video signal processing method, the method comprising:
[0006] Calculate the energy and voice activity detection information of all input channel audio signals;
[0007] Determining a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection;
[0008] When the target channels meet the quantity requirement, the target channels are audio enhanced, and the remaining channels are audio attenuated;
[0009] Mix the audio from all input channels and output them.
[0010] In some embodiments, calculating the audio signal energy of all input channels specifically includes:
[0011] Calculate the maximum energy of each audio sampling point in a single frame of the current channel;
[0012] Calculate the average value of the maximum energy of all frames within the preset duration of the current channel;
[0013] Perform quantization and shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;
[0014] Each input channel is used as the current channel in turn to obtain the audio signal energy of all input channels.
[0015] In some embodiments, calculating voice activity detection information of all input channels specifically includes:
[0016] Resample the audio signal of the current channel to a preset sampling frequency, divide it into multiple frequency bands according to the Mel-frequency cepstrum, and calculate the energy information in each frequency band;
[0017] The Gaussian mixture model is used to calculate the Gaussian probability density function of speech and noise in each sub-band respectively;
[0018] Calculate the log-likelihood ratio of speech and noise, and sum the log-likelihood ratios of all sub-bands;
[0019] Based on the relationship between a preset threshold and the sum of log-likelihood ratios of all sub-bands, the speech activity detection of the current channel is calibrated;
[0020] Each input channel is used as the current channel in turn to obtain voice activity detection information of all input channels.
[0021] In some embodiments, determining multiple target channels according to the calculation results of the audio signal energy and the voice activity specifically includes:
[0022] According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;
[0023] According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.
[0024] In some embodiments, when the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, specifically including:
[0025] When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels;
[0026] When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels;
[0027] The number of candidate channels is the difference between the target channel and the preset number.
[0028] In some embodiments, selecting a candidate channel from the remaining channels as a supplementary target channel specifically includes:
[0029] According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;
[0030] According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel;
[0031] When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.
[0032] The present invention also provides a multi-channel audio and video signal processing device, the device comprising:
[0033] A parameter calculation unit, used to calculate the audio signal energy and voice activity detection information of all input channels;
[0034] A channel selection unit, configured to determine a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection;
[0035] An audio processing unit, configured to perform audio enhancement on the target channel and audio attenuation on the remaining channels when the target channel meets the quantity requirement;
[0036] The result output unit is used to mix and output the audio of all input channels.
[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.
[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.
[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described above is implemented.
[0040] The multi-channel audio and video signal processing method and device provided by the present invention calculates the audio signal energy and voice activity detection information of all input channels, and determines multiple target channels according to the calculation results of the audio signal energy and the voice activity detection; when the target channels meet the quantity requirements, the target channels perform audio enhancement, the remaining channels perform audio attenuation, and the audio of all input channels is mixed and output. This method and device aims at the problem of noisy sound in the scenario where multiple people turn on the microphone at the same time during multi-person audio and video calls, audio and video teaching, or audio and video conferences. Through reasonable audio channel selection and audio enhancement, it makes it easier for users to obtain the information they need, improves the user's auditory experience, makes online communication smoother, and saves the user's time cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0042] Figure 1 It is one of the flow charts of the multi-channel audio and video signal processing method provided by the present invention;
[0043] Figure 2 This is the second flowchart of the multi-channel audio and video signal processing method provided by the present invention;
[0044] Figure 3 This is the third flow chart of the multi-channel audio and video signal processing method provided by the present invention;
[0045] Figure 4 This is the fourth flow chart of the multi-channel audio and video signal processing method provided by the present invention;
[0046] Figure 5 This is the fifth flow chart of the multi-channel audio and video signal processing method provided by the present invention;
[0047] Figure 6 This is the sixth flow chart of the multi-channel audio and video signal processing method provided by the present invention;
[0048] Figure 7 This is the effect diagram of audio enhancement and compression processing;
[0049] Figure 8 This is the effect diagram of compression after mixing;
[0050] Fig. 9It is a structural block diagram of a multi-channel audio and video signal processing device provided by the present invention;
[0051] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] For ease of understanding, Figure 1 The figure shows an online application scenario of the multi-channel audio and video signal processing method provided by the present invention. In this specific application scenario, the video window area of the client interface displays the multiple audio and video channels (i.e., all channels) received by the current user, and the on-stage user area displays the various audio and video channels (i.e., target channels) currently preparing to speak on stage. In this embodiment, the number of windows in the on-stage user area ranges from The purpose of the present invention is to select 6 channels from all video window channels for audio enhancement, and to perform audio attenuation on the remaining video window channels, so as to better obtain the information that needs attention.
[0054] The overall technical solution of the multi-channel audio and video signal processing method provided by the present invention is as follows Figure 2 First, the channels that need audio decoding are selected for the multiple input audio signals according to the energy information and the voice activity detection information, and then the channels that need audio enhancement are selected, and then the audio enhancement and audio attenuation are performed on the selected multiple audio channels, and finally the audio of all channels are mixed and output.
[0055] In a specific embodiment, Figure 3 As shown, the multi-channel audio and video signal processing method provided by the present invention comprises the following steps:
[0056] S310: Calculate audio signal energy and voice activity detection information of all input channels;
[0057] S320: determining a plurality of target channels according to the calculation result of the audio signal energy and the calculation result of the voice activity detection;
[0058] S330: When the target channel meets the quantity requirement, the target channel is audio enhanced, and the remaining channels are audio attenuated;
[0059] S340: Mix and output the audio of all input channels.
[0060] In step S310, the audio signal energy of all input channels is calculated, such as Figure 4 As shown, the specific steps include:
[0061] S410: Calculate the maximum energy of each audio sampling point in a single frame of the current channel;
[0062] S420: Calculate the average value of the maximum energy of all frames within the preset time length of the current channel;
[0063] S430: performing quantization shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;
[0064] S440: Taking each input channel as the current channel in turn to obtain audio signal energy of all input channels.
[0065] Specifically, in a specific usage scenario, when calculating the energy information of the audio signals of all channels, the purpose of calculating the audio energy is to report the audio energy information to the background, and then forward it to the receiving client by the background, so as to guide the routing strategy of the receiving end. The reporting of audio energy cannot be too frequent, otherwise the amount of data is too large, which will cause great pressure on the background. In this embodiment, the reporting period interval is selected to be once every 2 seconds, the period interval of the statistical volume is also 2 seconds, and the frame length of the audio frame is 20 milliseconds, so there are a total of 100 frames of audio data in 2 seconds.
[0066] In this embodiment, the method involves a method for calculating volume, which specifically includes the following steps:
[0067] First, calculate the maximum energy of the audio sampling points in a frame:
[0068]
[0069] In the formula, is the maximum value of the current audio frame sampling point, is the audio sampling point, the value range is , The length of the audio frame.
[0070] Then, we average the maximum values of the audio data frames within 2 seconds:
[0071]
[0072] In the formula, is the average value of the maximum value of the audio data frame within 2 seconds, It is the maximum value of the sampling point of the current audio frame.
[0073] Finally, calculate the volume:
[0074]
[0075] In the formula, The maximum volume within 2 seconds after quantization and shaping, with a value range of , It is the average value of the maximum value of the audio data frame within 2 seconds.
[0076] In step S310, the voice activity detection information of all input channels is calculated, such as Figure 5 As shown, the specific steps include:
[0077] S510: resampling the audio signal of the current channel to a preset sampling frequency, dividing it into multiple frequency bands according to the Mel-frequency cepstrum, and calculating the energy information in each frequency band;
[0078] S520: using a Gaussian mixture model to calculate the Gaussian probability density function of the speech and noise of each sub-band;
[0079] S530: Calculate the log likelihood ratio of speech and noise, and sum the log likelihood ratios of all sub-bands;
[0080] S540: calibrating the speech activity of the current channel based on a relationship between a preset threshold and the sum of log likelihood ratios of all sub-bands;
[0081] S550: Taking each input channel as the current channel in turn to obtain voice activity detection information of all input channels.
[0082] Still taking the above specific usage scenario as an example, voice activity detection is performed on the audio signals of all channels. The purpose of voice activity detection (VAD) is to distinguish between voice and non-voice. During audio transmission, VAD can guide the transmission of audio data. When the signal is judged to be voice, it is sent. When the signal is judged to be non-voice, it is stopped. This can save a lot of bandwidth and traffic.
[0083] This embodiment uses a statistical model method to perform voice activity detection, which specifically includes the following steps:
[0084] First, the audio signal is resampled to a sampling frequency of 8 kHz (Hertz), and the cutoff frequency corresponding to the signal with a sampling frequency of 8 kHz is 4 kHz. Then, the signal with a sampling frequency of 8 kHz is divided into multiple frequency bands according to the Mel cepstrum frequency, and the energy information in each frequency band is calculated. In this embodiment, the logarithmic energy is taken, and the calculation method is as follows:
[0085]
[0086] In the formula, is the subband marker, is the length mark of the subband, represents the length of the subband, represents the number of subbands, represents the logarithmic energy of the subband, Represents the sampling points within the subband.
[0087] Then, the Gaussian mixture model is used to calculate the Gaussian probability density functions of speech and noise in the N sub-bands respectively. The Gaussian probability density function of speech is calculated as follows:
[0088]
[0089]
[0090] In the formula, represents the Gaussian probability density function of speech, Indicates the number of subbands, represents the order of Gaussian mixture model, represents the logarithmic energy of the subband, , , They represent the mean, variance, and mixing weight of the Gaussian mixture model of speech respectively.
[0091] The Gaussian probability density function of noise is calculated as follows:
[0092]
[0093]
[0094] In the formula, represents the Gaussian probability density function of the noise, Indicates the number of subbands, represents the order of Gaussian mixture model, represents the logarithmic energy of the subband, , , They represent the mean, variance, and mixing weight of the Gaussian mixture model of noise respectively.
[0095] Then the log-likelihood ratio of speech and noise is calculated and summed over all subbands:
[0096]
[0097] In the formula, represents the sum of log-likelihood ratios of all subbands, Indicates the number of subbands, represents the Gaussian probability density function of speech, Gaussian probability density function representing the noise.
[0098] According to the result of VAD determined by the threshold, speech is marked as 1 and non-speech is marked as 0, as shown below:
[0099]
[0100] In the formula, represents the result of VAD, represents the sum of log-likelihood ratios of all subbands, is the VAD decision threshold.
[0101] Finally, update the mean, variance, and mixing weight of the Gaussian mixture model for the next calculation.
[0102] In step S320, multiple target channels are determined according to the calculation results of the audio signal energy and the calculation results of the voice activity detection, specifically including:
[0103] According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;
[0104] According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.
[0105] Specifically, in the selection of decoding channels, the audio codec of this embodiment adopts the opus codec. Audio decoding is performed on the client side, which consumes a lot of performance in the whole system. The system is a multi-channel system. If dozens of audio encoding signals are to be decoded at the same time, the equipment is not enough to support such a large performance pressure. And if there are too many decoding channels, it will be very time-consuming and affect the processing of subsequent modules, so selective decoding is required.
[0106] In order to take into account the performance of various low-end devices, it is found through experiments and calculations that decoding 20 audio encoding signals at the same time can basically cover various low-end devices on the market.
[0107] The principle of decoding channel selection is:
[0108] Select a channel with speech according to the VAD information in step S310;
[0109] All channels are sorted according to the energy information in step S310, and a decoding channel is selected from the highest energy to the lowest energy.
[0110] In step S330, when the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, which specifically includes:
[0111] When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels;
[0112] When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels;
[0113] The number of candidate channels is the difference between the target channel and the preset number.
[0114] In some embodiments, selecting a candidate channel from the remaining channels as a supplementary target channel specifically includes:
[0115] According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;
[0116] According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel;
[0117] When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.
[0118] When calculating the signal-to-noise ratio of the audio signals of all channels, the signal-to-noise ratio information of the audio signals is the information outputted by the noise reduction module at the encoding end, which reflects the ratio of the signal strength to the background noise strength. This embodiment uses this information to guide the routing strategy of the receiving end, and the calculation method is as follows:
[0119]
[0120] In the formula, represents the ratio of signal to noise, represents the power of the signal, Represents the power of the noise.
[0121] The calculation method is as follows:
[0122]
[0123] In the formula, Represents the sampling point value of the signal, Indicates the frame length of the audio frame.
[0124] The calculation method is as follows:
[0125]
[0126] In the formula, represents the power of the noise, represents the sampling point value of the noise, Indicates the frame length of the audio frame.
[0127] In the selection of enhanced channels, first of all, the definition of enhanced channels and non-enhanced channels is that according to the needs of the product, the sound of some channels needs to be amplified, while the sound of other channels needs to be reduced, so as to better focus on the audio information of the required channels. There are as many as dozens of channels with microphones turned on at the same time, which requires certain principles to select which channels' audio needs to be enhanced and which channels' audio needs to be attenuated.
[0128] The process of audio enhancement channel selection is as follows Figure 6 As shown:
[0129] First, check whether the number of users on the stage is more than 6 from the N audio channels. If the number of users on the stage is more than 6, the enhanced channels need to be selected from the users on the stage; if the number of users on the stage is less than 6, the users on the stage are selected for enhancement first, and then the users who have not come on stage are selected, so that the total number of channels selected to be enhanced is 6;
[0130] The principle of selection is to first select the channel with VAD of 1 (i.e., with speech), and then select from large to small according to the energy information sorting; if the energy difference is not large, then select according to the signal-to-noise ratio, and give priority to the channel with a high signal-to-noise ratio;
[0131] Then there is the issue of switching to enhanced channel selection. If other users want to speak, they can also switch to the enhanced channel. The principle of switching is that after entering the enhanced queue, the channel will not be reselected within 2 seconds; the channel that has not spoken for more than 4 seconds enters the selection queue and waits for re-selection of the enhanced channel.
[0132] Finally, the channel enhancement strategy of the channel enhancement and mixing modules is configured to the client through the server background. The parameters of the enhancement configuration are: 1dB, 3dB, 5dB. The parameters of the attenuation configuration are: -3dB, -5dB, -10dB. Attenuate the gain of all audio channels that need to be attenuated, and increase the gain of all audio channels that need to be enhanced. When increasing the gain, please note that the increase in gain may cause the volume value to exceed the maximum limit of 0dB, so this method performs compression processing on the part above -10dB after the gain to prevent popping, and the part below -10dB is increased with a fixed gain, such as Figure 4As shown in the figure, the horizontal axis is the energy value of the audio signal before compression processing, and the vertical axis is the energy value of the audio signal after compression processing. The volume value after multi-channel mixing may also exceed the maximum limit of 0dB, so this method performs compression processing on the part above -10dB after mixing, while the gain of the part below -10dB remains unchanged, as shown in the figure. Figure 5 As shown, the horizontal axis is the energy value of the audio signal before compression processing, and the vertical axis is the energy value of the audio signal after compression processing.
[0133] In the above specific implementation, the multi-channel audio and video signal processing method provided by the present invention calculates the audio signal energy and voice activity detection information of all input channels, and determines multiple target channels according to the calculation results of the audio signal energy and the voice activity detection; when the target channels meet the quantity requirements, the target channels perform audio enhancement, the remaining channels perform audio attenuation, and the audio of all input channels is mixed and output. This method aims to solve the problem of noisy sound when multiple people turn on their microphones at the same time during multi-person audio and video calls, audio and video teaching, or audio and video conferences. Through reasonable audio channel selection and audio enhancement, it makes it easier for users to obtain the information they need, improves the user's auditory experience, makes online communication smoother, and saves users' time costs.
[0134] In addition to the above method, the present invention also provides a multi-channel audio and video signal processing device, such as Fig. 9 As shown, the device comprises:
[0135] A parameter calculation unit 910, configured to calculate audio signal energy and voice activity detection information of all input channels;
[0136] A channel selection unit 920, configured to determine a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection;
[0137] An audio processing unit 930 is configured to perform audio enhancement on the target channel and audio attenuation on the remaining channels when the target channel meets the quantity requirement;
[0138] The result output unit 940 is used to mix and output the audio of all input channels.
[0139] In some embodiments, calculating the audio signal energy of all input channels specifically includes:
[0140] Calculate the maximum energy of each audio sampling point in a single frame of the current channel;
[0141] Calculate the average value of the maximum energy of all frames within the preset duration of the current channel;
[0142] Perform quantization and shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;
[0143] Each input channel is used as the current channel in turn to obtain the audio signal energy of all input channels.
[0144] In some embodiments, calculating voice activity detection information of all input channels specifically includes:
[0145] Resample the audio signal of the current channel to a preset sampling frequency, divide it into multiple frequency bands according to the Mel-frequency cepstrum, and calculate the energy information in each frequency band;
[0146] The Gaussian mixture model is used to calculate the Gaussian probability density function of speech and noise in each sub-band respectively;
[0147] Calculate the log-likelihood ratio of speech and noise, and sum the log-likelihood ratios of all sub-bands;
[0148] Based on the relationship between a preset threshold and the sum of log-likelihood ratios of all sub-bands, the speech activity of the current channel is calibrated;
[0149] Each input channel is used as the current channel in turn to obtain voice activity detection information of all input channels.
[0150] In some embodiments, determining multiple target channels according to the calculation results of the audio signal energy and the calculation results of the voice activity detection specifically includes:
[0151] According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;
[0152] According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.
[0153] In some embodiments, when the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, specifically including:
[0154] When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels;
[0155] When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels;
[0156] The number of candidate channels is the difference between the target channel and the preset number.
[0157] In some embodiments, selecting a candidate channel from the remaining channels as a supplementary target channel specifically includes:
[0158] According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;
[0159] According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel;
[0160] When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.
[0161] In the above specific implementation, the multi-channel audio and video signal processing device provided by the present invention calculates the audio signal energy and voice activity detection information of all input channels, and determines multiple target channels according to the calculation results of the audio signal energy and the voice activity detection; when the target channels meet the quantity requirements, the target channels perform audio enhancement, the remaining channels perform audio attenuation, and the audio of all input channels is mixed and output. This device aims at the problem of noisy sound in the scenario where multiple people turn on the microphone at the same time during multi-person audio and video calls, audio and video teaching or audio and video conferences. Through reasonable audio channel selection and audio enhancement, it makes it easier for users to obtain the information they need, improves the user's auditory experience, makes online communication smoother, and saves users' time costs.
[0162] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the above method.
[0163] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0164] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the above method.
[0165] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-channel audio and video signal processing method, characterized in that: The method comprises: Calculate the energy and voice activity detection information of all input channel audio signals; Determining a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection; When the target channels meet the quantity requirement, the target channels are audio enhanced, and the remaining channels are audio attenuated; Mix and output the audio of all input channels; Calculate the energy of all input channel audio signals, including: Calculate the maximum energy of each audio sampling point in a single frame of the current channel; Calculate the average value of the maximum energy of all frames within the preset duration of the current channel; Perform quantization and shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel; Taking each input channel as the current channel in turn to obtain audio signal energy of all input channels; Calculate the energy and voice activity detection information of all input channel audio signals, including: Resample the audio signal of the current channel to a preset sampling frequency, divide it into multiple frequency bands according to the Mel-frequency cepstrum, and calculate the energy information in each frequency band; The Gaussian mixture model is used to calculate the Gaussian probability density function of speech and noise in each sub-band respectively; Calculate the log-likelihood ratio of speech and noise, and sum the log-likelihood ratios of all sub-bands; Based on the relationship between a preset threshold and the sum of log-likelihood ratios of all sub-bands, the speech activity of the current channel is calibrated; Each input channel is used as the current channel in turn to obtain voice activity detection information of all input channels.
2. The multi-channel audio and video signal processing method according to claim 1, characterized in that: According to the calculation results of the audio signal energy and the calculation results of the voice activity detection, multiple target channels are determined, including: According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel; According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.
3. The multi-channel audio and video signal processing method according to claim 2, characterized in that: When the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, specifically including: When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels; When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels; The number of candidate channels is the difference between the target channel and the preset number.
4. The multi-channel audio and video signal processing method according to claim 3, wherein the step of selecting a candidate channel from the remaining channels as a supplementary target channel comprises: According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel; According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel; When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.
5. A multi-channel audio and video signal processing device, used to implement the method according to any one of claims 1 to 4, characterized in that: The device comprises: A parameter calculation unit, used to calculate the audio signal energy and voice activity detection information of all input channels; A channel selection unit, configured to determine a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection; An audio processing unit, configured to perform audio enhancement on the target channel and audio attenuation on the remaining channels when the target channel meets the quantity requirement; The result output unit is used to mix and output the audio of all input channels.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
