Gain compensation method and related device, equipment, system and storage medium

Abstract

The present application discloses a gain compensation method and related devices, equipment, system and storage medium, wherein the gain compensation method includes: obtaining the ambient audio collected by the sound pickup device in its place of use; replacing the invalid section in the ambient audio with the audio data of the preset frequency to obtain the audio to be tested; sending the audio to be tested to the sound card device; wherein the sound card device is used to detect the first energy intensity of the audio to be tested from each sound pickup device at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristics of the sound pickup device, and based on the third energy intensity of each sound pickup device, obtain the gain compensation coefficient of each sound pickup device. The above scheme can minimize the degree of manual participation in the process of determining the gain compensation coefficient and maximize the accuracy of the gain compensation coefficient.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a gain compensation method and related devices, equipment, systems and storage media. Background Art

[0002] Audio usually needs to be processed for gain in many applications to improve the effect of subsequent applications. Take speech recognition as an example. After the sound pickup device collects the user's voice, the user's voice can be recognized by using audio gain, speech recognition and other technologies in sequence. In multi-user scenarios, different speakers can even be distinguished to achieve role separation in the recognition process, and finally a conversation record can be formed for subsequent storage, review, retrieval and internal dissemination.

[0003] At present, the gain compensation coefficient of the pickup device usually needs to be manually adjusted by experienced debugging personnel through manual analysis when deploying the relevant system. This method is complex, difficult, and costly. After the deployment is completed, if adjustments are required during subsequent use, professional personnel are required to conduct on-site adjustments, which is a poor experience. In view of this, how to minimize the manual participation in the process of determining the gain compensation coefficient and improve the accuracy of the gain compensation coefficient as much as possible has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a gain compensation method and related devices, equipment, system and storage medium, which can reduce the manual participation in the process of determining the gain compensation coefficient as much as possible and improve the accuracy of the gain compensation coefficient as much as possible.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a gain compensation method, including: obtaining ambient audio collected by a sound pickup device in its use place; replacing an invalid section in the ambient audio with audio data of a preset frequency to obtain audio to be tested; wherein the energy intensity of the human voice signal in the invalid section is not higher than a first intensity threshold; sending the audio to be tested to a sound card device; wherein the sound card device is used to detect a first energy intensity of the audio to be tested from each sound pickup device at a preset frequency, and in response to the first energy intensity satisfying a first condition, obtain a third energy intensity for reflecting the gain characteristics of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be tested, and obtain a gain compensation coefficient for each sound pickup device based on the third energy intensity of each sound pickup device.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a gain compensation method, including: receiving audio to be tested from each sound pickup device; respectively detecting the first energy intensity of each audio to be tested at a preset frequency; in response to the first energy intensity satisfying the first condition, obtaining a third energy intensity reflecting the gain characteristics of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be tested; obtaining a gain compensation coefficient of each sound pickup device based on the third energy intensity of each sound pickup device; wherein the sound pickup device is used to replace the audio data with an invalid section of the preset frequency in the ambient audio collected at the place where it is used to obtain the audio to be tested, and send the audio to be tested to the sound card device, and the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold.

[0007] In order to solve the above technical problems, the third aspect of the present application provides a gain compensation device, including: an audio acquisition module, an audio replacement module and an audio sending module, the audio acquisition module is used to obtain the ambient audio collected by the sound pickup device in its use place; the audio replacement module is used to replace the invalid section of the ambient audio with the audio data of the preset frequency to obtain the audio to be tested; wherein the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold; the audio sending module is used to send the audio to be tested to a sound card device; wherein the sound card device is used to detect the first energy intensity of the audio to be tested from each sound pickup device at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristics of the sound pickup device, and based on the third energy intensity of each sound pickup device, obtain the gain compensation coefficient of each sound pickup device.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a gain compensation device, including: an audio receiving module, an intensity detection module, an intensity analysis module and a coefficient calculation module, the audio receiving module is used to receive the audio to be tested from each sound pickup device; the intensity detection module is used to respectively detect the first energy intensity of each audio to be tested at a preset frequency; the intensity analysis module is used to obtain a third energy intensity reflecting the gain characteristic of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be tested in response to the first energy intensity satisfying the first condition; the coefficient calculation module is used to obtain the gain compensation coefficient of each sound pickup device based on the third energy intensity of each sound pickup device; wherein the sound pickup device is used to replace the audio data with the preset frequency in the invalid section of the ambient audio collected at its use place to obtain the audio to be tested, and send the audio to be tested to the sound card device, and the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold.

[0009] In order to solve the above technical problems, the fifth aspect of the present application provides a sound pickup device, including a microphone array, a memory and a processor, the microphone array and the memory are coupled to the processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the gain compensation method in the above first aspect.

[0010] In order to solve the above technical problem, the sixth aspect of the present application provides a sound card device, including a memory and a processor coupled to each other, the memory stores program instructions, and the processor is used to execute the program instructions to implement the gain compensation method in the above second aspect.

[0011] In order to solve the above technical problems, the seventh aspect of the present application provides a gain compensation coefficient, including a sound card device and several sound pickup devices, and the several sound pickup devices are respectively connected to the sound card device; wherein the sound pickup device is used to replace the audio data of the invalid section of the ambient audio collected at the place where it is used with the preset frequency to obtain the audio to be tested, and send the audio to the sound card device, the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold, the sound card device is used to detect the first energy intensity of the audio to be tested from each sound pickup device at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristics of the sound pickup device, and based on the third energy intensity of each sound pickup device, obtain the gain compensation coefficient of each sound pickup device.

[0012] In order to solve the above technical problems, the eighth aspect of the present application provides a computer-readable storage medium, storing program instructions that can be executed by a processor, and the program instructions are used to implement the gain compensation method in the above first aspect, or to implement the gain compensation method in the above second aspect.

[0013] The above scheme obtains the ambient audio collected by the sound pickup device at its use place, and replaces the invalid section in the ambient audio with the audio data of the preset frequency to obtain the audio to be tested, and the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold, and sends the audio to be tested to the sound card device, and the sound card device is used to detect the first energy intensity of the audio to be tested from each sound pickup device at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristic of the sound pickup device, and based on the third energy intensity of each sound pickup device, obtain the gain compensation coefficient of each sound pickup device. On the one hand, the gain compensation coefficient can be automatically determined without manual participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be tested of the sound pickup device at the preset frequency meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, the manual participation in the process of determining the gain compensation coefficient can be reduced as much as possible, and the accuracy of the gain compensation coefficient can be improved as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of an embodiment of a gain compensation method of the present application;

[0015] Figure 2 It is a schematic diagram of an embodiment of a sound pickup method of a sound pickup device;

[0016] Figure 3 is a schematic diagram of a framework of an embodiment of a sound pickup device;

[0017] Figure 4 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S12;

[0018] Figure 5 is a schematic diagram of an embodiment of audio replacement;

[0019] Figure 6 is a flow chart of another embodiment of the gain compensation method of the present application;

[0020] Figure 7 It is a schematic diagram of the framework of an embodiment of the gain compensation device of the present application;

[0021] Figure 8 is a schematic diagram of a framework of another embodiment of the gain compensation device of the present application;

[0022] Fig. 9 It is a schematic diagram of the framework of an embodiment of the sound pickup device of the present application;

[0023] Fig.10 It is a schematic diagram of the framework of an embodiment of the sound card device of the present application;

[0024] Fig.11 It is a schematic diagram of the framework of an embodiment of the gain compensation system of the present application;

[0025] Fig.12 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0029] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of a gain compensation method of the present application.

[0030] Specifically, the following steps may be included:

[0031] Step S11: Acquire the ambient audio collected by the sound pickup device at the place where it is used.

[0032] In one implementation scenario, the ambient audio may include several audio clips, and the specific number of the audio clips may not be limited, such as 10, 20, 30, etc.; alternatively, in the process of determining the gain compensation coefficient, the sound pickup device may continue to collect audio. In this case, a set of a certain number (such as 20, 30, etc.) of continuous audio clips may be used as the ambient audio, and on this basis, subsequent replacement, sending, and other operations may be performed on it, and collection is continued to obtain the next set of clip sets until the gain compensation coefficient of each sound pickup device is determined.

[0033] In one implementation scenario, in order to facilitate subsequent operations, the ambient audio can be a digital signal, and subsequent replacement operations can be performed on this basis. This can reduce the possibility of gain inconsistency due to crosstalk between analog signals of adjacent sound pickup devices, which is conducive to further improving the consistency of gain compensation.

[0034] In one implementation scenario, in order to improve the audio quality of the ambient audio and to further improve the accuracy of determining the gain compensation coefficient, the sound pickup device may also include a plurality of microphones arranged in an array. For example, the sound pickup device may include 4 microphones, 6 microphones, 8 microphones, etc. arranged in an array, which are not limited here. On this basis, the analog audio collected by the plurality of microphones may be converted into digital audio, and the digital audio corresponding to each microphone may be windowed to obtain a plurality of first segments, and the first segments corresponding to the plurality of microphones may be fused using beamforming to obtain fused audio, and the fused audio includes a plurality of second segments, and then the fused segments are subjected to noise suppression to obtain ambient audio. The above method, based on the analog audio collected by the plurality of microphones respectively, performs a series of processing such as digital sampling, windowing, beamforming, and noise suppression, which can effectively suppress interference such as ambient noise and surrounding human voices in the use place, and is conducive to improving the audio quality of the ambient audio.

[0035] In a specific implementation scenario, please refer to Figure 2 , Figure 2 1 is a schematic diagram of an embodiment of a sound pickup method of a sound pickup device. It should be noted that: Figure 2 The figure shows a top view of the sound pickup process. Figure 2 As shown, the sound pickup device can be placed on the desktop, and the user can get close to the side of the desktop. The angle formed by the solid arrow represents the sound pickup range of the sound pickup device. Figure 2 The figure is merely a possible arrangement of the sound pickup device in actual application, and does not limit the actual arrangement of the sound pickup device.

[0036] In a specific implementation scenario, the sound pickup device may be provided with a CODEC (COder-DECoder) chip, and the CODEC chip may be integrated with an ADC (Analog to Digital Converter) and a DAC (Digital to Analog Converter). The ADC is used to convert an analog signal into a digital signal, and the DAC is used to convert a digital signal into an analog signal. In addition, in the process of converting an analog signal into a digital signal, a preset sampling rate may be used for digital sampling, and the preset sampling rate may include but is not limited to: 16kHz, 32kHz, etc., which are not limited here. Taking the preset sampling rate set to 16kHz as an example, it means that the analog signal is sampled 16,000 times per second. When the preset sampling rate is other values, it can be deduced by analogy, and examples are not given one by one here.

[0037] In a specific implementation scenario, please refer to Figure 3 , Figure 3 Schematic diagram of a frame of an embodiment of a sound pickup device. Figure 3 As shown, the sound pickup device may include a microphone, an analog-to-digital conversion circuit (ADC), a processor (i.e., Central Processing Unit, CPU), and a digital-to-analog conversion circuit (DAC). The number of ADCs is the same as the number of microphones. For example, the sound pickup device includes 4 microphones, and the sound pickup device also includes 4 ADCs, which are respectively used to convert the analog audio collected by the microphones into digital audio. For ease of description, the digital audio converted from the analog audio collected by the 4 microphones can be denoted as A1, A2, A3, and A4, respectively. In addition, the ADC and the CPU can be connected via an I2S interface, so that the above 4 digital audios A1, A2, A3, and A4 can be sent to the CPU for processing via the I2S interface.

[0038] In a specific implementation scenario, a window function can be used to window the digital audio to obtain a plurality of first segments. Specifically, the window function can be set according to the actual application needs. For example, the window length of the window function can be set to 1024 (i.e., 1024 sampling points), and the window height can be set to 1 (i.e., the amplitude of the digital audio is not changed after windowing). Of course, other window functions can also be set as needed, which is not limited here. For ease of description, the above window function can be expressed as:

[0039]

[0040] By using a window function such as that described in the above formula (1), the digital audio can be divided into first segments of fixed length for subsequent processing, and the set of segments after windowing can be collectively referred to as A. When other window functions are selected for windowing processing, the same can be applied, and examples are not given here one by one.

[0041] In a specific implementation scenario, as mentioned above, the set of fragments after windowing can be collectively referred to as A, and beamforming can be used to process A to suppress interference signals outside the beam to obtain signal B, where signal B is the fused audio, and the number of second fragments contained in the fused audio is the same as the number of first fragments after windowing of each digital audio channel. Exemplarily, N first fragments can be obtained after windowing of each digital audio channel, and beamforming can be used to perform fusion processing on the i-th first fragment in each digital audio channel to suppress interference signals outside the beam, and multiple signals can be merged into one channel to obtain the i-th second fragment in the fused audio. It should be noted that the specific process of the fusion processing can refer to the technical details of beamforming, which will not be repeated here.

[0042] In a specific implementation scenario, signal B (i.e., fused audio) can be subjected to environmental noise detection and environmental noise suppression to obtain signal C (i.e., environmental audio). It should be noted that the process of environmental noise detection and environmental noise suppression does not change the number of audio clips, that is, the number of audio clips remains unchanged before and after detection and suppression. In addition, the specific process of environmental noise detection and environmental noise suppression can refer to technical details such as the least mean square adaptive filter and the recursive least squares filter, which will not be repeated here.

[0043] Step S12: replacing the invalid section in the ambient audio with audio data of a preset frequency to obtain the audio to be tested.

[0044] In the embodiment of the present disclosure, the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold, and the first intensity threshold can be set according to the actual application needs. For example, in the case of high accuracy requirements for the gain compensation coefficient, the first intensity threshold can be set appropriately smaller, such as 0.5, 0.4, etc., or even 0; or, in the case of relatively loose accuracy requirements for the gain compensation coefficient, the first intensity threshold can be set appropriately larger, such as 0.7, 0.8, etc., which is not limited here.

[0045] In one implementation scenario, the preset frequency can be set according to actual application needs, such as 1kHz, 2kHz, 3kHz, etc., which is not limited here. In addition, since the audio to be tested needs to be sent to the sound card device for processing later, the preset frequency can also be set according to the detection sensitivity of the sound card device to various frequencies. For example, if the sound card device has a good detection sensitivity to 1kHz, the preset frequency can be set to 1kHz. Other situations can be deduced by analogy, and examples are not given one by one here.

[0046] In one implementation scenario, in order to facilitate subsequent sound card device processing, the duration of the invalid segment may not be longer than the duration threshold. That is, the ambient audio may contain at least one invalid segment, and the duration of each invalid segment should not be longer than the duration threshold. Specifically, the ambient audio may be subjected to human voice signal detection, and the moment when the human voice signal is first detected to be not higher than the first intensity threshold is used as the start time of the invalid segment, and the human voice signal is continuously detected and the duration from the current moment to the start moment is recorded. If the duration is equal to the duration threshold, and the human voice signal is still not higher than the first intensity threshold, the moment can be directly used as the end time of the invalid segment, or if the duration is less than the duration threshold, and the human voice signal is first detected to be higher than the first intensity threshold, the moment when the human voice signal is first detected to be higher than the first intensity threshold is used as the end time of the invalid segment. It should be noted that the duration threshold can be set according to actual application needs. For example, when the accuracy requirement of the gain compensation coefficient is high and the determination speed requirement of the gain compensation coefficient is relatively loose, the duration threshold can be set to be appropriately larger; or, when the accuracy requirement and the determination speed requirement of the gain compensation coefficient are both high, the duration threshold can be set to be moderate; or, when the accuracy requirement of the gain compensation coefficient is relatively loose and the determination speed requirement of the gain compensation coefficient is high, the duration threshold can be set to be appropriately smaller. The specific value of the duration threshold is not limited here.

[0047] In an implementation scenario, as described above, the ambient audio may include several audio clips, and the number of audio clips in the invalid segment is no more than the quantity threshold. That is, the ambient audio may include at least one invalid segment, and the audio clips contained in each invalid segment are no more than the quantity threshold. It should be noted that the quantity threshold can be set according to the actual application needs. For example, when the accuracy requirements of the gain compensation coefficient are high and the determination speed requirements of the gain compensation coefficient are relatively loose, the quantity threshold can be set appropriately larger, or, when the accuracy requirements and determination speed requirements of the gain compensation coefficient are both high, the quantity threshold can be set moderately, or, when the accuracy requirements of the gain compensation coefficient are relatively loose and the determination speed requirements of the gain compensation coefficient are high, the quantity threshold can be set appropriately smaller, and the specific value of the duration threshold is not limited here. Exemplarily, in order to balance the accuracy requirements of the gain compensation coefficient and the determination speed requirements of the gain compensation coefficient, the quantity threshold can be set to 15, 16, 17, etc., which are not limited here. The specific process of replacing the audio can refer to the relevant description in the following public embodiments, which will not be repeated here.

[0048] In one implementation scenario, before replacing the invalid segment in the ambient audio with the audio data of the preset frequency, it is also possible to first detect whether the current state satisfies the second condition, and in response to the current state satisfying the second condition, replace the invalid segment in the ambient audio with the audio data, and the current state includes the current time, and the second condition includes any one of the following: the current time is in a preset time period, and the current time exceeds the preset interval from the last execution of the embodiment of the gain compensation method of the present application. It should be noted that in order to eliminate noise interference as much as possible and to improve the accuracy of the gain compensation coefficient as much as possible, the preset time period can be set to a time period other than the normal working period (such as early morning, all day on weekends, etc.); or, in order to keep the gain compensation coefficient in a relatively accurate state at all times, the embodiment of the gain compensation method of the present application can be executed once at a preset interval to update the gain compensation coefficient of the pickup device, and the preset interval can be set to 1 week, 1 month, 1 quarter, half a year, 1 year, etc. according to actual application needs, and is not limited here. In the above method, before replacing the audio, it is first detected whether the current state meets the second condition, and the audio is replaced when the second condition is met. The current state includes the current time, and the second condition includes the current time being in a preset time period and the current time being more than a preset interval from the last execution of the gain compensation method. Therefore, noise interference can be eliminated as much as possible to improve the accuracy of the gain compensation coefficient as much as possible, so that the gain compensation coefficient can always be kept in a relatively accurate state.

[0049] In a specific implementation scenario, the current state may also include the power-on state of the sound pickup device, and the second condition may include: the sound pickup device switches from an unpowered state to a powered state, that is, when it is detected that the sound pickup device switches from an unpowered state to a powered state, an audio replacement operation may be triggered to obtain the gain compensation coefficient of the sound pickup device. Therefore, the gain compensation coefficient of the sound pickup device can be automatically obtained after power-on, which is conducive to further improving the automation.

[0050] In a specific implementation scenario, the sound pickup device may also be provided with a trigger key (e.g., a button, a knob, etc.), and the user may trigger the audio replacement operation through the trigger key (e.g., pressing a button, turning a knob, etc.). In other words, the current state may also include the trigger state of the trigger key, and the second condition may include: the trigger key is in a triggered state (e.g., pressing a button, turning a knob, etc.). Therefore, in addition to automatic triggering, the audio replacement may also be manually triggered through the trigger key, which is conducive to improving autonomy.

[0051] In a specific implementation scenario, it should be noted that, when the current state does not satisfy the second condition, the step of replacing the invalid segment in the ambient audio with the audio data of the preset frequency may not be performed. That is to say, when the current state does not satisfy the second condition, even if there is an invalid segment in the ambient audio, it will not be replaced, so that the sound pickup device can work normally during non-preset time periods, or when the coefficient update time has not arrived, or when the power-on state has not been switched, or when the trigger key has not been manually triggered.

[0052] Step S13: Send the audio to be tested to the sound card device.

[0053] In the disclosed embodiment, the sound card device is used to detect a first energy intensity of the audio to be tested from each sound pickup device at a preset frequency, and in response to the first energy intensity satisfying a first condition, obtain a third energy intensity for reflecting the gain characteristic of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be tested, and obtain a gain compensation coefficient of each sound pickup device based on the third energy intensity of each sound pickup device. The specific process can be referred to the following disclosed embodiment, which will not be described in detail here.

[0054] In an implementation scenario, such as Figure 3 As shown, the sound pickup device may further include a DAC, and the DAC is connected to the sound card device. After the audio to be tested is acquired, the audio to be tested may be converted into analog audio through the DAC and sent to the sound card device.

[0055] In addition, compared to adjusting the gain by a manual knob, the smaller the step size of the manual knob, the smaller the adjustable gain range, and the larger the step size of the manual knob, the larger the amplification factor, resulting in a larger error, so the manual knob inevitably has problems such as being restricted in applicable scenarios and limited adjustable gain range. In comparison, the embodiment disclosed in the present application can automatically calculate the gain compensation coefficient of each pickup device without a manual knob, so the above problems do not exist, which is conducive to expanding the adjustable gain range and meeting more scene requirements.

[0056] The above scheme obtains the ambient audio collected by the sound pickup device at its use place, and replaces the invalid section in the ambient audio with the audio data of the preset frequency to obtain the audio to be tested, and the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold, and sends the audio to be tested to the sound card device, and the sound card device is used to detect the first energy intensity of the audio to be tested from each sound pickup device at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristic of the sound pickup device, and based on the third energy intensity of each sound pickup device, obtain the gain compensation coefficient of each sound pickup device. On the one hand, the gain compensation coefficient can be automatically determined without manual participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be tested of the sound pickup device at the preset frequency meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, the manual participation in the process of determining the gain compensation coefficient can be reduced as much as possible, and the accuracy of the gain compensation coefficient can be improved as much as possible.

[0057] See also Figure 4 , Figure 4 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S12 in FIG. In the embodiment of the present disclosure, the ambient audio includes a plurality of audio clips, and the number of audio clips in the invalid section is no more than a quantity threshold. Specifically, the embodiment of the present disclosure includes the following steps:

[0058] Step S41: taking several audio segments as current segments respectively.

[0059] Specifically, taking the ambient audio including N audio segments as an example, the i-th audio segment can be used as the current segment, and i can be taken from 1 until i equals N. Other situations can be deduced by analogy, and no examples are given here one by one.

[0060] Step S42: Detect the energy intensity of the human voice signal in the current segment, and count the number of audio segments that are replaced continuously starting from the audio segment before the current segment.

[0061] In one implementation scenario, the human voice signal in the current segment may be detected by methods such as VAD (Voice Activity Detection). For the specific detection process, please refer to the technical details of detection methods such as VAD, which will not be described in detail here.

[0062] In one implementation scenario, as described above, the ambient audio C may be a digital signal sampled at a preset sampling rate. For example, taking the preset sampling rate as 16 kHz (i.e., 960,000 samples per minute), the energy intensity of the human voice signal within one minute may be expressed as:

[0063]

[0064] In the above formula (2), t represents the sampling point at each moment. Similarly, when the audio segment contains a preset number (e.g., 1024) of sampling points, C(t) in the above formula (2) can be replaced by C'(t), and C' represents the audio segment, and the upper limit value of the definite integral in the above formula (2) can be replaced by a preset value (e.g., 1024). Other situations can be deduced by analogy, and no examples are given here one by one.

[0065] In one implementation scenario, for ease of description, the order of the current segment in the ambient audio can be recorded as i, and the number of audio segments that have been replaced successively can be counted starting from the i-1th audio segment. For example, if the i-1th audio segment has been replaced with audio data of a preset frequency, but the i-2th audio segment has not been replaced with audio data of a preset frequency, the number of segments can be recorded as 1; if the i-1th audio segment and the i-2th audio segment have both been replaced with audio data of a preset frequency, but the i-3th audio segment has not been replaced with audio data of a preset frequency, the number of segments can be recorded as 2; other situations can be deduced in the same way, and examples are not given one by one here.

[0066] Step S43: In response to the energy intensity of the vocal signal in the current segment satisfying the intensity condition and the number of segments satisfying the number condition, the current segment is replaced with audio data.

[0067] In the disclosed embodiment, the intensity condition includes that the energy intensity of the vocal signal is not higher than the first intensity threshold, and the quantity condition includes that the number of segments is less than the quantity threshold. The specific setting methods of the first intensity threshold and the quantity threshold can be referred to the relevant description in the aforementioned disclosed embodiment and will not be repeated here.

[0068] In one implementation scenario, taking the quantity threshold set to 16 as an example, if the energy intensity of the human voice signal in the current segment is not higher than the first intensity threshold (such as 0, etc.) and the number of segments is less than 16 (such as 15, 14, 13, etc.), the current segment can be replaced with audio data of a preset frequency.

[0069] In one implementation scenario, still taking the quantity threshold set to 16 as an example, if the energy intensity of the human voice signal in the current segment is higher than the first intensity threshold (such as 0, etc.), regardless of the number of segments, the current segment will not be replaced by audio data of a preset frequency.

[0070] In one implementation scenario, still taking the number threshold set to 16 as an example, if the energy intensity of the human voice signal in the current segment is not higher than the first intensity threshold (such as 0, etc.) and the number of segments is equal to 16, it can be considered that a complete gain consistency automatic compensation signal has been sent, and the current segment is not replaced by audio data of a preset frequency; alternatively, the number of segments can be reset to 0, and then after the current segment is replaced with audio data of a preset frequency, the number of segments is updated to 1 to start a new round of gain consistency automatic compensation signal sending until the number of segments is updated to 16 again.

[0071] In an implementation scenario, please refer to Figure 5 , Figure 5 Schematic diagram of an embodiment of audio replacement. Figure 5 As shown, when 16 audio segments are replaced continuously, it can be regarded as completing a complete gain consistency automatic compensation signal transmission. On the contrary, when the number of continuously replaced audio segments is less than 15 and the energy intensity of the human voice signal in the current segment does not meet the intensity condition, the current segment can no longer be replaced, and the number of segments is reset to 0, and wait for the next audio segment in which the energy intensity of the human voice signal meets the intensity condition, and then replace the audio segment with audio data of a preset frequency, and update the number of segments to 1, and so on, and no examples are given here. In addition, if Figure 5 As shown, the audio segment is a digital signal containing a preset number of sampling points (e.g., 1024), and the audio data replacing the audio segment in the invalid segment is also a digital signal containing a preset number of sampling points (e.g., 1024). In the above manner, the audio segment is a digital signal containing a preset number of sampling points, and the audio data replacing the audio segment in the invalid segment is also a digital signal containing a preset number of sampling points, which can help improve the accuracy of the subsequent determination of the gain compensation coefficient.

[0072] The above scheme takes several audio segments as the current segment respectively, detects the energy intensity of the human voice signal in the current segment, and counts the number of audio segments that are replaced continuously starting from the audio segment before the current segment. On this basis, in response to the energy intensity of the human voice signal in the current segment satisfying the intensity condition and the number of segments satisfying the quantity condition, the current segment is replaced with audio data of a preset frequency. Therefore, during the replacement process, the audio segment can be used as the smallest unit for detection one by one, and a decision on whether to replace is made based on both the energy intensity and the number of segments, which is conducive to improving the efficiency and accuracy of audio replacement.

[0073] See also Figure 6 , Figure 6 It is a flow chart of another embodiment of the gain compensation method of the present application.

[0074] Specifically, the following steps may be included:

[0075] Step S61: receiving the audio to be tested from each sound pickup device.

[0076] In the embodiment of the present disclosure, the sound pickup device is used to replace the audio data of the preset frequency in the invalid section of the ambient audio collected at the place where it is used to obtain the audio to be tested, and send the audio to be tested to the sound card device. The energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold. For details, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0077] Step S62: respectively detecting the first energy intensity of each audio frequency to be tested at a preset frequency.

[0078] In one implementation scenario, Fourier transform can be used to transform the audio to be tested into the frequency domain to obtain the first energy intensity at the preset frequency in the frequency domain. The specific process of frequency domain conversion can refer to the technical details of Fourier transform, which will not be repeated here.

[0079] In one implementation scenario, each sound pickup device can be used as the current device, and the audio to be tested from the current device can be used as the current audio. Taking a total of 4 sound pickup devices as an example, the first sound pickup device, the second sound pickup device, the third sound pickup device, and the fourth sound pickup device can be used as the current device. Other situations can be deduced by analogy, and examples are not given one by one here. In addition, as described in the aforementioned disclosed embodiment, the ambient audio includes a number of audio clips, and the number of audio clips in the invalid section is no more than the quantity threshold. On this basis, the first energy intensity of each audio clip in the current audio at a preset frequency can be detected.

[0080] In a specific implementation scenario, please refer to the aforementioned disclosed embodiments and Figure 2The audio to be tested is a digital signal, which is converted into an analog signal by a DAC in a sound pickup device before being sent to the sound card device. Furthermore, the sound card device may also include an ADC. After receiving the signal sent by the sound pickup device, the ADC may be used to convert it into a digital signal, and then the digital signal may be windowed using the window function w(x) in the aforementioned disclosed embodiment, and then a Fourier transform may be performed on this basis.

[0081] In a specific implementation scenario, taking an 8-channel sound pickup device as an example, the sound card device can simultaneously pick up the voice signals output by the 8-channel sound pickup devices and send them to the ADC of the sound card device for analog-to-digital conversion to obtain digital signals D1, D2, ..., D8. First, the signals D1, D2, ..., D8 of the 8 channels are windowed by w(x), and then a fast Fourier transform is performed on each channel signal Di to obtain the frequency domain information of the signal, and finally the frequency domain information of each channel is monitored to obtain the first energy intensity of each audio clip at a preset frequency.

[0082] Step S63: in response to the first energy intensity satisfying the first condition, obtaining a third energy intensity reflecting the gain characteristic of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be measured.

[0083] In one implementation scenario, it is possible to detect whether the first energy intensities of a continuous number of threshold audio clips all meet the first condition. If so, the continuous number of threshold audio clips can be used as the segment to be analyzed, and based on the first energy intensity of the segment to be analyzed at the preset frequency and the second energy intensity of the segment to be analyzed as a whole, a third energy intensity reflecting the gain characteristics of the current device is obtained, and the first condition includes that the first energy intensity is higher than the second intensity threshold. In the above method, in response to the first energy intensities of a continuous number of threshold audio clips all meeting the first condition, the continuous number of threshold audio clips are used as the segment to be analyzed, and based on the first energy intensity of the segment to be analyzed at the preset frequency and the second energy intensity of the segment to be analyzed as a whole, a third energy intensity reflecting the gain characteristics of the current device is obtained, and the first condition includes that the first energy intensity is higher than the second intensity threshold. Therefore, the sound card device can send a complete gain consistency automatic compensation signal corresponding to one side of the pickup device, implement a gain compensation coefficient determination, which is conducive to improving gain consistency.

[0084] In a specific implementation scenario, taking the quantity threshold as 16 as an example, it can be detected whether there are 16 consecutive audio clips whose first energy intensity at the preset frequency is higher than the second intensity threshold. In addition, the second intensity threshold can be set according to actual application needs. For example, when the accuracy of the gain compensation coefficient is required to be high, the second intensity threshold can be set to be slightly larger, or when the accuracy of the gain compensation coefficient is required to be relatively loose, the second intensity threshold can be set to be moderate. The specific value of the second intensity threshold is not limited here.

[0085] In a specific implementation scenario, in order to further improve the accuracy of the gain compensation coefficient, energy detection can be performed on each audio clip at a non-harmonic frequency to obtain the fourth energy intensity of the audio clip at the non-harmonic frequency. On this basis, it can be detected whether the first energy intensity of a continuous number threshold audio clips all meet the first condition and whether the fourth energy intensity all meet the third condition, and the first condition includes that the first energy intensity is higher than the second intensity threshold, and the third condition includes that the fourth energy intensity is lower than the third intensity threshold. If satisfied, the continuous number threshold audio clips can be used as the segment to be analyzed. It should be noted that, similar to the second intensity threshold, the third intensity threshold can also be set according to actual application needs. For example, in the case of high requirements for the accuracy of the gain compensation coefficient, the third intensity threshold can be set slightly smaller, or, in the case of relatively loose requirements for the accuracy of the gain compensation coefficient, the third intensity threshold can be set moderately. The specific value of the third intensity threshold is not limited here.

[0086] In a specific implementation scenario, the average energy intensity of the section to be analyzed may be counted as the second energy intensity of the entire section to be analyzed.

[0087] In a specific implementation scenario, the difference between the first energy intensity and the second energy intensity can be used as the third energy intensity reflecting the gain characteristics of the current device. Taking the i-th sound pickup device as an example, by analyzing the audio to be tested from the i-th sound pickup device, it is possible to detect the segment to be analyzed in which the first energy intensities of a threshold number of consecutive audio segments in the audio to be tested all meet the first condition, and obtain the first energy intensity Vx of the segment to be analyzed at a preset frequency minus the second energy intensity Dx of the entire segment to be analyzed, and obtain the third energy intensity characterizing the gain characteristics of the sound pickup device.

[0088] In one implementation scenario, as described above, it is possible to detect whether the first energy intensities of a continuous number of threshold audio clips all satisfy the first condition. If there are no continuous number of threshold audio clips in the current audio whose first energy intensities all satisfy the first condition, the above-mentioned step of re-executing the step of receiving the audio to be tested from each sound pickup device and the subsequent steps can be re-executed until a third energy intensity reflecting the gain characteristics of the current device is obtained. The above-mentioned method, in the case where there are no continuous number of threshold audio clips in the current audio whose first energy intensities all satisfy the first condition, re-executes the step of receiving the audio to be tested from each sound pickup device and the subsequent steps until a third energy intensity reflecting the gain characteristics of the current device is obtained, which can be beneficial to improving the robustness of gain compensation.

[0089] In a specific implementation scenario, if the third energy intensity of the current device is obtained during this re-execution and the third energy intensity of the current device has been obtained before this re-execution, the third energy intensity of the current device can be updated to the third energy intensity obtained during this re-execution. In the above manner, in the process of obtaining the third energy intensity of each pickup device, the third energy intensity of each pickup device can be kept as the latest one, which is conducive to improving the accuracy of the gain compensation coefficient.

[0090] In a specific implementation scenario, if the third energy intensity of the current device is obtained during this re-execution and the third energy intensity of the current device has not been obtained before this re-execution, the third energy intensity of the current device can be set to the third energy intensity obtained during this re-execution.

[0091] Step S64: based on the third energy intensity of each sound pickup device, obtaining a gain compensation coefficient of each sound pickup device.

[0092] In one implementation scenario, the third energy intensity is positively correlated with the gain compensation coefficient, that is, the greater the third energy intensity, the greater the gain compensation coefficient, and conversely, the smaller the third energy intensity, the smaller the gain compensation coefficient. Therefore, it is possible to amplify useful signals and suppress irrelevant signals, which is beneficial to improving signal quality.

[0093] In a specific implementation scenario, the third energy intensity of one of the sound pickup devices can be selected as the reference energy intensity, and the ratio between the third energy intensity of each sound pickup device and the reference energy intensity can be used as the gain compensation coefficient of the corresponding sound pickup device. In the above manner, the gain compensation coefficient of each sound pickup device can be obtained through simple mathematical operations, which can help reduce the complexity of the gain compensation coefficient calculation.

[0094] In a specific implementation scenario, still taking 8 sound pickup devices as an example, after collecting the third energy intensities I1, I2, ..., I8 of the 8 sound pickup devices, illustratively, the gain compensation coefficients S1, S2, ..., S8 of each sound pickup device can be calculated with I1 as the reference energy intensity, which can be specifically expressed as:

[0095]

[0096] In the above formula (3), i represents the number of the sound pickup device, and its value range is between 1 and 8. Other situations can be deduced by analogy, and no examples are given here one by one.

[0097] In one implementation scenario, after obtaining the gain compensation coefficient of each sound pickup device, voice audio from each sound pickup device can be received, and the gain compensation coefficient of each sound pickup device is used to perform gain compensation on the voice audio of each sound pickup device to obtain the gain audio of each sound pickup device, and based on the gain audio of each sound pickup device, the voice recognition result of each sound pickup device is obtained. The above method, by using the gain compensation coefficient of each sound pickup device to perform gain compensation on the voice audio of each sound pickup device, and then performing voice recognition on this basis, is conducive to improving the accuracy of voice recognition and role differentiation.

[0098] In a specific implementation scenario, the speech recognition result may include the recognition text and the recognition role, that is, the text corresponding to the speech audio and the role (character) who spoke the speech audio. In addition, on this basis, a conversation record may be formed based on the speech recognition result. Further, in order to improve the recognition accuracy, the gain audio may be sent to the server for recognition.

[0099] In a specific implementation scenario, still taking 8 sound pickup devices as an example, 8 channels of voice audio can be recorded as D1, D2, ..., D8 respectively, and the gain audio corresponding to each sound pickup device can be expressed as:

[0100] E i =D i ×S i ……(4)

[0101] In the above formula (4), i represents the number of the sound pickup device, and its value range is between 1 and 8. i Indicates the gain audio corresponding to the i-th pickup device.

[0102] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0103] See also Figure 7 , Figure 7 : is a schematic diagram of a framework of an embodiment of a gain compensation device 70 of the present application. The gain compensation device 70 includes: an audio acquisition module 71, an audio replacement module 72 and an audio sending module 73, wherein the audio acquisition module 71 is used to acquire the ambient audio collected by the sound pickup device in its use place; the audio replacement module 72 is used to replace the audio data of the invalid section of the ambient audio with the preset frequency to obtain the audio to be tested; wherein the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold; the audio sending module 73 is used to send the audio to be tested to the sound card device; wherein the sound card device is used to detect the first energy intensity of the audio to be tested from each sound pickup device at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristics of the sound pickup device, and based on the third energy intensity of each sound pickup device, obtain the gain compensation coefficient of each sound pickup device.

[0104] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0105] In some disclosed embodiments, the ambient audio includes a plurality of audio segments, and the number of the audio segments in the invalid section is no more than a quantity threshold.

[0106] Therefore, the ambient audio includes several audio segments, and the number of audio segments in the invalid section is no more than the quantity threshold, and the accuracy requirement of the gain compensation coefficient and the speed requirement for determining the gain compensation coefficient can be balanced through the quantity threshold.

[0107] In some disclosed embodiments, the audio replacement module 72 includes a segment selection submodule for selecting several audio segments as the current segment respectively; the audio replacement module 72 includes a human voice detection submodule for detecting the energy intensity of the human voice signal in the current segment, the audio replacement module 72 includes a segment statistics submodule for counting the number of audio segments that are successively replaced starting from the previous audio segment of the current segment; the audio replacement module 72 includes a segment replacement submodule for replacing the current segment with audio data in response to the energy intensity of the human voice signal in the current segment satisfying the intensity condition and the number of segments satisfying the quantity condition; wherein the intensity condition includes that the energy intensity of the human voice signal is not higher than the first intensity threshold, and the quantity condition includes that the number of segments is less than the quantity threshold.

[0108] Therefore, several audio clips are taken as the current clip respectively, and the energy intensity of the human voice signal in the current clip is detected, and the number of audio clips that are continuously replaced is counted starting from the audio clip before the current clip. On this basis, in response to the energy intensity of the human voice signal in the current clip satisfying the intensity condition and the number of clips satisfying the quantity condition, the current clip is replaced with audio data of a preset frequency. Therefore, during the replacement process, the audio clip is used as the smallest unit for detection one by one, and a decision on whether to replace is made based on both the energy intensity and the number of clips, which is conducive to improving the efficiency and accuracy of audio replacement.

[0109] In some disclosed embodiments, the audio segment is a digital signal including a preset number of sampling points, and the audio data replacing the audio segment in the invalid segment is also a digital signal including a preset number of sampling points.

[0110] Therefore, the audio segment is a digital signal containing a preset number of sampling points, and the audio data replacing the audio segment in the invalid section is also a digital signal containing a preset number of sampling points, which can help improve the accuracy of subsequent determination of the gain compensation coefficient.

[0111] In some disclosed embodiments, the gain compensation device 70 may further include a state detection module for detecting whether the current state satisfies the second condition; the audio replacement module 72 is specifically used to replace the invalid segment in the ambient audio with audio data in response to the current state satisfying the second condition; wherein the current state includes the current time, and the second condition includes any one of the following: the current time is in a preset time period, and the current time exceeds a preset interval from the last execution of the gain compensation method.

[0112] Therefore, before replacing the audio, first check whether the current state satisfies the second condition, and perform audio replacement if the second condition is met. The current state includes the current time, and the second condition includes whether the current time is in a preset time period or the current time is beyond a preset interval from the last execution of the gain compensation method. Therefore, noise interference can be eliminated as much as possible to improve the accuracy of the gain compensation coefficient as much as possible, so that the gain compensation coefficient can always be kept in a relatively accurate state.

[0113] In some disclosed embodiments, the sound pickup device includes a plurality of microphones arranged in an array, and the ambient audio includes a plurality of audio segments. The audio acquisition module 71 includes an analog-to-digital conversion submodule for converting the analog audios collected by the plurality of microphones into digital audios respectively; the audio acquisition module 71 includes a windowing processing submodule for windowing the digital audios corresponding to the respective microphones to obtain a plurality of first segments; the audio acquisition module 71 includes a beamforming submodule for fusing the first segments corresponding to the plurality of microphones using beamforming to obtain fused audio; wherein the fused audio includes a plurality of second segments; the audio acquisition module 71 includes a noise suppression submodule for performing noise suppression on the fused audio to obtain ambient audio.

[0114] Therefore, based on the analog audio collected by several microphones, a series of processing such as digital sampling, windowing, beamforming and noise suppression are performed, which can effectively suppress interference from ambient noise, surrounding human voices, etc. in the place of use, and is conducive to improving the audio quality of ambient audio.

[0115] See also Figure 8 , Figure 8 1 is a schematic diagram of a framework of an embodiment of a gain compensation device 80 of the present application. The gain compensation device 80 includes: an audio receiving module 81, an intensity detection module 82, an intensity analysis module 83 and a coefficient calculation module 84, wherein the audio receiving module 81 is used to receive the audio to be tested from each sound pickup device; the intensity detection module 82 is used to respectively detect the first energy intensity of each audio to be tested at a preset frequency; the intensity analysis module 83 is used to obtain a third energy intensity reflecting the gain characteristic of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be tested in response to the first energy intensity satisfying the first condition; the coefficient calculation module 84 is used to obtain the gain compensation coefficient of each sound pickup device based on the third energy intensity of each sound pickup device; wherein the sound pickup device is used to replace the audio data of the preset frequency in the invalid section of the ambient audio collected at the place of use to obtain the audio to be tested, and send the audio to be tested to the sound card device, and the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold.

[0116] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0117] In some disclosed embodiments, the ambient audio includes a number of audio segments, the number of audio segments in the invalid segment is no more than a quantity threshold, the intensity detection module 82 includes a device selection submodule, which is used to respectively use each sound pickup device as the current device, and use the audio to be tested from the current device as the current audio; the intensity detection module 82 includes an intensity detection submodule, which is used to detect the first energy intensity of each audio segment in the current audio at a preset frequency; the intensity analysis module 83 includes a segment determination submodule, which is used to respond to the first energy intensity of a continuous number of threshold audio segments that all meet the first condition, and use the continuous number of threshold audio segments as the segment to be analyzed; the intensity analysis module 83 includes an intensity calculation submodule, which is used to obtain a third energy intensity reflecting the gain characteristics of the current device based on the first energy intensity of the segment to be analyzed at the preset frequency and the second energy intensity of the segment to be analyzed as a whole; wherein the first condition includes that the first energy intensity is higher than the second intensity threshold.

[0118] Therefore, in response to the fact that the first energy intensities of a consecutive number of threshold audio clips all satisfy the first condition, the consecutive number of threshold audio clips are used as segments to be analyzed, and based on the first energy intensity of the segment to be analyzed at the preset frequency and the second energy intensity of the segment to be analyzed as a whole, a third energy intensity reflecting the gain characteristics of the current device is obtained, and the first condition includes that the first energy intensity is higher than the second intensity threshold, so the sound card device can send a complete gain consistency automatic compensation signal to the pickup device side and implement a gain compensation coefficient determination, which is conducive to improving gain consistency.

[0119] In some disclosed embodiments, the gain compensation device 80 also includes a loop execution module for responding to the absence of a threshold number of consecutive audio segments in the current audio whose first energy intensities satisfy the first condition, combining the aforementioned audio receiving module 81, the intensity detection module 82 and the intensity analysis module 83 to re-execute the step of receiving the audio to be tested from each pickup device and subsequent steps until a third energy intensity reflecting the gain characteristics of the current device is obtained.

[0120] Therefore, when there are no first energy intensities of a threshold number of consecutive audio segments in the current audio that satisfy the first condition, re-execute the step of receiving the audio to be tested from each sound pickup device and subsequent steps until a third energy intensity reflecting the gain characteristics of the current device is obtained, which can help improve the robustness of gain compensation.

[0121] In some disclosed embodiments, the gain compensation device 80 also includes an intensity update module for updating the third energy intensity of the current device to the third energy intensity acquired during the current re-execution in response to acquiring the third energy intensity of the current device during the current re-execution and the third energy intensity of the current device having been acquired before the current re-execution.

[0122] Therefore, in the process of acquiring the third energy intensity of each sound pickup device, the third energy intensity of each sound pickup device can be kept to be the latest acquired, which is conducive to improving the accuracy of the gain compensation coefficient.

[0123] In some disclosed embodiments, the third energy intensity is positively correlated with the gain compensation coefficient.

[0124] Therefore, useful signals can be amplified and irrelevant signals can be suppressed, which is beneficial to improving signal quality.

[0125] In some disclosed embodiments, the coefficient calculation module 84 includes an intensity selection submodule for selecting the third energy intensity of one of the pickup devices as the reference energy intensity; the coefficient calculation module 84 includes an intensity ratio submodule for respectively using the ratio between the third energy intensity of each pickup device and the reference energy intensity as the gain compensation coefficient of the corresponding pickup device.

[0126] Therefore, the gain compensation coefficient of each sound pickup device can be obtained through simple mathematical operations, which can help reduce the complexity of gain compensation coefficient calculation.

[0127] In some disclosed embodiments, the gain compensation device 80 also includes a speech receiving module for receiving speech audio from each sound pickup device; the gain compensation device 80 also includes a speech gain module for performing gain compensation on the speech audio of each sound pickup device using the gain compensation coefficient of each sound pickup device to obtain the gain audio of each sound pickup device; the gain compensation device 80 also includes a speech recognition module for obtaining a speech recognition result of each sound pickup device based on the gain audio of each sound pickup device.

[0128] Therefore, by using the gain compensation coefficient of each sound pickup device to perform gain compensation on the voice audio of each sound pickup device, and then performing voice recognition on this basis, it is beneficial to improve the accuracy of voice recognition and role differentiation.

[0129] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a sound pickup device 90 of the present application. The sound pickup device 90 includes a microphone array 91, a memory 92 and a processor 93. The microphone array 91 and the memory 92 are coupled to the processor 93. The memory 92 stores program instructions. The processor 93 is used to execute the program instructions to implement the steps in the above-mentioned gain compensation method embodiment.

[0130] Specifically, the processor 93 is used to control itself, the microphone array 91, and the memory 92 to implement the steps in the above-mentioned gain compensation method embodiment. The processor 93 can also be called a CPU (Central Processing Unit). The processor 93 may be an integrated circuit chip with signal processing capabilities. The processor 93 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 93 can be implemented by an integrated circuit chip.

[0131] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0132] In some disclosed embodiments, the sound pickup device 90 further includes an analog-to-digital converter (not shown) and a digital-to-analog converter (not shown), the input end of the analog-to-digital converter is connected to the microphone array 91, the output end of the analog-to-digital converter is connected to the input end of the processor 93, the input end of the digital-to-analog converter is connected to the output end of the processor 93, and the output end of the digital-to-analog converter is used to connect to the sound card device. Figure 2 As well as the related descriptions in the aforementioned disclosed embodiments, they will not be repeated here.

[0133] See also Fig.10, Fig.10 1 is a schematic diagram of a sound card device 100 of the present application. The sound card device 100 includes a memory 101 and a processor 102 coupled to each other. The memory 101 stores program instructions. The processor 102 is used to execute the program instructions to implement the steps in the above gain compensation method embodiment.

[0134] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned physical condition scoring method embodiments. The processor 102 can also be referred to as a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip having signal processing capabilities. The processor 102 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 102 may be implemented by an integrated circuit chip.

[0135] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0136] See also Fig.11 , Fig.111 is a schematic diagram of a framework of an embodiment of a gain compensation system 110 of the present application. The gain compensation system 110 includes a sound card device 111 and a plurality of sound pickup devices 112, and the plurality of sound pickup devices 112 are connected to the sound card device 111, respectively. It should be noted that the sound pickup device 112 is used to replace the audio data of the preset frequency in the invalid section of the ambient audio collected at the place of use thereof to obtain the audio to be tested, and send the audio to be tested to the sound card device 111, and the energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold. The sound card device 111 is used to detect the first energy intensity of the audio to be tested from each sound pickup device 112 at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristics of the sound pickup device 112, and based on the third energy intensity of each sound pickup device 112, obtain the gain compensation coefficient of each sound pickup device 112. In addition, the steps respectively performed by the sound card device 111 and the sound pickup device 112 in the gain compensation process can refer to the relevant descriptions in the aforementioned disclosed embodiments, and will not be repeated here. Fig.11 As described in the aforementioned disclosed embodiments, in order to realize voice recognition and role differentiation, the sound card device 111 can also be connected to a server to realize voice recognition and role differentiation through the server. For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0137] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0138] See also Fig.12 , Fig.12 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 120 of the present application. The computer-readable storage medium 120 stores program instructions 121 that can be executed by a processor, and the program instructions 121 are used to implement the steps in any of the above-mentioned gain compensation method embodiments.

[0139] The above scheme, on the one hand, can automatically complete the determination of the gain compensation coefficient without human participation, which is conducive to reducing the complexity, difficulty and cost of determining the gain compensation coefficient, and is conducive to improving the user experience. On the other hand, since each sound pickup device uses the same method to replace the audio, and when the first energy intensity of the audio to be measured at the preset frequency of the sound pickup device meets certain conditions, the sound card device uses the same method to analyze the coefficient of each sound pickup device, which is conducive to improving the consistency of gain compensation. Therefore, it is possible to reduce the degree of human participation in the process of determining the gain compensation coefficient as much as possible, and to improve the accuracy of the gain compensation coefficient as much as possible.

[0140] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0141] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A gain compensation method, characterized in that: include: Obtaining the ambient audio collected by the sound pickup device at the place where it is used; Replacing the invalid section in the ambient audio with audio data of a preset frequency to obtain the audio to be tested; wherein the energy intensity of the human voice signal in the invalid section is not higher than a first intensity threshold; The audio to be tested is sent to a sound card device; wherein the sound card device is used to detect a first energy intensity of the audio to be tested from each of the sound pickup devices at the preset frequency, and in response to the first energy intensity satisfying a first condition, a third energy intensity for reflecting the gain characteristic of the sound pickup device is obtained based on the first energy intensity and the second energy intensity of the audio to be tested, and a gain compensation coefficient of each of the sound pickup devices is obtained based on the third energy intensity of each of the sound pickup devices, the second energy intensity is the overall energy intensity of the section to be analyzed in the audio to be tested, the third energy intensity is obtained by subtracting the first energy intensity of the section to be analyzed at the preset frequency from the second energy intensity, and the third energy intensity is positively correlated with the gain compensation coefficient.

2. The method according to claim 1, characterized in that The ambient audio includes a plurality of audio segments, and the number of the audio segments in the invalid section is no greater than a quantity threshold.

3. The method according to claim 2, characterized in that The step of replacing the invalid section in the ambient audio with audio data of a preset frequency to obtain the audio to be tested includes: Taking the plurality of audio segments as current segments respectively; Detecting the energy intensity of the human voice signal in the current segment, and counting the number of audio segments that are replaced continuously starting from the audio segment before the current segment; In response to the energy intensity of the human voice signal in the current segment satisfying an intensity condition and the number of segments satisfying a number condition, replacing the current segment with the audio data; The intensity condition includes that the energy intensity of the human voice signal is not higher than the first intensity threshold, and the quantity condition includes that the number of the segments is less than the quantity threshold.

4. The method according to claim 2, characterized in that: The audio segment is a digital signal including a preset number of sampling points, and the audio data replacing the audio segment in the invalid section is also a digital signal including the preset number of sampling points.

5. The method according to claim 1, characterized in that Before replacing the invalid segment in the ambient audio with audio data of a preset frequency, the method further includes: Detect whether the current state satisfies the second condition; In response to the current state satisfying the second condition, replacing the invalid segment in the ambient audio with the audio data; The current state includes the current time, and the second condition includes any one of the following: the current time is within a preset time period, and the current time exceeds a preset interval from the last execution of the gain compensation method.

6. The method according to claim 1, characterized in that The sound pickup device includes a plurality of microphones arranged in an array, and the ambient audio includes a plurality of audio clips. The step of obtaining the ambient audio collected by the sound pickup device at the place where the sound pickup device is used includes: Converting the analog audio collected by the microphones into digital audio respectively; Performing windowing on the digital audio corresponding to each of the microphones to obtain a plurality of first segments; Using beamforming to fuse the first segments corresponding to the plurality of microphones to obtain fused audio; wherein the fused audio includes the plurality of second segments; Noise suppression is performed on the fused audio to obtain the ambient audio.

7. A gain compensation method, characterized in that: include: Receive the audio to be tested from each pickup device; Respectively detecting the first energy intensity of each of the audio frequencies to be tested at a preset frequency; In response to the first energy intensity satisfying a first condition, obtaining a third energy intensity reflecting a gain characteristic of the sound pickup device based on the first energy intensity and a second energy intensity of the audio to be measured; Based on the third energy intensity of each of the sound pickup devices, obtaining a gain compensation coefficient of each of the sound pickup devices; Among them, the sound pickup device is used to replace the invalid segment in the ambient audio collected at the place where it is used with the audio data of the preset frequency to obtain the audio to be tested, and send the audio to be tested to the sound card device, the energy intensity of the human voice signal in the invalid segment is not higher than the first intensity threshold, the second energy intensity is the overall energy intensity of the segment to be analyzed in the audio to be tested, the third energy intensity is obtained by subtracting the first energy intensity of the segment to be analyzed at the preset frequency from the second energy intensity, and the third energy intensity is positively correlated with the gain compensation coefficient.

8. The method according to claim 7, characterized in that The ambient audio includes a plurality of audio segments, the number of the audio segments in the invalid section is no more than a quantity threshold, and the respectively detecting the first energy intensity of each of the audios to be tested at a preset frequency includes: Using each of the sound pickup devices as a current device, and using the audio to be tested from the current device as the current audio; Detecting a first energy intensity of each of the audio segments in the current audio at the preset frequency; In response to the first energy intensity satisfying a first condition, obtaining a third energy intensity reflecting a gain characteristic of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be measured, comprises: In response to the first energy intensities of the threshold number of consecutive audio segments all satisfying the first condition, taking the threshold number of consecutive audio segments as segments to be analyzed; Based on the first energy intensity of the to-be-analyzed section at the preset frequency and the second energy intensity of the to-be-analyzed section as a whole, obtaining a third energy intensity reflecting the gain characteristic of the current device; The first condition includes that the first energy intensity is higher than a second intensity threshold.

9. The method according to claim 8, characterized in that The method further comprises: In response to the fact that there are not a plurality of consecutive audio segments in the current audio whose first energy intensities satisfy the first condition, the step of receiving the audio to be tested from each sound pickup device and subsequent steps are re-executed until a third energy intensity reflecting the gain characteristic of the current device is obtained.

10. The method according to claim 9, characterized in that During each re-execution, the method further includes: In response to obtaining the third energy intensity of the current device during the re-execution and having obtained the third energy intensity of the current device before the re-execution, the third energy intensity of the current device is updated to the third energy intensity obtained during the re-execution.

11. The method according to claim 7, characterized in that The step of obtaining a gain compensation coefficient of each of the sound pickup devices based on the third energy intensity of each of the sound pickup devices comprises: selecting a third energy intensity of one of the sound pickup devices as a reference energy intensity; The ratio between the third energy intensity of each of the sound pickup devices and the reference energy intensity is respectively used as the gain compensation coefficient of the corresponding sound pickup device.

12. The method according to claim 7, characterized in that After obtaining the gain compensation coefficient of each of the sound pickup devices based on the third energy intensity of each of the sound pickup devices, the method further includes: Receiving voice audio from each of the sound pickup devices; Performing gain compensation on the voice audio of each sound pickup device using the gain compensation coefficient of each sound pickup device to obtain the gain audio of each sound pickup device; Based on the gain audio of each of the sound pickup devices, the speech recognition results of each of the sound pickup devices are obtained.

13. A gain compensation device, characterized in that: include: An audio collection module, used to obtain the ambient audio collected by the sound collection device at the place where it is used; An audio replacement module, used to replace the invalid section in the ambient audio with audio data of a preset frequency to obtain the audio to be tested; wherein the energy intensity of the human voice signal in the invalid section is not higher than a first intensity threshold; An audio sending module is used to send the audio to be tested to a sound card device; wherein the sound card device is used to detect a first energy intensity of the audio to be tested from each of the sound pickup devices at the preset frequency, and in response to the first energy intensity satisfying a first condition, obtain a third energy intensity for reflecting the gain characteristic of the sound pickup device based on the first energy intensity and the second energy intensity of the audio to be tested, and obtain a gain compensation coefficient of each of the sound pickup devices based on the third energy intensity of each of the sound pickup devices, wherein the second energy intensity is the overall energy intensity of the section to be analyzed in the audio to be tested, and the third energy intensity is obtained by subtracting the first energy intensity of the section to be analyzed at the preset frequency from the second energy intensity, and the third energy intensity is positively correlated with the gain compensation coefficient.

14. A gain compensation device, characterized in that: include: An audio receiving module, used for receiving the audio to be tested from each sound pickup device; An intensity detection module, used to respectively detect the first energy intensity of each of the audio frequencies to be tested at a preset frequency; an intensity analysis module, configured to obtain, in response to the first energy intensity satisfying a first condition, a third energy intensity reflecting a gain characteristic of the sound pickup device based on the first energy intensity and a second energy intensity of the audio to be measured; A coefficient calculation module, used for obtaining a gain compensation coefficient of each of the sound pickup devices based on the third energy intensity of each of the sound pickup devices; Among them, the sound pickup device is used to replace the invalid segment in the ambient audio collected at the place where it is used with the audio data of the preset frequency to obtain the audio to be tested, and send the audio to be tested to the sound card device, the energy intensity of the human voice signal in the invalid segment is not higher than the first intensity threshold, the second energy intensity is the overall energy intensity of the segment to be analyzed in the audio to be tested, the third energy intensity is obtained by subtracting the first energy intensity of the segment to be analyzed at the preset frequency from the second energy intensity, and the third energy intensity is positively correlated with the gain compensation coefficient.

15. A sound pickup device, characterized in that: The invention comprises a microphone array, a memory and a processor, wherein the microphone array and the memory are coupled to the processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the gain compensation method according to any one of claims 1 to 6.

16. The device according to claim 15, characterized in that The sound pickup device also includes an analog-to-digital converter and a digital-to-analog converter, the input end of the analog-to-digital converter is connected to the microphone array, the output end of the analog-to-digital converter is connected to the input end of the processor, the input end of the digital-to-analog converter is connected to the output end of the processor, and the output end of the digital-to-analog converter is used to connect to a sound card device.

17. A sound card device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the gain compensation method according to any one of claims 7 to 12.

18. A gain compensation system, characterized in that: It includes a sound card device and a plurality of sound pickup devices, wherein the plurality of sound pickup devices are respectively connected to the sound card device; The sound pickup device is used to replace an invalid section in the ambient audio collected at the place where it is used with audio data of a preset frequency to obtain the audio to be tested, and send the audio to be tested to the sound card device. The energy intensity of the human voice signal in the invalid section is not higher than the first intensity threshold. The sound card device is used to detect the first energy intensity of the audio to be tested from each of the sound pickup devices at the preset frequency, and in response to the first energy intensity satisfying the first condition, based on the first energy intensity and the second energy intensity of the audio to be tested, obtain a third energy intensity for reflecting the gain characteristic of the sound pickup device, and based on the third energy intensity of each of the sound pickup devices, obtain a gain compensation coefficient of each of the sound pickup devices, the second energy intensity is the energy intensity of the entire section to be analyzed in the audio to be tested, the third energy intensity is obtained by subtracting the first energy intensity of the section to be analyzed at the preset frequency from the second energy intensity, and the third energy intensity is positively correlated with the gain compensation coefficient.

19. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the gain compensation method according to any one of claims 1 to 12.

Images