Audio codec device, method and system with beamforming function

Abstract

The present application provides an audio codec device, method and system with a beamforming function. By integrating the beamforming function into the audio codec, the use process of the microphone beamforming technology is greatly simplified, so that the user only needs to input the relative position of the microphone, and the audio codec can automatically perform beamforming processing; at the same time, based on the relative position of the microphone, the parameter configuration can also be automatically calculated, so that the system can adapt to a variety of microphone array layouts, and get rid of the problem that the beamforming technology is highly dependent on the microphone array configuration; in addition, the codec with beamforming function also alleviates the demand for computing power in the prior art, significantly reduces the computing burden of the main processor, reduces the overall cost of the product and simplifies the design and production process.

Description

Technical Field

[0001] The present application relates to the field of audio coding and decoding, and in particular to an audio coding and decoding device, method and system with beamforming function. Background Art

[0002] The audio codec system is a core technology in the field of communications and media. Its main function is to encode and decode audio signals for efficient transmission and storage between different devices and platforms. The audio codec system compresses audio files, reduces storage space and transmission bandwidth requirements, while maintaining the original audio quality. It is widely used in mobile communications, online music streaming, and film production. With the advancement of technology, the combination of audio codec systems and signal processing technology has become a major trend, which improves the efficiency and quality of codecs and enables them to process complex audio signals more intelligently. This combination promotes new applications and innovations, such as 3D audio rendering and personalized audio experience, opening up new possibilities for the future development of audio technology. Therefore, the integration of audio codec systems and signal processing technology is the key to promoting industry development and meeting user needs.

[0003] Microphone beamforming technology is an advanced signal processing method that captures sound directionally through an array of multiple microphones. This technology adjusts the phase and amplitude of each microphone signal so that the sound signal from a specific direction is enhanced, while the noise and interference from other directions are effectively suppressed. Microphone beamforming technology is widely used in conference systems, smart speakers, hearing aids, and noise suppression systems. The importance of this technology lies in that it can significantly improve the quality of audio capture, especially in multi-noise environments or far-field audio capture scenarios. For example, in a conference room environment, microphone beamforming can help capture the speaker's voice more clearly while reducing the interference of background noise.

[0004] Problems with existing microphone beamforming technology include high entry barriers, high dependence on array configuration, and high demand for computing power. First of all, conventional audio equipment usually does not include beamforming functions, and the beamforming function is highly dependent on professionals, which increases the cost of product development. Secondly, microphone beamforming depends on the layout and number of microphone arrays. Different application scenarios correspond to a variety of different arrays, which means that once the use environment or purpose is changed, the microphone array may need to be redesigned and reconfigured. Finally, microphone beamforming requires higher computing resources to perform real-time analysis and processing of signals collected by multiple microphones, which requires higher computing power, resulting in increased product research and development. Summary of the invention

[0005] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide an audio codec device, method and system with beamforming function, which is used to solve the problems of high entry threshold, high dependence on array configuration and high demand for computing power in the microphone beamforming technology of the prior art.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an audio codec device with a beamforming function, including: a signal acquisition unit, which is used to collect near-end audio analog signals through one or more pairs of audio acquisition devices; each audio acquisition device collects one of the near-end audio analog signals; an analog-to-digital conversion unit, which is electrically connected to the signal acquisition unit; the analog-to-digital conversion unit is used to perform analog-to-digital conversion operations, data matching operations and data caching operations on the near-end audio analog signals to generate corresponding near-end audio digital signals, and output the near-end audio digital signals to a beam processing unit; a beam processing unit, which is electrically connected to the analog-to-digital conversion unit; the beam processing unit includes two input ends and one output end, and the two input ends respectively input the near-end audio digital signal and the position information of the audio acquisition device sent by the control interface; the beam processing unit generates a beam signal through the near-end audio digital signal and the position information of the audio acquisition device, and outputs the beam signal to the audio interface through the output end.

[0007] In some embodiments of the first aspect of the present application, the beam processing unit includes: a Fourier transform module: which includes an input end and an output end; the input end is connected to the analog-to-digital conversion unit, and the output end is connected to the sound source direction estimation module; the Fourier transform module converts the received near-end audio digital signal from the time domain to the frequency domain to generate a frequency domain signal of the near-end audio digital signal, and sends the frequency domain signal of the near-end audio digital signal to the sound source direction estimation module; a sound source direction estimation module: which includes two input ends and an output end; the input end is connected to the Fourier transform module and the control interface, and the output end is connected to the beamforming module; the sound source direction estimation module performs a sound source directional operation based on the position information of the audio acquisition device and the frequency domain signal of the near-end audio digital signal to generate sound source directional information; and sends the frequency domain signal of the near-end audio digital signal to the sound source direction estimation module. The frequency domain signal of the digital signal and the sound source directional information are sent to the beamforming module; the beamforming module: includes two input ends and one output end; the two input ends are respectively connected to the sound source direction estimation module and the control interface, and the output end is connected to the inverse Fourier transform module; the beamforming module performs a beamforming operation based on the position information of the audio acquisition device, the sound source directional information, and the frequency domain signal of the near-end audio digital signal to generate a beam intermediate signal, and sends the beam intermediate signal to the inverse Fourier transform module; the inverse Fourier transform module: includes an input end and an output end; the input end is connected to the beamforming module, and the output end is connected to the audio interface; the inverse Fourier transform module converts the received beam intermediate signal from the frequency domain to the time domain to generate a beam signal, and sends the beam signal to the audio interface.

[0008] In some embodiments of the first aspect of the present application, the process of the sound source direction estimation module performing the sound source directional operation includes: receiving the frequency domain signal of the near-end audio digital signal sent by the Fourier transform module, and the position information of the audio acquisition device sent by the control interface; based on the position information of the audio acquisition device, calculating the sound source directional delay and the maximum distance of the audio acquisition device; based on the sound source directional delay and the maximum distance, calculating the sound source direction angle as the sound source directional information, and sending the sound source directional information to the beamforming module.

[0009] In some embodiments of the first aspect of the present application, the process of calculating the sound source directional delay and the maximum distance of the audio acquisition device includes: based on the position information of the audio acquisition device, selecting the corresponding frequency domain signals collected by a pair of audio acquisition devices with the farthest distance as the sound source reference signals, and calculating the position distance difference between the current pair of audio acquisition devices as the maximum distance; constructing a cross-correlation function of the sound source reference signals, and calculating the sound source directional delay between the sound source reference signals based on the cross-correlation function.

[0010] In some embodiments of the first aspect of the present application, the process of the beamforming module performing a beamforming operation includes: receiving the frequency domain signal and sound source orientation information of the near-end audio digital signal sent by the sound source direction estimation module, and the position information of the audio acquisition device sent by the control interface; based on the position information of the audio acquisition device, calculating the beamforming coefficient of the near-end audio digital signal; based on the beamforming coefficient, generating a beam signal through the frequency domain signal of the near-end audio digital signal.

[0011] In some embodiments of the first aspect of the present application, the process of calculating the beamforming coefficient of the near-end audio digital signal includes: calculating the sound wave arrival delay based on the position information of the audio acquisition device, and calculating the sound wave arrival vector of each audio acquisition device according to the sound wave arrival delay; calculating the covariance matrix of the near-end audio digital signal based on the frequency domain signal of the near-end audio digital signal; and calculating the beamforming coefficient of the near-end audio digital signal based on the covariance matrix and the sound wave arrival vector.

[0012] In some embodiments of the first aspect of the present application, the analog-to-digital conversion unit includes: an analog-to-digital conversion module: used to perform an analog-to-digital conversion operation on an analog signal of a near-end audio signal to generate an analog-to-digital intermediate signal; a data matching module: used to perform a format conversion matching operation on the analog-to-digital intermediate signal output by the analog-to-digital conversion module to generate a digital signal of the near-end audio signal; a data cache module: used to perform delay matching and data cache operations on the digital signal of the near-end audio signal.

[0013] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an audio codec method with beamforming function, which is applied to an audio codec, including: collecting near-end audio analog signals through one or more pairs of audio acquisition devices; each audio acquisition device collects one of the near-end audio analog signals; performing analog-to-digital conversion operations, data matching operations and data caching operations on the near-end audio analog signals to generate corresponding near-end audio digital signals; receiving the position information of the audio acquisition device, generating a beam signal through the near-end audio digital signal and the position information of the audio acquisition device, and outputting the beam signal to an audio interface.

[0014] In some embodiments of the second aspect of the present application, the process of generating a beam signal through a near-end audio digital signal and the position information of an audio collection device includes: converting the received near-end audio digital signal from the time domain to the frequency domain to generate a frequency domain signal of the near-end audio digital signal; performing a sound source orientation operation based on the position information of the audio collection device and the frequency domain signal of the near-end audio digital signal to generate sound source orientation information; performing a beamforming operation on the frequency domain signal of the near-end audio digital signal based on the position information of the audio collection device and the sound source orientation information to generate a beam intermediate signal; converting the beam intermediate signal from the frequency domain to the time domain to generate a beam signal.

[0015] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides an audio codec system with beamforming function, including: a main processor and the audio codec device with sound source directional function as described above.

[0016] As described above, the field of audio codec of the present application, in particular, relates to an audio codec device, method and system with beamforming function, which has the following beneficial effects: the present invention greatly simplifies the use process of microphone beamforming technology by integrating the beamforming function into the audio codec. The user only needs to input the relative position of the microphone, and the audio codec can automatically perform beamforming processing; at the same time, the parameter configuration can also be automatically calculated based on the relative position of the microphone, so that the system can adapt to a variety of microphone array layouts, and get rid of the problem that the beamforming technology is highly dependent on the microphone array configuration; in addition, the codec with beamforming function also alleviates the demand for computing power in the prior art, significantly reduces the computing burden of the main processor, reduces the overall cost of the product and simplifies the design and production process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the structure of external connections in an embodiment of an audio codec device with beamforming function of the present application is shown.

[0018] Figure 2 A schematic diagram of the structure of the internal connection in an embodiment of an audio codec device with beamforming function of the present application is shown.

[0019] Figure 3 A schematic diagram of the structure of an analog-to-digital conversion unit in an embodiment of an audio codec device with a beamforming function of the present application is shown.

[0020] Figure 4 A schematic diagram of the structure of a beamforming unit in an embodiment of an audio codec device with a beamforming function of the present application is shown.

[0021] Figure 5A schematic diagram of the structure of a digital-to-analog conversion unit in an embodiment of an audio codec device with beamforming function of the present application is shown.

[0022] Figure 6 A flowchart of a digital circuit implementation process in an embodiment of an audio codec device with beamforming function of the present application is shown.

[0023] Figure 7 A flowchart of an embodiment of an audio encoding and decoding method with beamforming function of the present application is shown.

[0024] Figure 8 A structural schematic diagram of an embodiment of an audio codec system with beamforming function of the present application is shown. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0026] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0027] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0028] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0029] In order to solve the problems in the above-mentioned background technology, the present invention provides an audio codec device, method and system with beamforming function, which aims to solve the problems of high entry threshold, high dependence on array configuration and high demand for computing power in the microphone beamforming technology of the prior art. At the same time, in order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the invention.

[0030] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0031] <1> Beamforming: A technique that adjusts the sensor elements in an array to enhance or suppress sound signals in a specific direction.

[0032] <2> MVDR beamforming coefficients: Coefficients used in minimum variance distortionless response beamforming techniques to calculate the weights for each sensor in the array to maximize the energy of the desired signal and minimize the effects of interference and noise.

[0033] <3> Fourier transform: A mathematical method for converting a time domain signal into a frequency domain signal, a technique that can decompose a signal into different frequency components.

[0034] <4> Cross-correlation function: A function used to measure the correlation between two signals, which can help determine the similarity and delay between the signals.

[0035] <5> Sound source direction estimation: A technology that estimates the direction of the sound signal's source by processing and analyzing the received sound signal.

[0036] <6> Sound wave arrival vector: A vector that describes the path of a sound wave, including the distance from the source to the receiver and the direction of propagation.

[0037] The embodiments of the present invention provide an audio codec device with a beamforming function, an audio codec method with a beamforming function, and an audio codec system for implementing the audio codec device with a beamforming function. As far as the implementation of the audio codec device with a beamforming function is concerned, the embodiments of the present invention will describe an exemplary implementation scenario of the audio codec device with a beamforming function.

[0038] like Figure 1 and Figure 2 FIG. 1 is a schematic diagram showing the structure of an audio codec device with beamforming function in an embodiment of the present invention. The audio codec method with beamforming function in this embodiment mainly includes the following structures.

[0039] In one embodiment of the present invention, an audio codec device with a beamforming function includes: a signal acquisition unit, which is used to collect near-end audio analog signals through one or more pairs of audio acquisition devices; each audio acquisition device collects one of the near-end audio analog signals; an analog-to-digital conversion unit, which is electrically connected to the signal acquisition unit; the analog-to-digital conversion unit is used to perform analog-to-digital conversion operations, data matching operations, and data caching operations on the near-end audio analog signals to generate corresponding near-end audio digital signals, and output the near-end audio digital signals to a beam processing unit; a beam processing unit, which is electrically connected to the analog-to-digital conversion unit; the beam processing unit includes two input ends and one output end, and the two input ends respectively input the near-end audio digital signal and the position information of the audio acquisition device sent by the control interface; the beam processing unit generates a beam signal through the near-end audio digital signal and the position information of the audio acquisition device, and outputs the beam signal to the audio interface through the output end.

[0040] In one embodiment of the present invention, an audio codec refers to a device that encodes / decodes audio data, converting audio data from one format to another format to reduce data calculation, transmission and storage costs. The audio codec includes three parts. The first part is the uplink (recording) path, which includes an analog-to-digital conversion unit, a data format conversion unit, a data cache unit, and a data interface; the second part is the downlink (playback) path, which includes a data cache unit, a data format conversion unit, a mixing unit (multiple audio streams), and a digital-to-analog conversion unit; the third part is a control interface. The audio codec system of the present invention adds a beamforming (BF, BeamForming) processing unit to the architecture of the traditional audio codec system, thereby realizing real-time beamforming in the encoding and decoding link, thereby lowering the threshold for using beamforming, simplifying the use process and reducing the cost of use.

[0041] In one embodiment of the present invention, the audio input device refers to an audio acquisition device for an audio codec to obtain an audio stream from the outside, wherein the audio input device that can be used includes but is not limited to a digital microphone, an analog microphone, a sensor, or other forms of sound-to-electric conversion devices. The audio input device converts an acoustic signal into an electrical signal, and its input is a sound wave signal and its output is a voltage signal. The input audio collected by the audio input device used in the present invention is a multi-channel signal, so as to facilitate the subsequent beamforming operation.

[0042] In one embodiment of the present invention, the control interface serves as a channel for the main processor to send commands to the audio codec and transmit microphone position information. At the same time, the audio codec also feeds back the operating status to the main processor through the control interface. The interface protocol used by the control interface includes but is not limited to I2C protocol, Soundwire protocol, Slimbus protocol, HDA and other interface protocols. The user can also communicate control instructions with the codec through the main processor and the control interface. In particular, in the present invention, the control interface is also used to send the microphone position information to the beamforming module.

[0043] In one embodiment of the present invention, the data interface is used as a channel for audio data interaction between the main processor and the audio codec. The functions implemented include sending the original audio stream to the main processor through the data interface, and the main processor sending the remote audio signal or the locally stored audio signal to the audio codec through the data interface. The interface protocols used include but are not limited to the I2C protocol, Soundwire protocol, Slimbus protocol, HDA and other interface protocols.

[0044] In one embodiment of the present invention, the main processor refers to a SOC (System On Chip) processing chip, which is responsible for the main control function of the device. It receives and processes the audio data of the audio codec upstream, or transmits the audio data to the audio codec downstream channel for further processing. At the same time, it monitors the status of the audio codec through the control interface, or sends parameter configuration to it.

[0045] In one embodiment of the present invention, audio output refers to the audio codec outputting the audio stream to an external sound playback device, which may be a receiver, a speaker or an earphone, or a combination of the above three, and may be a single channel or a multi-channel, to complete the function of converting an electrical signal into an acoustic signal. Its input is a voltage signal, and its output is an acoustic wave signal.

[0046] like Figure 3The following shows a schematic structural diagram of an analog-to-digital conversion unit in an embodiment of the present invention. The analog-to-digital conversion unit includes: an analog-to-digital conversion module: for performing an analog-to-digital conversion operation on the analog signal of the proximal audio signal to generate an analog-to-digital intermediate signal; a data matching module: for performing a format conversion and matching operation on the analog-to-digital intermediate signal output by the analog-to-digital conversion module to generate a digital signal of the proximal audio signal; a data caching module: for performing a delay matching and data caching operation on the digital signal of the proximal audio signal.

[0047] In an embodiment of the present invention, the analog-to-digital conversion module is mainly responsible for converting an external audio input signal into a digital signal that meets the processing requirements of the current system. The analog-to-digital conversion module includes: an analog-to-digital conversion module, a data matching module, and a data caching module. The input signal of this module is an audio input signal (analog audio signal), and the output is a digital signal that meets the processing requirements of the current system. The input signal of this module is a multi-channel signal, and the number of channels of the output signal is the same as that of the input signal. The analog-to-digital conversion module collects the analog audio stream input by the audio input, and through sampling and quantization, converts it into a digital audio stream so that subsequent processing is based on digital discrete signals.

[0048] In an embodiment of the present invention, the analog-to-digital conversion module collects the analog audio stream input by the audio input, and through sampling and quantization, converts it into a digital audio stream so that subsequent processing is based on digital discrete signals.

[0049] x(n) = x(nT), -∞ < n < ∞ (Formula 1)

[0050] x q (n) = Q[x(n)] (Formula 2)

[0051] Formula 1 shows the process of sampling the analog audio stream in the analog-to-digital conversion module, where the input analog continuous signal x(t) is discretely sampled at a sampling period T, and the output is x(n); Formula 2 shows the process of quantization operation. Specifically, the output x(n) of the sampling process is discretized in amplitude through the quantization function Q, and the output is x_q(n). After sampling and quantization, the conversion of the analog continuous signal into the output digital discrete signal is achieved. The input of this module is a multi-channel audio input signal (analog audio signal), and the output is an analog-to-digital intermediate signal.

[0052] In an embodiment of the present invention, the data matching module performs a certain format conversion on the converted digital audio stream as required, including the matching of the sampling rate, the matching of the signal bit width, etc.

[0053]

[0054]

[0055] Wherein, Formula 3 is the filtering and sampling rate conversion of the module input signal, x(n) is the output of the analog-to-digital conversion module, M is the downsampling change factor, I is the upsampling change factor, h(k) is the unit impulse response function, and the output y d (n) or y u (n). Formula 4 is the signal bit width matching. According to the positive or negative of the shift bit width B, it is determined whether the bit width matching is left shift amplification or right shift reduction. The input signal of this module is the analog-to-digital intermediate signal, and the output is a digital signal that meets the current system processing requirements.

[0056] In one embodiment of the present invention, the data buffer module is used to buffer a certain amount of data in the audio interface to avoid repeated sampling or loss of audio data due to jitter introduced by system clock design defects on both sides of the interface. The input of this module is a digital signal, and there is no output data stream.

[0057] Figure 4 A structural schematic diagram of a beam processing unit in an embodiment of the present invention is shown, and the beam processing unit includes: a Fourier transform module: which includes an input end and an output end; the input end is connected to the analog-to-digital conversion unit, and the output end is connected to the sound source direction estimation module; the Fourier transform module converts the received near-end audio digital signal from the time domain to the frequency domain to generate a frequency domain signal of the near-end audio digital signal, and sends the frequency domain signal of the near-end audio digital signal to the sound source direction estimation module; a sound source direction estimation module: which includes two input ends and an output end; the input end is connected to the Fourier transform module and the control interface, and the output end is connected to the beam forming module; the sound source direction estimation module performs a sound source directional operation based on the position information of the audio acquisition device and the frequency domain signal of the near-end audio digital signal to generate sound source directional information; and The frequency domain signal of the near-end audio digital signal and the sound source directional information are sent to the beamforming module; the beamforming module: includes two input ends and one output end; the two input ends are respectively connected to the sound source direction estimation module and the control interface, and the output end is connected to the inverse Fourier transform module; the beamforming module performs a beamforming operation based on the position information of the audio acquisition device, the sound source directional information, and the frequency domain signal of the near-end audio digital signal to generate a beam intermediate signal, and sends the beam intermediate signal to the inverse Fourier transform module; the inverse Fourier transform module: includes an input end and an output end; the input end is connected to the beamforming module, and the output end is connected to the audio interface; the inverse Fourier transform module converts the received beam intermediate signal from the frequency domain to the time domain to generate a beam signal, and sends the beam signal to the audio interface.

[0058] In one embodiment of the present invention, the beamforming unit implements the technology of directional transmission or reception of sound by using multiple microphones or speakers. Through beamforming, it is possible to focus or suppress sound in a specific direction, thereby improving the accuracy of speech recognition, improving the quality of voice communication, and reducing the impact of background noise in a noisy environment. The principle of beamforming is to use multiple microphones or speakers to receive or send sound signals, and synthesize a suitable beam pattern through a signal processing algorithm, so that the sound is enhanced in a specific direction and suppressed in other directions, so as to improve the clarity and recognizability of the sound signal.

[0059] In this embodiment, the beamforming unit sequentially inputs a short-time Fourier transform (STFT) circuit, a sound source direction estimation circuit, a minimum variance distortion-free response (MVDR) beamforming circuit, and an inverse STFT circuit. It should be noted that the design of the beamforming unit fully considers the needs of real-time processing, and realizes dynamic positioning and separation of sound sources in the environment by designing hardware circuit modules suitable for a variety of application scenarios, thereby providing users with an efficient and accurate audio processing solution.

[0060] In one embodiment of the present invention, the following microphone array configuration operation is performed before the near-end audio is input to perform Fourier transform (STFT), which is used to determine the basic configuration of the audio acquisition device, that is, the microphone array. When the microphone array is composed of N microphones, each microphone has a unique two-dimensional spatial position coordinate, the position coordinate of the microphone array is p i =(x i ,y i ), where i represents the i-th microphone, and i=1,2,…,N.

[0061] Furthermore, after configuring the audio acquisition device with a microphone array, the collected near-end audio signal is input into the Fourier transform module to convert the time domain signal into the frequency domain. Specifically, for each microphone, i, the time domain signal x i [n] Perform a short-time Fourier transform (STFT) to convert it to a frequency domain representation. The STFT is performed independently on each microphone signal and is calculated as follows.

[0062]

[0063] Among them, X i [m,k] represents the frequency domain signal of the ith microphone at time frame m and frequency point k. w[nm] is used to localize the signal x i [n] is the window function, L and N are the lengths of the window function and discrete Fourier transform (DFT), respectively.

[0064] In one embodiment of the present invention, the process of the sound source direction estimation module performing the sound source directional operation includes: receiving the frequency domain signal of the near-end audio digital signal sent by the Fourier transform module, and the position information of the audio acquisition device sent by the control interface; based on the position information of the audio acquisition device, calculating the sound source directional delay and the maximum distance of the audio acquisition device; based on the sound source directional delay and the maximum distance, calculating the sound source direction angle as the sound source directional information, and sending the sound source directional information to the beamforming module.

[0065] Furthermore, the process of calculating the sound source directional delay and the maximum distance of the audio acquisition device includes: based on the position information of the audio acquisition device, selecting the corresponding frequency domain signals collected by a pair of audio acquisition devices with the farthest distance as the sound source reference signal, and calculating the position distance difference between the current pair of audio acquisition devices as the maximum distance; constructing a cross-correlation function of the sound source reference signals, and calculating the sound source directional delay between the sound source reference signals based on the cross-correlation function.

[0066] In one embodiment of the present invention, the process of the sound source direction estimation module performing the sound source directional operation includes selecting a pair of microphones with the farthest distance from a plurality of audio input devices. Specifically, assuming that any two microphones in the array are p i and p j , whose coordinates are (x i ,y j ) and (x j ,y i ). Then p i and p j The Euclidean distance between is shown below.

[0067]

[0068] By calculating the Euclidean distance d between any two microphones ij , select the pair of microphones with the longest distance and record them as p a and p b .

[0069] Further, when selecting a pair of microphones p that are the farthest apart a and p b After the frequency domain signal is obtained, a and p b The delay estimation process includes cross-correlation function calculation and time difference of arrival calculation (TDOA). The cross-correlation function is used to determine the time delay or time difference between two signals. TDOA is used to determine the time difference of arrival of signals between different syllable collection devices. The cross-correlation function is calculated as follows: for each frequency point k, calculate p a and p b Signal Xa [m,k] and X b The cross-correlation function between [m,k].

[0070]

[0071] As shown in Formula 7, X a [m,k] and X b [m,k] represents microphone p a and p b The frequency domain signal at frequency point k, Represents X b The complex conjugate of [m,k].

[0072] The TODA estimation method is as follows: when the cross-correlation function C ab When [m,k] reaches its maximum value, p a and p b The time delay Δm(k) between them can be expressed as shown in Formula 8.

[0073] Δm(k)=arg max m |C ab [m,k]| (Formula 8)

[0074] The discrete time delay is then converted to the actual time delay τ using Equation 9 ab (k).

[0075]

[0076] where f s Indicates the sampling frequency of the signal.

[0077] In this embodiment, the actual time delay τ is calculated. ab (k) and the microphone p with the shortest distance a and p b After calculating the distance d between them, the sound source direction θ is further estimated.

[0078]

[0079] Where c is the speed of sound. Formula 10 uses the difference in the propagation time of sound waves between different microphones to estimate the direction of the sound source.

[0080] In one embodiment of the present invention, the process of the beamforming module performing a beamforming operation includes: receiving the frequency domain signal and sound source orientation information of the near-end audio digital signal sent by the sound source direction estimation module, and the position information of the audio acquisition device sent by the control interface; based on the position information of the audio acquisition device, calculating the beamforming coefficient of the near-end audio digital signal; based on the beamforming coefficient, generating a beam signal through the frequency domain signal of the near-end audio digital signal.

[0081] Furthermore, the process of calculating the beamforming coefficient of the near-end audio digital signal includes: calculating the sound wave arrival delay based on the position information of the audio acquisition device, and calculating the sound wave arrival vector of each audio acquisition device according to the sound wave arrival delay; calculating the covariance matrix of the near-end audio digital signal based on the frequency domain signal of the near-end audio digital signal; and calculating the beamforming coefficient of the near-end audio digital signal based on the covariance matrix and the sound wave arrival vector.

[0082] In one embodiment of the present invention, the specific process of the beam forming module in performing beam forming includes: calculating the time delay τ by a digital circuit i (θ), calculate the sound wave arrival vector a(k,θ) through the digital circuit, calculate the covariance matrix through the digital circuit, calculate the MVDR beamforming coefficient through the digital circuit, and form a beam model through the digital circuit.

[0083] Exemplarily, firstly, the vector a(k,θ) of the sound wave reaching each microphone is calculated according to the estimated sound source direction θ.

[0084] Equation 11 shows how to calculate the sound wave arrival vector.

[0085]

[0086] Among them, f is the frequency corresponding to frequency point k, f s is the sampling frequency.

[0087] Then, for each frequency point k, the covariance matrix is ​​calculated. The covariance matrix R(k) is represented as the average of the outer products of the signal vectors of all microphones at that frequency.

[0088]

[0089] where X[m,k] is the signal vector of all microphones at time frame m and frequency point k, X H [m,k] is its conjugate transpose.

[0090] Secondly, the minimum variance distortionless response (MVDR) beamforming coefficients are calculated based on the covariance matrix and the sound wave arrival vector. Specifically, for each frequency point k and the estimated sound source direction θ, the beamforming coefficients w(k,θ) are calculated by formula 13.

[0091]

[0092] Where R -1 (k) is the inverse matrix of the covariance matrix, and a(k,θ) is the sound wave arrival vector.

[0093] The signal vector Y[m,k] of all microphones at each frequency point k is obtained by combining the above beamforming coefficients, as shown in Formula 14.

[0094] Y[m,k]=w H (k,θ)X[m,k](Formula 14)

[0095] Finally, the frequency domain signal is converted to the time domain. In this embodiment, the weighted frequency domain signal Y[m,k] is converted back to the time domain signal y[n] through an inverse short-time Fourier transform (ISTFT), as shown in formula 15.

[0096]

[0097] Figure 5 The structural schematic diagram of the digital-to-analog conversion unit in one embodiment of the present invention is shown. The downlink path of the audio codec system also includes: a data cache unit, a data matching unit, a digital-to-analog conversion module and a mixing unit. When the downlink path receives the remote audio data stream sent by the main chip through the audio interface, it is first input into the data cache module to ensure stable transmission and processing of the data. At the same time, the data cache can also improve the transmission efficiency and processing speed of the data, and reduce data loss and delay. Subsequently, the digital signal of the cached data is input into the data matching unit to perform format matching, synchronization matching and quality matching, etc., to ensure that the input and output data are matched and synchronized, and avoid data distortion and errors.

[0098] Furthermore, the digital signal generates a mixing unit after the matching operation of the data matching unit. The mixing unit is used to mix and superimpose multiple downstream audio streams such as the remote audio stream transmitted by the audio codec main processor through the data interface or the locally stored audio stream, as shown in Formula 16.

[0099]

[0100] Wherein, formula (5) indicates that M input audio streams x(n) are accumulated and the mixed audio stream y(n) is output as a mixed signal.

[0101] Finally, the mixed signal is input into the digital-to-analog conversion unit to convert the digital audio stream processed by the codec into an analog audio stream.

[0102]

[0103] v = ky(t) (Formula 18)

[0104] Among them, formula 17 indicates that the sampling rate is increased to the analog processing sampling rate through the interpolation process, and the interpolation fitting is performed between the digital sample points. Formula 18 indicates the process of converting the digital signal into the analog signal. The input digital discrete signal y(n) is multiplied by the proportional factor k to obtain the output analog voltage signal v(t), realizing the function of converting the discrete digital signal into the continuous analog signal.

[0105] In one embodiment of the present invention, the present invention protects an audio codec device with a beamforming function, such as a digital circuit for audio codec with a beamforming function. Figure 6 A flowchart of the digital circuit implementation process of the beamforming unit in one embodiment of the present invention is shown. First, the floating-point code of the beamforming algorithm corresponding to the beamforming unit described above is fixed-pointed to improve the calculation efficiency and reduce the errors that may occur in floating-point operations. Then, the fixed-pointed code is input into the code simulation platform to perform performance simulation of the beamforming, and the algorithm code is adjusted according to the simulation results to prevent excessive performance loss of the beamforming, and generate fixed-point code with acceptable performance loss. The simulation languages ​​used include but are not limited to C language, C++ language and Matlab language.

[0106] Furthermore, the fixed-point code verified by simulation is encoded using Verilog to generate the RTL code of beamforming. In order to ensure the consistency of the RTL code of beamforming with the algorithm fixed-point code, UVM methodology or EDA tools are used for consistency verification. Among them, UVM (Universal Verification Methodology) is a methodology for verifying integrated circuit design. It is based on the verification methodology of System Verilog and aims to improve the reusability, maintainability and scalability of the verification environment. If the consistency check fails, the RTL code is modified; if the consistency check passes, the DC tool, constraint file and library file are used to generate the beamforming netlist based on the RTL code of beamforming, and then the constraint modification is performed according to the netlist design specification. Finally, the beamforming netlist is post-simulated, the netlist data of the remaining structure of the codec is added, and the final total layout file is generated using the layout tool so that the layout file can be finally delivered to the manufacturer for production.

[0107] In this embodiment, the DC tool is a synthesis tool developed by Synopsys, which is used to synthesize RTL-level design code into a gate-level netlist. Constraint files are usually used to constrain the design during synthesis or layout and routing. The constraint file contains constraints on timing, area, power consumption, etc. to ensure that the design meets specific performance and functional requirements in actual implementation. The library file contains the standard cell library (such as gates, triggers, etc.), I / O library and other commonly used logic elements used in the design. These library files contain the physical information of various standard cells for use in the synthesis and layout and routing process.

[0108] like Figure 7 As shown, a flowchart of an audio codec method with beamforming function in an embodiment of the present invention is shown, which is applied to an audio codec. The audio codec method with beamforming function in this embodiment mainly includes the following steps:

[0109] Step S71: collecting near-end audio analog signals through one or more pairs of audio collection devices; each audio collection device collects one near-end audio analog signal.

[0110] Step S72: performing analog-to-digital conversion, data matching and data buffering operations on the near-end audio analog signal to generate a corresponding near-end audio digital signal.

[0111] Step S73: receiving the location information of the audio acquisition device, generating a beam signal through the near-end audio digital signal and the location information of the audio acquisition device, and outputting the beam signal to the audio interface.

[0112] In one embodiment of the present invention, the process of generating a beam signal using a near-end audio digital signal and position information of an audio collection device includes: converting a received near-end audio digital signal from a time domain to a frequency domain to generate a frequency domain signal of the near-end audio digital signal; performing a sound source orientation operation based on the position information of the audio collection device and the frequency domain signal of the near-end audio digital signal to generate sound source orientation information; performing a beamforming operation on the frequency domain signal of the near-end audio digital signal based on the position information of the audio collection device and the sound source orientation information to generate a beam intermediate signal; and converting the beam intermediate signal from the frequency domain to the time domain to generate a beam signal.

[0113] like Figure 8 , which shows a flowchart of an audio codec system interaction process with beamforming function in an embodiment of the present invention. In this embodiment, the audio codec system 800 with beamforming function includes: a main processor 801 and an audio codec device 802 with sound source directional function as described above.

[0114] It should be noted that: the audio codec system with beamforming function provided in the above embodiment only uses the division of the above program modules as an example when performing the beamforming function of audio codec. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the system is divided into different program modules to complete all or part of the processing described above. In addition, the audio codec system with beamforming function provided in the above embodiment and the audio codec method embodiment with beamforming function belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0115] In summary, the present application provides an audio codec device, method and system with a beamforming function. The present invention provides a method for improving beamforming efficiency through an audio codec. By integrating the beamforming function into the audio codec, the use process of the microphone beamforming technology is greatly simplified. The user only needs to input the relative position of the microphone, and the audio codec can automatically perform beamforming processing; at the same time, the parameter configuration can also be automatically calculated based on the relative position of the microphone, so that the system can adapt to a variety of microphone array layouts, and get rid of the problem that the beamforming technology is highly dependent on the microphone array configuration; in addition, the codec with a beamforming function also alleviates the demand for computing power in the prior art, significantly reduces the computing burden of the main processor, reduces the overall cost of the product and simplifies the design and production process. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0116] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An audio codec device with beamforming function, characterized in that: include: A signal acquisition unit is used to acquire near-end audio analog signals through one or more pairs of audio acquisition devices; each audio acquisition device acquires one near-end audio analog signal; an analog-to-digital conversion unit, electrically connected to the signal acquisition unit; the analog-to-digital conversion unit is used to perform an analog-to-digital conversion operation, a data matching operation, and a data caching operation on the near-end audio analog signal to generate a corresponding near-end audio digital signal, and output the near-end audio digital signal to the beam processing unit; A beam processing unit is electrically connected to the analog-to-digital conversion unit; the beam processing unit includes two input terminals and one output terminal, the two input terminals respectively input the near-end audio digital signal and the position information of the audio acquisition device sent by the control interface; the beam processing unit generates a beam signal through the near-end audio digital signal and the position information of the audio acquisition device, and outputs the beam signal to the audio interface through the output terminal; including: a sound source direction estimation module: including two input terminals and one output terminal; the input terminal is connected to the Fourier transform module and the control interface, and the output terminal is connected to the beam forming module; the sound source direction estimation module is based on the audio acquisition device The method comprises the steps of: performing a sound source directional operation based on the position information of the audio acquisition device and the frequency domain signal of the near-end audio digital signal to generate the sound source directional information; and sending the frequency domain signal of the near-end audio digital signal and the sound source directional information to the beamforming module; the beamforming module comprises two input terminals and one output terminal; the two input terminals are respectively connected to the sound source direction estimation module and the control interface, and the output terminal is connected to the inverse Fourier transform module; the beamforming module performs a beamforming operation based on the position information of the audio acquisition device, the sound source directional information, and the frequency domain signal of the near-end audio digital signal to generate a beam intermediate signal, and sending the beam intermediate signal to the inverse Fourier transform module; The beam processing unit includes a plurality of hardware circuit modules, and the plurality of hardware circuit modules include: a short-time Fourier transform circuit, a sound source direction estimation circuit, a minimum variance distortion-free response beamforming circuit, and an inverse short-time Fourier transform circuit.

2. The audio codec device with beamforming function according to claim 1, characterized in that: The beam processing unit comprises: Fourier transform module: comprising an input end and an output end; the input end is connected to the analog-to-digital conversion unit, and the output end is connected to the sound source direction estimation module; the Fourier transform module converts the received near-end audio digital signal from the time domain to the frequency domain to generate a frequency domain signal of the near-end audio digital signal, and sends the frequency domain signal of the near-end audio digital signal to the sound source direction estimation module; An inverse Fourier transform module: the inverse Fourier transform module comprises an input end and an output end; the input end is connected to the beamforming module, and the output end is connected to the audio interface; the inverse Fourier transform module converts the received beam intermediate signal from the frequency domain to the time domain to generate a beam signal, and sends the beam signal to the audio interface.

3. The audio codec device with beamforming function according to claim 1, characterized in that: The process of the sound source direction estimation module performing the sound source directional operation includes: Receiving the frequency domain signal of the near-end audio digital signal sent by the Fourier transform module and the position information of the audio acquisition device sent by the control interface; Based on the position information of the audio acquisition device, the directional delay of the sound source and the maximum distance of the audio acquisition device are calculated; Based on the sound source directional delay and the maximum distance, the sound source direction angle is calculated as the sound source directional information, and the sound source directional information is sent to the beamforming module.

4. The audio codec device with beamforming function according to claim 3, characterized in that: The process of calculating the directional delay of the sound source and the maximum distance of the audio capture device includes: Based on the position information of the audio acquisition devices, select the corresponding frequency domain signals collected by the pair of audio acquisition devices with the farthest distance as the sound source reference signal, and calculate the position distance difference between the current pair of audio acquisition devices as the maximum distance; A cross-correlation function of the sound source reference signals is constructed, and the sound source directional delay between the sound source reference signals is calculated based on the cross-correlation function.

5. The audio codec device with beamforming function according to claim 2, characterized in that: The process of the beamforming module performing the beamforming operation includes: Receiving the frequency domain signal and the sound source orientation information of the near-end audio digital signal sent by the sound source direction estimation module, and the position information of the audio acquisition device sent by the control interface; Calculate the beamforming coefficient of the near-end audio digital signal based on the position information of the audio acquisition device; A beam signal is generated by a frequency domain signal of the near-end audio digital signal based on the beamforming coefficient.

6. The audio codec device with beamforming function according to claim 5, characterized in that: The process of calculating the beamforming coefficients of the near-end audio digital signal includes: Calculate the sound wave arrival delay based on the position information of the audio acquisition device, and calculate the sound wave arrival vector of each audio acquisition device according to the sound wave arrival delay; Calculating a covariance matrix of the near-end audio digital signal based on the frequency domain signal of the near-end audio digital signal; Based on the covariance matrix and the sound wave arrival vector, a beamforming coefficient of the near-end audio digital signal is calculated.

7. The audio codec device with beamforming function according to claim 1, characterized in that: The analog-to-digital conversion unit includes: Analog-to-digital conversion module: used for performing an analog-to-digital conversion operation on an analog signal of a near-end audio signal to generate an analog-to-digital intermediate signal; A data matching module: used for performing a format conversion and matching operation on the analog-to-digital intermediate signal output by the analog-to-digital conversion module to generate a digital signal of a near-end audio signal; Data cache module: used to perform delay matching and data cache operations on the digital signal of the near-end audio signal.

8. An audio codec method with beamforming function, applied to the audio codec device with sound source directional function as claimed in any one of claims 1 to 7, characterized in that: The audio codec device includes a beam processing unit, the beam processing unit includes a plurality of hardware circuit modules, the plurality of hardware circuit modules include: a short-time Fourier transform circuit, a sound source direction estimation circuit, a minimum variance distortion-free response beamforming circuit and an inverse short-time Fourier transform circuit, and the method includes: The near-end audio analog signal is collected by one or more pairs of audio collection devices; each audio collection device collects one of the near-end audio analog signals; Performing an analog-to-digital conversion operation, a data matching operation, and a data buffering operation on the near-end audio analog signal to generate a corresponding near-end audio digital signal; Receive the position information of the audio acquisition device, generate a beam signal through the near-end audio digital signal and the position information of the audio acquisition device, and output the beam signal to the audio interface; the process of generating the beam signal includes: converting the received near-end audio digital signal from the time domain to the frequency domain to generate a frequency domain signal of the near-end audio digital signal; performing a sound source directional operation based on the position information of the audio acquisition device and the frequency domain signal of the near-end audio digital signal to generate sound source directional information; performing a beamforming operation on the frequency domain signal of the near-end audio digital signal based on the position information of the audio acquisition device and the sound source directional information to generate a beam intermediate signal; converting the beam intermediate signal from the frequency domain to the time domain to generate a beam signal.

9. An audio codec system with beamforming function, characterized in that: include: A main processor and an audio codec device with a sound source directional function as claimed in any one of claims 1 to 7.

Images