Far-field microphone sound capture optimization method and related equipment

Abstract

The invention provides a far-field microphone sound capture optimization method and related equipment, and the method comprises the steps: taking an obtained audio collected by a near-field microphone array as an optimization target, taking the audio data collected by a far-field microphone as an optimized object, and collecting near-field and far-field audio data as training data to train a constructed composite network, thereby achieving the sound capture optimization of the near-field microphone array and the far-field microphone array. The obtained audio optimization network comprises a time domain denoising model, a frequency domain spatial enhancement model and a time-frequency fusion model, and the time domain denoising model is pre-trained under a meta-learning framework to obtain general optimization model parameters of near and far fields; the time domain signals are further denoised through the time domain denoising model, the frequency domain signals are enhanced through the frequency domain space enhancement model, the denoised signals and the enhanced signals are fused through the time frequency fusion model, and finally far-field audio data to be optimized are input into the audio optimization network. And the optimized audio data approximate to the near-field tone quality is obtained, so that the quality of the audio signal is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a far-field microphone sound capture optimization method and related equipment. Background Art

[0002] In modern speech processing systems, sound capture optimization technology plays a vital role, especially in far-field environments, where microphones need to capture high-quality speech signals under complex conditions such as noise and reverberation. Far-field microphone sound capture optimization technology aims to improve the clarity and intelligibility of long-distance speech signals, which is of great significance for applications such as speech recognition, voice control, and conference systems. Traditional far-field microphone speech enhancement technology mainly relies on signal processing and basic statistical model methods, such as directional beamforming, spatial filtering, spectral subtraction, and generalized gamma distribution models. With the development of new technologies such as machine learning and deep learning, modern speech enhancement technology has gradually surpassed traditional methods and can achieve higher speech enhancement effects in more complex and changeable environments. By combining traditional methods with emerging technologies, better speech enhancement effects can be obtained in practical applications.

[0003] Traditional far-field microphone speech enhancement technology works well in simple processing environments, but its performance is limited in complex noise environments and dynamically changing scenarios. For example, directional beamforming enhances signals in a specific direction by weighting and phase-adjusting the signals in the microphone array, but in dynamic environments, the signal direction and noise characteristics change frequently, resulting in unsatisfactory enhancement effects. Methods such as spatial filtering and spectral subtraction also have similar problems and are difficult to cope with changing noise environments. In recent years, with the development of artificial intelligence and deep learning technologies, speech enhancement technology has made significant progress. Transformer, as an important deep learning model, is widely used in speech enhancement tasks. The Transformer model achieves end-to-end speech enhancement by learning the relationship between clean speech signals and noisy speech signals. However, the Transformer model is prone to problems such as excessive complexity during training, resulting in poor model stability. Summary of the invention

[0004] The present invention provides a far-field microphone sound capture optimization method and related equipment, the purpose of which is to improve the quality of audio signals.

[0005] In order to achieve the above object, the present invention provides a far-field microphone sound capture optimization method, comprising:

[0006] Step 1: Obtain original audio data for training, where the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone;

[0007] Step 2, generating multiple tasks through the original audio data, inputting all the tasks into the constructed composite network to train the composite network, and obtaining an audio optimization network, wherein the composite network includes a time domain denoising model for denoising the time domain signal in the input data, a frequency domain spatial enhancement model for enhancing the frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, and the time domain denoising model is obtained by pre-training under a meta-learning framework;

[0008] Step 3: Input the audio data to be optimized into the audio optimization network for optimization to obtain the optimized audio data.

[0009] More specifically, step 1 includes:

[0010] Determine multiple positioning points in the same space;

[0011] The near-field microphone array is used to collect spherical wave audio data at each positioning point in turn to obtain the original audio signal;

[0012] Using the geometric acoustic method, the distance attenuation, phase and signal of the original audio signal are adjusted to obtain a spherical wave audio signal;

[0013] The spherical wave audio signal is converted into plane wave audio data by using beamforming technology, delay-sum beamforming and signal superposition method to obtain label audio;

[0014] Far-field microphones are used to synchronously collect audio data from each positioning point.

[0015] Further, the time domain denoising model includes a mapping module and an encoding module;

[0016] The input of the encoding module is the input of the time domain denoising model;

[0017] The output end of the encoding module is connected to the input end of the mapping module;

[0018] The output end of the mapping module is connected to the first input end of the multimodal fusion network;

[0019] The encoding module extracts time domain features from the input audio data to obtain a high-dimensional time domain feature vector;

[0020] The mapping module transforms the high-dimensional time-domain feature vector to obtain denoised frequency data and inputs it into the multimodal fusion module.

[0021] Further, the frequency domain spatial enhancement model includes a discrete Fourier transform module, a recursive matrix dot product module, an inverse Fourier transform module, and a linear mapping module connected in sequence;

[0022] The input of the discrete Fourier transform module is the input of the frequency domain spatial enhancement model;

[0023] The output end of the linear mapping module is connected to the second output end of the multimodal fusion module;

[0024] The discrete Fourier transform module transforms the input audio data into a frequency domain signal;

[0025] The recursive matrix dot product module extracts frequency domain features from the frequency domain signal and performs matrix calculation to obtain an enhanced frequency domain feature vector;

[0026] The inverse Fourier transform module transforms the enhanced frequency domain feature vector into a time domain signal;

[0027] The linear mapping module maps the time domain signal to obtain enhanced audio data and input it into the multimodal fusion module.

[0028] Further, the multimodal fusion model includes a concatenation module, a first fully connected module, a ReLU activation function module, and a second fully connected module connected in sequence;

[0029] The first input end of the splicing module is connected to the output end of the mapping module, and the second input end of the splicing module is connected to the output end of the linear mapping module;

[0030] The splicing module splices the denoised audio data and the enhanced audio data to obtain spliced ​​audio data;

[0031] The first fully connected module and the ReLU activation function module weight the concatenated audio data to obtain weighted audio data;

[0032] The second fully connected module linearly maps the weighted audio data to obtain optimized audio data.

[0033] Specifically, pre-training under the meta-learning framework includes:

[0034] M layers of encoding modules are set to process M pre-tasks, where the pre-task is a task of training audio data collected at a positioning point;

[0035] Set the initial parameters of each layer of the encoding module to θ0, and the optimization goal of each pre-task to minimize the loss function

[0036] Through the gradient descent algorithm formula Update the parameters and update the updated parameters θ after each pre-task is completed. i As the initial parameter θ for the next pre-task i+1 =θ i ;

[0037] Until all pre-tasks are completed, the updated parameter θ corresponding to the completion of the last pre-task is m Freeze, meta-learning ends, and the time domain denoising model is obtained.

[0038] The present invention also provides a far-field microphone sound capture optimization device, comprising:

[0039] An acquisition module is used to acquire raw audio data for training, where the raw audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone;

[0040] A training module, used to generate multiple tasks through original audio data, input all tasks into the constructed composite network to train the composite network, and obtain an audio optimization network, wherein the composite network includes a time domain denoising model for denoising the time domain signal in the input data, a frequency domain space enhancement model for enhancing the frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, wherein the time domain denoising model is obtained by pre-training under a meta-learning framework;

[0041] The optimization module is used to input the audio data to be optimized into the audio optimization network for optimization to obtain the optimized audio data.

[0042] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a far-field microphone sound capture optimization method when executing the computer program.

[0043] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a far-field microphone sound capture optimization method is implemented.

[0044] The above scheme of the present invention has the following beneficial effects:

[0045] The present invention obtains original audio data for training, wherein the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone; generates multiple tasks through the original audio data, inputs all the tasks into a constructed composite network to train the composite network, and obtains an audio optimization network, wherein the composite network includes a time domain denoising model for denoising a time domain signal in the input data, a frequency domain space enhancement model for enhancing a frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, wherein the time domain denoising model is obtained by pre-training under a meta-learning framework; inputs the audio data to be optimized into the audio optimization network for optimization, and obtains the optimized audio data; and Compared with the existing technology, the present invention uses the audio collected by the near-field microphone array as the optimization target and the audio data collected by the far-field microphone as the optimized object. At the same time, the near-field and far-field audio data are collected as training data to train the composite network, and the time domain denoising model is pre-trained through meta-learning training to optimize the parameters of the composite network. The time domain denoising model is used to denoise the time domain signal in the input data, and the frequency domain spatial enhancement model is used to enhance the frequency domain signal in the input data. Then, the denoised signal and the enhanced signal are fused through a multimodal fusion model. The audio data is denoised and enhanced from the perspectives of the time domain and frequency domain, respectively, which significantly improves the denoising and enhancement effects of the audio, thereby improving the quality of the audio signal.

[0046] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0048] Figure 2 is a system principle block diagram of an embodiment of the present invention;

[0049] Figure 3 Schematic diagram of converting a spherical wave into a plane wave in an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a mapping module in an embodiment of the present invention;

[0051] Figure 5 Schematic diagram of a recursive matrix dot product module in an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of a multimodal fusion model in an embodiment of the present invention;

[0053] Figure 7 Schematic diagram of a far-field microphone sound capture optimization device in an embodiment of the present invention;

[0054] Figure 8 Schematic diagram of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0057] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0059] In view of the existing problems, the present invention provides a far-field microphone sound capture optimization method and related equipment.

[0060] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a far-field microphone sound capture optimization method, comprising:

[0061] Step 1: Obtain original audio data for training, where the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone;

[0062] Step 2, generating multiple tasks through the original audio data, inputting all the tasks into the constructed composite network to train the composite network, and obtaining an audio optimization network, wherein the composite network includes a time domain denoising model for denoising the time domain signal in the input data, a frequency domain spatial enhancement model for enhancing the frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, and the time domain denoising model is obtained by pre-training under a meta-learning framework;

[0063] Step 3: Input the audio data to be optimized into the audio optimization network for optimization to obtain the optimized audio data.

[0064] Specifically, step 1 includes:

[0065] Determine multiple positioning points in the same space;

[0066] The near-field microphone array is used to collect spherical wave audio data at each positioning point in turn to obtain the original audio signal;

[0067] Using the geometric acoustic method, the distance attenuation, phase and signal of the original audio signal are adjusted to obtain a spherical wave audio signal;

[0068] The spherical wave audio signal is converted into plane wave audio data by using beamforming technology, delay-sum beamforming and signal superposition method to obtain label audio;

[0069] Far-field microphones are used to synchronously collect audio data from each positioning point.

[0070] In the embodiment of the present invention, for each positioning point (for example, in a large conference room, studio, etc.), data collection is first performed, the purpose of which is to collect clear label audio with a near-field microphone array and collect original audio with a far-field microphone. The characteristics of near-field and far-field audio are different, one is a spherical wave and the other is a plane wave. Since the optimization target object of the embodiment of the present invention is the audio collected in the far field, it is necessary to convert the clear label audio into a waveform, such as Figure 3 As shown, it is convenient for subsequent processing. The conversion process is as follows:

[0071] Starting from a certain positioning point, a high-precision near-field microphone array is used to collect the spherical wave audio data of the positioning point, and the original audio signal is represented as Y;

[0072] Using the geometric acoustics method, the distance attenuation, phase and signal of the original audio signal are adjusted to obtain the processed spherical wave audio signal Y d ;

[0073] The beamforming technology is used to convert the spherical wave audio data into plane wave audio data, and the converted plane wave data Y is obtained through the delay summation beamforming and signal superposition method. bf , that is, the label audio y, the calculation formula is as follows:

[0074] According to the distance d between the i-th near-field microphone in the array and the sound source i , assuming that the signal attenuation is inversely proportional to the square of the distance, adjust the signal amplitude to:

[0075]

[0076] Assuming the speed of sound is v, according to the distance d i The phase shift is calculated by the frequency f as:

[0077]

[0078] The collected signal is adjusted according to the attenuation factor and phase change to obtain the processed spherical wave audio signal Y d for:

[0079]

[0080] The spherical wave audio data received by each near-field microphone in the array Calculate the time delay τ from the sound source to the microphone j for:

[0081]

[0082] Finally, the signal delay step and alignment superposition are performed to obtain the plane wave audio data Y bf for:

[0083]

[0084] Based on the plane wave audio data, the label audio y is:

[0085]

[0086] Among them, d j represents the distance from the sound source to the jth microphone, and L represents the length of the time window.

[0087] Use far-field microphones to synchronously collect audio data containing noise from each positioning point X noise,i , the data is divided by selecting the first c audio data as the training set and the remaining audio data as the test set samples.

[0088] Specifically, pre-training is performed under the meta-learning framework, including:

[0089] Set M-layer encoding modules {E1, E2, …, E m} Process M pre-tasks, where the pre-task is to train the audio data collected from a positioning point;

[0090] Set the initial parameters of each layer of the encoding module to θ0, and the optimization goal of each pre-task to minimize the loss function

[0091] Through the gradient descent algorithm formula Update the parameters and update the updated parameters θ after each pre-task is completed. i As the initial parameter θ for the next pre-task i+1 =θ i ;

[0092] Until all pre-tasks are completed, the updated parameter θ corresponding to the completion of the last pre-task is m Freeze, meta-learning ends, and the time domain denoising model is obtained.

[0093] In the embodiment of the present invention, before generating multiple tasks using original audio data, the method further includes:

[0094] For the audio data X in the training set noise,i Perform time domain-frequency domain processing to obtain the time domain audio signal X time and frequency domain audio signal X freq They are used as input data for the time domain denoising model and frequency domain spatial enhancement model in the composite network respectively.

[0095] It should be noted that when the time domain audio signal X time Before inputting the time domain denoising model, patching processing must be performed first, and it is converted into k time domain frame blocks of length p by the moving average method, that is, X time ∈R 1×L →X patch ∈R k×p .

[0096] In the embodiment of the present invention, the design of the loss function is an important part of the fusion network optimization. The embodiment of the present invention uses the mean square error as the loss function to measure the difference between the optimized audio data and the label audio converted by the near-field microphone array. The specific formula is:

[0097]

[0098] in, represents the labeled audio, that is, the high-quality audio data processed by geometric acoustic methods, beamforming technology and transformation methods. L represents the length of the sample time window. By minimizing this loss function, the network can learn more accurate audio features, thereby improving the denoising and enhancement effects.

[0099] Most preferably, the time domain denoising model includes a mapping module and an encoding module;

[0100] The input of the encoding module is the input of the time domain denoising model;

[0101] The output end of the encoding module is connected to the input end of the mapping module;

[0102] The output end of the mapping module is connected to the first input end of the multimodal fusion network;

[0103] The encoding module extracts time domain features from the input audio data to obtain a high-dimensional time domain feature vector;

[0104] The mapping module transforms the high-dimensional time-domain feature vector to obtain denoised frequency data and inputs it into the multimodal fusion module.

[0105] In the embodiment of the present invention, a multi-head attention mechanism is set inside the encoding module to perform multi-head attention on the time domain frame block X. patch Perform global feature extraction to obtain a high-dimensional time domain feature vector H, and then transform the high-dimensional time domain feature vector H into denoised frequency data of the original dimension through the mapping module. The mapping module includes input layer, hidden layer and output layer, such as Figure 4 As shown, the specific formula is as follows:

[0106]

[0107] Q i =W Q x i +b Q ,i∈[1,k]

[0108] K i =W K x i +b K ,i∈[1,k]

[0109] V i =W V x i +b V ,i∈[1,k]

[0110]

[0111] H=Concat(h1,h2,…,hk )W0

[0112]

[0113] Among them, S represents the step size, W Q , W K , W V Represents the weight matrix of the linear transformation, x i represents the i-th time domain frame block, x i ∈X patch , b Q 、b K 、b V is the bias term, d k represents the dimension of the key vector, softmax(·) is used to ensure weight distribution, Concat(·) is used to aggregate the multi-head attention results, and L(·) represents the mapping function, which maps the high-dimensional feature vector to the denoised frequency.

[0114] Most preferably, the frequency domain spatial enhancement model includes a discrete Fourier transform module, a recursive matrix dot product module, an inverse Fourier transform module, and a linear mapping module connected in sequence;

[0115] The input of the discrete Fourier transform module is the input of the frequency domain spatial enhancement model;

[0116] The output end of the linear mapping module is connected to the second input end of the multimodal fusion module;

[0117] The discrete Fourier transform module transforms the input audio data into a frequency domain signal;

[0118] The recursive matrix dot product module extracts frequency domain features from the frequency domain signal and performs matrix calculation to obtain an enhanced frequency domain feature vector;

[0119] The inverse Fourier transform module transforms the enhanced frequency domain feature vector into a time domain signal;

[0120] The linear mapping module maps the time domain signal to obtain enhanced audio data and input it into the multimodal fusion module.

[0121] In the embodiment of the present invention, the audio data X of the frequency domain spatial enhancement model is input. noise,i First, the frequency domain signal X is transformed through the discrete Fourier transform module fre q, and then the local information and matrix calculation of the frequency domain signal are deeply extracted through the recursive matrix dot product module to fully learn the data features and obtain the enhanced frequency domain feature vector H freq Then, it is transformed through the inverse Fourier module to obtain the time domain signal, and finally mapped through the linear mapping module to obtain the enhanced audio data The specific formula is as follows:

[0122]

[0123] Y[k]=X[k]·H[k]+B[k]

[0124] Y m [k] = Y m-1 [k]·H[k]+B[k]

[0125]

[0126] Where X[k] represents the spectrum graph, w[z] represents the window function, X[t+z] represents the original audio signal of the current window, and Z represents the window length of the discrete Fourier transform. represents the complex exponential kernel of Fourier transform, B[k] represents the bias term, z[n] represents the time domain signal, Z[k] represents the frequency domain signal, N represents the signal length, n represents the signal index, and FNN(·) represents the flattening function.

[0127] In the embodiment of the present invention, Figure 5 As shown, the recursive matrix dot product module includes 7 layers of matrix dot product blocks and residual blocks. The input end of the first layer of matrix dot product blocks and the first input end of the residual block are connected to the output end of the discrete Fourier transform module. The 7 layers of matrix dot product blocks are connected in sequence and the output end of each layer of matrix dot product blocks is connected to the input end of the residual block. The output end of the residual block is connected to the input end of the inverse Fourier transform module for deep extraction of local information and matrix calculation of frequency domain signals. Each layer of matrix dot product blocks performs nonlinear transformation through the ReLU activation function when performing dot product operations to extract richer features, which can fully learn data features and obtain the enhanced frequency domain feature vector H. freq ; This multi-level dot product residual network can achieve the effect of frequency domain convolution, and thus fully extract the local and global features of the frequency domain signal.

[0128] Most preferably, Figure 6 As shown, the multimodal fusion model includes a splicing module, a first fully connected module, a ReLU activation function module, and a second fully connected module connected in sequence;

[0129] The first input end of the splicing module is connected to the output end of the mapping module, and the second input end of the splicing module is connected to the output end of the linear mapping module;

[0130] The splicing module splices the denoised audio data and the enhanced audio data to obtain spliced ​​audio data;

[0131] The first fully connected module and the ReLU activation function module weight the concatenated audio data to obtain weighted audio data;

[0132] The second fully connected module linearly maps the weighted audio data to obtain optimized audio data.

[0133] In the embodiment of the present invention, the multimodal fusion model first removes the noise of the frequency data and enhanced audio data The feature vectors are concatenated to form a more comprehensive feature representation X combined for:

[0134]

[0135] The purpose of learning through multiple fully connected layers is to weight the time domain and frequency domain features according to their different contributions, and the activation function is used to enhance the nonlinear expression ability of the network; the final fusion feature X combined Optimized audio data obtained through linear mapping for:

[0136]

[0137] Among them, f fused Represents the multimodal fusion function. Through this multi-level feature fusion method, the model can more deeply explore the correlation between time domain and frequency domain signals, and finally achieve the complete sound capture optimization task.

[0138] The embodiment of the present invention obtains original audio data for training, wherein the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone; generates multiple tasks through the original audio data, inputs all the tasks into a constructed composite network to train the composite network, and obtains an audio optimization network, wherein the composite network includes a time domain denoising model for denoising a time domain signal in the input data, a frequency domain space enhancement model for enhancing a frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, wherein the time domain denoising model is obtained by pre-training under a meta-learning framework; inputs the audio data to be optimized into the audio optimization network for optimization, and obtains the optimized audio data; and Compared with the prior art, the embodiment of the present invention uses the audio collected by the near-field microphone array as the optimization target and the audio data collected by the far-field microphone as the optimized object, and collects the near-field and far-field audio data as training data to train the composite network, and pre-trains the time domain denoising model through meta-learning training to optimize the parameters of the composite network, denoises the time domain signal in the input data through the time domain denoising model, and enhances the frequency domain signal in the input data through the frequency domain spatial enhancement model, and then fuses the denoised signal and the enhanced signal through the multimodal fusion model, and denoises and enhances the audio data from the perspectives of time domain and frequency domain, respectively, which significantly improves the denoising and enhancement effects of the audio, thereby improving the quality of the audio signal.

[0139] The embodiment of the present invention also provides a far-field microphone sound capture optimization device, such as Figure 7 As shown, the far-field microphone sound capture optimization device 100 includes:

[0140] An acquisition module 101 is used to acquire original audio data for training, where the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone;

[0141] A training module 102 is used to generate multiple tasks through original audio data, input all tasks into the constructed composite network to train the composite network, and obtain an audio optimization network, wherein the composite network includes a time domain denoising model for denoising the time domain signal in the input data, a frequency domain spatial enhancement model for enhancing the frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, wherein the time domain denoising model is obtained by pre-training under a meta-learning framework;

[0142] The optimization module 103 is used to input the audio data to be optimized into the audio optimization network for optimization to obtain the optimized audio data.

[0143] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0144] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0145] An embodiment of the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a far-field microphone sound capture optimization method when executing the computer program.

[0146] like Figure 8 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 8 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps in any of the above-mentioned method embodiments when executing the computer program D102. Alternatively, the processor D100 implements the functions of each module / unit in the above-mentioned device embodiments when executing the computer program D102.

[0147] The terminal device D10 may be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will appreciate that Figure 8 This is only an example of the terminal device D10 and does not constitute a limitation on the terminal device D10. The terminal device D10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0148] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0149] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0150] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0151] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0152] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a far-field microphone sound capture optimization method is implemented.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0154] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A far-field microphone sound capture optimization method, characterized in that: include: Step 1: Obtain original audio data for training, where the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone; Step 2, generating multiple tasks through the original audio data, inputting all the tasks into the constructed composite network to train the composite network, and obtaining an audio optimization network, wherein the composite network includes a time domain denoising model for denoising the time domain signal in the input data, a frequency domain space enhancement model for enhancing the frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, wherein the time domain denoising model is obtained by pre-training under a meta-learning framework; Step 3: input the audio data to be optimized into the audio optimization network for optimization to obtain optimized audio data.

2. The far-field microphone sound capture optimization method according to claim 1, characterized in that: The step 1 comprises: Determine multiple positioning points in the same space; The near-field microphone array is used to collect spherical wave audio data at each positioning point in turn to obtain the original audio signal; Using a geometric acoustic method, the distance attenuation, phase and signal of the original audio signal are adjusted to obtain a spherical wave audio signal; The spherical wave audio signal is converted into plane wave audio data by using beamforming technology, delay-sum beamforming and signal superposition method to obtain label audio; Far-field microphones are used to synchronously collect audio data from each positioning point.

3. The far-field microphone sound capture optimization method according to claim 2, characterized in that: The time domain denoising model includes a mapping module and an encoding module; The input end of the encoding module is the input end of the time domain denoising model; The output end of the encoding module is connected to the input end of the mapping module; The output end of the mapping module is connected to the first input end of the multimodal fusion network; The encoding module extracts time domain features from the input audio data to obtain a high-dimensional time domain feature vector; The mapping module transforms the high-dimensional time-domain feature vector to obtain denoised frequency data and inputs the denoised frequency data into the multimodal fusion module.

4. The far-field microphone sound capture optimization method according to claim 2, characterized in that: The frequency domain spatial enhancement model includes a discrete Fourier transform module, a recursive matrix dot product module, an inverse Fourier transform module, and a linear mapping module which are connected in sequence; The input end of the discrete Fourier transform module is the input end of the frequency domain spatial enhancement model; The output end of the linear mapping module is connected to the second output end of the multimodal fusion module; The discrete Fourier transform module transforms the input audio data into a frequency domain signal; The recursive matrix dot product module extracts frequency domain features from the frequency domain signal and performs matrix calculation to obtain an enhanced frequency domain feature vector; The inverse Fourier transform module transforms the enhanced frequency domain feature vector into a time domain signal; The linear mapping module maps the time domain signal to obtain enhanced audio data and inputs the enhanced audio data into the multimodal fusion module.

5. The far-field microphone sound capture optimization method according to claim 4, characterized in that: The multimodal fusion model includes a splicing module, a first fully connected module, a ReLU activation function module, and a second fully connected module connected in sequence; The first input end of the splicing module is connected to the output end of the mapping module, and the second input end of the splicing module is connected to the output end of the linear mapping module; The splicing module splices the denoised audio data and the enhanced audio data to obtain spliced ​​audio data; The first fully connected module and the ReLU activation function module weight the concatenated audio data to obtain weighted audio data; The second fully connected module performs linear mapping on the weighted audio data to obtain optimized audio data.

6. The far-field microphone sound capture optimization method according to claim 5, characterized in that: By pre-training under the meta-learning framework, including: Setting M layers of encoding modules to process M pre-tasks, wherein the pre-task is a task of training audio data collected from a positioning point; Set the initial parameters of each layer of encoding module to θ0, and the optimization goal of each pre-task to minimize the loss function Through the gradient descent algorithm formula Update the parameters and update the updated parameters θ after each pre-task is completed. i As the initial parameter θ for the next pre-task i+1 =θ i ; Until all pre-tasks are completed, the updated parameter θ corresponding to the completion of the last pre-task is m Freeze, meta-learning ends, and the time domain denoising model is obtained.

7. A far-field microphone sound capture optimization device, characterized in that: include: An acquisition module, used to acquire original audio data for training, wherein the original audio data includes label audio collected by a near-field microphone array and audio data collected by a far-field microphone; A training module, used to generate multiple tasks through the original audio data, input all the tasks into the constructed composite network to train the composite network, and obtain an audio optimization network, wherein the composite network includes a time domain denoising model for denoising the time domain signal in the input data, a frequency domain space enhancement model for enhancing the frequency domain signal in the input data, and a multimodal fusion model for fusing the denoised signal and the enhanced signal, wherein the time domain denoising model is obtained by pre-training under a meta-learning framework; The optimization module is used to input the audio data to be optimized into the audio optimization network for optimization to obtain the optimized audio data.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the far-field microphone sound capture optimization method as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the far-field microphone sound capture optimization method as described in any one of claims 1 to 6 is implemented.

Images