Audio processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an audio processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring target audio data collected by a microphone array, wherein the target audio data is multi-channel audio data; pre-processing the target audio data to obtain pre-processed audio data, the pre-processed audio data being suitable for being input into a pre-trained audio processing model; processing the preprocessed audio data through a pre-trained audio processing model to obtain pickup data in at least two directions, the pre-trained audio processing model comprising a coding network and at least two decoding networks respectively connected to the coding network, the directions of the pickup data output by different decoding networks in the at least two decoding networks are different. According to the technical scheme provided by the embodiment of the invention, the separation effect of the pickup data in different directions can be better, and the probability of voice confusion of the pickup data in multiple directions is reduced, so that the auditory experience is improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] Currently, more and more electronic devices are equipped with microphone arrays. The electronic devices need to process the audio data collected by the microphones to achieve directional sound pickup, thereby providing a better auditory experience.

[0003] In related technologies, electronic devices process audio data collected by microphone arrays through beamforming algorithms to obtain sound pickup data in a specified direction. The implementation principle of the beamforming algorithm is to adjust the weight of each microphone so that the signal from a specific direction is strengthened at the output end, while the signals from other directions are suppressed or offset.

[0004] In the related art, when an electronic device needs to pick up sounds from multiple directions, there is a problem of voice confusion, and the sound pickup data from multiple directions are not separated cleanly. Summary of the invention

[0005] The present application proposes an audio processing method, device, electronic device and storage medium.

[0006] In a first aspect, an embodiment of the present application provides an audio processing method, which is applied to an electronic device, wherein the electronic device is provided with a microphone array, and the method comprises: obtaining target audio data collected by the microphone array, wherein the target audio data is multi-channel audio data; preprocessing the target audio data to obtain preprocessed audio data, wherein the preprocessed audio data is suitable for inputting a pre-trained audio processing model; processing the preprocessed audio data by a pre-trained audio processing model to obtain sound pickup data in at least two directions, wherein the pre-trained audio processing model comprises an encoding network and at least two decoding networks respectively connected to the encoding network, wherein directions of sound pickup data output by different decoding networks in the at least two decoding networks are different.

[0007] In a second aspect, an embodiment of the present application provides an audio processing device, which includes: a data acquisition module, used to acquire target audio data collected by a microphone array, the target audio data being multi-channel audio data; a preprocessing module, used to preprocess the target audio data to obtain preprocessed audio data, the preprocessed audio data being suitable for inputting a pre-trained audio processing model; an audio processing module, used to process the preprocessed audio data through a pre-trained audio processing model to obtain sound pickup data in at least two directions, the pre-trained audio processing model including an encoding network and at least two decoding networks respectively connected to the encoding network, and the directions of sound pickup data output by different decoding networks in the at least two decoding networks are different.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that computer program instructions are stored in the computer-readable storage medium, and the computer program instructions can be called by a processor to execute the method described in the first aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when instructions in the computer program product are executed, is used to implement the method described in the first aspect.

[0011] Compared with the prior art, the technical solution provided in the embodiment of the present application pre-processes the multi-channel audio data collected by the microphone array to obtain pre-processed audio data, and the audio processing model performs audio processing on the pre-processed audio data to obtain sound pickup data in at least two directions, wherein the audio processing model includes at least two decoding networks, and different decoding networks are used to output sound pickup data in different directions. The above-mentioned at least two decoding networks are independent of each other and do not interfere with each other. Therefore, when multi-directional sound pickup is realized based on the audio processing model, the separation effect of sound pickup data in different directions can be better, and the probability of voice confusion in sound pickup data in multiple directions can be reduced, thereby improving the auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 It is a schematic diagram of the implementation environment provided by the embodiment of the present application.

[0014] Figure 2 It is a schematic diagram of a microphone array provided in an embodiment of the present application.

[0015] Figure 3 It is a structural diagram of an audio processing model provided by an embodiment of the present application.

[0016] Figure 4 This is a flowchart of an audio processing method provided by an embodiment of the present application.

[0017] Figure 5This is a flowchart of an audio processing method provided by another embodiment of the present application.

[0018] Figure 6 This is a flowchart of the training process of the audio processing model provided by an embodiment of the present application.

[0019] Figure 7 It is a block diagram of an audio processing device provided by an embodiment of the present application.

[0020] Figure 8 It is a structural block diagram of an electronic device provided by an embodiment of the present application.

[0021] Fig. 9 It is a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0023] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0024] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes an electronic device 100, which can be a smart phone, a tablet computer, a smart speaker, a smart home appliance, a smart monitoring device, etc.

[0025] The electronic device 100 is provided with a microphone array 200, and the microphone array 200 may be a circular array or a linear array, which is not limited in the present embodiment. In the present embodiment, the microphone array 200 is only described as a circular array. Figure 2 , which shows a schematic diagram of a microphone array 200 provided by an embodiment of the present application. Figure 2 In the embodiment, the microphone array 200 includes 8 microphones, and the 8 microphones are arranged in a circle.

[0026] In an embodiment of the present application, the electronic device 100 also includes an audio processing device, which includes a pre-trained audio processing model. Specifically, the electronic device 100 first pre-processes the multi-channel audio data collected by the microphone array to obtain pre-processed audio data suitable for inputting the above-mentioned audio processing model, and the audio processing model performs audio processing on the pre-processed audio data to obtain sound pickup data in at least two directions. In an embodiment of the present application, the audio processing model includes at least two decoding networks, and different decoding networks are used to output sound pickup data in different directions. The above-mentioned at least two decoding networks are independent of each other and do not interfere with each other. Compared with the speech confusion problem caused by multi-directional sound pickup through beamforming algorithm in the related art, the technical solution provided in the embodiment of the present application can make the separation effect of sound pickup data in different directions better, reduce the probability of speech confusion in sound pickup data in multiple directions, thereby improving the auditory experience.

[0027] Please refer to Figure 3 , which shows a structural diagram of a pre-trained audio processing model 300 provided in one embodiment of the present application. The audio processing model 300 is obtained by training a neural network using multiple training audio data. In the embodiment of the present application, the neural network can be a standard U-net network, a DC-UNet network, etc., which is not limited in the embodiment of the present application.

[0028] The audio processing model 300 includes an encoding network 310 and at least two decoding networks, and the at least two decoding networks are connected to the encoding network 310 respectively. The number of decoding networks can be specifically determined according to the number of areas where sound needs to be picked up. Specifically, the number of decoding networks is the same as the number of areas where sound needs to be picked up. For example, if the electronic device needs to pick up sound in area A and area B respectively, the audio processing model included in the electronic device should include two decoding networks. Figure 3 In the embodiment, the audio processing model 300 includes two decoding networks, namely a first decoding network 320 and a second decoding network 330. The first decoding network 320 is used to achieve directional sound pickup for area A, and the second decoding network 330 is used to achieve directional sound pickup for area B.

[0029] Each decoding network includes a decoding layer, a mask layer, a first operation layer, a pooling layer, and a second operation layer connected in sequence. The decoding layer is used to decode the encoded data. The network structure of the decoding layer included in different decoding networks may be the same or different. In addition, the parameters of the decoding layers included in different decoding networks are usually different. The mask layer is used to mask the output result of the decoding layer. The mask matrices corresponding to the mask layers included in different decoding networks are different, so as to achieve sound pickup in different areas. The first operation layer is used to operate on the output result of the mask layer, the real spectrum data obtained in the preprocessing process, and the imaginary spectrum data. The pooling layer is used to perform pooling on the output result of the first operation layer, and the above-mentioned pooling process may be mean pooling, maximum pooling, etc. In the embodiment of the present application, only the pooling process is described as mean pooling. The network structure of the pooling layer included in different decoding networks may be the same or different, and the parameters of the pooling layer included in different decoding networks are usually different. The second operation layer is used to operate on the output result of the pooling layer, so as to obtain the sound pickup data in the specified direction corresponding to the decoding network.

[0030] exist Figure 3 In the embodiment, the first decoding network 320 includes a mask layer A, a first operation layer A, a pooling layer A, and a second operation layer A. The second decoding network 330 includes a mask layer B, a first operation layer B, a pooling layer B, and a second operation layer B.

[0031] Please refer to Figure 4 , which shows a flow chart of an audio processing method provided by an embodiment of the present application. The method includes the following process.

[0032] S401, obtaining target audio data collected by the microphone array.

[0033] The target audio data is multi-channel audio data. Figure 2 Taking the microphone array 200 shown as an example, the microphone array 200 includes 8 microphones arranged in a circular shape, and the target audio data collected by the microphone array 200 is 8-channel audio data.

[0034] S402, preprocessing the target audio data to obtain preprocessed audio data.

[0035] The pre-processed audio data is suitable for inputting into a pre-trained audio processing model. That is, the pre-processed audio data meets the input requirements of the above audio processing model, such as data type requirements, data format requirements, etc. The pre-processing process will be described in the following embodiments.

[0036] S403, performing audio processing on the pre-processed audio data by using a pre-trained audio processing model to obtain sound pickup data in at least two directions.

[0037] The pre-trained audio processing model includes an encoding network and at least two decoding networks respectively connected to the encoding network, and the directions of the sound pickup data output by different decoding networks in the at least two decoding networks are different. It should be noted that the number of decoding networks is the same as the number of directions to be picked up. Figure 3 For example, the first decoding network 320 outputs the sound pickup data of area A, and the second decoding network 330 outputs the decoding network of area B. In the embodiment of the present application, different decoding networks are independent of each other and do not interfere with each other. Therefore, when multi-directional sound pickup is implemented based on the audio processing model, the separation effect of sound pickup data from different directions can be better, and the probability of voice confusion of sound pickup data from multiple directions can be reduced, thereby improving the auditory experience.

[0038] In summary, the technical solution provided in the embodiment of the present application pre-processes the multi-channel audio data collected by the microphone array to obtain pre-processed audio data, and the audio processing model performs audio processing on the pre-processed audio data to obtain sound pickup data in at least two directions, wherein the audio processing model includes at least two decoding networks, and different decoding networks are used to output sound pickup data in different directions. The above-mentioned at least two decoding networks are independent of each other and do not interfere with each other. Therefore, when multi-directional sound pickup is realized based on the audio processing model, the separation effect of sound pickup data in different directions can be better, and the probability of voice confusion in sound pickup data in multiple directions can be reduced, thereby improving the auditory experience.

[0039] Please refer to Figure 5 , which shows a flow chart of an audio processing method provided by an embodiment of the present application. The method includes the following process.

[0040] S501, obtaining target audio data collected by a microphone array.

[0041] The target audio data is multi-channel audio data.

[0042] S502: pre-process the target audio data to obtain pre-processed audio data.

[0043] The pre-processed audio data is suitable for inputting into a pre-trained audio processing model. In some embodiments, the pre-processing process may include the following steps.

[0044] S5021, perform Fourier transform on the target audio data to obtain the real spectrum and imaginary spectrum corresponding to the audio data of each channel.

[0045] Fourier transform is a mathematical method for converting time domain signals into frequency domain signals. The electronic device performs Fourier transform on the audio data of each channel in the target audio data to obtain the real spectrum and imaginary spectrum corresponding to the audio data of each channel.

[0046] Optionally, the electronic device processes the audio data x of the first channel k1 (n) Perform a short-time Fourier transform (STFT) to obtain X k1 (τ, f).

[0047] Among them, X k1 (τ, f) can be expressed by the following first calculation formula:

[0048]

[0049] Among them, X k1 (τ, f) is the transformation result of Fourier transform on the audio data of the first channel; N is the window length of Fourier transform, which can be equal to the number of Fourier transform points, such as 1024; n is the nth sampling point in the window; w(τ) is the window function, which can be a Hamming window; τ is the position point where the window function slides on the time axis; exp() is an exponential function with the natural constant e as the base; f is the frequency.

[0050] The electronic device converts the first calculation formula into Expanding it into complex form, we get the following second calculation formula:

[0051]

[0052] in, That is, the real spectrum corresponding to the audio data of the first channel. That is, the imaginary spectrum corresponding to the audio data of the first channel. In addition, the size of the real spectrum and the imaginary spectrum is [ch,,nfft / 2,T], where ch is the number of channels and T is the number of frames, such as [8,512,8].

[0053] The process of the electronic device performing Fourier transform on the audio data of other channels can refer to the process of performing Fourier transform on the audio data of the first channel, which will not be described in detail here.

[0054] When the target audio data is eight-channel audio data, the real spectrum generated includes 8, which can be expressed as Re k = {Re k1 ,Re k2 ,Re k3 ,Re k4 ,Re k5 ,Re k6 ,Re k7 ,Re k8}. Similarly, the generated imaginary number spectrum also includes 8, which can be expressed as Im k ={Im k1 ,Im k2 ,Imk3 ,Im k4 ,Im k5 ,Im k6 ,Im k7 ,Im k8}.

[0055] S5022: Calculate the amplitude spectrum of the audio data of each channel according to the real spectrum and the imaginary spectrum corresponding to the audio data of each channel.

[0056] The electronic device calculates the amplitude spectrum of the audio data of the first channel by the following third calculation formula:

[0057]

[0058] Among them, Mag(x k1 ) represents the amplitude spectrum of the audio data of the first channel, Re(x k1 ) represents the real spectrum corresponding to the audio data of the first channel, Im(x k1 ) represents the imaginary spectrum corresponding to the audio data of the first channel.

[0059] The calculation process of the amplitude spectrum of the audio data of other channels can refer to the calculation process of the amplitude spectrum of the audio data of the first channel, which will not be repeated here.

[0060] S5023, compress the amplitude spectrum of the audio data of each channel to obtain a compressed amplitude spectrum of the audio data of each channel.

[0061] The electronic device compresses the amplitude spectrum of the audio data of the first channel by using the following fourth calculation formula to obtain a compressed amplitude spectrum of the audio data of each first channel:

[0062] Mag compress (x k1 )=Mag(x k1 ) c .

[0063] Among them, Mag compress (x k1 ) represents the compressed amplitude spectrum of the audio data of the first channel, c is the compression coefficient, and its value is set according to experiments or experience, such as 0.5.

[0064] The calculation process of the compressed amplitude spectrum of the audio data of other channels can refer to the calculation process of the compressed amplitude spectrum of the audio data of the first channel, which will not be repeated here. The compressed amplitude spectrum of the audio data of each channel constitutes the preprocessed audio data.

[0065] S503, encoding the pre-processed audio data through a coding network to obtain encoded data.

[0066] The encoding network is used to extract features from the preprocessed audio data and gradually convert the preprocessed audio data into a set of feature maps. These feature maps can reflect various information of the target audio data, such as loudness, timbre, etc.

[0067] S504: Decode the encoded data using at least two decoding networks respectively, and use the sound pickup data respectively output by the at least two decoding networks as the sound pickup data in at least two directions.

[0068] In some embodiments, the decoding network includes a mask layer, a first operation layer, a pooling layer, and a second operation layer connected in sequence, and the mask matrices corresponding to the mask layers included in different decoding networks are different.

[0069] In this embodiment, S504 is specifically implemented as follows:

[0070] S5041, for each decoding network, masking is performed on the encoded data through a mask layer to obtain mask data.

[0071] Masking refers to a technique for selecting, filtering or hiding a specific portion of data. In an embodiment of the present application, masking can be used to select audio data in a specified direction corresponding to the decoding network and filter out audio data in other directions outside the specified direction.

[0072] Optionally, the mask layer corresponds to a mask matrix, the dimension of the mask matrix is ​​the same as the dimension of the encoded data, and the value of the position corresponding to the audio data in the specified direction in the mask matrix is ​​a first value, and the value of the position corresponding to the audio data in other directions in the mask matrix is ​​a second value. Exemplarily, the first value is 1, the second value is 0, and the audio processing model multiplies the encoded data by the mask matrix to obtain the mask data.

[0073] by Figure 3 For example, when the target audio data is eight-channel audio data, the mask data output by the first decoding network includes

[0074] S5042, performing operation processing on the mask data through the first operation layer to obtain first operation data.

[0075] The audio processing model multiplies the mask data by the real spectrum corresponding to the audio data of each channel through the first operation layer, and multiplies the mask data by the imaginary spectrum corresponding to the mask data of each channel, to obtain first operation data.

[0076] The audio processing model multiplies the mask data by the real spectrum corresponding to the audio data of each channel through the following fifth calculation formula to obtain the first operator data:

[0077]

[0078] When the target audio data is eight-channel audio data, the fifth calculation formula can be expressed as:

[0079]

[0080] The audio processing model multiplies the mask data by the imaginary spectrum corresponding to the audio data of each channel through the following sixth calculation formula to obtain the second operator data:

[0081]

[0082] When the target audio data is eight-channel audio data, the sixth calculation formula can be expressed as:

[0083]

[0084] The first operator data and the second operator data constitute the first operation data.

[0085] S5043, performing pooling processing on the first operation data through a pooling layer to obtain pooled data.

[0086] The role of the pooling layer is to reduce the amount of calculation by reducing the size of the feature map and extract important features in the feature map. In the embodiment of the present application, the pooling layer can convert the multi-channel first operation data into single-channel data. The pooling process can be a maximum pooling process or a mean pooling process. In the embodiment of the present application, only the pooling process is used as an example for explanation.

[0087] The audio processing model performs pooling processing on the first operator data through the following seventh calculation formula to obtain a single-channel real spectrum

[0088]

[0089] Where ch is the channel number.

[0090] The audio processing model performs pooling processing on the second operator data through the following eighth calculation formula to obtain a single-channel imaginary spectrum

[0091]

[0092] The above single-channel real spectrum and single-channel imaginary spectrum constitute pooled data, whose size is [1,nfft / 2,T].

[0093] S5044, performing operation processing on the pooled data through the second operation layer to obtain the sound pickup data of the specified direction corresponding to the decoding network.

[0094] The audio processing model performs inverse Fourier transform on the single-channel real spectrum and the single-channel imaginary spectrum through the second operation layer to obtain the sound pickup data in the A direction corresponding to the first decoding network.

[0095] The processing process of the encoded data by other decoding networks can refer to the above S5041-S5044 and will not be repeated here.

[0096] In summary, the technical solution provided in the embodiment of the present application pre-processes the multi-channel audio data collected by the microphone array to obtain pre-processed audio data, and the audio processing model performs audio processing on the pre-processed audio data to obtain sound pickup data in at least two directions, wherein the audio processing model includes at least two decoding networks, and different decoding networks are used to output sound pickup data in different directions. The above-mentioned at least two decoding networks are independent of each other and do not interfere with each other. Therefore, when multi-directional sound pickup is realized based on the audio processing model, the separation effect of sound pickup data in different directions can be better, and the probability of voice confusion in sound pickup data in multiple directions can be reduced, thereby improving the auditory experience.

[0097] Please refer to Figure 6 , which shows a flow chart of the training process of the audio processing model provided by an embodiment of the present application. The training process of the audio processing model can be completed by the electronic device, or it can be pushed to the electronic device after completion by the server, which is not limited in the embodiment of the present application. The training process of the audio processing model includes the following steps.

[0098] S601, obtaining a plurality of training audio data.

[0099] The amount of training audio data can be determined according to the accuracy requirements of the audio processing model. For example, the higher the accuracy requirements of the audio processing model, the more training audio data there is. Each of the multiple training audio data corresponds to label information, and the label information corresponding to the training audio data includes sound pickup data in at least two directions determined based on the training audio data. The following is an explanation of how to obtain the training audio data.

[0100] Optionally, multiple training audio data are obtained based on the audio collection results of multiple sound source data by simulating the microphone array based on the room impulse response; wherein the multiple sound source data are single-channel audio data; based on the playback positions of the multiple sound source data, the label information corresponding to the multiple training audio data is determined.

[0101] Room Impulse Response (RIR) characterizes the system response of a room system and can be used for room equalization and calculation of room acoustic parameters. In an embodiment of the present application, the method used to simulate the audio acquisition results of a microphone array for multiple sound source data based on the room impulse response includes but is not limited to: methods based on wave acoustics, such as the finite element method and the boundary element method, methods based on ray acoustics, such as the virtual source method and the path tracing method, methods based on RIR statistical models, such as the Schroder statistical model, methods based on deep learning, and the like.

[0102] When one or more sound source data are played at different locations, the electronic device collects corresponding audio data based on the room impulse response simulation microphone array to obtain multiple training audio data. In addition, the electronic device detects whether the playback position of the sound source data is in the area to be picked up by the microphone array. If so, the sound data of the area to be picked up is also the above-mentioned sound source data. If not, the sound data of the area to be picked up is empty.

[0103] In some embodiments, when there are multiple areas to be picked up by the microphone array, the playback position of the sound source data needs to be reasonably set so that the multiple training audio data obtained by simulation are more in line with the actual scene. Optionally, there is at least one first training audio data in the training audio data, and the first training audio data is the audio acquisition result of the microphone array simulated by the room impulse response when the playback position of the sound source data is in the shielded area outside the area to be picked up; there is also at least one second training audio data in the training audio data, and the second training audio data is the audio acquisition result of the microphone array simulated by the room impulse response when the playback positions of multiple sound source data are respectively in multiple areas to be picked up; there is also at least one third training audio data in the training audio data, and the third training audio data is the audio acquisition result of the microphone array simulated by the room impulse response when the playback position of a sound source data is in one area to be picked up. The playback position of the above-mentioned sound source data can simulate multiple scenarios such as synchronous sounding of multiple sound sources and asynchronous sounding of multiple sound sources, so that the audio processing model trained subsequently has better performance when facing the above-mentioned scenarios.

[0104] by Figure 2 Taking the embodiment as an example, area A and area B are the areas to be picked up, and other areas are shielded areas. The electronic device can set the playback position of the sound source data in the following manner.

[0105] (1) Play a sound source data in the shielded area. At this time, the label information corresponding to the simulated training audio data is a two-channel silent audio, that is, L k = {l 1 =0, l 2 =0}.

[0106] (2) Play a sound source data s in area A i , at this time, the label information corresponding to the simulated training audio data is L k = {l 1 =s i , l 2 =0}, play a sound source data s in area B i , at this time, the label information corresponding to the simulated training audio data is L k = {l 1 =0, l 2 =s i}.

[0107] (3) Play a sound source data s in area A i , play a sound source data s in area B j At this time, the label information corresponding to the simulated training audio data is L k = {l 1 =s i , l 2 =s j}.

[0108] S602, input the training audio data into the neural network, and the neural network outputs the sound pickup results corresponding to the training audio data.

[0109] It should be noted that the training audio data also needs to be preprocessed to be suitable for input into the neural network.

[0110] S603: Determine a relative error based on the sound pickup result corresponding to the training audio data and the label information of the training audio data.

[0111] Optionally, the electronic device determines sub-errors corresponding to the sound pickup data in at least two directions based on the recognition results corresponding to the training audio data and the label information of the training audio data; and determines the relative error based on the sub-errors corresponding to the sound pickup data in at least two directions.

[0112] For any direction, the electronic device compares the sound pickup data of that direction in the tag information with the sound pickup data of that direction in the sound pickup result to obtain the sub-error corresponding to the sound pickup data in that direction, and then calculates the sub-errors corresponding to the sound pickup data in the above at least two directions based on the mean square error or the square absolute error to obtain the final error.

[0113] S604, updating the parameters in the neural network based on the relative error, and restarting the step of outputting the training audio data to the neural network until the iteration stop condition is met.

[0114] The electronic device uses a back propagation algorithm based on the relative error to update the parameters in the neural network. The condition for stopping iteration includes at least one of the following: the relative error is less than the preset error, and the number of iterations is greater than the preset number. The number of iterations can be understood as the number of times the parameters in the neural network are updated. The preset error and the preset number can be determined according to the accuracy requirements of the audio processing model. Specifically, the higher the accuracy requirements of the audio processing model, the smaller the preset error and the greater the preset number. After determining that the condition for stopping iteration is met, the electronic device determines the current neural network that has undergone multiple parameter adjustments as the final audio processing model.

[0115] In summary, the embodiment of the present application provides a training process for an audio processing model. The trained audio processing model has decoding networks that are independent of each other and do not interfere with each other. Different decoding networks can output sound pickup data from different directions in the audio data collected by the microphone array, which can make the separation of sound pickup data from different directions better and reduce the probability of voice confusion in sound pickup data from multiple directions, thereby improving the auditory experience.

[0116] Please refer to Figure 7 , which shows a structural block diagram of an audio processing device provided by an embodiment of the present application. The audio processing device includes: a data acquisition module 710, a pre-processing module 720, and an audio processing module 730.

[0117] The data acquisition module 710 is used to acquire target audio data collected by the microphone array, where the target audio data is multi-channel audio data.

[0118] The preprocessing module 720 is used to preprocess the target audio data to obtain preprocessed audio data, where the preprocessed audio data is suitable for inputting into a pre-trained audio processing model.

[0119] The audio processing module 730 is used to process the pre-processed audio data through a pre-trained audio processing model to obtain sound pickup data in at least two directions. The pre-trained audio processing model includes an encoding network and at least two decoding networks respectively connected to the encoding network. The directions of the sound pickup data output by different decoding networks in the at least two decoding networks are different.

[0120] In some embodiments, the audio processing module 730 is used to encode the pre-processed audio data through an encoding network to obtain encoded data; decode the encoded data through at least two decoding networks respectively, and use the picked-up data output by the at least two decoding networks as the picked-up data in at least two directions.

[0121] In some embodiments, the decoding network includes a mask layer, a first operation layer, a pooling layer, and a second operation layer connected in sequence; the mask matrices corresponding to the mask layers included in different decoding networks are different; the audio processing module 730 is used to perform mask processing on the decoded data through the mask layer for each decoding network to obtain mask data; perform operation processing on the mask data through the first operation layer to obtain first operation data; perform pooling processing on the first operation data through the pooling layer to obtain pooled data; perform operation processing on the pooled data through the second operation layer to obtain the sound pickup data of the specified direction corresponding to the decoding network.

[0122] In some embodiments, the device further includes: a model training module (not shown in the figure). The model training module is used to obtain multiple training audio data, each of the multiple training audio data corresponds to label information, and the label information corresponding to the training audio data includes at least two directions of sound pickup data determined based on the training audio data; input the training audio data into the neural network, and the neural network outputs the sound pickup results corresponding to the training audio data; determine the relative error based on the sound pickup results corresponding to the training audio data and the label information of the training audio data; update the parameters in the neural network based on the relative error, and restart the execution from the step of outputting the training audio data to the neural network until the stop iteration condition is met.

[0123] In some embodiments, the model training module is used to determine the sub-errors corresponding to the sound pickup data in at least two directions based on the recognition results corresponding to the training audio data and the label information of the training audio data; and determine the relative error based on the sub-errors corresponding to the sound pickup data in at least two directions.

[0124] In some embodiments, the model training module is used to simulate the audio collection results of a microphone array for multiple sound source data through a room impulse response to obtain multiple training audio data; wherein the multiple sound source data are single-channel audio data; based on the playback positions of the multiple sound source data, determine the label information corresponding to the multiple training audio data respectively.

[0125] In some embodiments, the preprocessing module 720 is used to perform Fourier transform on the target audio data to obtain the real spectrum and imaginary spectrum corresponding to the audio data of each channel; calculate the amplitude spectrum of the audio data of each channel according to the real spectrum and imaginary spectrum corresponding to the audio data of each channel; compress the amplitude spectrum of the audio data of each channel to obtain the compressed amplitude spectrum of the audio data of each channel; wherein the preprocessed audio data includes the compressed amplitude spectrum of the audio data of each channel.

[0126] In summary, the technical solution provided in the embodiment of the present application pre-processes the multi-channel audio data collected by the microphone array to obtain pre-processed audio data, and the audio processing model performs audio processing on the pre-processed audio data to obtain sound pickup data in at least two directions, wherein the audio processing model includes at least two decoding networks, and different decoding networks are used to output sound pickup data in different directions. The above-mentioned at least two decoding networks are independent of each other and do not interfere with each other. Therefore, when multi-directional sound pickup is realized based on the audio processing model, the separation effect of sound pickup data in different directions can be better, and the probability of voice confusion in sound pickup data in multiple directions can be reduced, thereby improving the auditory experience.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0128] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0129] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0130] See also Figure 8 , which shows that the embodiment of the present application also provides an electronic device 800, which includes: one or more processors 810, a memory 820, and one or more applications. Among them, the one or more applications are stored in the memory 820 and are configured to be executed by the one or more processors 810, and the one or more applications are configured to execute the method described in the above embodiment.

[0131] The processor 810 may include one or more processing cores. The processor 810 uses various interfaces and lines to connect various parts of the entire battery management system, and executes various functions and processes data of the battery management system by running or executing instructions, programs, code sets or instruction sets stored in the memory 820, and calling data stored in the memory 820. Optionally, the processor 810 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 810 can integrate one or more combinations of a central processing unit 810 (CPU), a graphics processing unit 810 (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 810, but may be implemented separately through a communication chip.

[0132] The memory 820 may include a random access memory 820 (Random Access Memory, RAM), and may also include a read-only memory 820 (Read-Only Memory). The memory 820 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the electronic device during use (such as a phone book, audio and video data, chat record data), etc.

[0133] See also Fig. 9 , which shows that an embodiment of the present application also provides a computer-readable storage medium 900, in which computer program instructions 910 are stored. The computer program instructions 910 can be called by a processor to execute the method described in the above embodiment.

[0134] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for computer program instructions 910 for executing any method steps in the above method. These computer program instructions 910 may be read from or written to one or more computer program products. The computer program instructions 910 may be compressed in an appropriate form.

[0135] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. An audio processing method, characterized in that: Applied to an electronic device, the electronic device is provided with a microphone array, and the method comprises: Acquire target audio data collected by the microphone array, wherein the target audio data is multi-channel audio data; Preprocessing the target audio data to obtain preprocessed audio data, wherein the preprocessed audio data is suitable for inputting into a pre-trained audio processing model; The pre-processed audio data is processed by the pre-trained audio processing model to obtain sound pickup data in at least two directions, wherein the pre-trained audio processing model includes an encoding network and at least two decoding networks respectively connected to the encoding network, and directions of sound pickup data output by different decoding networks in at least two of the decoding networks are different.

2. The method according to claim 1, characterized in that The pre-processed audio data is processed by the pre-trained audio processing model to obtain sound pickup data in at least two directions, including: Performing encoding processing on the pre-processed audio data through the encoding network to obtain encoded data; The encoded data is decoded by at least two decoding networks respectively, and the sound pickup data respectively output by at least two decoding networks are used as the sound pickup data in at least two directions.

3. The method according to claim 2, characterized in that The decoding network includes a mask layer, a first operation layer, a pooling layer, and a second operation layer connected in sequence; the mask matrices corresponding to the mask layers included in different decoding networks are different; The step of respectively decoding the encoded data by at least two decoding networks and using the sound pickup data respectively output by the at least two decoding networks as the sound pickup data in the at least two directions comprises: For each of the decoding networks, masking the encoded data through the mask layer to obtain mask data; Performing operation processing on the mask data through the first operation layer to obtain first operation data; Performing pooling processing on the first operation data through the pooling layer to obtain pooled data; The pooled data is processed by the second operation layer to obtain the sound pickup data of the specified direction corresponding to the decoding network.

4. The method according to any one of claims 1 to 3, characterized in that The training process of the audio processing model includes: Acquire a plurality of training audio data, each of the plurality of training audio data corresponding to label information, and the label information corresponding to the training audio data includes sound pickup data in at least two directions determined based on the training audio data; Inputting the training audio data into a neural network, and having the neural network output a sound pickup result corresponding to the training audio data; Determining a relative error based on a sound pickup result corresponding to the training audio data and label information of the training audio data; The parameters in the neural network are updated based on the relative error, and the step of outputting the training audio data to the neural network is restarted until a stop iteration condition is met.

5. The method according to claim 4, characterized in that The determining the relative error based on the sound pickup result corresponding to the training audio data and the label information of the training audio data includes: Determine the sub-errors corresponding to the sound pickup data in the at least two directions respectively based on the recognition result corresponding to the training audio data and the label information of the training audio data; The relative error is determined based on the sub-errors respectively corresponding to the sound pickup data in the at least two directions.

6. The method according to claim 4, characterized in that The acquiring of a plurality of training audio data comprises: Simulating the audio collection results of the microphone array on the multiple sound source data through the room impulse response to obtain the multiple training audio data; wherein the multiple sound source data are single-channel audio data; Based on the playback positions of the plurality of sound source data, label information corresponding to the plurality of training audio data is determined.

7. The method according to any one of claims 1 to 3, characterized in that The preprocessing of the target audio data to obtain preprocessed audio data includes: Performing Fourier transform on the target audio data to obtain a real spectrum and an imaginary spectrum corresponding to the audio data of each channel; Calculate the amplitude spectrum of the audio data of each channel according to the real spectrum and the imaginary spectrum corresponding to the audio data of each channel; The amplitude spectrum of the audio data of each channel is compressed to obtain a compressed amplitude spectrum of the audio data of each channel; wherein the pre-processed audio data includes the compressed amplitude spectrum of the audio data of each channel.

8. An audio processing device, characterized in that: The device comprises: A data acquisition module is used to acquire target audio data collected by a microphone array, wherein the target audio data is multi-channel audio data; A preprocessing module, used to preprocess the target audio data to obtain preprocessed audio data, wherein the preprocessed audio data is suitable for inputting a pre-trained audio processing model; An audio processing module is used to process the pre-processed audio data through the pre-trained audio processing model to obtain sound pickup data in at least two directions, wherein the pre-trained audio processing model includes an encoding network and at least two decoding networks respectively connected to the encoding network, and the directions of the sound pickup data output by different decoding networks in at least two of the decoding networks are different.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein one or more of the applications are stored in the memory and configured to be executed by one or more of the processors, and one or more of the applications are configured to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and the computer program instructions can be called by a processor to execute the method according to any one of claims 1 to 7.

Images