A model training method, a sound signal processing method, and a device

Abstract

The embodiment of the present application provides a model training method, a sound signal processing method, and a device, which relate to the field of signal processing technology. The specific implementation scheme is: obtain multiple sample pairs; perform Fourier transform on the noise-free sound signal and the noisy sound signal in each sample pair respectively to obtain a first frequency domain signal and a second frequency domain signal; use the initial prediction model to process the second frequency domain signal to obtain a first prediction vector; multiply the first prediction vector corresponding to the sample pair with the first frequency domain signal and the second frequency domain signal respectively to obtain a first output vector and a second output vector; calculate the model loss value based on the difference between the first output vector and the second output vector corresponding to each of the multiple sample pairs; adjust the model parameters of the prediction model based on the model loss value. It can be seen that through this scheme, the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array can be accurately predicted.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a model training method, a sound signal processing method, and a device. Background Art

[0002] An acoustic imager, also known as an acoustic camera, is a special device that uses a microphone array to measure the distribution of sound fields within a certain range. It can be used to measure the location of the sound emitted by an object and the state of sound radiation. It uses microphone array technology to determine the location of the sound source, and through image visualization, it presents the collected sound in the form of a color contour map, using color and brightness to represent the strength of the sound, effectively showing the distribution of the sound source in space; combined with a camera, that is, the sound image is superimposed on the video image, forming a cloud map that can intuitively analyze the noise of the object being measured.

[0003] When the acoustic imager performs acoustic imaging, it is necessary to set the frequency range of the sound signal that needs to be imaged, so that after the original sound signal is collected, the original sound signal is filtered through the set frequency range to obtain the required target frequency band signal. In subsequent data processing, processing can be performed based on the target frequency band signal to determine the final cloud map. At present, users are usually required to manually set the frequency range of the sound signal that needs to be imaged. Since users do not understand the frequency range of the sound signal that needs to be imaged, it is easy for the set frequency range to not cover the frequency range of the sound signal that needs to be imaged, resulting in poor imaging effect of the cloud map, that is, the position of the sound source represented in the cloud map is inaccurate. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model training method, a sound signal processing method, and a device to accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of a microphone array. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0006] Acquire multiple sample pairs; wherein each sample pair includes: a noise-free sound signal collected by the microphone array and a noisy sound signal obtained by adding noise to the noise-free sound signal;

[0007] For each sample pair, Fourier transform is performed on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and Fourier transform is performed on the noisy sound signal in the sample pair to obtain a second frequency domain signal;

[0008] Using the initial prediction model to process the second frequency domain signal corresponding to the sample pair, a first prediction vector corresponding to the sample pair is obtained; wherein the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0009] Multiplying the first prediction vector corresponding to the sample pair with the first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and multiplying the first prediction vector corresponding to the sample pair with the second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair;

[0010] Calculating a model loss value based on differences between the first output vector and the second output vector corresponding to each of the plurality of sample pairs;

[0011] Based on the model loss value, the model parameters of the prediction model are adjusted.

[0012] Optionally, before performing Fourier transform on the noise-free sound signal in the sample pair for each sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal, the method further includes:

[0013] Determine a plurality of directions to be used; wherein each direction to be used is: a direction from a real position represented by a predetermined pixel point in the sample visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the sample visible light image is an image captured by a camera at the center position of the microphone array;

[0014] For each sample pair, performing Fourier transform on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal, including:

[0015] For each sample pair, beamforming processing is performed on the noise-free sound signal in the sample pair according to the multiple directions to be used to obtain a first beam signal corresponding to each direction to be used, and beamforming processing is performed on the noisy sound signal in the sample pair according to the multiple directions to be used to obtain a second beam signal corresponding to each direction to be used;

[0016] A first beam signal in the sample pair is Fourier transformed to obtain a first frequency domain signal, and a second beam signal in the sample pair is Fourier transformed to obtain a second frequency domain signal.

[0017] Optionally, calculating the model loss value based on the difference between the first output vector and the second output vector corresponding to each of the plurality of sample pairs includes:

[0018] For each sample pair, based on the difference between the first output vector and the second output vector corresponding to the sample pair, calculate the signal-to-noise ratio loss value and / or the normalized loss value corresponding to the sample pair;

[0019] Based on the obtained signal-to-noise ratio loss value and / or the normalized loss value, a model loss value is calculated.

[0020] Optionally, the signal-to-noise ratio loss value corresponding to any sample pair is calculated using a first formula; the first formula is:

[0021] ;

[0022] and / or,

[0023] The normalized loss value corresponding to any sample pair is calculated using the second formula; the second formula is:

[0024] ;

[0025] in, is the signal-to-noise ratio loss value corresponding to the sample pair, is the normalized loss value corresponding to the sample pair, is the first output vector corresponding to the sample pair, is the second output vector corresponding to the sample pair, represents the mean operation, represents the mean square error operation, represents the L2 norm.

[0026] In a second aspect, an embodiment of the present application provides a method for processing a sound signal, the method comprising:

[0027] Receiving the sound signal to be detected collected by the microphone array;

[0028] Performing Fourier transform on the sound signal to be detected to obtain a frequency domain signal to be processed;

[0029] The frequency domain signal to be processed is processed using a prediction model trained by the model training method described in any one of the above items to obtain a second prediction vector; wherein the elements in the second prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0030] The frequency points corresponding to the first preset number of elements whose element values ​​in the second prediction vector are sorted from high to low are used as the frequency points to be used;

[0031] The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected to obtain the sound signal used to generate the sound image.

[0032] Optionally, before performing Fourier transform on the sound signal to be detected to obtain the frequency domain signal to be processed, the method further includes:

[0033] Determine a plurality of directions to be used; wherein each direction to be used is: a direction from a real position represented by a predetermined pixel point in the visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the visible light image is an image captured by a camera at the center position of the microphone array;

[0034] The performing Fourier transform on the sound signal to be detected to obtain a frequency domain signal to be processed includes:

[0035] Performing beamforming processing on the sound signal to be detected according to the multiple directions to be used, to obtain a third beam signal corresponding to each direction to be used;

[0036] Perform Fourier transform on the third beam signal to obtain a frequency domain signal to be processed.

[0037] Optionally, selecting the sound signal corresponding to the frequency point to be used from the sound signal to be detected to obtain the sound signal for generating the sonogram includes:

[0038] The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected, and a preset window function is used to perform windowing processing on the selected sound signal to obtain a sound signal for generating an acoustic image.

[0039] Optionally, the method further comprises:

[0040] An acoustic image is generated based on the sound signal corresponding to the frequency point to be used, and is superimposed with the visible light image collected by the camera at the center position of the microphone array to obtain a cloud image.

[0041] In a third aspect, an embodiment of the present application provides a model training device, the device comprising:

[0042] An acquisition module is used to acquire multiple sample pairs, wherein each sample pair includes: a noise-free sound signal collected by a microphone array and a noisy sound signal obtained after the noise-free sound signal is processed by adding noise;

[0043] A first transform module is used for performing Fourier transform on the noise-free sound signal in each sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal;

[0044] A first prediction module is used to process the second frequency domain signal corresponding to the sample pair using an initial prediction model to obtain a first prediction vector corresponding to the sample pair; wherein the elements in the first prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0045] an output module, configured to multiply a first prediction vector corresponding to the sample pair with a first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and to multiply the first prediction vector corresponding to the sample pair with a second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair;

[0046] A calculation module, configured to calculate a model loss value based on a difference between the first output vector and the second output vector corresponding to each of the plurality of sample pairs;

[0047] An adjustment module is used to adjust the model parameters of the prediction model based on the model loss value.

[0048] Optionally, the device further comprises:

[0049] A first direction determination module is used to determine a plurality of directions to be used before the first transformation module performs Fourier transform on the noise-free sound signal in the sample pair for each sample pair to obtain a first frequency domain signal, and performs Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal; wherein each direction to be used is: the direction from the real position represented by a predetermined pixel point in the sample visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the sample visible light image is an image captured by the camera at the center position of the microphone array;

[0050] The first transformation module comprises:

[0051] A first beamforming submodule is used to perform beamforming processing on the noise-free sound signal in the sample pair according to the multiple directions to be used for each sample pair, so as to obtain a first beam signal corresponding to each direction to be used, and to perform beamforming processing on the noisy sound signal in the sample pair according to the multiple directions to be used, so as to obtain a second beam signal corresponding to each direction to be used;

[0052] The first Fourier transform submodule is used to perform Fourier transform on the first beam signal in the sample pair to obtain a first frequency domain signal, and to perform Fourier transform on the second beam signal in the sample pair to obtain a second frequency domain signal.

[0053] Optionally, the calculation module includes:

[0054] A first calculation submodule, configured to calculate, for each sample pair, a signal-to-noise ratio loss value corresponding to the sample pair and / or a normalized loss value based on a difference between a first output vector and a second output vector corresponding to the sample pair;

[0055] The second calculation submodule is used to calculate the model loss value based on the obtained signal-to-noise ratio loss value and / or the normalized loss value.

[0056] Optionally, the signal-to-noise ratio loss value corresponding to any sample pair is calculated using a first formula; the first formula is: ;

[0057] and / or,

[0058] The normalized loss value corresponding to any sample pair is calculated using the second formula; the second formula is: ;

[0059] in, is the signal-to-noise ratio loss value corresponding to the sample pair, is the normalized loss value corresponding to the sample pair, is the first output vector corresponding to the sample pair, is the second output vector corresponding to the sample pair, represents the mean operation, represents the mean square error operation, represents the L2 norm.

[0060] In a fourth aspect, an embodiment of the present application provides a sound signal processing device, the device comprising:

[0061] A receiving module, used for receiving the sound signal to be detected collected by the microphone array;

[0062] A second transformation module, used for performing Fourier transformation on the sound signal to be detected to obtain a frequency domain signal to be processed;

[0063] A second prediction module is used to process the frequency domain signal to be processed using a prediction model trained by any of the above-mentioned model training methods to obtain a second prediction vector; wherein the elements in the second prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0064] A determination module, configured to use frequency points corresponding to the first preset number of elements whose element values ​​in the second prediction vector are sorted from high to low as the frequency points to be used;

[0065] The selection module is used to select the sound signal corresponding to the frequency point to be used from the sound signal to be detected, and obtain the sound signal used to generate the sonogram.

[0066] Optionally, the device further comprises:

[0067] A second direction determination module is used to determine a plurality of directions to be used before the second transformation module performs Fourier transform on the sound signal to be detected to obtain the frequency domain signal to be processed; wherein each direction to be used is: the direction from the real position represented by a predetermined pixel point in the visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the visible light image is an image captured by the camera at the center position of the microphone array;

[0068] The second transformation module comprises:

[0069] A second beamforming submodule, configured to perform beamforming processing on the sound signal to be detected according to the multiple directions to be used, so as to obtain a third beam signal corresponding to each direction to be used;

[0070] The second Fourier transform submodule is used to perform Fourier transform on the third beam signal to obtain a frequency domain signal to be processed.

[0071] Optionally, the selection module is specifically used to:

[0072] The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected, and a preset window function is used to perform windowing processing on the selected sound signal to obtain a sound signal for generating an acoustic image.

[0073] Optionally, the device further comprises:

[0074] The superposition module is used to generate an acoustic image based on the sound signal corresponding to the frequency point to be used, and to superimpose it with the visible light image collected by the camera at the center position of the microphone array to obtain a cloud image.

[0075] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0076] Memory, used to store computer programs;

[0077] The processor is used to implement any of the above-mentioned model training methods or sound signal processing methods when executing the program stored in the memory.

[0078] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the model training method or the sound signal processing method described above is implemented.

[0079] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the above-mentioned model training methods or sound signal processing methods.

[0080] Beneficial effects of the embodiments of the present application:

[0081] In the solution provided by the embodiment of the present application, since the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array, the first output vector is obtained by multiplying the first prediction vector with the first frequency domain signal, and the second output vector is obtained by multiplying the first prediction vector with the second frequency domain signal, the first frequency domain signal is a frequency domain signal converted from a noise-free sound signal, and the second frequency domain signal is a frequency domain signal converted from a noisy sound signal, therefore, the first output vector represents the signal power after the signal strength of each frequency component in the noise-free sound signal is adjusted, and the second output vector represents the signal power after the signal strength of each frequency component in the noisy sound signal is adjusted. Since the frequency components of the noisy sound signal have a higher signal-to-noise ratio closer to the frequency components of the noise-free sound signal, if the first output vector and the second output vector are closer, it means that after processing the first frequency domain signal and the second frequency domain signal using the first prediction vector, the first output vector and the second output vector that pay more attention to the signal-to-noise ratio distribution of the real sound signal can be obtained. Therefore, based on the difference between the first output vector and the second output vector corresponding to each of the multiple sample pairs, the model loss value is calculated, and based on the model loss value, the model parameters of the prediction model are adjusted, so that the first prediction vector output by the prediction model can reflect the signal-to-noise ratio distribution of the actual sound signal, that is, the prediction model can accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array.

[0082] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0084] Figure 1 A cloud map generated by an audio-visual instrument provided in an embodiment of the present application;

[0085] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram of a preset grid matrix provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of a microphone array and a camera provided in an embodiment of the present application;

[0088] Figure 5 A schematic diagram of a model training method for performing beamforming processing provided in an embodiment of the present application;

[0089] Figure 6 A flowchart of a sound signal processing method provided in an embodiment of the present application;

[0090] Figure 7 A schematic diagram of a specific example of the sound signal processing method provided in an embodiment of the present application;

[0091] Figure 8 A schematic diagram of a specific example of generating a cloud map based on frequency points to be used provided in an embodiment of the present application;

[0092] Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0093] Fig.10 A schematic diagram of the structure of a sound signal processing device provided in an embodiment of the present application;

[0094] Fig.11 A schematic diagram of an electronic device for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0096] At present, the commonly used acoustic imagers on the market need to set the frequency range of the sound signal to be imaged during acoustic imaging. If the set frequency range does not cover the frequency range of the sound signal to be imaged, the imaging effect of the cloud map will be inaccurate. Figure 1 As shown in the figure, it is a cloud map generated by an audio-visual device. The real sound signal in the scene shown in the figure is emitted by the hair dryer shown in the frame 1 in the figure. Since the user does not understand the frequency range of the sound signal of the hair dryer, the frequency range is set to the frequency range of noise. At this time, the position of the color contour spectrum in the cloud map is at the position of the ground shown in the frame 2 in the figure, resulting in inaccurate sound source positioning and poor imaging effect.

[0097] In order to achieve accurate automatic acoustic imaging even when the user does not know the frequency range of the sound signal, the embodiments of the present application provide a model training method, a sound signal processing method, and a device.

[0098] The model training method provided in the embodiment of the present application can be applied to various electronic devices, such as personal computers, servers, and other devices with data processing capabilities. The sound signal processing method provided in the embodiment of the present application can be applied to various electronic devices, such as personal computers, servers, audio-visual devices, and other devices with data processing capabilities. In addition, it is understood that the model training method and sound signal processing method provided in the embodiment of the present disclosure can be implemented by software, hardware, or a combination of software and hardware.

[0099] Among them, a model training method provided in an embodiment of the present application may include the following steps:

[0100] Acquire multiple sample pairs; wherein each sample pair includes: a noise-free sound signal collected by the microphone array and a noisy sound signal obtained by adding noise to the noise-free sound signal;

[0101] For each sample pair, Fourier transform is performed on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and Fourier transform is performed on the noisy sound signal in the sample pair to obtain a second frequency domain signal;

[0102] The second frequency domain signal corresponding to the sample pair is processed using the initial prediction model to obtain a first prediction vector corresponding to the sample pair; wherein the elements in the first prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0103] Multiplying the first prediction vector corresponding to the sample pair with the first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and multiplying the first prediction vector corresponding to the sample pair with the second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair;

[0104] Calculate the model loss value based on the difference between the first output vector and the second output vector corresponding to each of the multiple sample pairs;

[0105] Based on the model loss value, the model parameters of the prediction model are adjusted.

[0106] In the solution provided by the embodiment of the present application, since the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array, the first output vector is obtained by multiplying the first prediction vector with the first frequency domain signal, and the second output vector is obtained by multiplying the first prediction vector with the second frequency domain signal, the first frequency domain signal is a frequency domain signal converted from a noise-free sound signal, and the second frequency domain signal is a frequency domain signal converted from a noisy sound signal, therefore, the first output vector represents the signal power after the signal strength of each frequency component in the noise-free sound signal is adjusted, and the second output vector represents the signal power after the signal strength of each frequency component in the noisy sound signal is adjusted. Since the frequency components of the noisy sound signal have a higher signal-to-noise ratio closer to the frequency components of the noise-free sound signal, if the first output vector and the second output vector are closer, it means that after processing the first frequency domain signal and the second frequency domain signal using the first prediction vector, the first output vector and the second output vector that pay more attention to the signal-to-noise ratio distribution of the real sound signal can be obtained. Therefore, based on the difference between the first output vector and the second output vector corresponding to each of the multiple sample pairs, the model loss value is calculated, and based on the model loss value, the model parameters of the prediction model are adjusted, so that the first prediction vector output by the prediction model can reflect the signal-to-noise ratio distribution of the actual sound signal, that is, the prediction model can accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array.

[0107] The model training method provided in the embodiment of the present application is introduced below in conjunction with the accompanying drawings. Figure 2 As shown, the method includes steps S201-S206:

[0108] S201, obtaining a plurality of sample pairs, wherein each sample pair comprises: a noise-free sound signal collected by a microphone array and a noisy sound signal obtained by adding noise to the noise-free sound signal;

[0109] In this embodiment, the noise-free sound signal and the noisy sound signal in each sample pair are both multi-channel sound signals, and the number of channels of the noise-free sound signal and the noisy sound signal is the number of microphones in the microphone array. It can be understood that the microphone array is composed of multiple microphones, and each microphone collects sound to obtain a single-channel sound signal, so that the microphone array can collect multi-channel sound signals. Exemplarily, if the microphone array includes 64 microphones, the microphone array can collect 64-channel sound signals.

[0110] For example, in practical applications, a microphone array can be used to collect sound signals in a quiet environment to obtain a noise-free sound signal. It is understandable that in practical applications, since it is difficult to collect noise-free sound signals, when constructing multiple sample pairs, a noise-free sound signal can be collected first, and then multiple noise-free sound signals can be generated through multi-channel simulation. Each noise-free sound signal is processed by adding noise to obtain a noisy sound signal corresponding to the noise-free sound signal.

[0111] Exemplarily, the noise-adding process to the noise-free sound signal may be to add the common stationary noise in the factory, or non-stationary noise such as knocking, to the noise-free sound signal. It is understandable that by adding non-stationary noise as the noisy sound signal to train the prediction model during the training process, the prediction model can accurately predict the signal-to-noise ratio corresponding to each frequency point in a harsh environment such as a factory.

[0112] S202, for each sample pair, performing Fourier transform on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal;

[0113] In order to better analyze the frequency components of the noise-free sound signal and the noisy sound signal, the sound signal in each sample pair is converted into a frequency domain signal. In this embodiment, the noise-free sound signal and the noisy sound signal in the time domain are converted into the frequency domain using Fourier transform to obtain a first frequency domain signal and a second frequency domain signal corresponding to each sample pair.

[0114] Exemplarily, if the number of microphones in the microphone array is 64, the microphone array collects 64 channels of noise-free sound signals, and each sample pair includes 64 channels of noise-free sound signals and 64 channels of noisy sound signals. The noise-free sound signals in the sample pair are Fourier transformed, that is, the noise-free sound signals of each channel are Fourier transformed, and finally a 64-channel first frequency domain signal is obtained. Correspondingly, the noisy sound signals in the sample pair are Fourier transformed, and a 64-channel second frequency domain signal is also obtained.

[0115] It is understandable that since the microphones in the microphone array are omnidirectional pickup microphones, that is, all noise in the environment will be collected by the microphones, if the multi-channel sound signals collected by the microphone array are directly Fourier transformed, the signal quality of the obtained frequency domain signal is poor, which leads to the poor signal quality of the frequency domain signal subsequently sent to the prediction model, which will affect the training effect of the prediction model. Therefore, in order to improve the training effect of the model, in the embodiment of the present application, the noise-free sound signal and the noisy sound signal in the sample pair can also be beamformed separately to suppress the direction of the interfering sound source and improve the signal-to-noise ratio of the frequency domain signal input to the prediction model.

[0116] It should be noted that, for the sake of clarity of the solution layout, the following embodiments introduce the manner of performing beamforming processing on the noise-free sound signal and the noisy sound signal in the sample pair respectively, which will not be repeated here.

[0117] S203, using the initial prediction model to process the second frequency domain signal corresponding to the sample pair to obtain a first prediction vector corresponding to the sample pair; wherein the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0118] In this embodiment, the initial prediction model can be CNN (Convolutional Neural Networks), LSTM (Long Short-Term Memory) or DCCRN (Deep Complex Convolution Recurrent Network), etc.

[0119] Exemplarily, the second frequency domain signal corresponding to each sample pair can be organized into a matrix form and input into the initial prediction model, so as to process the second frequency domain signal corresponding to the sample pair using the initial prediction model. Wherein, each row in the matrix represents the second frequency domain signal of a channel, and each column represents the amplitude characteristic of a frequency component in the second frequency domain signal, that is, the amplitude characteristic of a frequency point.

[0120] The output of the prediction model is the first prediction vector corresponding to the sample pair, and each element in the first prediction vector represents the signal-to-noise ratio corresponding to a frequency point. The number of elements included in the first prediction vector is related to the frequency response range of the microphone array, that is, if the frequency response range of the microphone array is 20Hz-20kHz, the number of elements included in the first prediction vector is 19800, so that the signal-to-noise ratio of each frequency component of the second frequency domain signal can be predicted.

[0121] Exemplarily, if the second frequency domain signal corresponding to a sample pair includes second frequency domain signals of 64 channels and the frequency response range of the microphone array is 20Hz-20kHz, the second frequency domain signal corresponding to the sample pair can be organized as a matrix of 64×19800 dimensions and input into the initial prediction model. The first prediction vector corresponding to the sample pair output by the prediction model is a vector of 19800×1 dimensions.

[0122] It can be understood that since the second frequency domain signal input to the prediction model contains the amplitude characteristics of each frequency point, and the signal power is related to the amplitude of the frequency domain signal, the prediction model can predict the signal-to-noise ratio corresponding to each frequency point based on the amplitude characteristics of each frequency point.

[0123] In addition, it should be noted that the elements in the first prediction vector may be normalized elements, that is, the value of the signal-to-noise ratio represented by each element in the first prediction vector is between 0 and 1.

[0124] S204, multiplying the first prediction vector corresponding to the sample pair with the first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and multiplying the first prediction vector corresponding to the sample pair with the second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair;

[0125] It can be understood that, since each element in the first prediction vector represents the signal-to-noise ratio corresponding to a frequency point, the first prediction vector corresponding to each sample pair is multiplied by the first frequency domain signal corresponding to the sample pair, and the first frequency signal can be weighted using the element value of each element in the first prediction vector, and the obtained first output vector represents the signal power after the signal strength of each frequency component is adjusted. Similarly, the first prediction vector corresponding to each sample pair is multiplied by the second frequency domain signal corresponding to the sample pair, and the second frequency signal can be weighted using the element value of each element in the first prediction vector, and the obtained second output vector represents the signal power after the signal strength of each frequency component is adjusted.

[0126] Exemplarily, if the first frequency domain signal is a 64×n matrix, 64 is the number of channels, n is the number of frequency points, and the first prediction vector is n×1, then the first frequency domain signal and the first prediction vector are multiplied to obtain a 64×1 first output vector, and each element in the first output vector represents the signal power corresponding to a channel. And the signal power corresponding to each channel is the signal power after the signal strength of the predicted frequency component with a high signal-to-noise ratio is enhanced.

[0127] S205, calculating a model loss value based on differences between the first output vector and the second output vector corresponding to each of the plurality of sample pairs;

[0128] It can be understood that, since the first output vector is obtained by multiplying the first prediction vector and the first frequency domain signal, the first frequency domain signal is a frequency domain signal converted from a noise-free sound signal, and the second output vector is obtained by multiplying the first prediction vector and the second frequency domain signal, and the second frequency domain signal is a frequency domain signal converted from a noisy sound signal, the first output vector represents the signal power corresponding to the noise-free sound signal, and the second output vector represents the signal power corresponding to the noisy sound signal. Therefore, the smaller the difference between the first output vector and the second output vector corresponding to each sample pair, the closer the signal power of the noisy sound signal and the noise-free sound signal contained in the sample pair is, that is, the greater the signal-to-noise ratio of the noisy sound signal. Then, based on the difference between the first output vector and the second output vector corresponding to each of the multiple sample pairs, the model loss value is calculated, and the model parameters of the prediction model can be adjusted later by minimizing the model loss value, so that the signal-to-noise ratio represented by each element in the first prediction vector output by the prediction model is more accurate.

[0129] Optionally, in one implementation, the model loss value is calculated based on the difference between the first output vector and the second output vector corresponding to each of the plurality of sample pairs, including steps A1-A2:

[0130] A1, for each sample pair, based on the difference between the first output vector and the second output vector corresponding to the sample pair, calculate the signal-to-noise ratio loss value and / or the normalized loss value corresponding to the sample pair;

[0131] It can be understood that by calculating the signal-to-noise ratio loss value corresponding to the sample pair and subsequently adjusting the model parameters by minimizing the signal-to-noise ratio loss value, the signal-to-noise ratio represented by each element in the first prediction vector output by the trained prediction model can be made more accurate. By calculating the normalized loss value corresponding to the sample pair and subsequently adjusting the model parameters by minimizing the normalized loss value, the signal-to-noise ratio represented by each element in the first prediction vector output by the trained prediction model can be made more uniform without large jumps.

[0132] Exemplarily, the signal-to-noise ratio loss value corresponding to any sample pair is calculated using the first formula; the first formula is: ;

[0133] The normalized loss value corresponding to any sample pair is calculated using the second formula; the second formula is: ;

[0134] in, is the signal-to-noise ratio loss value corresponding to the sample pair, is the normalized loss value corresponding to the sample pair, is the first output vector corresponding to the sample pair, is the second output vector corresponding to the sample pair, represents the mean operation, represents the mean square error operation, represents the L2 norm.

[0135] It should be noted that the coefficient 20log in the first formula is used to convert the signal power into decibel (dB) units. In the first formula, and The smaller the difference is, the smaller the signal-to-noise ratio loss value is. Then, by minimizing the signal-to-noise ratio loss value, the first output vector and the second output vector can be made closer. It can be understood that since the first output vector is obtained by multiplying the first prediction vector and the first frequency domain signal, and the second output vector is obtained by multiplying the second prediction vector and the second frequency domain signal, if the first output vector and the second output vector are closer, it means that after processing the first frequency domain signal and the second frequency domain signal using the first prediction vector, the first output vector and the second output vector that pay more attention to the signal-to-noise ratio distribution of the real sound signal can be obtained. Then, based on the signal-to-noise ratio loss value, the model parameters of the prediction model are adjusted, so that the first prediction vector output by the prediction model can reflect the signal-to-noise ratio distribution of the real sound signal, that is, the signal-to-noise ratio output by the first prediction vector can be accurate.

[0136] It can be understood that the second formula is a formula for normalizing the difference between the first output vector and the second output vector, so that the deviation between the difference and the mean of the difference is expressed in units of standard deviation. By minimizing the normalized loss value, the difference between the differences between the first output vector and the second output vector can be made smaller, so that the signal-to-noise ratio represented by each element in the first prediction vector can be made more uniform without large jumps.

[0137] A2, calculating the model loss value based on the obtained signal-to-noise ratio loss value and / or the normalized loss value.

[0138] After obtaining the signal-to-noise ratio loss value and / or the normalized loss value corresponding to each sample pair in the above step A1, the sum of the obtained signal-to-noise ratio loss values ​​can be directly used as the model loss value, or the sum of the obtained normalized loss values ​​can be used as the model loss value, or the obtained signal-to-noise ratio loss value and the normalized loss value can be weighted and summed to obtain the model loss value, which is reasonable. Exemplarily, in a specific implementation, the model loss value can be calculated by the following formula:

[0139] ;

[0140] in, Represents the model loss value, Represents the sum of the signal-to-noise ratio loss values ​​corresponding to each sample pair, Represents the sum of the normalized loss values ​​corresponding to each sample pair.

[0141] S206: Adjust the model parameters of the prediction model based on the model loss value.

[0142] In this embodiment, the model parameters of the prediction model can be adjusted by minimizing the back propagation of the model loss value, so that the signal-to-noise ratio of each frequency point predicted by the adjusted prediction model is more accurate. Then, the prediction model can be used to select frequency points with high signal-to-noise ratio from the frequency points included in the frequency response range of the microphone. It can be understood that after adjusting the model parameters of the prediction model, it is possible to continue to return to step S201 until the predetermined training end condition is reached and the training is stopped. The predetermined training end condition can be that a predetermined number of training times is reached, or the model loss value is less than a preset threshold, which is reasonable. Among them, the predetermined number of training times and the preset threshold are set by relevant technical personnel based on experience. It should be noted that the embodiment of the present application does not limit the method of adjusting the model parameters of the prediction model based on the model loss value. For example, in actual applications, the model parameters of the prediction model can also be adjusted based on the model loss value using the gradient descent method.

[0143] In the solution provided by the embodiment of the present application, since the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array, the first output vector is obtained by multiplying the first prediction vector with the first frequency domain signal, and the second output vector is obtained by multiplying the first prediction vector with the second frequency domain signal, the first frequency domain signal is a frequency domain signal converted from a noise-free sound signal, and the second frequency domain signal is a frequency domain signal converted from a noisy sound signal, therefore, the first output vector represents the signal power after the signal strength of each frequency component in the noise-free sound signal is adjusted, and the second output vector represents the signal power after the signal strength of each frequency component in the noisy sound signal is adjusted. Since the frequency components of the noisy sound signal have a higher signal-to-noise ratio closer to the frequency components of the noise-free sound signal, if the first output vector and the second output vector are closer, it means that after processing the first frequency domain signal and the second frequency domain signal using the first prediction vector, the first output vector and the second output vector that pay more attention to the signal-to-noise ratio distribution of the real sound signal can be obtained. Therefore, based on the difference between the first output vector and the second output vector corresponding to each of the multiple sample pairs, the model loss value is calculated, and based on the model loss value, the model parameters of the prediction model are adjusted, so that the first prediction vector output by the prediction model can reflect the signal-to-noise ratio distribution of the actual sound signal, that is, the prediction model can accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array.

[0144] Furthermore, since the trained prediction model can accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array, the trained prediction model can be used to predict the signal-to-noise ratio of each frequency point in the sound signal to be detected, and the frequency point with a high signal-to-noise ratio can be determined. Since the sound signal at the sound source position has a higher signal-to-noise ratio, the use of frequency points with a high signal-to-noise ratio to generate an acoustic image can improve the accuracy of the sound source position in the obtained acoustic image.

[0145] Optionally, in another embodiment of the present application, before performing Fourier transform on the noise-free sound signal in the sample pair for each sample pair in the above step S202 to obtain the first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain the second frequency domain signal, the method further includes:

[0146] Determine a plurality of directions to be used; wherein each direction to be used is: a direction from a real position represented by a predetermined pixel point in the sample visible light image to a center position of the microphone array; each predetermined pixel point is a center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the sample visible light image is an image captured by a camera at the center position of the microphone array;

[0147] In this embodiment, the direction between a first coordinate of a predetermined pixel point in the sample visible light image converted from the image coordinate system to the world coordinate system and a second coordinate of the center position of the microphone array in the world coordinate system is the direction to be used.

[0148] The preset matrix grid can be a 3×3 or 4×4 matrix grid. The embodiment of the present application does not limit the size of the matrix grid on the basis of ensuring that the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array. It can be understood that by determining multiple directions to be used and subsequently using the multiple directions to be used to perform beamforming processing on the multi-channel sound signals collected by the microphone array, it is not only possible to suppress the sounds in the surrounding non-target directions, but also to reduce the number of channels of the frequency domain signals that need to be input into the prediction model later, thereby reducing the complexity of the algorithm.

[0149] Since there are many microphones in the sound imager, if beamforming is required in all directions, the calculation complexity is too complicated and difficult to implement on hardware devices. Therefore, in this embodiment, the number of directions to be used can be determined by comprehensively considering the computing power and the size of the rectangular image area obtained by division.

[0150] Exemplarily, in one implementation, the preset matrix grid is as follows Figure 3 As shown in FIG. , it is a 3×3 matrix grid. After dividing the sample visible light image using the matrix grid, 9 rectangular image regions can be obtained, such as Figure 3 As shown by the numbers ① to ⑨ in FIG. 1 . The center point of each rectangular image area is a predetermined pixel point, and the direction from the real position represented by the predetermined pixel point to the center position of the microphone array is a direction to be used. Figure 3 The division method shown can obtain 9 directions to be used, so that the multi-channel sound signals collected by the microphone array can be subsequently beamformed in 9 directions to obtain beams in 9 directions. It can be understood that in order to make the obtained 9 beams in the directions to be used cover the entire picture area of ​​the sample visible light image, each beam needs to be specifically designed. For example, in order to make Figure 3 The beams in the directions to be used corresponding to the rectangular image areas marked with numbers ①, ②, and ④ can overlap, and a specific number of microphones can be selected, such as using only a certain number of microphones in the inner circle, to achieve wider beam coverage. That is, during beamforming processing, only the sound signals collected by some microphones can be selected for beamforming processing. The fewer the number of microphones selected, the wider the beam formed.

[0151] For example, the microphone array used in the embodiment of the present application is as follows: Figure 4As shown in FIG. 1 , the microphone array is a 64-microphone array, that is, it includes 64 microphones. Each small dot numbered 1-64 in the figure represents a microphone. A camera is set at the center of the microphone array. Figure 4 As shown in the black rectangular area at the center, the camera is used to collect sample visible light images. It can be understood that ensuring that the center of the microphone array and the center of the camera are at the same coordinate position can effectively improve the calibration accuracy of the sound source position.

[0152] Accordingly, in this embodiment, for each sample pair in step S202, Fourier transform is performed on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and Fourier transform is performed on the noisy sound signal in the sample pair to obtain a second frequency domain signal, including steps B1-B2:

[0153] B1, for each sample pair, performing beamforming processing on the noise-free sound signal in the sample pair according to multiple directions to be used, to obtain a first beam signal corresponding to each direction to be used, and performing beamforming processing on the noisy sound signal in the sample pair according to multiple directions to be used, to obtain a second beam signal corresponding to each direction to be used;

[0154] It can be understood that the role of beamforming processing is to merge multi-channel sound signals to suppress sound signals in non-target directions and enhance sound signals in target directions. Exemplarily, in practical applications, a delayed-sum-based beamforming algorithm or an MVDR (Minimum Variance Distortionless Response) algorithm can be used to perform beamforming processing. Specifically, according to the pitch angles and azimuth angles corresponding to the multiple directions to be used, the angle parameters required in the beamforming algorithm are determined, so that the noise-free sound signal and the noisy sound signal in the sample pair are respectively beamformed according to the multiple directions to be used, and the corresponding first beam signal and second beam signal are obtained. It should be noted that the embodiments of the present application do not limit the specific beamforming algorithm used.

[0155] B2, performing Fourier transform on the first beam signal in the sample pair to obtain a first frequency domain signal, and performing Fourier transform on the second beam signal in the sample pair to obtain a second frequency domain signal.

[0156] It can be understood that by performing beamforming processing on the noise-free sound signal and the noisy sound signal in the sample pair, and then performing Fourier transform on the obtained first beam signal and the second beam signal, the signal quality of the frequency domain signal subsequently sent to the prediction model can be improved. Thus, the training effect of the model can be improved.

[0157] For example, Figure 5 As shown in , this is the model training method for beamforming processing. Figure 5 As shown, first use the microphone array to obtain a clean signal (corresponding to the noise-free sound signal mentioned above), and then perform multi-channel simulation on the clean signal to obtain a noisy signal (corresponding to the noisy sound signal mentioned above). Perform feature extraction on the noisy signal, that is, perform beamforming processing on the noisy signal, and then perform FFT (Fast Fourier Transform) on the beam signal obtained after the beamforming processing and input it into the CNN network (corresponding to the prediction model mentioned above) to obtain the output Mask vector. The elements in the Mask vector represent the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array. Multiply the noisy signal by the Mask vector after FFT (as shown in Figure 5 The symbol " ”), get the first vector (corresponding to the second output vector above), and multiplying the clean signal by the Mask vector after FFT to obtain the second vector (corresponding to the first output vector above). and The difference between the two is used to calculate the Loss (corresponding to the model loss value in the above text), and then the Loss is used to adjust the model parameters of the CNN network until the training end conditions are met.

[0158] It can be seen that through this solution, the signal quality of the frequency domain signal sent to the prediction model can be improved, thereby improving the training effect of the prediction model, so that the prediction model can more accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array.

[0159] Corresponding to the above-mentioned model training method embodiment, the present application embodiment also provides a sound signal processing method, such as Figure 6 As shown, steps S601-S605 are included:

[0160] S601, receiving a sound signal to be detected collected by a microphone array;

[0161] S602, performing Fourier transform on the sound signal to be detected to obtain a frequency domain signal to be processed;

[0162] In this embodiment, when it is necessary to determine the location of a sound source in a scene, the sound signal to be detected collected by the microphone array can be received first. It is understandable that due to different collection scenes, the sound signal to be detected may not include a noise signal, but may include a noise signal, such as wind noise or other noise signals.

[0163] After obtaining the sound signal to be detected, the sound signal to be detected can be subjected to Fourier transform, and the sound signal to be detected can be converted from the time domain to the frequency domain, so that the frequency components of the sound signal to be detected can be subsequently analyzed to determine the frequency components with the highest signal-to-noise ratio, so that the sound source position representing the true sound signal can be subsequently determined based on the frequency components with the highest signal-to-noise ratio.

[0164] It can be understood that since the microphones in the microphone array are omnidirectional pickup microphones, that is, all noise in the environment will be collected by the microphones, if the multi-channel sound signals collected by the microphone array are directly subjected to Fourier transform, the signal quality of the obtained frequency domain signals will be poor, which will lead to the signal quality of the subsequent frequency domain signals sent to the prediction model will be poor. Therefore, in order to improve the signal quality of the frequency domain signals sent to the prediction model, the sound signals to be detected can be first subjected to beamforming processing to improve the signal-to-noise ratio of the frequency domain signals input into the prediction model.

[0165] Optionally, in one implementation, before Fourier transform is performed on the sound signal to be detected to obtain the frequency domain signal to be processed, the method further includes:

[0166] Determine a plurality of directions to be used; wherein each direction to be used is: a direction from a real position represented by a predetermined pixel point in the visible light image to a center position of the microphone array; each predetermined pixel point is a center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the visible light image is an image captured by a camera at the center position of the microphone array;

[0167] It should be noted that the method of determining multiple directions to be used can refer to the relevant description of the above step S202, which will not be repeated here.

[0168] Accordingly, in this implementation, Fourier transform is performed on the sound signal to be detected to obtain a frequency domain signal to be processed, including steps C1 and C2:

[0169] C1, performing beamforming processing on the sound signal to be detected according to multiple directions to be used, to obtain a third beam signal corresponding to each direction to be used;

[0170] C2, performing Fourier transform on the third beam signal to obtain a frequency domain signal to be processed.

[0171] It should be noted that the specific implementation of steps C1-C2 can refer to the relevant description of the above steps B1-B2, which will not be repeated here.

[0172] S603, using the prediction model trained based on the above model training method to process the frequency domain signal to be processed, to obtain a second prediction vector; wherein the elements in the second prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0173] It can be understood that since the prediction model trained according to the above model training method can accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array, the trained prediction model can be used to process the frequency domain signal to be processed to obtain a second prediction vector.

[0174] S604, taking frequency points corresponding to the first preset number of elements whose element values ​​in the second prediction vector are sorted from high to low as frequency points to be used;

[0175] S605 , selecting a sound signal corresponding to a frequency point to be used from the sound signal to be detected, and obtaining a sound signal for generating an acoustic image.

[0176] This embodiment does not limit the preset number. In practical applications, the preset number is set by relevant technical personnel based on experience.

[0177] It can be understood that, since the second prediction vector output by the prediction model trained according to the above-mentioned model training method can reflect the signal-to-noise ratio distribution of the real sound signal, the frequency points corresponding to the first preset number of elements in the second prediction vector whose element values ​​are sorted from high to low are the frequency points with the highest signal-to-noise ratio, and the sound signal corresponding to the real sound source position has a higher signal-to-noise ratio. Therefore, by using the frequency points corresponding to the first preset number of elements in the second prediction vector whose element values ​​are sorted from high to low as the frequency points to be used, and selecting the sound signals corresponding to the frequency points to be used from the sound signals to be detected as the sound signals for generating the sound image, the accuracy of the color contour map representing the sound source position in the generated sound image can be improved, that is, the accuracy of acoustic imaging can be improved.

[0178] Optionally, in one implementation, selecting a sound signal corresponding to a frequency point to be used from the sound signal to be detected to obtain a sound signal for generating an acoustic image includes:

[0179] The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected, and the selected sound signal is subjected to windowing processing using a preset window function to obtain a sound signal for generating an acoustic image.

[0180] It is understandable that, since the frequency points to be used are discrete frequency points, after selecting the sound signal corresponding to the frequency points to be used, the selected sound signal can also be smoothed, that is, the selected sound signal is windowed using a preset window function. Among them, the sound signal corresponding to the frequency point to be used is a sound signal in the time domain. Exemplarily, the preset window function can be a Hanning window, or it can be other window functions, such as a triangular window, which is not limited in the embodiments of the present application. It is understandable that the Hanning window is also called a raised cosine window, which adds a cosine term on the basis of a rectangular window so that the signal transitions to zero more smoothly in the time domain, thereby reducing spectrum leakage and aliasing.

[0181] For example, Figure 7 As shown, it is a schematic diagram of a specific example of the sound signal processing method provided by an embodiment of the present application. The black origin in the figure represents the microphone included in the microphone array, and the microphone array includes n microphones, namely Mic_1, Mic_2, ..., Mic_n. The sound signal collected by the microphone is first subjected to feature extraction, that is, beamforming processing, and then the signal obtained after beamforming is converted to the frequency domain and input into the prediction model to predict the SNR (Signal-to-Noise Ratio) at different frequency points. Finally, post-processing is performed, that is, the sound signals at several frequency points with the highest predicted SNR are windowed using a preset window function to obtain a sound signal for generating an acoustic image.

[0182] The solution provided by the embodiment of the present application, since the prediction model obtained through training can accurately predict the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array, therefore, by using the prediction model obtained through training to predict the signal-to-noise ratio of each frequency point in the sound signal to be detected, the frequency points with high signal-to-noise ratio can be determined. Since the sound signal at the sound source position has a higher signal-to-noise ratio, the sound signal corresponding to the frequency point with a high signal-to-noise ratio is selected from the sound signal to be detected to generate an acoustic image, which can improve the accuracy of the sound source position in the obtained acoustic image. In addition, the present solution can automatically select the required frequency points using a deep learning model, and use the sound signals corresponding to the automatically selected frequency points as the sound signals for generating acoustic images, without the need for the user to manually set the frequency range of the sound signal to be imaged, thereby achieving an acoustic imaging function based on automatic frequency selection.

[0183] Optionally, in another embodiment of the present application, the above-mentioned sound signal processing method further includes:

[0184] An acoustic image is generated based on the sound signal corresponding to the frequency point to be used, and is superimposed with the visible light image collected by the camera at the center position of the microphone array to obtain a cloud map.

[0185] It can be understood that, since the frequency points to be used obtained according to the above steps S601-S605 are the preset number of frequency points with the highest signal-to-noise ratio, the sound image generated based on the sound signal corresponding to the frequency points to be used can improve the accuracy of the sound source position in the obtained acoustic image. Then, the sound image generated based on the sound signal corresponding to the frequency points to be used is superimposed on the visible light image collected by the camera at the center position of the microphone array, and the obtained cloud map has a better imaging effect, that is, the obtained cloud map can intuitively and accurately show the location of the sound source.

[0186] For example, Figure 8 As shown in FIG. , it is a schematic diagram of a specific example of generating a cloud map based on the frequency points to be used. Figure 8 As shown, the microphone array includes n microphones, namely Mic_1, Mic_2, ..., Mic_n. The microphone array collects sound signals, and automatically selects the frequencies of the collected sound signals, that is, automatically selects the frequencies according to the relevant contents of the above steps S601-S605, and obtains the sound signals at the frequency points to be used. A steering vector is generated based on the sound signals at the frequency points to be used, that is, beamforming is performed. The camera collects image information in the scene, and performs angle conversion according to the image information, so as to determine the angle of the beam to be generated based on the image information during the generation of the steering vector. Next, an energy map is generated according to the steering vector (that is, an acoustic image is generated), and image processing is performed based on the image information, that is, color display is performed through the RGB (RedGreenBlue) color model to obtain a visible light image. The generated energy map and the visible light image are then superimposed (as shown in FIG. 1 ). Figure 8 The symbol " ”), and a cloud map is obtained. It can be seen that the combination of automatic frequency selection and acoustic imaging reduces the professional requirements for personnel using the acoustic imager, and can improve the effect of acoustic imaging when the user does not understand the frequency range of the sound signal.

[0187] It can be seen that through this solution, a cloud map with better imaging effect can be obtained.

[0188] Corresponding to the above-mentioned model training method embodiment, the present application embodiment also provides a model training device, such as Fig. 9 As shown, the device comprises:

[0189] The acquisition module 910 is used to acquire multiple sample pairs, wherein each sample pair includes: a noise-free sound signal collected by the microphone array and a noisy sound signal obtained by adding noise to the noise-free sound signal;

[0190] A first transform module 920 is configured to perform a Fourier transform on the noise-free sound signal in each sample pair to obtain a first frequency domain signal, and to perform a Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal;

[0191] A first prediction module 930 is used to process the second frequency domain signal corresponding to the sample pair using the initial prediction model to obtain a first prediction vector corresponding to the sample pair; wherein the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0192] An output module 940 is used to multiply the first prediction vector corresponding to the sample pair with the first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and to multiply the first prediction vector corresponding to the sample pair with the second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair;

[0193] A calculation module 950, configured to calculate a model loss value based on a difference between the first output vector and the second output vector corresponding to each of the plurality of sample pairs;

[0194] The adjustment module 960 is used to adjust the model parameters of the prediction model based on the model loss value.

[0195] Optionally, the device further comprises:

[0196] A first direction determination module is used to determine a plurality of directions to be used before the first transformation module 920 performs Fourier transform on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and performs Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal for each sample pair; wherein each direction to be used is: the direction from the real position represented by a predetermined pixel point in the sample visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the sample visible light image is an image captured by the camera at the center position of the microphone array;

[0197] The first transformation module 920 includes:

[0198] A first beamforming submodule is used to perform beamforming processing on the noise-free sound signal in the sample pair according to the multiple directions to be used for each sample pair, so as to obtain a first beam signal corresponding to each direction to be used, and to perform beamforming processing on the noisy sound signal in the sample pair according to the multiple directions to be used, so as to obtain a second beam signal corresponding to each direction to be used;

[0199] The first Fourier transform submodule is used to perform Fourier transform on the first beam signal in the sample pair to obtain a first frequency domain signal, and to perform Fourier transform on the second beam signal in the sample pair to obtain a second frequency domain signal.

[0200] Optionally, the calculation module 950 includes:

[0201] A first calculation submodule, configured to calculate, for each sample pair, a signal-to-noise ratio loss value corresponding to the sample pair and / or a normalized loss value based on a difference between a first output vector and a second output vector corresponding to the sample pair;

[0202] The second calculation submodule is used to calculate the model loss value based on the obtained signal-to-noise ratio loss value and / or the normalized loss value.

[0203] Optionally, the signal-to-noise ratio loss value corresponding to any sample pair is calculated using a first formula; the first formula is: ;

[0204] and / or,

[0205] The normalized loss value corresponding to any sample pair is calculated using the second formula; the second formula is: ;

[0206] in, is the signal-to-noise ratio loss value corresponding to the sample pair, is the normalized loss value corresponding to the sample pair, is the first output vector corresponding to the sample pair, is the second output vector corresponding to the sample pair, represents the mean operation, represents the mean square error operation, represents the L2 norm.

[0207] Corresponding to the above-mentioned embodiment of the sound signal processing method, the embodiment of the present application also provides a sound signal processing device, such as Fig.10 As shown, the device comprises:

[0208] The receiving module 1010 is used to receive the sound signal to be detected collected by the microphone array;

[0209] The second transformation module 1020 is used to perform Fourier transformation on the sound signal to be detected to obtain a frequency domain signal to be processed;

[0210] A second prediction module 1030 is used to process the frequency domain signal to be processed using a prediction model trained by any of the above-mentioned model training methods to obtain a second prediction vector; wherein the elements in the second prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array;

[0211] A determination module 1040 is configured to use frequency points corresponding to the first preset number of elements whose element values ​​in the second prediction vector are sorted from high to low as frequency points to be used;

[0212] The selection module 1050 is used to select the sound signal corresponding to the frequency point to be used from the sound signal to be detected, and obtain the sound signal for generating the sonogram.

[0213] Optionally, the device further comprises:

[0214] A second direction determination module is used to determine a plurality of directions to be used before the second transformation module 1020 performs Fourier transform on the sound signal to be detected to obtain the frequency domain signal to be processed; wherein each direction to be used is: the direction from the real position represented by a predetermined pixel point in the visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the visible light image is an image captured by the camera at the center position of the microphone array;

[0215] The second transformation module 1020 includes:

[0216] A second beamforming submodule, configured to perform beamforming processing on the sound signal to be detected according to the multiple directions to be used, so as to obtain a third beam signal corresponding to each direction to be used;

[0217] The second Fourier transform submodule is used to perform Fourier transform on the third beam signal to obtain a frequency domain signal to be processed.

[0218] Optionally, the selection module 1050 is specifically configured to:

[0219] The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected, and a preset window function is used to perform windowing processing on the selected sound signal to obtain a sound signal for generating an acoustic image.

[0220] Optionally, the device further comprises:

[0221] The superposition module is used to generate an acoustic image based on the sound signal corresponding to the frequency point to be used, and to superimpose it with the visible light image collected by the camera at the center position of the microphone array to obtain a cloud image.

[0222] In the technical solution of this application, the operations involved in acquiring, storing, using, processing, transmitting, providing and disclosing sound signals and visible light images are all carried out with authorization.

[0223] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0224] Memory 1101, used for storing computer programs;

[0225] The processor 1102 is used to implement any of the above-mentioned model training methods or steps of the sound signal processing method when executing the program stored in the memory 1101.

[0226] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0227] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0228] The communication interface is used for communication between the above electronic device and other devices.

[0229] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0230] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0231] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods or sound signal processing methods are implemented.

[0232] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method or sound signal processing method in the above-mentioned embodiments.

[0233] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0234] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0235] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0236] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire multiple sample pairs; wherein each sample pair includes: a noise-free sound signal collected by the microphone array and a noisy sound signal obtained by adding noise to the noise-free sound signal; For each sample pair, Fourier transform is performed on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and Fourier transform is performed on the noisy sound signal in the sample pair to obtain a second frequency domain signal; Using the initial prediction model to process the second frequency domain signal corresponding to the sample pair, a first prediction vector corresponding to the sample pair is obtained; wherein the elements in the first prediction vector represent: the signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array; Multiplying the first prediction vector corresponding to the sample pair with the first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and multiplying the first prediction vector corresponding to the sample pair with the second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair; Calculating a model loss value based on differences between the first output vector and the second output vector corresponding to each of the plurality of sample pairs; Based on the model loss value, the model parameters of the prediction model are adjusted.

2. The method according to claim 1, characterized in that Before performing Fourier transform on the noise-free sound signal in the sample pair for each sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal, the method further includes: Determine a plurality of directions to be used; wherein each direction to be used is: a direction from a real position represented by a predetermined pixel point in the sample visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the sample visible light image is an image captured by a camera at the center position of the microphone array; For each sample pair, performing Fourier transform on the noise-free sound signal in the sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal, including: For each sample pair, beamforming processing is performed on the noise-free sound signal in the sample pair according to the multiple directions to be used to obtain a first beam signal corresponding to each direction to be used, and beamforming processing is performed on the noisy sound signal in the sample pair according to the multiple directions to be used to obtain a second beam signal corresponding to each direction to be used; A first beam signal in the sample pair is Fourier transformed to obtain a first frequency domain signal, and a second beam signal in the sample pair is Fourier transformed to obtain a second frequency domain signal.

3. The method according to claim 1 or 2, characterized in that: Calculating a model loss value based on differences between the first output vector and the second output vector corresponding to each of the plurality of sample pairs includes: For each sample pair, based on the difference between the first output vector and the second output vector corresponding to the sample pair, calculate the signal-to-noise ratio loss value and / or the normalized loss value corresponding to the sample pair; Based on the obtained signal-to-noise ratio loss value and / or the normalized loss value, a model loss value is calculated.

4. The method according to claim 3, characterized in that: The signal-to-noise ratio loss value corresponding to any sample pair is calculated using the first formula; the first formula is: ; and / or, The normalized loss value corresponding to any sample pair is calculated using the second formula; the second formula is: ; in, is the signal-to-noise ratio loss value corresponding to the sample pair, is the normalized loss value corresponding to the sample pair, is the first output vector corresponding to the sample pair, is the second output vector corresponding to the sample pair, represents the mean operation, represents the mean square error operation, represents the L2 norm.

5. A sound signal processing method, characterized in that: The method comprises: Receiving the sound signal to be detected collected by the microphone array; Performing Fourier transform on the sound signal to be detected to obtain a frequency domain signal to be processed; The frequency domain signal to be processed is processed using a prediction model trained by the model training method according to any one of claims 1 to 4 to obtain a second prediction vector; wherein the elements in the second prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array; The frequency points corresponding to the first preset number of elements whose element values ​​in the second prediction vector are sorted from high to low are used as the frequency points to be used; The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected to obtain the sound signal used to generate the sound image.

6. The method according to claim 5, characterized in that Before performing Fourier transform on the sound signal to be detected to obtain the frequency domain signal to be processed, the method further includes: Determine a plurality of directions to be used; wherein each direction to be used is: a direction from a real position represented by a predetermined pixel point in the visible light image to the center position of the microphone array; each predetermined pixel point is the center point of a rectangular image area obtained by dividing the visible light image according to a preset matrix grid; the number of rectangular image areas obtained by division is less than the number of microphones included in the microphone array, and the visible light image is an image captured by a camera at the center position of the microphone array; The performing Fourier transform on the sound signal to be detected to obtain a frequency domain signal to be processed includes: Performing beamforming processing on the sound signal to be detected according to the multiple directions to be used, to obtain a third beam signal corresponding to each direction to be used; Perform Fourier transform on the third beam signal to obtain a frequency domain signal to be processed.

7. The method according to claim 5 or 6, characterized in that: The step of selecting the sound signal corresponding to the frequency point to be used from the sound signal to be detected to obtain the sound signal used to generate the sound image includes: The sound signal corresponding to the frequency point to be used is selected from the sound signal to be detected, and a preset window function is used to perform windowing processing on the selected sound signal to obtain a sound signal for generating an acoustic image.

8. The method according to claim 5 or 6, characterized in that: The method further comprises: An acoustic image is generated based on the sound signal corresponding to the frequency point to be used, and is superimposed with the visible light image collected by the camera at the center position of the microphone array to obtain a cloud image.

9. A model training device, characterized in that: The device comprises: An acquisition module is used to acquire multiple sample pairs, wherein each sample pair includes: a noise-free sound signal collected by a microphone array and a noisy sound signal obtained after the noise-free sound signal is processed by adding noise; A first transform module is used for performing Fourier transform on the noise-free sound signal in each sample pair to obtain a first frequency domain signal, and performing Fourier transform on the noisy sound signal in the sample pair to obtain a second frequency domain signal; A first prediction module is used to process the second frequency domain signal corresponding to the sample pair using an initial prediction model to obtain a first prediction vector corresponding to the sample pair; wherein the elements in the first prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array; an output module, configured to multiply a first prediction vector corresponding to the sample pair with a first frequency domain signal corresponding to the sample pair to obtain a first output vector corresponding to the sample pair, and to multiply the first prediction vector corresponding to the sample pair with a second frequency domain signal corresponding to the sample pair to obtain a second output vector corresponding to the sample pair; A calculation module, configured to calculate a model loss value based on a difference between the first output vector and the second output vector corresponding to each of the plurality of sample pairs; An adjustment module is used to adjust the model parameters of the prediction model based on the model loss value.

10. A sound signal processing device, characterized in that: The device comprises: A receiving module, used for receiving the sound signal to be detected collected by the microphone array; A second transformation module is used to perform Fourier transformation on the sound signal to be detected to obtain a frequency domain signal to be processed; A second prediction module, configured to process the frequency domain signal to be processed using a prediction model trained by the model training method according to any one of claims 1 to 4 to obtain a second prediction vector; wherein the elements in the second prediction vector represent: a signal-to-noise ratio corresponding to each frequency point in the frequency response range of the microphone array; A determination module, configured to use frequency points corresponding to the first preset number of elements whose element values ​​in the second prediction vector are sorted from high to low as the frequency points to be used; The selection module is used to select the sound signal corresponding to the frequency point to be used from the sound signal to be detected, and obtain the sound signal used to generate the sonogram.

Images