A sound perception method, device, equipment and computer readable storage medium

Abstract

The application discloses a sound perception method, device, equipment and computer readable storage medium, comprising: obtaining each first mixed signal reflected after the radar transmits each first frequency modulation pulse signal; by respectively performing discrete Fourier transform on each first mixed signal, a first phase set corresponding to each first distance unit in a first phase matrix is extracted; according to the first phase set corresponding to each first distance unit, a first sound signal containing noise is extracted from each first mixed signal; the first compressed signal is obtained by compressing the first sound signal; the target sound perception model trained based on the generative adversarial network is used for sound perception of the first compressed signal. The application guarantees the processing ability of the target sound perception model to the baseline fluctuation and nonlinear mapping of the original sound perception signal, reduces the calculation complexity of the model, and improves the robustness of the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a sound perception method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Radar is a system that uses radio waves to detect targets. It is widely used in automotive, industrial inspection, and security monitoring. Millimeter-wave radar is a type of radar that has all-weather operation capabilities and is not easily affected by environmental factors such as rain and smoke. Many millimeter-wave radar signal processing methods rely primarily on visual features, making it difficult to fully utilize the rich acoustic information contained in the signal. Developing technologies based on the acoustic characteristics of millimeter-wave radar can provide a new dimension and improve the overall performance of the system.

[0003] Currently, many non-machine learning methods for sound perception have been proposed. For target recognition, traditional methods often result in false alarms or missed detections because there are other non-sound signal targets in a scene besides sound signals. In the sound perception process using millimeter-wave radar, issues such as the slow sampling rate leading to high-frequency loss of sound signals, non-linear mapping between vibration signals and speech, and robustness in complex multi-source, multipath scenarios make it difficult to achieve high-quality sound perception and reconstruction using traditional non-machine learning methods. Therefore, machine learning methods for sound processing have emerged. However, existing machine learning methods require extensive training, resulting in large datasets and high computational complexity.

[0004] In summary, how to effectively solve the problems of current sound perception methods, such as difficulty in achieving high-quality sound perception and reconstruction, excessive data volume, and high computational complexity, is an urgent issue that needs to be addressed by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a sound perception method that ensures the target sound perception model's ability to handle problems such as baseline fluctuations and nonlinear mapping in the original sound perception signal, reduces the model's computational complexity, and improves the model's robustness. Another purpose of this application is to provide a sound perception device, equipment, and computer-readable storage medium.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] A sound perception method, comprising:

[0008] Acquire the first mixed signals obtained by reflecting each first frequency modulated pulse signal after the radar transmits each first frequency modulated pulse signal;

[0009] By performing discrete Fourier transform on each of the first mixed signals, the first phase set corresponding to each first distance unit in the first phase matrix is ​​extracted.

[0010] Based on the first phase set corresponding to each first distance unit, a first sound signal containing noise is extracted from each first mixed signal;

[0011] The first sound signal is compressed and sensed to obtain a first compressed signal;

[0012] The target sound perception model trained based on generative adversarial network is used to perform sound perception on the first compressed signal.

[0013] In one specific embodiment of this application, the method further includes a training process for the target sound perception model, which includes:

[0014] The radar transmits each second frequency-modulated pulse signal and acquires the second mixed signal obtained by reflecting each second frequency-modulated pulse signal.

[0015] By performing discrete Fourier transform on each of the second mixed signals, the second phase set corresponding to each second distance unit in the second phase matrix is ​​extracted.

[0016] Based on the second phase set corresponding to each second distance unit, a second sound signal containing noise is extracted from each second mixed signal;

[0017] The second sound signal is compressed and sensed to obtain a second compressed signal;

[0018] Obtain the original signal corresponding to the second sound signal;

[0019] The target sound perception model is obtained by training an initial sound perception model based on a generative adversarial network using the second compressed signal and the original signal.

[0020] In one specific embodiment of this application, a second sound signal containing noise is extracted from each second mixed signal based on the second phase set corresponding to each second distance unit, including:

[0021] Calculate the prediction energy error corresponding to each second distance unit based on the second phase set corresponding to each second distance unit;

[0022] Calculate the prediction energy error threshold based on each prediction energy error;

[0023] The second sound signal is extracted from each second mixed signal based on the predicted energy error threshold and each predicted energy error.

[0024] In one specific embodiment of this application, a second sound signal containing noise is extracted from each second mixed signal based on the second phase set corresponding to each second distance unit, including:

[0025] The stability of the linear prediction coding coefficients corresponding to each second distance unit is calculated based on the second phase set corresponding to each second distance unit.

[0026] Calculate the stability threshold of the linear predictive coding coefficients based on the stability of each linear predictive coding coefficient;

[0027] The second sound signal is extracted from each second mixed signal based on the stability threshold of the linear predictive coding coefficients and the stability of each linear predictive coding coefficient.

[0028] In one specific embodiment of this application, a second sound signal containing noise is extracted from each second mixed signal based on the phase set corresponding to each second distance unit, including:

[0029] Calculate the prediction energy error and linear prediction coding coefficient stability for each second distance unit based on the second phase set corresponding to each second distance unit.

[0030] Calculate the prediction energy error threshold based on each prediction energy error, and calculate the linear prediction coding coefficient stability threshold based on the stability of each linear prediction coding coefficient.

[0031] The second sound signal is extracted from each second mixed signal based on the prediction energy error, the stability of each linear prediction coding coefficient, the prediction energy error threshold, and the stability threshold of the linear prediction coding coefficient.

[0032] In one specific embodiment of this application, the second audio signal is extracted from each second mixed signal based on each prediction energy error, each linear prediction coding coefficient stability, the prediction energy error threshold, and the linear prediction coding coefficient stability threshold, including:

[0033] Select target distance cells from each second distance cell whose prediction energy error is less than the prediction energy error threshold and whose linear prediction coding coefficient stability is less than the linear prediction coding coefficient stability threshold;

[0034] The signal corresponding to the target distance unit is determined as the second sound signal.

[0035] In one specific embodiment of this application, the target sound perception model is obtained by training an initial sound perception model based on a generative adversarial network using the second compressed signal and the original signal, including:

[0036] The second compressed signal is input into the generator in the initial sound perception model constructed based on the generative adversarial network to obtain the uncompressed temporal generator output signal;

[0037] The output signal of the time-domain generator is converted from the time domain to the frequency domain to obtain the output signal of the frequency-domain generator.

[0038] The original signal is transformed from the time domain to the frequency domain to obtain the original frequency domain signal;

[0039] The output signal of the frequency domain generator and the original frequency domain signal are input into the discriminator in the initial sound perception model constructed based on the generative adversarial network, so as to perform adversarial training on the generator and the discriminator to obtain the target sound perception model.

[0040] The number and size of each network layer in the generator and the discriminator are determined through sensitivity analysis.

[0041] A sound sensing device, comprising:

[0042] The first mixed signal acquisition module is used to acquire each first mixed signal reflected after the radar transmits each first frequency modulation pulse signal;

[0043] The first phase set extraction module is used to extract the first phase set corresponding to each first distance unit in the first phase matrix by performing discrete Fourier transform on each first mixed signal respectively.

[0044] The first sound signal extraction module is used to extract a first sound signal containing noise from each first mixed signal based on the first phase set corresponding to each first distance unit.

[0045] The first compressed signal acquisition module is used to perform compression sensing on the first sound signal to obtain the first compressed signal;

[0046] The sound perception module is used to perform sound perception on each first compressed signal using a target sound perception model trained based on a generative adversarial network.

[0047] A sound sensing device, comprising:

[0048] Memory, used to store computer programs;

[0049] A processor for executing the computer program to implement the steps of the sound perception method as described above.

[0050] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the sound perception method as described above.

[0051] The sound perception method provided in this application acquires each first mixed signal reflected after the radar transmits each first frequency-modulated pulse signal; extracts the first phase set corresponding to each first distance unit in the first phase matrix by performing discrete Fourier transform on each first mixed signal; extracts a first sound signal containing noise from each first mixed signal according to the first phase set corresponding to each first distance unit; performs compressed sensing on the first sound signal to obtain a first compressed signal; and uses a target sound perception model trained based on a generative adversarial network to perform sound perception on the first compressed signal.

[0052] As can be seen from the above technical solution, by performing Discrete Fourier Transform on each of the acquired first mixed signals, the sparsity of each signal is ensured. Based on the first phase set corresponding to each extracted first distance unit, a first sound signal containing noise is extracted from each of the first mixed signals. By performing compressed sensing on the first sound signal, the dimensionality of each signal is significantly reduced, thus reducing the computational complexity of the model. The sound perception method based on a combination of compressed sensing and generative adversarial networks not only ensures the target sound perception model's ability to handle issues such as baseline fluctuations and nonlinear mappings in the original sound perception signal, but also reduces the model's computational complexity and improves its robustness.

[0053] Accordingly, this application also provides a sound sensing device, equipment, and computer-readable storage medium corresponding to the above-mentioned sound sensing method, which have the above-mentioned technical effects, and will not be repeated here. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating one implementation of the sound perception method in this application.

[0056] Figure 2 This is a flowchart illustrating another implementation of the sound perception method in this application.

[0057] Figure 3 This is a structural block diagram of a sound sensing device according to an embodiment of this application;

[0058] Figure 4 This is a structural block diagram of a sound sensing device according to an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of the specific structure of a device provided in an embodiment of this application. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] See Figure 1 , Figure 1 This is a flowchart illustrating one implementation of the sound perception method in this application. The method may include the following steps:

[0062] S101: Acquire the first mixed signals obtained by reflecting each first frequency modulation pulse signal emitted by the radar.

[0063] The radar (such as millimeter radar) transmits first frequency modulated pulse signals (chirp), which are reflected back to the radar receiver after being reflected by multiple objects in the scene. The radar receives the first mixed signals reflected after transmitting each first frequency modulated pulse signal, and obtains the first mixed signals reflected after transmitting each first frequency modulated pulse signal.

[0064] Each first mixed signal may include a time-domain reflection signal of the first frequency-modulated pulse signal reflected back by sound, and may also include a time-domain reflection signal of the first frequency-modulated pulse signal reflected back by an object.

[0065] S102: By performing discrete Fourier transform on each of the first mixed signals, the first phase set corresponding to each first distance unit in the first phase matrix is ​​extracted.

[0066] After acquiring the first mixed signals reflected from the radar's first frequency-modulated pulse signal, the first phase set corresponding to each first range unit in the first phase matrix is ​​extracted by performing discrete Fourier transform on each of the first mixed signals.

[0067] S103: Extract the first sound signal containing noise from each first mixed signal according to the first phase set corresponding to each first distance unit.

[0068] After extracting the first phase set corresponding to each first distance unit in the first phase matrix, a first sound signal containing noise is extracted from each first mixed signal based on the first phase set corresponding to each first distance unit. For example, the prediction energy error and / or linear prediction coding coefficient stability corresponding to each first distance unit can be calculated based on the first phase set corresponding to each first distance unit, and the first sound signal containing noise can be extracted by referring to one or both of the two indicators corresponding to each first distance unit.

[0069] S104: Compress and sense the first sound signal to obtain the first compressed signal.

[0070] After extracting the first sound signal containing noise from each of the first mixed signals, the first sound signal is subjected to compressed sensing to obtain the first compressed signal. The high-dimensional first sound signals are converted into low-dimensional first compressed signals through compressed sensing.

[0071] S105: Use the target sound perception model trained based on generative adversarial network to perform sound perception on each of the first compressed signals.

[0072] Based on a target sound perception model pre-trained using a generative adversarial network (GAN), a compressed sound signal is obtained by compressing and sensing the first sound signal. Then, the GAN-based model is used to perform sound perception on the compressed signal, thereby extracting the noise-free sound signal. This enables sound source identification and high-quality sound perception under complex conditions.

[0073] As can be seen from the above technical solution, by performing Discrete Fourier Transform on each of the acquired first mixed signals, the sparsity of each signal is ensured. Based on the first phase set corresponding to each extracted first distance unit, a first sound signal containing noise is extracted from each of the first mixed signals. By performing compressed sensing on the first sound signal, the dimensionality of each signal is significantly reduced, thus reducing the computational complexity of the model. The sound perception method based on a combination of compressed sensing and generative adversarial networks not only ensures the target sound perception model's ability to handle issues such as baseline fluctuations and nonlinear mappings in the original sound perception signal, but also reduces the model's computational complexity and improves its robustness.

[0074] It should be noted that, based on the above embodiments, this application also provides corresponding improvement solutions. In subsequent embodiments, steps that are the same as or corresponding to those in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other. These improvements will not be elaborated upon in the following improved embodiments.

[0075] See Figure 2 , Figure 2 This is a flowchart illustrating another implementation of the sound perception method in this application. The method may include the following steps:

[0076] S201: Transmit each second frequency modulation pulse signal through radar, and acquire the second mixed signal obtained by reflecting each second frequency modulation pulse signal.

[0077] When collecting sound signals, the radar transmits each second frequency modulation pulse signal and acquires the second mixed signal obtained by reflecting each second frequency modulation pulse signal.

[0078] It should be noted that the first and second frequency modulation pulse signals are not distinguished by magnitude or order; they are merely used to differentiate between the frequency modulation pulse signals transmitted by the radar during the model application phase and the frequency modulation pulse signals transmitted by the radar during the model training phase.

[0079] It should also be noted that the first and second mixed signals do not have any difference in magnitude or order; they are only used to distinguish between the mixed signals reflected during the model application phase and the mixed signals reflected during the model training phase.

[0080] Each second mixed signal may include the reflected signal of the first frequency-modulated pulse signal reflected back by sound, and may also include the reflected signal of the first frequency-modulated pulse signal reflected back by an object.

[0081] S202: By performing discrete Fourier transform on each of the second mixed signals, the second phase set corresponding to each second distance unit in the second phase matrix is ​​extracted.

[0082] After acquiring the second mixed signals obtained by reflecting each second frequency-modulated pulse signal, the second phase set corresponding to each second distance unit in the second phase matrix is ​​extracted by performing a Discrete Fourier Transform on each second mixed signal. Alternatively, the second mixed signals can be converted into discrete signals using a low-pass filter and analog-to-digital conversion, and then the distance to all objects can be identified and phase information extracted using a Fast Fourier Transform (FFT).

[0083] The phase matrix can be represented by the following formula:

[0084] ;

[0085] Where N represents N frequency-modulated pulse signals, The pulse repetition period is represented by M, which represents the total number of sampling points for one frequency-modulated pulse signal. This represents the phase of the mixed signal measured at the time of transmission of the nth frequency-modulated pulse signal in the qth distance unit.

[0086] It should be noted that the "first" and "second" in the first phase matrix and the second phase matrix do not have any difference in size or order; they are only used to distinguish between the phase matrix extracted during the model application stage and the phase matrix extracted during the model training stage.

[0087] It should also be noted that the first and second distance units in the first and second distance units do not have any difference in size or order. They are only used to distinguish the distance units in the phase matrix extracted during the model application stage from the distance units in the phase matrix extracted during the model training stage.

[0088] It should also be noted that the first and second phase sets do not have any difference in size or order; they are only used to distinguish between the phase sets corresponding to the distance units in the phase matrix extracted during the model application phase and the phase sets corresponding to the distance units in the phase matrix extracted during the model training phase.

[0089] S203: Extract the second sound signal containing noise from each second mixed signal according to the second phase set corresponding to each second distance unit.

[0090] After extracting the second phase set corresponding to each second distance unit in the second phase matrix, the second sound signal containing noise is extracted from each second mixed signal based on the second phase set corresponding to each second distance unit. For example, the features of each second mixed signal can be obtained through linear predictive coding (lpc), and the second sound signal can be extracted from each second mixed signal by performing feature analysis on each second mixed signal.

[0091] In one specific embodiment of this application, step S203 may include the following steps:

[0092] Step 1: Calculate the prediction energy error corresponding to each second distance unit based on the second phase set corresponding to each second distance unit;

[0093] Step 2: Calculate the prediction energy error threshold based on each prediction energy error;

[0094] Step 3: Extract the second sound signal from each mixed signal based on the predicted energy error threshold and each predicted energy error.

[0095] For ease of description, the three steps above can be combined for explanation.

[0096] After extracting the phase set corresponding to each second distance unit in the second phase matrix, the prediction energy error corresponding to each second distance unit is calculated based on the second phase set. A prediction energy error threshold is then calculated based on each prediction energy error, such as by calculating the first mean and first standard deviation of each prediction energy error. The prediction energy error threshold is then calculated based on the prediction energy error threshold and each prediction energy error. Finally, the second sound signal is extracted from each second mixed signal based on the prediction energy error threshold and each prediction energy error. Compared to noise signals, sound signals have relatively low prediction error energy. By extracting the second sound signal from each second mixed signal based on the prediction energy error threshold and each prediction energy error, rapid and accurate extraction of the sound signal is achieved.

[0097] The formula for calculating the prediction error energy of the q-th distance cell is as follows:

[0098] ;

[0099] in, Y is the prediction error energy of the q-th distance cell, Y is the total number of frames, and Z is the frame length. This represents the z-th LPC coefficient in the w-th frame. L is the LPC order, which is a set value. This represents the phase of the mixed signal measured at the nth chirp emission time in the qth distance cell. This indicates the pulse repetition period. M is the total number of sampling points in a chirp. .

[0100] In one specific embodiment of this application, step S203 may include the following steps:

[0101] Step 1: Calculate the stability of the linear prediction coding coefficients corresponding to each second distance unit based on the second phase set corresponding to each second distance unit;

[0102] Step 2: Calculate the stability threshold of the linear predictive coding coefficients based on the stability of each linear predictive coding coefficient;

[0103] Step 3: Extract the second sound signal from each second mixed signal based on the stability threshold of the linear predictive coding coefficients and the stability of each linear predictive coding coefficient.

[0104] For ease of description, the three steps above can be combined for explanation.

[0105] After extracting the second phase set corresponding to each second distance unit in the second phase matrix, the stability of the linear predictive coding coefficients (LRDs) for each second distance unit is calculated based on the second phase set. A LRD stability threshold is then calculated based on the stability of each LRD. Finally, the second sound signal is extracted from each second mixed signal based on the LRD stability threshold and the stability of each LRD. Compared to periodic vibration signals, sound signals have lower LRD stability. By extracting the sound signal from each second mixed signal based on the LRD stability threshold and the stability of each LRD, rapid and accurate sound signal extraction is achieved.

[0106] The formula for calculating the stability of the linear prediction coding coefficients of the q-th distance unit is as follows:

[0107] ;

[0108] in, For the stability of the linear prediction coding coefficients of the q-th distance unit, It is a very small number.

[0109] In one specific embodiment of this application, step S203 may include the following steps:

[0110] Step 1: Calculate the prediction energy error and linear prediction coding coefficient stability for each second distance unit based on the phase set corresponding to each second distance unit;

[0111] Step 2: Calculate the prediction energy error threshold based on each prediction energy error, and calculate the linear prediction coding coefficient stability threshold based on the stability of each linear prediction coding coefficient;

[0112] Step 3: Extract the second sound signal from each second mixed signal based on the prediction energy error, the stability of each linear prediction coding coefficient, the prediction energy error threshold, and the stability threshold of the linear prediction coding coefficient.

[0113] For ease of description, the three steps above can be combined for explanation.

[0114] The prediction energy error and linear prediction coding coefficient stability for each second distance unit are calculated based on the second phase set corresponding to each second distance unit. A prediction energy error threshold is calculated based on each prediction energy error, and a linear prediction coding coefficient stability threshold is calculated based on each linear prediction coding coefficient stability. The second audio signal is then extracted from each second mixed signal based on these thresholds. The accuracy and filtering efficiency of audio signal extraction are further improved by calculating the prediction energy error threshold and the linear prediction coding coefficient stability threshold.

[0115] In one specific embodiment of this application, extracting a second audio signal from each second mixed signal based on each prediction energy error, each linear prediction coding coefficient stability, a prediction energy error threshold, and a linear prediction coding coefficient stability threshold may include the following steps:

[0116] Step 1: Select target distance cells from each second distance cell whose prediction energy error is less than the prediction energy error threshold and whose linear prediction coding coefficient stability is less than the linear prediction coding coefficient stability threshold;

[0117] Step 2: Determine the signal corresponding to the target distance unit as the second sound signal.

[0118] For ease of description, the two steps above can be combined for explanation.

[0119] After calculating the prediction energy error threshold and the linear prediction coding coefficient stability threshold, target distance units with prediction energy errors less than the prediction energy error threshold and linear prediction coding coefficient stability less than the linear prediction coding coefficient stability threshold are selected from each second distance unit. The signal corresponding to the target distance unit is determined as the second sound signal, which further improves the accuracy and selection efficiency of sound signal extraction.

[0120] Each distance cell has a prediction error energy and linear prediction coding coefficient stability. The mean values ​​of the prediction error energy and linear prediction coding coefficient stability are calculated for all distance cells. and standard deviation Then set a constant. Then the lower threshold can be obtained. and upper threshold :

[0121] ;

[0122] ;

[0123] in, It can be set to 5.

[0124] Signal rejection can also be performed using upper limits for prediction energy error and linear prediction coding coefficient stability. For example, if the prediction energy error of a range cell is lower than the lower limit of the prediction energy error threshold, but its linear prediction coding coefficient stability is higher than the upper limit of the linear prediction coding coefficient stability threshold, then the signal corresponding to that range cell is determined to be a non-audio signal and rejected. Similarly, if the linear prediction coding coefficient stability of a range cell is lower than the lower limit of the linear prediction coding coefficient stability threshold, but its prediction energy error is higher than the upper limit of the prediction energy error threshold, then the signal corresponding to that range cell is also determined to be a non-audio signal and rejected.

[0125] S204: Compress and sense the second sound signal to obtain the second compressed signal.

[0126] After extracting the second sound signal from each of the second mixed signals, the second sound signal is subjected to compressed sensing to obtain the second compressed signal. Compressed sensing converts the high-dimensional sound signal into a low-dimensional second compressed signal.

[0127] Compressed sensing projects high-dimensional signals onto low-dimensional ones using an observation matrix. If the signal is sparse, it can be reconstructed from a number of sampling points far below the requirements of the sampling theorem. Signal sparsity can occur in two ways: either the signal itself is sparse, or the signal is transformed into another domain through a transform domain, where it may already be sparse. Common transform domains include the Fourier transform domain and the wavelet transform domain.

[0128] Such as signals The expression is as follows:

[0129] ;

[0130] in, , It is a sparse base. yes In another domain representation, It is sparse. It is a measurement matrix. It is a perception matrix. These are observed values.

[0131] S205: Obtain the original signal corresponding to the second sound signal.

[0132] After compressing and sensing the second sound signal to obtain the compressed signal, the original signals corresponding to the second sound signal are then acquired. The original signal refers to the noise-free signal emitted by the sound source of the second sound signal.

[0133] S206: The initial sound perception model constructed based on the generative adversarial network is trained using the second compressed signal and the original signal to obtain the target sound perception model.

[0134] After compressing and sensing the second sound signal to obtain the compressed signal, and acquiring the corresponding original signals, the initial sound perception model based on a generative adversarial network is trained using the compressed signal and the original signals to obtain the target sound perception model. Through generative adversarial training, a target sound perception model is obtained that can quickly extract noise-free sound signals from noisy sound signals, improving the model training speed and achieving rapid and accurate sound signal extraction.

[0135] Generative Adversarial Networks (GANs) are widely used in fields such as image processing. A standard GAN consists of a generator... and discriminator The generator is composed of two parts. The generator takes random Gaussian white noise as input and aims to generate fake data that resembles real data. The discriminator takes either real data or fake data generated by the generator as input and aims to distinguish between the two. The generator and discriminator are trained adversarially to ensure that the data generated by the generator network is as realistic as possible. Through continuous training, an optimal model is eventually reached.

[0136] The generative adversarial network (GAN) architecture includes convolutional layers, transposed convolutional layers, skip connections, and nonlinear activation functions. Sensitivity analysis was performed on the choice of nonlinear activation function. Because the Rectified Linear Unit (ReLU) can result in a negative input and a zero output, the corresponding weights cannot be updated, causing the neuron to "die." Therefore, the Parametric Rectified Linear Unit (PReLU) was used. PReLU introduces learnable parameters, allowing the neural network to automatically adjust the slope of the negative half-axis, eliminating the need for manual slope setting like Leaky ReLU and avoiding the "death" problem of ReLU neurons. The formula for PReLU is shown below:

[0137] ;

[0138] in, For PReLU input values, It is a learnable parameter.

[0139] The generator in a Generative Adversarial Network (GAN) can be configured with five convolutional layers (conv_1, conv_2, conv_3, conv_4, and conv_5) and five transposed convolutional layers (tconv_1, tconv_2, tconv_3, tconv_4, and tconv_5). It uses ten non-linear activation functions, five skip layers, and one fully connected layer. The stride can be set to 1 or 2 depending on the requirements, and the number of convolutional kernels can be chosen between 16 and 256. In the convolutional layers, downsampling is performed, reducing the signal length after each convolution and increasing the number of channels, essentially transforming it into a matrix. In the transposed convolutional layers, upsampling is performed, increasing the signal length after the transposed convolution and decreasing the number of channels. The kernel size is set to 5*1. The GAN generator first performs convolution processing, adding a non-linear activation function after each convolution, and the transposed convolution is processed similarly. This is because in deep networks, information may be gradually lost during layer-by-layer propagation, and gradients may gradually decay during backpropagation. Therefore, after the convolutional layers conv_1, conv_2, conv_3, and conv_4, the transposed convolutional layers tconv_1, tconv_2, tconv_3, and tconv_4 are connected via skip connections.

[0140] The discriminator consists of five convolutional layers and corresponding non-linear activation functions. It also includes a reformatting process to transform the matrix into a long vector, and finally a fully connected layer to output a single value.

[0141] Traditional GAN ​​structures suffer from training instability and gradient vanishing problems. The emergence of Wasserstein GAN (WGAN) addresses this issue. WGAN uses the Wasserstein distance to measure the difference in probability distributions between real and generated samples. By replacing Jensen-Shannon divergence with Wasserstein distance, WGAN is less prone to gradient vanishing and mode collapse, resulting in more stable training. (Assuming real samples...) Conforms to probability distribution ,noise Conforms to probability distribution Based on the Wasserstein distance, the objective function of WGAN can be written as:

[0142] ;

[0143] in, Expressing expectations, It consists of 1-Lipschitz functions. In WGAN, the discriminator is forced to satisfy the Lipschitz constraint by weight pruning. Although weight pruning is easy to implement, it makes optimization difficult and can lead to an ill-conditioned surface even if the optimization is successful.

[0144] To address the issue of WGAN sometimes failing to converge and to achieve higher quality generation, a gradient penalty term, called WGAN-GP, is added, as follows:

[0145] ;

[0146] in, It is a hyperparameter. It is an interpolated sample. It follows a uniform distribution from 0 to 1. yes gradient term, Real variables and generated variables The convex combination of WGAN-GP and GAN-GP achieves Lipschitz constraints more gently through gradient penalty, avoiding the drawbacks of weight pruning. Compared with GAN and WGAN, WGAN-GP is more stable and produces higher-quality samples. Therefore, the embodiments of this application can train the model based on the initial sound perception model constructed by WGAN, thereby improving the stability of the model, improving the quality of samples, and thus improving the accuracy of the trained target sound perception model in perceiving sound signals.

[0147] In one specific embodiment of this application, step S205 may include the following steps:

[0148] Step 1: Input the second compressed signal into the generator in the initial sound perception model built on generative adversarial network to obtain the uncompressed temporal generator output signal;

[0149] Step 2: Convert the output signal of the time-domain generator from the time domain to the frequency domain to obtain the output signal of the frequency-domain generator;

[0150] Step 3: Convert the original signal from the time domain to the frequency domain to obtain the original signal in the frequency domain;

[0151] Step 4: Input the output signal of the frequency domain generator and the original frequency domain signal into the discriminator in the initial sound perception model built on the generative adversarial network, so as to perform adversarial training on the generator and the discriminator to obtain the target sound perception model.

[0152] The number and size of each network layer in the generator and discriminator were determined through sensitivity analysis.

[0153] For ease of description, the four steps above can be combined for explanation.

[0154] Sensitivity analysis was used to pre-determine the number and size of each layer in the generator and discriminator of the initial sound perception model built based on a generative adversarial network. First, the initial number of layers and size were set. Then, sensitivity analysis was performed on each layer to observe its effect. Excessive sensitivity could lead to poor denoising, while insufficient sensitivity could result in loss of sound signal detail. Finally, sensitivity analysis was used to confirm the final model, making it more suitable for the current scenario.

[0155] The second compressed signal is input into the generator of the initial sound perception model built on a generative adversarial network (GAN) to obtain the uncompressed time-domain generator output signal. This time-domain generator output signal is then converted to the frequency domain to obtain the frequency-domain generator output signal. The original signal is then converted back to the frequency domain to obtain the original frequency-domain signal. The frequency-domain generator output signal and the original frequency-domain signal are then input into the discriminator of the initial sound perception model built on the GAN, thus enabling adversarial training of the generator and discriminator to obtain the target sound perception model. By pre-determining the number and size of each network layer through sensitivity analysis, good denoising performance is ensured while reducing the probability of audio signal loss. By converting both the time-domain generator output signal and the original signal from the time domain to the frequency domain, and using the compressed signal as the generator input and the discriminator input as the frequency-domain signal, the sparsity of the signal is guaranteed due to its stronger sparsity and lack of alignment requirements. This reduces the complexity of model training and further improves the training speed.

[0156] S207: Acquire the first mixed signals obtained by reflecting each first frequency modulation pulse signal emitted by the radar.

[0157] S208: By performing discrete Fourier transform on each of the first mixed signals, the first phase set corresponding to each first distance unit in the first phase matrix is ​​extracted.

[0158] S209: Extract the first sound signal containing noise from each first mixed signal according to the first phase set corresponding to each first distance unit.

[0159] S210: Compress and sense the first sound signal to obtain the first compressed signal.

[0160] S211: Use the target sound perception model trained based on generative adversarial network to perform sound perception on each first compressed signal.

[0161] Corresponding to the above method embodiments, this application also provides a sound sensing device, and the sound sensing device described below can be referred to in correspondence with the sound sensing method described above.

[0162] See Figure 3 , Figure 3 This is a structural block diagram of a sound sensing device according to an embodiment of this application. The device may include:

[0163] The first mixed signal acquisition module 31 is used to acquire each first mixed signal obtained by reflection after the radar transmits each first frequency modulation pulse signal;

[0164] The first phase set extraction module 32 is used to extract the first phase set corresponding to each first distance unit in the first phase matrix by performing discrete Fourier transform on each first mixed signal respectively.

[0165] The first sound signal extraction module 33 is used to extract a first sound signal containing noise from each first mixed signal according to the first phase set corresponding to each first distance unit;

[0166] The first compressed signal acquisition module 34 is used to perform compression sensing on the first sound signal to obtain the first compressed signal;

[0167] The sound perception module 35 is used to perform sound perception on each first compressed signal using a target sound perception model trained based on a generative adversarial network.

[0168] As can be seen from the above technical solution, by performing Discrete Fourier Transform on each of the acquired first mixed signals, the sparsity of each signal is ensured. Based on the first phase set corresponding to each extracted first distance unit, a first sound signal containing noise is extracted from each of the first mixed signals. By performing compressed sensing on the first sound signal, the dimensionality of each signal is significantly reduced, thus reducing the computational complexity of the model. The sound perception method based on a combination of compressed sensing and generative adversarial networks not only ensures the target sound perception model's ability to handle issues such as baseline fluctuations and nonlinear mappings in the original sound perception signal, but also reduces the model's computational complexity and improves its robustness.

[0169] In one specific embodiment of this application, the device may further include a model training module, which may include:

[0170] The second mixed signal acquisition submodule is used to transmit each second frequency modulation pulse signal through the radar and acquire the second mixed signal obtained by reflecting each second frequency modulation pulse signal.

[0171] The second phase set extraction submodule is used to extract the second phase set corresponding to each second distance unit in the second phase matrix by performing discrete Fourier transform on each second mixed signal respectively.

[0172] The second sound signal extraction submodule is used to extract a second sound signal containing noise from each second mixed signal based on the second phase set corresponding to each second distance unit.

[0173] The second compressed signal acquisition submodule is used to perform compression sensing on the second sound signal to obtain the second compressed signal;

[0174] The original signal acquisition submodule is used to acquire the original signal corresponding to the second sound signal;

[0175] The target sound perception model acquisition submodule is used to train the initial sound perception model based on the generative adversarial network using the second compressed signal and the original signal to obtain the target sound perception model.

[0176] In one specific embodiment of this application, the sound signal extraction submodule may include:

[0177] The prediction energy error calculation unit is used to calculate the prediction energy error corresponding to each second distance unit based on the second phase set corresponding to each second distance unit.

[0178] The prediction energy error threshold calculation unit is used to calculate the prediction energy error threshold based on each prediction energy error.

[0179] The first audio signal filtering unit is used to extract the second audio signal from each second mixed signal based on the prediction energy error threshold and each prediction energy error.

[0180] In one specific embodiment of this application, the sound signal extraction submodule may include:

[0181] The linear prediction coding coefficient stability calculation unit is used to calculate the stability of the linear prediction coding coefficients corresponding to each second distance unit based on the second phase set corresponding to each second distance unit.

[0182] The linear predictive coding coefficient stability threshold calculation unit is used to calculate the stability threshold of the linear predictive coding coefficient based on the stability of each linear predictive coding coefficient.

[0183] The second audio signal filtering unit is used to extract the second audio signal from each second mixed signal based on the stability threshold of the linear predictive coding coefficients and the stability of each linear predictive coding coefficient.

[0184] In one specific embodiment of this application, the sound signal extraction submodule may include:

[0185] The energy error and stability calculation unit is used to calculate the prediction energy error and linear prediction coding coefficient stability of each second distance unit based on the second phase set corresponding to each second distance unit.

[0186] The threshold calculation unit is used to calculate the prediction energy error threshold based on each prediction energy error, and to calculate the linear prediction coding coefficient stability threshold based on the stability of each linear prediction coding coefficient.

[0187] The third audio signal extraction unit is used to extract the second audio signal from each second mixed signal based on each prediction energy error, each linear prediction coding coefficient stability, the prediction energy error threshold, and the linear prediction coding coefficient stability threshold.

[0188] In one specific embodiment of this application, the third audio signal extraction unit may include:

[0189] The distance cell filtering sub-unit is used to filter target distance cells from each second distance cell whose prediction energy error is less than the prediction energy error threshold and whose linear prediction coding coefficient stability is less than the linear prediction coding coefficient stability threshold.

[0190] The sound signal determination subunit is used to determine the signal corresponding to the target distance unit as the second sound signal.

[0191] In one specific embodiment of this application, the target sound perception model acquisition submodule may include:

[0192] The temporal generator output signal acquisition unit is used to input the second compressed signal into the generator in the initial sound perception model constructed based on the generative adversarial network to obtain the uncompressed temporal generator output signal.

[0193] The frequency domain generator output signal acquisition unit is used to convert the time domain generator output signal from the time domain to the frequency domain to obtain the frequency domain generator output signal.

[0194] The frequency domain original signal acquisition unit is used to convert the original signal from the time domain to the frequency domain to obtain the frequency domain original signal;

[0195] The target sound perception model acquisition unit is used to input the output signal of the frequency domain generator and the original frequency domain signal into the discriminator in the initial sound perception model constructed based on the generative adversarial network, so as to perform adversarial training on the generator and the discriminator to obtain the target sound perception model.

[0196] The number and size of each network layer in the generator and discriminator were determined through sensitivity analysis.

[0197] For the method embodiments described above, see [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the sound sensing device provided in this application, which may include:

[0198] Memory 332 is used to store computer programs;

[0199] The processor 322 is used to implement the steps of the sound perception method in the above method embodiments when executing a computer program.

[0200] For details, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the specific structure of a sound sensing device provided in this embodiment. The sound sensing device can vary significantly depending on its configuration or performance. It may include a processor (central processing unit, CPU) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the sound sensing device 301.

[0201] The sound sensing device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0202] The steps in the sound perception method described above can be implemented by the structure of the sound perception device.

[0203] Corresponding to the above method embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0204] The system acquires the first mixed signals reflected after each first frequency modulated pulse signal is emitted by the radar; it extracts the first phase set corresponding to each first range unit in the first phase matrix by performing discrete Fourier transform on each first mixed signal; it extracts the first sound signal containing noise from each first mixed signal based on the first phase set corresponding to each first range unit; it compresses and senses the first sound signal to obtain the first compressed signal; and it uses the target sound perception model trained based on generative adversarial network to perform sound perception on the first compressed signal.

[0205] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0207] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0208] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A sound perception method, characterized in that, include: Acquire the first mixed signals obtained by reflecting each first frequency modulated pulse signal after the radar transmits each first frequency modulated pulse signal; By performing discrete Fourier transform on each of the first mixed signals, the first phase set corresponding to each first distance unit in the first phase matrix is ​​extracted. Based on the first phase set corresponding to each first distance unit, a first sound signal containing noise is extracted from each first mixed signal; The first sound signal is compressed and sensed to obtain a first compressed signal; The target sound perception model trained based on generative adversarial network is used to perform sound perception on the first compressed signal. It also includes the training process of the target sound perception model, which includes: The radar transmits each second frequency-modulated pulse signal and acquires the second mixed signal obtained by reflecting each second frequency-modulated pulse signal. By performing discrete Fourier transform on each of the second mixed signals, the second phase set corresponding to each second distance unit in the second phase matrix is ​​extracted. Based on the second phase set corresponding to each second distance unit, a second sound signal containing noise is extracted from each second mixed signal; The second sound signal is compressed and sensed to obtain a second compressed signal; Obtain the original signal corresponding to the second sound signal; The target sound perception model is obtained by training an initial sound perception model based on a generative adversarial network using the second compressed signal and the original signal. Based on the phase set corresponding to each second distance unit, a second sound signal containing noise is extracted from each second mixed signal, including: Calculate the prediction energy error and linear prediction coding coefficient stability for each second distance unit based on the second phase set corresponding to each second distance unit. Calculate the prediction energy error threshold based on each prediction energy error, and calculate the linear prediction coding coefficient stability threshold based on the stability of each linear prediction coding coefficient. The second sound signal is extracted from each second mixed signal based on the prediction energy error, the stability of each linear prediction coding coefficient, the prediction energy error threshold, and the stability threshold of the linear prediction coding coefficient.

2. The sound perception method according to claim 1, characterized in that, Based on the second phase set corresponding to each second distance unit, a second sound signal containing noise is extracted from each second mixed signal, including: Calculate the prediction energy error corresponding to each second distance unit based on the second phase set corresponding to each second distance unit; Calculate the prediction energy error threshold based on each prediction energy error; The second sound signal is extracted from each second mixed signal based on the predicted energy error threshold and each predicted energy error.

3. The sound perception method according to claim 1, characterized in that, Based on the second phase set corresponding to each second distance unit, a second sound signal containing noise is extracted from each second mixed signal, including: The stability of the linear prediction coding coefficients corresponding to each second distance unit is calculated based on the second phase set corresponding to each second distance unit. Calculate the stability threshold of the linear predictive coding coefficients based on the stability of each linear predictive coding coefficient; The second sound signal is extracted from each second mixed signal based on the stability threshold of the linear predictive coding coefficients and the stability of each linear predictive coding coefficient.

4. The sound perception method according to claim 1, characterized in that, Based on each prediction energy error, each linear prediction coding coefficient stability, the prediction energy error threshold, and the linear prediction coding coefficient stability threshold, the second audio signal is extracted from each second mixed signal, including: Select target distance cells from each second distance cell whose prediction energy error is less than the prediction energy error threshold and whose linear prediction coding coefficient stability is less than the linear prediction coding coefficient stability threshold; The signal corresponding to the target distance unit is determined as the second sound signal.

5. The sound perception method according to claim 1, characterized in that, The target sound perception model is obtained by training an initial sound perception model based on a generative adversarial network using the second compressed signal and the original signal, including: The second compressed signal is input into the generator in the initial sound perception model constructed based on the generative adversarial network to obtain the uncompressed temporal generator output signal; The output signal of the time-domain generator is converted from the time domain to the frequency domain to obtain the output signal of the frequency-domain generator. The original signal is converted from the time domain to the frequency domain to obtain the original frequency domain signal; The output signal of the frequency domain generator and the original frequency domain signal are input into the discriminator in the initial sound perception model constructed based on the generative adversarial network, so as to perform adversarial training on the generator and the discriminator to obtain the target sound perception model. The number and size of each network layer in the generator and the discriminator are determined through sensitivity analysis.

6. A sound sensing device, characterized in that, include: The first mixed signal acquisition module is used to acquire each first mixed signal reflected after the radar transmits each first frequency modulation pulse signal; The first phase set extraction module is used to extract the first phase set corresponding to each first distance unit in the first phase matrix by performing discrete Fourier transform on each first mixed signal respectively. The first sound signal extraction module is used to extract a first sound signal containing noise from each first mixed signal based on the first phase set corresponding to each first distance unit. The first compressed signal acquisition module is used to perform compression sensing on the first sound signal to obtain the first compressed signal; The sound perception module is used to perform sound perception on each first compressed signal using a target sound perception model trained based on a generative adversarial network. It also includes a model training module, which includes: The second mixed signal acquisition submodule is used to transmit each second frequency modulation pulse signal through the radar and acquire the second mixed signal obtained by reflecting each second frequency modulation pulse signal. The second phase set extraction submodule is used to extract the second phase set corresponding to each second distance unit in the second phase matrix by performing discrete Fourier transform on each second mixed signal respectively. The second sound signal extraction submodule is used to extract a second sound signal containing noise from each second mixed signal based on the second phase set corresponding to each second distance unit. The second compressed signal acquisition submodule is used to perform compression sensing on the second sound signal to obtain the second compressed signal; The original signal acquisition submodule is used to acquire the original signal corresponding to the second sound signal; The target sound perception model acquisition submodule is used to train the initial sound perception model based on the generative adversarial network using the second compressed signal and the original signal to obtain the target sound perception model. The audio signal extraction submodule includes: The energy error and stability calculation unit is used to calculate the prediction energy error and linear prediction coding coefficient stability of each second distance unit based on the second phase set corresponding to each second distance unit. The threshold calculation unit is used to calculate the prediction energy error threshold based on each prediction energy error, and to calculate the linear prediction coding coefficient stability threshold based on the stability of each linear prediction coding coefficient. The third audio signal extraction unit is used to extract the second audio signal from each second mixed signal based on each prediction energy error, each linear prediction coding coefficient stability, the prediction energy error threshold, and the linear prediction coding coefficient stability threshold.

7. A sound sensing device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the sound perception method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the sound perception method as described in any one of claims 1 to 5.

Images