Signal processing method and device and computer readable storage medium

Abstract

The invention discloses a signal processing method and device and a computer readable storage medium, and relates to the technical field of signal processing, and the method comprises the steps: obtaining an original signal to be subjected to signal processing, and the signal type of the original signal comprises a bone conduction signal; extracting signal features of the original signal; the signal features are input into a pre-trained frequency band expansion model to obtain a target bandwidth signal, the frequency band expansion model comprises a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network which are connected in sequence, the low-frequency feature analysis network is used for analyzing the fusion signal features, and the high-frequency signal prediction network is used for predicting the fusion signal features; the high-frequency signal prediction network is used for predicting high-frequency signals, and the fusion network is used for fusing the high-frequency signals and original signals. According to the method, the definition of the bone conduction signal is improved, so that the processed bone conduction signal has better hearing feeling.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a signal processing method, device and computer-readable storage medium. Background Art

[0002] Bone conduction signals (hereinafter referred to as bone conduction signals) refer to signals transmitted using the vibration characteristics of fixed media (such as bones and skulls). Bone conduction signals can usually be picked up by sensors, such as bone conduction (Voice Pick Up, VPU) sensors, which pick up bone conduction signals by converting the vibration of solid media into electrical signals. Therefore, bone conduction signals have the inherent ability to suppress environmental noise, which is a favorable physical property for voice enhancement under low signal-to-noise ratio conditions. In recent years, bone conduction technology has been widely used in voice communication, assistive listening devices, virtual reality, military applications, underground mining and other fields.

[0003] However, due to the special characteristics of bone conduction signals, the bone conduction signals usually picked up still have the following disadvantages: compared with the clear air conduction signals (referred to as air conduction signals), the picked up bone conduction signals have significant information loss in the high frequency range, and their energy distribution is limited to below 2kHz. This may greatly reduce the clarity of the signal and make the bone conduction signal sound dull.

[0004] Therefore, how to improve the clarity of bone conduction signals to improve the hearing experience of bone conduction signals is a technical problem that needs to be solved urgently in the field of this technology. Summary of the invention

[0005] The main purpose of the present application is to provide a signal processing method, device and computer-readable storage medium, aiming to solve the technical problem of how to improve the clarity of bone conduction signals to improve the hearing experience of bone conduction signals.

[0006] To achieve the above object, the present application provides a signal processing method, the signal processing method comprising:

[0007] Acquire an original signal to be processed, wherein the signal type of the original signal includes a bone conduction signal;

[0008] Extracting signal features of the original signal, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, the low-frequency amplitude vector represents the signal amplitude feature of the original signal, and the low-frequency phase value vector represents the phase value feature of the original signal;

[0009] The signal features are input into a pre-trained frequency band extension model to obtain a target bandwidth signal, wherein the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, the low-frequency feature analysis network is used to analyze and fuse the signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal.

[0010] In one embodiment, the step of inputting the signal feature into a pre-trained frequency band extension model to obtain a target bandwidth signal includes:

[0011] The signal features are analyzed and fused by the low-frequency feature analysis network to obtain fused feature data, and a high-frequency signal is predicted based on the fused feature data by the high-frequency signal prediction network, wherein the high-frequency signal represents a signal within a preset high-frequency range;

[0012] The high-frequency signal and the original signal are fused through the fusion network to obtain a target broadband signal.

[0013] In one embodiment, the high-frequency signal prediction network includes a high-frequency amplitude calculation subnetwork and a high-frequency phase value calculation subnetwork, and the step of predicting the high-frequency signal based on the fused feature data through the high-frequency signal prediction network includes:

[0014] Predicting a high-frequency amplitude vector based on the fused feature data by the high-frequency amplitude calculation subnetwork, wherein the high-frequency amplitude vector includes at least one signal amplitude;

[0015] Predicting a high-frequency phase value vector based on the fused feature data by the high-frequency phase value calculation subnetwork, wherein the high-frequency phase value vector includes at least one phase value, and the phase value corresponds to the signal amplitude one by one;

[0016] The high-frequency amplitude value vector and the high-frequency phase value vector are determined to be high-frequency signals.

[0017] In one embodiment, the fusion network includes an amplitude filter subnetwork and a phase filter subnetwork, and the step of fusing the high-frequency signal and the original signal through the fusion network to obtain a target broadband signal includes:

[0018] The high-frequency amplitude vector in the high-frequency signal is fused with the low-frequency amplitude vector through the amplitude filtering subnetwork to obtain a target broadband amplitude vector;

[0019] The high-frequency phase value vector in the high-frequency signal is fused with the low-frequency phase value vector through the phase value filtering subnetwork to obtain a target broadband phase value vector;

[0020] The target wideband amplitude vector and the target wideband phase value vector are determined as a target wideband signal.

[0021] In one embodiment, after the step of obtaining the original signal to be processed, the method further includes:

[0022] Performing voice activity detection on the original signal to obtain a voice activity detection result;

[0023] If the voice activity detection result indicates that there is voice activity, the step of extracting the signal feature of the original signal is continued.

[0024] In one embodiment, the step of extracting the signal feature of the original signal includes:

[0025] If the original signal is a time series signal, Fourier transform is performed on the original signal to obtain a target frequency domain signal;

[0026] The signal amplitude of the target signal frequency point in the target frequency domain signal is calculated to obtain a low-frequency amplitude vector, and the phase value of the target signal frequency point in the target frequency domain signal is calculated to obtain a low-frequency phase value vector, wherein the target signal frequency point includes the signal frequency point in the target frequency domain signal within a preset low-frequency range.

[0027] In one embodiment, the step of analyzing the fused signal features through the low-frequency feature analysis network to obtain fused feature data includes:

[0028] Acquire an associated frequency domain signal associated with the target frequency domain signal, and acquire an associated low-frequency amplitude value vector and an associated low-frequency phase value vector corresponding to the associated frequency domain signal, wherein the associated frequency domain signal includes a preset number of frame frequency domain signals before the target frequency domain signal;

[0029] Combining the associated low-frequency amplitude vector with the low-frequency amplitude vector to obtain an amplitude matrix, and combining the associated low-frequency phase value vector with the low-frequency phase value vector to obtain a phase value matrix;

[0030] The amplitude matrix and the phase value matrix are input into the low-frequency feature analysis network to obtain fused feature data.

[0031] In one embodiment, the low-frequency feature analysis network includes a first depth-wise separable convolutional network and a second depth-wise separable convolutional network connected in sequence, and the high-frequency signal prediction network includes a convolutional layer, a gated recurrent unit, and an activation layer connected in sequence.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a signal processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the signal processing method as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a program for implementing the signal processing method is stored on the computer-readable storage medium. The program for implementing the signal processing method is executed by a processor to implement the steps of the signal processing method as described above.

[0034] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the signal processing method as described above are implemented.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] The original signal to be processed is obtained, wherein the signal type of the original signal includes a bone conduction signal; the signal features of the original signal are extracted, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, wherein the low-frequency amplitude vector represents the signal amplitude features of the original signal, and the low-frequency phase value vector represents the phase value features of the original signal; the signal features are input into a pre-trained frequency band extension model to obtain a target bandwidth signal, wherein the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, wherein the low-frequency feature analysis network is used to analyze the fusion signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal. In this way, the embodiment of the present application designs a new frequency band extension model. Specifically, the frequency band extension model is composed of a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, and the frequency band extension model for extending the signal frequency band is pre-trained. For the original signal that needs to be frequency band extended, the high-frequency signal is predicted based on its phase value feature and the signal amplitude feature, so that the frequency band extension is completed through the high-frequency signal prediction network. After the original signal and the high-frequency signal are fused through the fusion network, the extended high-frequency signal is fused into the original signal, thereby realizing the frequency band extension of the original signal, so that the narrow-band original signal is extended to a high-frequency range signal to obtain a target broadband signal, and the signal type of the original signal includes a bone conduction signal. Therefore, after the bone conduction signal is processed by the signal processing method proposed in the embodiment of the present application, the frequency band of the bone conduction signal can be expanded, thereby improving the speech clarity of the bone conduction signal, so that it has a better sense of hearing. In addition, the embodiment of the present application predicts the high-frequency signal based on the phase value feature and the signal amplitude feature, and utilizes the mutual influence information between the two features, that is, the correlation, so that the predicted high-frequency signal is closer to the real signal, further improving the speech clarity of the processed signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 This is a flow chart of the first embodiment of the signal processing method of the present application;

[0040] Figure 2This is a schematic diagram of a low-frequency feature analysis network structure involved in an embodiment of a signal processing method of the present application;

[0041] Figure 3 This is a schematic diagram of a prediction network structure involved in an embodiment of a signal processing method of the present application;

[0042] Figure 4 A brief schematic diagram of a signal processing flow involved in an embodiment of a signal processing method of the present application;

[0043] Figure 5 This is a schematic diagram of the data processing flow of the frequency band extension model in the signal processing method of this application;

[0044] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the signal processing device in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0047] Air conduction signals (hereinafter referred to as air conduction signals) are signals conducted through the air. Both air conduction signals and bone conduction signals can be picked up by sensors. For example, taking a microphone as a sensor for picking up signals, a microphone that picks up air conduction signals is referred to as an air conduction microphone, and a microphone that picks up bone conduction signals is referred to as a bone conduction microphone.

[0048] Usually, the signal picked up by the air conduction microphone has a complete frequency band, but has almost no noise resistance, and speech recognition difficulties caused by ambient noise are also very common. The bone conduction microphone uses the wearer's skull to transmit sound, which will strengthen the energy of the wearer's voice and weaken the energy of non-wearers and surrounding sounds. This has great advantages over air conduction microphones, but the data collected by bone conduction microphones is mainly concentrated in the low-frequency area of ​​the frequency domain, and the high-frequency area information will be lost, making the user's voice sound dull, unlike the data collected by air conduction microphones, which are brighter and more personalized in hearing. In addition, bone conduction microphones can ensure the privacy of wearable product users, that is, when users wear the product and speak quietly, they can also capture the speaker's voice more accurately to avoid social embarrassment. The bone conduction microphone at the nose pad of AR glasses can isolate most of the noise, but the high-frequency area information of the user's voice needs to be restored to be closer to human hearing, which also provides convenience for subsequent speech transcription, speech recognition and other applications.

[0049] Based on this, the main solution of the present application is: obtaining the original signal to be processed, the signal type of the original signal includes a bone conduction signal; extracting the signal features of the original signal, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, the low-frequency amplitude vector represents the signal amplitude features of the original signal, and the low-frequency phase value vector represents the phase value features of the original signal; inputting the signal features into a pre-trained frequency band extension model to obtain a target bandwidth signal, wherein the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, the low-frequency feature analysis network is used to analyze the fusion signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal.

[0050] The present application has designed a new frequency band extension model. Specifically, the frequency band extension model is composed of a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, and the frequency band extension model for extending the signal frequency band is pre-trained. For the original signal that needs to be frequency band extended, the high-frequency signal is predicted based on its phase value feature and signal amplitude feature, so that the frequency band extension is completed through the high-frequency signal prediction network. The original signal and the high-frequency signal are fused through the fusion network, and the extended high-frequency signal is fused into the original signal, thereby realizing the frequency band extension of the original signal, so that the narrow-band original signal is extended to a high-frequency range signal to obtain a target broadband signal, and the signal type of the original signal includes a bone conduction signal. Therefore, after the bone conduction signal is processed based on the signal processing method proposed in the embodiment of the present application, the frequency band of the bone conduction signal can be extended, thereby improving the speech clarity of the bone conduction signal, so that it has a better sense of hearing. In addition, the embodiment of the present application predicts the high-frequency signal based on the phase value feature and the signal amplitude feature, and uses the mutual influence information between the two features, that is, the correlation, so that the predicted high-frequency signal is closer to the real signal, and the speech clarity of the processed signal is further improved. .

[0051] It should be noted that the execution subject of each embodiment of the signal processing method of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a signal processing device capable of realizing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, etc. Exemplarily, the signal processing method of the present application is described and illustrated using AR glasses as the execution subject.

[0052] Based on this, the present application proposes a signal processing method of the first embodiment, please refer to Figure 1 , the signal processing method comprises steps S10 to S30:

[0053] Step S10, obtaining an original signal to be processed, wherein the signal type of the original signal includes a bone conduction signal;

[0054] It should be noted that the original signal may be a signal to be band-expanded, and further, may be a signal collected by the AR glasses, or may be a signal collected by other devices and sent to the AR glasses. In a specific embodiment, a bone conduction microphone is provided at the nose pad of the AR glasses, and the original signal is a signal collected by the bone conduction microphone.

[0055] Further, the original signal may be a frame signal, that is, the signal is subjected to frequency band expansion processing with the frame as an atom to improve the precision of the frequency band expansion. Exemplarily, the original signal is taken as a frame signal as an example to illustrate and describe various embodiments of the present application.

[0056] Furthermore, considering that the AR glasses may not be able to process the collected or received original signals immediately, a first buffer area can be set. After the AR glasses collect or receive the original signals, the collected or received original signals are stored in the first buffer area, and then one frame of the original signal is obtained from the first buffer area in sequence for processing.

[0057] Furthermore, the first buffer area can be a circular queue, using an array of a fixed size to store data, and the head and tail of the queue are connected to form a ring. The characteristic of this queue is that when the tail of the queue reaches the end of the array, it will loop back to the beginning of the array. After the AR glasses collect or receive the original signal, the signal is stored at the end of the queue, and the original signal is obtained from the head of the queue for processing.

[0058] It should be noted that the signal type of the original signal includes but is not limited to bone conduction signal.

[0059] Step S20, extracting signal features of the original signal, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, the low-frequency amplitude vector represents the signal amplitude feature of the original signal, and the low-frequency phase value vector represents the phase value feature of the original signal;

[0060] It should be noted that the signal picked up by the sensor is usually a time domain signal. Based on this, when the original signal is a time domain signal, the original signal in the time domain can be converted to the frequency domain to extract the signal features of the low-frequency amplitude vector and the low-frequency phase value vector based on the original signal in the frequency domain. Specifically, the original signal in the time domain can be converted to the frequency domain by Fourier transform.

[0061] The low-frequency amplitude vector represents the signal amplitude characteristics of the original signal, and the low-frequency phase value vector represents the phase value characteristics of the original signal. Specifically, the low-frequency amplitude vector may include the signal amplitudes of some or all signal frequency points in the original signal in the frequency domain, and the low-frequency phase value vector may include the phase values ​​of some or all signal frequency points in the original signal in the frequency domain.

[0062] Furthermore, the signal amplitude in the low-frequency amplitude vector corresponds to the phase value in the low-frequency phase value vector one-to-one, and the signal amplitude and phase values ​​with a corresponding relationship are the signal amplitude and phase values ​​of the same signal frequency point. Exemplarily, assuming that the signal amplitude and phase values ​​of N signal frequency points in the original signal in the frequency domain are extracted, the signal amplitudes of the N signal frequency points are respectively [N_F1, N_F2...N_Fi...N_Fn], and the phase values ​​of the N signal frequency points are respectively [N_f1, N_f2...N_fi...N_fn], then the low-frequency amplitude vector is [N_F1, N_F2...N_Fi...N_Fn], the low-frequency phase value vector is [N_f1, N_f2...N_fi...N_fn], and N_Fi corresponds to N_fi.

[0063] It can be understood that after Fourier transform of N sampling points, the Fourier transform results of N signal points can be obtained. Each sampling point corresponds to a signal frequency point after Fourier transform, and the value after Fourier transform corresponding to each signal frequency point (recorded as FT value), and the signal amplitude is the modulus of the FT value corresponding to the signal frequency point. The phase value refers to the angle (angle) of the complex representation of the signal at the signal frequency point, which represents the phase offset of the signal at the frequency component. For example, assuming that the complex form signal corresponding to a certain signal frequency point is represented by x=a+bi, where a is the real part of the signal, b is the imaginary part of the signal, and i is the imaginary unit, then the signal amplitude of the signal frequency point is The phase value is arctan2(b,a).

[0064] Step S30, inputting the signal features into a pre-trained frequency band extension model to obtain a target bandwidth signal, wherein the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, the low-frequency feature analysis network is used to analyze and fuse the signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal.

[0065] The signal features are input into a pre-trained frequency band extension model to obtain a target bandwidth signal, which is the output of the frequency band extension model.

[0066] Optionally, the step of inputting the signal feature into a pre-trained frequency band extension model to obtain a target bandwidth signal comprises:

[0067] Step S301, analyzing and fusing the signal features through the low-frequency feature analysis network to obtain fused feature data;

[0068] The fused feature data may specifically be the output of a low-frequency feature analysis network.

[0069] The low-frequency feature analysis network is used to analyze the fusion signal features. Specifically, the low-frequency feature analysis network may include one or more convolutional layers connected in sequence. Figure 2 As shown, the low-frequency feature analysis network includes a first deep separable convolutional network and a second deep separable convolutional network connected in sequence. The deep separable convolutional network consists of a deep convolutional layer and a point-by-point convolutional layer, which independently calculates, fuses, independently calculates, and fuses the signal amplitude and phase values. This structure can reduce the number of parameters and computational complexity of the model, and at the same time can extract the temporal characteristics between different phonemes of the signal to improve the effectiveness of the extracted fusion feature data, thereby improving the temporal continuity of the subsequent predicted high-frequency signal.

[0070] Step S302, predicting a high-frequency signal based on the fused feature data through the high-frequency signal prediction network, wherein the high-frequency signal represents a signal within a preset high-frequency range;

[0071] The high-frequency signal may specifically be the output of a high-frequency signal prediction network.

[0072] The high-frequency signal prediction network is used to predict high-frequency signals. In a specific implementation, refer to Figure 3 As shown, the high-frequency signal prediction network includes a convolutional layer, a gated recurrent unit, and an activation layer connected in sequence.

[0073] In the convolution layer, considering that many hardware acceleration libraries and frameworks have been optimized for 3*3 convolution kernels, efficient algorithms such as Fast Fourier Transform (FFT) have been implemented for 3x3 convolution kernels, which can significantly improve computing efficiency. Therefore, 3*3 convolution kernels are preferably used in the convolution layer to facilitate calling the relevant library functions of the chip platform hardware computing acceleration to improve computing efficiency.

[0074] The Gated Recurrent Unit (GRU) is a variant of the Recurrent Neural Network (RNN). It can effectively capture the long-distance dependencies of time series when processing sequence data, solving the gradient vanishing or exploding problem encountered by traditional RNN when dealing with long-distance dependencies. Based on this, GRU is set in the high-frequency signal prediction network to extract the association between the phonemes of the signal in the time dimension.

[0075] In the activation layer, a nonlinear activation function, such as the hyperbolic tangent tanh function, can be used to introduce nonlinear characteristics to the high-frequency signal prediction network, so that the network can learn and simulate more complex functions.

[0076] The high frequency signal represents a signal within a preset high frequency range. Specifically, the preset high frequency range may be a range set in advance by relevant personnel according to actual needs, such as 2 kHz to 8 kHz. This embodiment does not specifically limit this.

[0077] Step S303: fusing the high-frequency signal with the original signal through the fusion network to obtain a target broadband signal.

[0078] The target broadband signal may be specifically the output of a fusion network. The fusion network is used to fuse the high-frequency signal with the original signal. In a specific implementation, the fusion network includes a fully connected layer.

[0079] It should be noted that the fusion network fuses the high-frequency signal with the original signal. Specifically, since the high-frequency signal is a signal in the frequency domain, the fusion network fuses the high-frequency signal with the original signal in the frequency domain to obtain the target broadband signal.

[0080] Furthermore, the total number of signal frequencies in the target broadband signal is consistent with the total number of signal frequencies in the original signal in the frequency domain, so as to ensure that the sampling rate of the signal after the generated target broadband signal is converted into the time domain is consistent with the sampling rate of the original signal, that is, the consistency of the signal before processing and the signal after processing is guaranteed, that is, the consistency of the signal before frequency band expansion and the signal after frequency band expansion is guaranteed.

[0081] In addition, after obtaining the target broadband signal, the target broadband signal can be subjected to inverse Fourier transform processing to convert the target broadband signal in the frequency domain into the time domain. Further, the target broadband signal in the time domain can be output. For example, in a call scenario, the target broadband signal in the time domain can be output through a speaker so that the AR glasses can receive the voice of the other end.

[0082] This embodiment obtains an original signal to be processed, wherein the signal type of the original signal includes a bone conduction signal; extracts signal features of the original signal, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, wherein the low-frequency amplitude vector represents the signal amplitude feature of the original signal, and the low-frequency phase value vector represents the phase value feature of the original signal; inputs the signal features into a pre-trained band extension model to obtain a target bandwidth signal, wherein the band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, wherein the low-frequency feature analysis network is used to analyze the fusion signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal. In this way, the present embodiment designs a new frequency band extension model. Specifically, the frequency band extension model is composed of a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, and the frequency band extension model for extending the signal frequency band is pre-trained. For the original signal that needs to be frequency band extended, the high-frequency signal is predicted based on its phase value feature and the signal amplitude feature, so that the frequency band extension is completed through the high-frequency signal prediction network. After the original signal and the high-frequency signal are fused through the fusion network, the extended high-frequency signal is fused into the original signal, thereby realizing the frequency band extension of the original signal, so that the narrow-band original signal is extended to a high-frequency range signal to obtain a target broadband signal, and the signal type of the original signal includes a bone conduction signal. Therefore, after the bone conduction signal is processed based on the signal processing method proposed in the present embodiment, the frequency band of the bone conduction signal can be expanded, thereby improving the speech clarity of the bone conduction signal, so that it has a better sense of hearing. In addition, the embodiment of the present application predicts the high-frequency signal based on the phase value feature and the signal amplitude feature, and utilizes the mutual influence information between the two features, that is, the correlation, so that the predicted high-frequency signal is closer to the real signal, further improving the speech clarity of the processed signal.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the high-frequency signal prediction network includes a high-frequency amplitude calculation subnetwork and a high-frequency phase value calculation subnetwork, and the step of predicting the high-frequency signal based on the fused feature data through the high-frequency signal prediction network includes:

[0084] Step A10, predicting a high-frequency amplitude vector based on the fused feature data through the high-frequency amplitude calculation subnetwork, wherein the high-frequency amplitude vector includes at least one signal amplitude;

[0085] The high-frequency amplitude vector may specifically be the output of the high-frequency amplitude calculation subnetwork.

[0086] Step A20, predicting a high-frequency phase value vector based on the fused feature data through the high-frequency phase value calculation subnetwork, wherein the high-frequency phase value vector includes at least one phase value, and the phase value corresponds to the signal amplitude one by one;

[0087] The high-frequency phase value vector may specifically be the output of the high-frequency phase value calculation subnetwork.

[0088] It should be noted that the high-frequency amplitude calculation subnetwork and the high-frequency phase value calculation subnetwork have the same network structure. In a specific implementation, the high-frequency amplitude calculation subnetwork and the high-frequency phase value calculation subnetwork both include sequentially connected convolutional layers, GRU layers, and activation layers.

[0089] The signal amplitude in the high-frequency amplitude vector corresponds to the phase value in the high-frequency phase value vector one by one, and the corresponding signal amplitude and phase value represent an extended signal frequency point.

[0090] Step A30, determining that the high-frequency amplitude value vector and the high-frequency phase value vector are high-frequency signals.

[0091] In this embodiment, a high-frequency amplitude calculation subnetwork and a high-frequency phase value calculation subnetwork are set to predict the high-frequency amplitude vector and the high-frequency phase value vector respectively, which is equivalent to independently predicting the high-frequency amplitude vector and the high-frequency phase value vector based on the fused feature data. As a result, the relevant characteristics of the fused feature data can be better utilized to predict the corresponding feature vectors, improve the model performance, and it is easier to understand and explain the behavior of each calculation subnetwork to enhance the interpretability and customizability of the model. The high-frequency amplitude calculation subnetwork and the high-frequency phase value subnetwork can also be tuned separately, which helps to stabilize the training process.

[0092] In a possible implementation, the fusion network includes an amplitude filter subnetwork and a phase value filter subnetwork, and the step of fusing the high-frequency signal with the original signal through the fusion network to obtain a target broadband signal includes:

[0093] Step B10, fusing the high-frequency amplitude vector in the high-frequency signal with the low-frequency amplitude vector through the amplitude filtering subnetwork to obtain a target broadband amplitude vector;

[0094] Step B20, fusing the high-frequency phase value vector in the high-frequency signal with the low-frequency phase value vector through the phase value filtering subnetwork to obtain a target broadband phase value vector;

[0095] It should be noted that, considering that the fully connected layer can perform weighted summation of all activated features of the previous layer, thereby integrating feature information from the previous layer, in the regression task, the fully connected layer can directly output continuous values ​​for prediction. Based on this, both the amplitude filter subnetwork and the phase value filter subnetwork can be a fully connected layer.

[0096] The target broadband amplitude vector may specifically be the output of the amplitude filter subnetwork. Similarly, the target broadband phase value vector may specifically be the output of the phase value filter subnetwork.

[0097] Furthermore, after obtaining the high-frequency amplitude vector and the high-frequency phase value vector, the high-frequency amplitude vector can be connected to the low-frequency amplitude vector, that is, the above two amplitude vectors are connected into one vector. Specifically, the high-frequency amplitude vector is placed after the low-frequency amplitude vector to combine into a broadband amplitude vector; similarly, the high-frequency phase value vector is connected to the low-frequency phase value vector, that is, the two phase value vectors are connected into one vector: the high-frequency phase value vector is placed after the low-frequency phase value vector to combine into a broadband phase value vector, the broadband amplitude vector is input into the amplitude filtering subnetwork output to obtain the target broadband amplitude vector, and the broadband phase value vector is input into the phase value filtering subnetwork to obtain the target broadband phase value vector.

[0098] Step B30, determining the target broadband amplitude vector and the target broadband phase value vector as a target broadband signal.

[0099] In this embodiment, the amplitude filtering subnetwork and the phase value filtering subnetwork are set to fuse the amplitude vector and the phase value vector respectively, thereby making it easier to understand and explain the behavior of each filtering subnetwork to enhance the interpretability and customizability of the model, and the amplitude filtering subnetwork and the phase value filtering subnetwork can be tuned separately, which helps to stabilize the training process.

[0100] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above description, and will not be described in detail later. On this basis, after the step of obtaining the original signal to be processed, the method further includes:

[0101] Step C10, performing voice activity detection on the original signal to obtain a voice activity detection result;

[0102] There are many ways to detect voice activity, which are not limited here. For example, in one embodiment, the average energy of the original signal can be calculated; if the average energy is greater than or equal to the set threshold, it is determined that the original signal has voice activity, and if the average energy is less than the set threshold, the original signal is smoothed; the average energy and average zero-crossing rate of the smoothed voice signal are calculated, and if the average energy after smoothing is greater than or equal to the set threshold, and the average zero-crossing rate T is within a preset range, such as 30≤T≤70, it is determined that the original signal has voice activity, and if the average energy after smoothing is less than the set threshold, or the average zero-crossing rate T is not within the preset range, it is determined that the original signal does not have voice activity.

[0103] Step C20: If the voice activity detection result indicates that there is voice activity, continue to execute the step of extracting the signal features of the original signal.

[0104] If the voice activity detection result indicates that there is no voice activity, no processing may be performed.

[0105] It should be noted that if a first buffer area for caching the original signal to be processed is set in the AR glasses, the original signal can be obtained from the first buffer area, and step C10 is executed to perform voice activity detection on the original signal.

[0106] As another implementation, after the AR glasses collect or receive the original signal, voice activity detection may be performed on the original signal, and when the voice activity detection result indicates the presence of voice activity, the original signal may be stored in the first buffer area. After the AR glasses subsequently obtain a frame of the original signal from the first buffer area, there is no need to perform voice activity detection again. After extracting the signal features of the obtained original signal, the signal features may be input into the pre-trained frequency band extension model.

[0107] In this embodiment, before the original signal is band expanded, voice activity detection is first performed on the signal. If the voice activity detection result indicates that there is voice activity in the original signal, that is, there is human voice, the original signal is band expanded. In this way, only the original signal with voice activity is band expanded, thereby achieving effective band expansion. When there is no voice activity in the original signal, the band expansion model does not need to be run, thereby reducing the running time of the band expansion model, thereby reducing the power consumption of the AR glasses and improving the battery life of the AR glasses.

[0108] In a possible implementation, the step of extracting the signal feature of the original signal includes:

[0109] Step D10, if the original signal is a time series signal, performing Fourier transform on the original signal to obtain a target frequency domain signal;

[0110] Step D20, calculate the signal amplitude of the target signal frequency point in the target frequency domain signal to obtain a low-frequency amplitude vector, and calculate the phase value of the target signal frequency point in the target frequency domain signal to obtain a low-frequency phase value vector, wherein the target signal frequency point includes the signal frequency point in the target frequency domain signal within a preset low-frequency range.

[0111] The preset low frequency range may be a frequency range set in advance according to actual needs, such as 0 Hz to 2 kHz, and this embodiment does not impose any specific restrictions on this. Further, the upper limit of the preset low frequency range is less than or equal to the lower limit of the preset high frequency range.

[0112] In this embodiment, the signal amplitude of the target signal frequency point within the preset low-frequency range is calculated to obtain a low-frequency amplitude vector, and the phase value of the target signal frequency point within the preset low-frequency range is calculated to obtain a low-frequency phase value vector. In this way, the number of signal amplitudes and phase values ​​that need to be calculated can be reduced, and the amount of input data of the subsequent model can be reduced, thereby reducing the amount of calculation of the model to improve the efficiency of frequency band expansion.

[0113] In a possible implementation, the step of analyzing the fused signal features through the low-frequency feature analysis network to obtain fused feature data includes:

[0114] Step E10, obtaining an associated frequency domain signal associated with the target frequency domain signal, and obtaining an associated low-frequency amplitude value vector and an associated low-frequency phase value vector corresponding to the associated frequency domain signal, wherein the associated frequency domain signal includes a preset number of frame frequency domain signals before the target frequency domain signal;

[0115] It should be noted that the preset number is a number set in advance according to actual needs, and this embodiment does not impose specific restrictions on this. If the number of signal frames before the target frequency domain signal is less than the preset number of frames, zero padding processing can be performed before the first frame of the target frequency domain signal until the number of frames of the target frequency domain signal is padded to the preset number of frames.

[0116] It should be noted that the preset number of frequency domain signals before the target frequency domain signal may specifically mean that the timing of the associated frequency domain signal is before the timing of the target frequency domain signal. It can be understood that the timing may specifically refer to the timing in the time domain after the frequency domain signal is converted into the time domain.

[0117] Furthermore, a second cache area can be set in the AR glasses. Each time the signal features of the original signal are extracted, the extracted signal features can be stored in the second cache area, so that the low-frequency amplitude vector and the low-frequency phase value vector of the associated frequency domain signal, that is, the associated low-frequency amplitude vector and the associated low-frequency phase value vector, can be obtained from the second cache area.

[0118] Furthermore, the second buffer area may also be a ring queue, which is not specifically limited in this embodiment.

[0119] Step E20, combining the associated low-frequency amplitude vector and the low-frequency amplitude vector to obtain an amplitude matrix, and combining the associated low-frequency phase value vector and the low-frequency phase value vector to obtain a phase value matrix;

[0120] The associated low-frequency amplitude vector and the low-frequency amplitude vector are combined to obtain an amplitude matrix, that is, the amplitude matrix includes all elements in the associated amplitude vector and the low-frequency amplitude vector. In a specific embodiment, each row (column) of the amplitude matrix is ​​an associated low-frequency amplitude vector or a low-frequency amplitude vector. For example, assuming that the associated low-frequency amplitude vectors are [a1, a2...an], [b1, b2...bn], and the low-frequency amplitude vector is [c1, c2...cn], then the amplitude matrix can be

[0121] Similarly, the associated low-frequency phase value vector and the low-frequency phase value vector are combined to obtain a phase value matrix, that is, the phase value matrix includes all elements in the associated phase value vector and the low-frequency phase value vector. In a specific embodiment, each row (for example) in the phase value matrix is ​​an associated low-frequency phase value vector or a low-frequency phase value vector.

[0122] Step E30, inputting the amplitude matrix and the phase value matrix into the low-frequency feature analysis network to obtain fused feature data.

[0123] In this embodiment, the associated low-frequency amplitude vector and the low-frequency amplitude vector are combined to obtain an amplitude matrix, and the associated low-frequency phase value vector and the low-frequency phase value vector are combined to obtain a phase value matrix, and the amplitude matrix and the phase value matrix are input into the low-frequency feature analysis network to obtain fused feature data. In this way, the high-frequency signal is predicted based on the signal characteristics corresponding to the target frequency domain signal and the associated frequency domain signal before the current frame, so that the high-frequency signal can be predicted based on the time series relationship between the target frequency domain signal and the associated frequency domain signal, so that the predicted high-frequency signal has good time series continuity.

[0124] For example, in order to help understand the technical concept or technical principle of the signal processing method after the present embodiment is combined with the first embodiment and the second embodiment, a specific embodiment is now listed, with reference to Figure 4 As shown, in this specific embodiment, the signal processing flow is:

[0125] Step 1: Use the VAD (Voice Activity Detection) algorithm to detect whether there is human voice in each frame of bone conduction microphone data (i.e., original signal). If there is human voice, update the current bone conduction microphone data into the cache (i.e., the first cache area), and reduce the current frame index value CI by one; at the same time, read the bone conduction microphone data corresponding to the CI value from the cache, perform short-time Fourier transform (STFT) calculation on the read bone conduction microphone data, output the complex number of each frequency point, and then calculate the signal amplitude and phase value of each frequency point from the complex number of each frequency point, combine the signal amplitude value of each frequency point to obtain a low-frequency amplitude vector, and combine the phase value of each frequency point to obtain a low-frequency phase value vector.

[0126] Step 2, input the low-frequency phase value vector and the low-frequency amplitude vector into the pre-trained bone conduction high-frequency extension generation model (i.e., the frequency band extension model). The bone conduction high-frequency extension generation model has an input cache, which actually takes the amplitude matrix and phase value matrix corresponding to the past N-1 frame associated frequency domain signals (N represents the preset number, N≥1) and the target frequency domain signal of the current frame as input. The high-frequency amplitude vector and high-frequency phase value vector after bandwidth extension can be calculated through the bone conduction high-frequency extension generation model; the high-frequency amplitude vector and high-frequency phase value vector of each frequency point (or frequency band) output by the bone conduction high-frequency extension generation model are used to calculate the complex value at each frequency point (or frequency band), and then the data in the time domain of the current frame after frequency band extension can be obtained through inverse Fourier transform (inverse STFT) calculation. At this time, the current frame index value CI is increased by one.

[0127] Step 3: If CI is greater than the threshold, that is, CI is greater than the maximum number of frames that can be stored in the cache, return to the human voice judgment part to wait for the arrival of the original signal, otherwise continue to process the next frame of bone conduction microphone data in the cache.

[0128] Among them, the data processing flow of the bone conduction high-frequency expansion generation model is as follows Figure 5 As shown in the figure, the low-frequency feature analysis network jointly calculates the amplitude and phase value of the bone conduction microphone data in the frequency domain, specifically: the amplitude matrix and the phase value matrix are spliced ​​in the channel dimension, and a two-layer deep separable convolutional network is used to complete the independent calculation, fusion calculation, re-independent calculation, and re-fusion calculation of the amplitude and phase values ​​to output the fusion feature data; the low-frequency feature analysis network integrates the amplitude information and phase value information of the bone conduction microphone data in the frequency domain, and fully considers the mutual influence between the amplitude and phase values ​​in the frequency domain.

[0129] After calculation by the low-frequency feature analysis network, they are respectively input into the high-frequency amplitude calculation subnetwork and the high-frequency phase value calculation subnetwork. The high-frequency amplitude calculation subnetwork and the high-frequency phase value calculation subnetwork are both composed of convolutional layers, GRU layers and tanh activation layers. The output is the high-frequency amplitude vector of the current frame in the preset high-frequency range and the high-frequency phase value vector of the current frame in the preset high-frequency range, that is, the output high-frequency signal.

[0130] The amplitude vector output by the high-frequency amplitude calculation subnetwork (the high-frequency amplitude vector of the current frame) is connected to the amplitude vector at the end of the input cache of the bone conduction high-frequency extension generation model (the low-frequency amplitude vector of the current frame), that is, the above two amplitude vectors are connected into one vector, and the high-frequency amplitude vector is placed after the low-frequency amplitude vector to form a broadband amplitude vector; similarly, the high-frequency phase value vector output by the high-frequency phase value calculation subnetwork is connected to the low-frequency phase value vector at the end of the input cache of the bone conduction high-frequency extension generation model, that is, the two phase value vectors are connected into one vector: the high-frequency phase value vector is placed after the low-frequency phase value vector to form a broadband phase value vector.

[0131] The broadband amplitude vector is input into the amplitude filtering subnetwork, which is a fully connected layer, and then the expanded amplitude vector, i.e., the target broadband amplitude vector, is output; the broadband phase value vector is input into the phase value filtering subnetwork, which is a fully connected layer, and then the expanded phase value vector, i.e., the target broadband phase value vector, is output, thereby completing the prediction of the bone conduction high-frequency extension generation model.

[0132] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the signal processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0133] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those of the above-mentioned first embodiment, second embodiment and third embodiment can be referred to the above introduction, and will not be repeated in the following. On this basis, the method further includes:

[0134] Step S100, obtaining a pre-built frequency band extension model and obtaining preset training data, wherein the training data includes narrow-band air conduction speech data and bone conduction noise data;

[0135] The frequency band extension model may be a model for frequency band extension constructed in advance by relevant personnel. As an example, the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network. The low-frequency feature analysis network is used to analyze and fuse features, the high-frequency signal prediction network is used to predict high-frequency signals, and the fusion network is used to fuse the high-frequency signal with the original signal to be band expanded.

[0136] The training data may be data prepared in advance by relevant personnel for training the frequency band extension model. The training data includes narrowband air conduction speech data and bone conduction noise data, wherein the narrowband air conduction speech data is data containing only valid signals, and the bone conduction noise data is data containing only noise signals.

[0137] Due to the conduction characteristics of bone conduction signals, there are few sources of noise, so the types of bone conduction noise data that need to be collected are relatively few. This reduces the cost of collecting bone conduction noise data to a certain extent, that is, the cost of preparing training data, and also reduces the application scenarios that the frequency band extension model needs to generalize.

[0138] Furthermore, the training data also includes broadband air conduction speech data, and the broadband air conduction speech data is used as a training label for the frequency band extension model, that is, the expected output of the model.

[0139] Step S200, mixing the narrowband air conduction speech data and the bone conduction noise data with at least one preset signal-to-noise ratio to obtain simulated bone conduction data;

[0140] It should be noted that relevant personnel may preset one or more signal-to-noise ratios, where the signal-to-noise ratio refers to the ratio of an effective signal to a noise signal. Specifically, the signal-to-noise ratio of simulated bone conduction data may be calculated in a known manner, which is not described in detail in this embodiment.

[0141] The simulated bone conduction data is data used to simulate a bone conduction signal, that is, used to simulate a bone conduction signal in an actual environment.

[0142] It should be noted that, in a preferred embodiment, narrow-band air conduction speech data and bone conduction noise data are mixed with multiple signal-to-noise ratios to obtain multiple simulated bone conduction data with different signal-to-noise ratios, so as to further simulate the diversity of bone conduction signals in the actual environment, so as to simulate the bone conduction signals actually picked up by AR glasses in different application scenarios, and enhance the generalization ability of the frequency band extension model.

[0143] Step S300: training the frequency band extension model based on the simulated bone conduction data to obtain the pre-trained frequency band extension model.

[0144] The frequency band extension model is trained based on the simulated bone conduction data. Specifically, the simulated bone conduction data can be used as the input of the frequency band extension model (in this case, the frequency band extension model also includes a preprocessing network before the low-frequency feature analysis network, and the preprocessing network is used to convert the simulated bone conduction data into the frequency domain, and extract the signal features of the simulated bone conduction data converted into the frequency domain, and input the extracted signal features into the low-frequency feature analysis network), or, after converting the simulated bone conduction data into the frequency domain, extract the signal features of the simulated bone conduction data in the frequency domain, and use the signal features as the input of the frequency band extension model. The signal features include a low-frequency amplitude vector and a low-frequency phase value vector, the low-frequency amplitude vector represents the signal amplitude features of the simulated bone conduction data, specifically including the signal amplitude of at least one signal frequency point in the simulated bone conduction data in the frequency domain, and the low-frequency phase value vector represents the phase value features of the simulated bone conduction data, specifically including the phase value of at least one signal frequency point in the simulated bone conduction data in the frequency domain.

[0145] Exemplarily, after converting the simulated bone conduction data into the frequency domain, the signal features of the simulated bone conduction data in the frequency domain are extracted, and the signal features are used as input of the frequency band extension model to illustrate and describe the various embodiments of the present application.

[0146] In this embodiment, the narrowband air conduction speech data and the bone conduction noise data are mixed with at least one preset signal-to-noise ratio to obtain simulated bone conduction data, and a frequency band extension model is trained based on the simulated bone conduction data to obtain a pre-trained frequency band extension model. In this way, this embodiment simulates the bone conduction signal in a manner of mixing narrowband air conduction speech data with bone conduction noise data. The air conduction speech data has a public data set. Therefore, before training the frequency band extension model, only the bone conduction noise data needs to be collected. Due to its own conduction characteristics, the bone conduction signal has few sources of noise, and the types of bone conduction noise data that need to be collected are relatively small. The collection cost of bone conduction noise data is low, thereby reducing the preparation cost of training data. The frequency band extension cost is trained with simulated bone data in a transfer learning manner, thereby reducing the training cost of the frequency band extension model. Moreover, since the air conduction speech data has a public data set, the data scale of the public data set is usually large, so that after mixing the narrowband air conduction speech data and the bone conduction noise data, a large-scale simulated bone conduction data can be obtained. The frequency band extension model is trained based on large-scale data, which can reduce the overfitting risk of the frequency band extension model, improve the generalization ability of the frequency band extension model, and make the frequency band extension result of the frequency band extension model more stable.

[0147] Based on the first embodiment, the second embodiment, the third embodiment and / or the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those of the above-mentioned embodiments 1, 2, 3 and 4 can be referred to the above introduction and will not be described in detail later. On this basis, the step of obtaining the preset training data includes:

[0148] Step A100, obtaining preset broadband air conduction speech data and preset bone conduction noise data;

[0149] It should be noted that the broadband air-conduction voice data may be public air-conduction voice data, and relevant personnel may obtain the public broadband air-conduction voice data from certain channels in advance.

[0150] Step A200, sequentially downsampling, upsampling and low-pass filtering the broadband air conduction speech data to obtain narrowband air conduction speech data, wherein the sampling rate of the narrowband air conduction speech data is the same as the sampling rate of the broadband air conduction speech data;

[0151] It should be noted that the broadband air conduction speech data may be subjected to one or more downsampling, upsampling and low-pass filtering processes in sequence. The sampling rate of each downsampling may be different. For example, when the sampling rate of the broadband air conduction speech data is 16000 Hz, the broadband air conduction speech data may be downsampled to 6000 Hz, 8000 Hz, 10000 Hz and 12000 Hz respectively.

[0152] The sampling rate of the narrowband air conduction speech data is the same as the sampling rate of the broadband air conduction speech data, that is, the sampling rate of the speech data obtained after upsampling is consistent with the sampling rate of the broadband air conduction speech data. For example, the original sampling rate of the air conduction speech data is 16000HZ, and the broadband air conduction speech data is downsampled to 6000HZ, 8000HZ, 10000HZ and 12000HZ respectively. The 6000HZ speech data is upsampled to 16000HZ and then low-pass filtered, the 8000HZ speech data is upsampled to 16000HZ and then low-pass filtered, the 10000HZ speech data is upsampled to 16000HZ and then low-pass filtered, and the 12000HZ speech data is upsampled to 16000HZ and then low-pass filtered. All speech data obtained after low-pass filtering constitute the narrowband air conduction speech data.

[0153] Step A300: determining that the narrowband air conduction speech data and the bone conduction noise data are preset training data.

[0154] The narrowband air conduction speech data and the bone conduction noise data are determined as the preset training data. Further, the broadband air conduction speech data may be determined as the preset training data, and the broadband air conduction speech data is used as a training label.

[0155] It can be understood that the signal value of the actual bone conduction signal in the high frequency band is not cut off to zero, but there is also a signal in the high frequency band, but the signal may be less. Therefore, in this embodiment, the broadband air conduction speech data is sequentially downsampled, upsampled and low-pass filtered, and the sampling rate of the narrowband air conduction speech data is the same as the sampling rate of the broadband air conduction speech data, so that the narrowband air conduction speech data has a full-band signal, and then the simulated bone conduction data obtained by mixing the narrowband air conduction speech data with the bone conduction noise data can better simulate the actual bone conduction signal, thereby improving the training effect of the model.

[0156] In a possible implementation manner, after the step of training the frequency band extension model based on the simulated bone conduction data, the method further includes:

[0157] Step B100, obtaining preset verification data, wherein the verification data includes bone conduction voice data and broadband air conduction voice data;

[0158] It should be noted that the broadband air conduction speech data in the verification data used to verify the frequency band extension model and the broadband air conduction speech data in the training data used to train the frequency band extension model may not be the same data. Specifically, the broadband air conduction speech data in the training data may be obtained from a public air conduction speech data set, and there is no need to collect it again, and the broadband air conduction speech data in the verification data may be collected in advance by relevant personnel.

[0159] It should be noted that, unless otherwise specified, in the embodiments of the present application, when broadband air-conduction speech data is described, it refers to broadband air-conduction speech data in the training data.

[0160] Furthermore, the broadband air conduction voice data and bone conduction voice data in the verification data are derived from the same voice data and can be collected by professionals. For example, a certain voice data can be played through a sound source in a quiet environment, and the user wears AR glasses, and a bone conduction microphone is provided at the nose pad of the AR glasses. Data is collected through the bone conduction microphone at the nose pad of the AR glasses, and data is collected through the air conduction microphone at the same time, and then broadband air conduction voice data is obtained based on the data collected by the air conduction microphone, and bone conduction voice data is obtained based on the data collected by the bone conduction microphone.

[0161] Step B200, in each round of training of the frequency band extension model, evaluating the model evaluation value of the frequency band extension model according to the verification data;

[0162] In each round of training of the frequency band extension model, the model evaluation value of the frequency band extension model is evaluated according to the verification data, that is, the frequency band extension model obtained in this round of training is evaluated according to the verification data to obtain the model evaluation value.

[0163] Specifically, the model can be evaluated by known model evaluation methods, such as STOI (Short-Time ObjectiveIntelligibility) evaluation, PESQ (Perceptual Evaluation of SpeechQuality) evaluation, etc. This implementation does not impose specific restrictions on this, nor does it elaborate on the specific evaluation process.

[0164] Step B300: After the training of the frequency band extension model is completed, the frequency band extension model corresponding to the optimal model evaluation value among the model evaluation values ​​is selected as the pre-trained frequency band extension model.

[0165] It should be noted that rules for evaluating the optimal model evaluation value can be set in advance, such as the maximum value among the model evaluation values ​​being the optimal model evaluation value, etc., and the optimal model evaluation value among the model evaluation values ​​is determined based on the preset rules to select the frequency band extension model corresponding to the optimal model evaluation value as the pre-trained frequency band extension model.

[0166] In this embodiment, bone conduction speech data is used as verification data, so that the performance of the frequency band extension model on the bone conduction signal can be evaluated, and the frequency band extension model corresponding to the optimal model evaluation value is selected as the pre-trained frequency band extension model, that is, the model with the best performance on the real bone conduction speech data is selected, so that the selected frequency band extension model has a better frequency band extension effect in the actual usage scenario.

[0167] Furthermore, after obtaining the pre-trained band extension model, the pre-trained band extension model can be deployed on the AR glasses, so that the AR glasses can use the band extension model deployed thereon to perform band extension. It should be noted that during model training, the high-frequency signal prediction network includes a convolutional layer, a normalization layer, a gated recurrent unit, and an activation layer connected in sequence. After the training is completed, there is no need to deploy the normalization layer. During the model training process, a batch normalization (BN) layer is connected after the convolution layer to reduce the training difficulty of the band extension model without increasing the computing cost on the chip side (adding a layer of batch normalization after the convolution calculation can constrain the output value range, thereby accelerating network convergence).

[0168] Based on the first embodiment, the second embodiment, the third embodiment, the fourth embodiment and / or the fifth embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment, the third embodiment, the fourth embodiment and the fifth embodiment can be referred to the above introduction, and will not be described in detail later. On this basis, the step of training the frequency band extension model based on the simulated bone conduction data to obtain the pre-trained frequency band extension model includes:

[0169] Step C100, inputting the amplitude vector and phase value vector of the simulated bone conduction data into the frequency band extension model to obtain a generation result;

[0170] Step C200, calculating a first loss value based on the generated result and a preset result label;

[0171] The preset result label can be set in advance. Specifically, considering that time domain signals are usually collected when preparing training data, the broadband air conduction speech data in the training data can be Fourier transformed, and after converting it to the frequency domain, the signal amplitude and phase values ​​corresponding to each signal frequency point are calculated, and the signal amplitude and phase values ​​corresponding to each signal frequency point are determined to be the preset result label.

[0172] The generated result and the preset result label can be input into the first preset loss function, and the first loss value can be output. The first preset loss function can be any loss function set in advance. In a specific embodiment, the first preset loss function is Among them, LGfreq_sp represents the first loss value, T represents the total number of frames, and F represents the total number of signal frequencies. Indicates the signal amplitude of the fth signal frequency point in the tth frame in the preset result label. Indicates the signal amplitude of the fth signal frequency point in the tth frame in the generated result, Indicates the phase value of the fth signal frequency point in the tth frame in the preset result label. Indicates the phase value of the fth signal frequency point in the tth frame in the generated result label.

[0173] Step C300, converting the generated result into time domain data, and / or calculating the Mel-band energy of the time domain data;

[0174] Mel-frequency band energy refers to the energy of each frequency band of a signal calculated in the Mel-frequency domain. The Mel-frequency band energy can be calculated in a known manner, which will not be described in detail in this embodiment.

[0175] Specifically, the generated result may be subjected to inverse Fourier transform, thereby converting the generated result into the time domain, that is, obtaining time domain data.

[0176] Step C400, calculating a second loss value based on the time domain data and a preset time domain label, and / or calculating a third loss value based on the Mel band energy and a preset energy label;

[0177] The preset time domain label may specifically be broadband air conduction speech data in the training data, and the preset energy label may be set in advance. Specifically, the preset energy label may be obtained by calculating the Mel-band energy of the preset time domain label.

[0178] The time domain data and the preset time domain label can be input into the second preset loss function, and the second loss value can be output. The second preset loss function can be any loss function set in advance. In a specific embodiment, the second preset loss function is Among them, L G_time represents the second loss value, N represents the total number of sampling points, y n Represents the sampling value of the nth sampling point in the preset time domain label, Represents the sample value of the nth sampling point in the time domain data.

[0179] The Mel band energy and the preset energy label can be input into a third preset loss function, and a third loss value can be output. The third preset loss function can be any loss function set in advance. In a specific embodiment, the third preset loss function is in, represents the third loss value, B represents the total number of frequency bands calculated by Mel, and y t,b represents the energy in the bth frequency band of the tth frame in the preset energy label, Represents the energy in the bth frequency band of the tth frame in the Mel-band energy.

[0180] Step C500, optimizing the frequency band extension model based on at least two loss values ​​among the first loss value, the second loss value and the third loss value, returning to execute the step of inputting the amplitude vector and the phase value vector of the simulated bone conduction data into the frequency band extension model to obtain the step of generating the result, until the preset training end condition is met, thereby obtaining the pre-trained frequency band extension model.

[0181] Specifically, the target loss value can be obtained by weighted summing at least two loss values: The frequency band extension model is optimized based on the target loss value, such as optimizing model parameters of the frequency band extension model with the purpose of reducing the target loss value.

[0182] In a preferred embodiment, the band extension model is optimized based on the first loss value, the second loss value and the third loss value. In this case, the target loss value can be expressed by the formula: in, represents the target loss value, λ sp represents the preset weight corresponding to the first loss value, λ mel represents the preset weight corresponding to the third loss value, λ tIn this way, the frequency band extension model is optimized based on the best direction of the first loss value, the second loss value and the third loss value, so that the output result of the frequency point extension model has good performance in the frequency domain, the time domain and the Mel band, further improving the extension quality of the frequency band extension model.

[0183] In this embodiment, the frequency band extension model is optimized based on at least two loss values ​​among the first loss value, the second loss value and the third loss value. In this way, the performance of the output result of the frequency band extension model in at least two aspects is evaluated, so that the output result of the frequency band extension model has better performance in at least two domains, namely, the frequency domain, the time domain or the Mel band domain, thereby improving the extension quality of the frequency band extension model.

[0184] Based on the first embodiment, the second embodiment, the third embodiment, the fourth embodiment, the fifth embodiment and / or the sixth embodiment of the present application, in the seventh embodiment of the present application, the same or similar contents as those in the above-mentioned embodiments 1, 2, 3, 4, 5 and 6 can be referred to the above introduction, and will not be repeated in the following. On this basis, the step of training the frequency band extension model based on the simulated bone conduction data to obtain the pre-trained frequency band extension model includes:

[0185] Step D100, obtaining at least one preset identification network, and performing adversarial training on the identification network and the frequency band extension model based on each of the simulated bone conduction data to obtain the pre-trained frequency band extension model;

[0186] The identification network includes at least one frequency identification network and a Mel identification network. Each of the frequency identification networks is used to identify whether input data with different sampling rates is real data, and the Mel identification network is used to identify whether input Mel frequency band energy data is real data.

[0187] The identification network is a classification network set in advance for identifying whether the input of this network is real data. Relevant personnel can set the specific network structure of each identification network in advance, and no specific restrictions are made here.

[0188] Furthermore, the loss function of each identification network can be designed as L Dk =(max(0,1-Dk(y))2+(max(0,1+Dk(G(x)))2, where Dk represents the kth discriminator network, and Dk(A) represents the output of the kth discriminator network when input A (A is a general term). G(x) is the generated result of the output of the frequency band extension model, and y is the broadband air-conducted speech data in the training data, which aims to promote D k (y) is close to label value 1, D k (G(x)) is close to the label value 0.

[0189] It should be noted that the sampling rate of the input data of each frequency identification network is different. In a specific embodiment, the frequency identification network includes a 2000HZ identification network, a 4000HZ identification network, an 8000HZ identification network and a 16000HZ identification network, wherein the sampling rate of the input data of the 2000HZ identification network is 2000HZ, the sampling rate of the input data of the 4000HZ identification network is 4000HZ, the sampling rate of the input data of the 6000HZ identification network is 6000HZ, the sampling rate of the input data of the 8000HZ identification network is 8000HZ, and the sampling rate of the input data of the 16000HZ identification network is 16000HZ. Assuming that the sampling rate of the generation result of the frequency band extension model is 16000HA, the generation result can be downsampled to 8000HZ, 4000HZ, and 2000HZ and then input into the corresponding frequency identification network.

[0190] The Mel identification network is used to identify whether the input Mel band energy data is real data, that is, the input of the Mel identification network is the Mel band energy. For example, the Mel band energy of the generated result of the band extension model can be calculated and input into the Mel identification network.

[0191] Based on each simulated bone conduction data, adversarial training is performed on the discrimination network and the frequency band extension model. Specifically, during the adversarial training process, the frequency band extension model can be optimized based on the loss value of each discrimination network to achieve adversarial training of the discrimination network and the frequency band extension model. In a specific embodiment, the adversarial training process integrates functions, and the loss function of the frequency band extension model can be designed as in, represents the comprehensive loss value of all identification networks, and α is the preset weight. For example, M represents the total number of identification networks, l represents the lth layer of the identification network, and L k represents the total number of layers of the k-th identification network minus one (the total number of layers excluding the output layer), γ is the preset weight, Represents the output of the lth layer of the kth discriminator network when input A. In this way, the loss of each layer of the discriminator network allows the generation result of the band extension model to be as close to the true label as possible, operating in the layer-by-layer feature domain of the discriminator network to focus on matching on semantic features. At the same time, since the final output loss of the discriminator model is calculated, when there is no available information in the degraded low-frequency band signal, it ensures that the band extension model is also creative at high frequencies.

[0192] In this embodiment, a discriminant network is used to train the band extension model adversarially. On the one hand, the discriminant network is trained to perform deep discrimination between the generation result of the band extension model and the true label value in the Mel domain at different sampling rates, thereby increasing the distance between the generation result of the band extension model and the true label value. On the other hand, the output loss value of the discriminant network is back-propagated to the band extension model, so that the band extension model can deceive the discriminant network at the output, thereby decreasing the distance between the generation result of the band extension model and the true label, thereby forming a competitive relationship and optimizing the band extension model to the maximum extent.

[0193] In a possible implementation manner, before the step of acquiring at least one preset identification network, the method further includes:

[0194] Step E100, training the frequency band extension model based on the simulated bone conduction data until a preset training termination condition is met, and then executing the step of obtaining at least one preset identification network and performing adversarial training on the identification network and the frequency band extension model based on each of the simulated bone conduction data.

[0195] The preset training termination condition may be a condition set in advance, and the preset training termination condition may be satisfied before the preset training termination condition during the model training process. For example, the preset training termination condition is that the number of iterations reaches 10,000 times, and the preset training termination condition may be that the number of iterations reaches 5,000 times.

[0196] Considering that the adversarial training process has a large amount of computation, the training process is relatively complicated. Based on this, in this embodiment, the frequency band extension model is first trained conventionally based on the simulated bone conduction data until the preset training termination condition is met, and then the frequency band extension model is trained adversarially based on the identification network. In this way, it is equivalent to using the adversarial training method to tune the previously conventionally trained frequency band extension model, which can reduce the number of adversarial trainings and thus reduce the overall training complexity of the frequency band extension model.

[0197] In addition, an embodiment of the present application also proposes a signal processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the signal processing method as described above.

[0198] refer to Figure 6, which shows a schematic diagram of the structure of a signal processing device suitable for implementing an embodiment of the present application. The signal processing device in the embodiment of the present application may also include, but is not limited to, mobile terminals such as AR glasses, VR glasses, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The signal processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0199] like Figure 6 As shown, the signal processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the signal processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the signal processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a signal processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0200] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0201] The signal processing device provided in the embodiment of the present application adopts the signal processing method in the above embodiment to solve how to improve the clarity of the bone conduction signal to improve the hearing experience of the bone conduction signal. Compared with the prior art, the beneficial effects of the signal processing device provided in the present application are the same as the beneficial effects of the signal processing method provided in the above embodiment, and other technical features in the signal processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0202] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0203] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0204] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the signal processing method in the above-mentioned embodiment.

[0205] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0206] The computer-readable storage medium may be included in the signal processing device; or may exist independently without being assembled into the signal processing device.

[0207] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the signal processing device, the signal processing device: obtains the original signal to be processed, and the signal type of the original signal includes a bone conduction signal; extracts the signal features of the original signal, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, the low-frequency amplitude vector represents the signal amplitude features of the original signal, and the low-frequency phase value vector represents the phase value features of the original signal; inputs the signal features into a pre-trained frequency band extension model to obtain a target bandwidth signal, wherein the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, the low-frequency feature analysis network is used to analyze and fuse the signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal.

[0208] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0210] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0211] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned signal processing method, and can solve how to improve the clarity of the bone conduction signal to improve the hearing of the bone conduction signal. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the signal processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0212] In addition, an embodiment of the present application further proposes a computer program product, including a signal processing program, which implements the steps of the signal processing method described above when executed by a processor.

[0213] The specific implementation of the computer program product of the present application is basically the same as the above-mentioned signal processing method embodiments, and will not be repeated here.

[0214] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0215] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0216] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0217] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A signal processing method, characterized in that: The signal processing method comprises the following steps: Acquire an original signal to be processed, wherein the signal type of the original signal includes a bone conduction signal; Extracting signal features of the original signal, wherein the signal features include a low-frequency amplitude vector and a low-frequency phase value vector, the low-frequency amplitude vector represents the signal amplitude feature of the original signal, and the low-frequency phase value vector represents the phase value feature of the original signal; The signal features are input into a pre-trained frequency band extension model to obtain a target bandwidth signal, wherein the frequency band extension model includes a low-frequency feature analysis network, a high-frequency signal prediction network and a fusion network connected in sequence, the low-frequency feature analysis network is used to analyze and fuse the signal features, the high-frequency signal prediction network is used to predict the high-frequency signal, and the fusion network is used to fuse the high-frequency signal with the original signal.

2. The signal processing method according to claim 1, characterized in that: The step of inputting the signal feature into a pre-trained frequency band extension model to obtain a target bandwidth signal comprises: The signal features are analyzed and fused by the low-frequency feature analysis network to obtain fused feature data, and a high-frequency signal is predicted based on the fused feature data by the high-frequency signal prediction network, wherein the high-frequency signal represents a signal within a preset high-frequency range; The high-frequency signal and the original signal are fused through the fusion network to obtain a target broadband signal.

3. The signal processing method according to claim 2, characterized in that: The high-frequency signal prediction network includes a high-frequency amplitude calculation subnetwork and a high-frequency phase value calculation subnetwork. The step of predicting the high-frequency signal based on the fused feature data through the high-frequency signal prediction network includes: Predicting a high-frequency amplitude vector based on the fused feature data by the high-frequency amplitude calculation subnetwork, wherein the high-frequency amplitude vector includes at least one signal amplitude; Predicting a high-frequency phase value vector based on the fused feature data by the high-frequency phase value calculation subnetwork, wherein the high-frequency phase value vector includes at least one phase value, and the phase value corresponds to the signal amplitude one by one; The high-frequency amplitude value vector and the high-frequency phase value vector are determined to be high-frequency signals.

4. The signal processing method according to claim 2, characterized in that: The fusion network includes an amplitude filter subnetwork and a phase value filter subnetwork, and the step of fusing the high-frequency signal with the original signal through the fusion network to obtain a target broadband signal includes: The high-frequency amplitude vector in the high-frequency signal is fused with the low-frequency amplitude vector through the amplitude filtering subnetwork to obtain a target broadband amplitude vector; The high-frequency phase value vector in the high-frequency signal is fused with the low-frequency phase value vector through the phase value filtering subnetwork to obtain a target broadband phase value vector; The target wideband amplitude vector and the target wideband phase value vector are determined as a target wideband signal.

5. The signal processing method according to claim 2, characterized in that: The step of analyzing the fused signal features through the low-frequency feature analysis network to obtain fused feature data comprises: Acquire an associated frequency domain signal associated with the target frequency domain signal, and acquire an associated low-frequency amplitude value vector and an associated low-frequency phase value vector corresponding to the associated frequency domain signal, wherein the associated frequency domain signal includes a preset number of frame frequency domain signals before the target frequency domain signal; Combining the associated low-frequency amplitude vector with the low-frequency amplitude vector to obtain an amplitude matrix, and combining the associated low-frequency phase value vector with the low-frequency phase value vector to obtain a phase value matrix; The amplitude matrix and the phase value matrix are input into the low-frequency feature analysis network to obtain fused feature data.

6. The signal processing method according to claim 1, characterized in that: After the step of obtaining the original signal to be processed, the method further comprises: Performing voice activity detection on the original signal to obtain a voice activity detection result; If the voice activity detection result indicates that there is voice activity, then the step of extracting the signal feature of the original signal is continued.

7. The signal processing method according to claim 1, characterized in that: The step of extracting the signal feature of the original signal comprises: If the original signal is a time series signal, Fourier transform is performed on the original signal to obtain a target frequency domain signal; The signal amplitude of the target signal frequency point in the target frequency domain signal is calculated to obtain a low-frequency amplitude vector, and the phase value of the target signal frequency point in the target frequency domain signal is calculated to obtain a low-frequency phase value vector, wherein the target signal frequency point includes the signal frequency point in the target frequency domain signal within a preset low-frequency range.

8. The signal processing method according to any one of claims 1 to 7, characterized in that: The low-frequency feature analysis network includes a first depth-separable convolutional network and a second depth-separable convolutional network connected in sequence, and the high-frequency signal prediction network includes a convolutional layer, a gated recurrent unit, and an activation layer connected in sequence.

9. A signal processing device, characterized in that: The signal processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the signal processing method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the signal processing method according to any one of claims 1 to 8 are implemented.

Images