Audio signal encoding methods, devices, and chips

Abstract

This application relates to an audio signal encoding method, apparatus, and chip. The method involves using a neural network encoder to extract temporal features from the audio signal to be encoded, obtaining a temporal feature map with a first time resolution; and using the neural network encoder to extract frequency domain features from the audio signal, obtaining a frequency domain feature map with a second time resolution. Then, the second time resolution of the frequency domain feature map and the first time resolution of the temporal feature map are aligned to obtain an aligned frequency domain feature map, which retains the first time resolution. Finally, feature vectors located at the same time position in the temporal feature map and the aligned frequency domain feature map are fused to obtain the signal encoding features. In this method, by aligning the frequency domain feature map to a higher time resolution before fusing it with the temporal feature map, the completeness of feature representation is improved.

Description

Technical Field

[0001] This application relates to the field of signal processing technology, and in particular to an audio signal encoding method, apparatus and chip. Background Technology

[0002] With the rapid development of deep learning and digital signal processing technologies, neural network audio codecs have been widely used in fields such as speech synthesis, speech enhancement, audio compression, and real-time communication.

[0003] Currently, the architecture of neural network audio codec systems mainly follows the paradigm of "encoder-latent space-decoder". The encoder extracts features from the input audio signal to obtain a latent space representation, which is then quantized and reconstructed into an audio waveform by the decoder. Therefore, the feature extraction capability of the encoder in the codec system directly affects the representation accuracy of the latent space representation, and thus affects the reconstruction quality of the audio signal by the system.

[0004] However, the encoder's ability to represent audio signals is limited in related technologies, which affects the overall reconstruction quality and speech intelligibility of the encoding and decoding system. Summary of the Invention

[0005] Therefore, it is necessary to provide an audio signal encoding method, apparatus, and chip that can improve the encoder's ability to represent audio signals, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides an audio signal encoding method, the method comprising:

[0007] A neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and a neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0008] Align the second time resolution of the frequency domain feature map with the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map, which has the first time resolution.

[0009] The signal coding features are obtained by fusing the feature vectors at the same time position in the time domain feature map and the aligned frequency domain feature map.

[0010] In some embodiments, the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map are aligned to obtain an aligned frequency domain feature map, including:

[0011] By using the prediction subnetwork in the pre-defined alignment network, the feature similarity between the time domain feature map and the frequency domain feature map within a local time window is analyzed, and the time offset of each time position in the frequency domain feature map is predicted.

[0012] By aligning the mapping subnetwork in the network, the time coordinates of each time position in the frequency domain feature map are mapped to the target time axis according to each time offset, thus obtaining the mapped coordinates of each time position in the frequency domain feature map on the target time axis; the target time axis is the time axis corresponding to the first time resolution.

[0013] Based on each mapped coordinate and the frequency domain feature map, determine the aligned frequency domain feature map.

[0014] In some embodiments, the alignment network further includes a difference processing subnetwork that determines the aligned frequency domain feature map based on each mapped coordinate and the frequency domain feature map, including:

[0015] The frequency domain feature map is interpolated and sampled according to each mapping coordinate by the interpolation subnetwork to obtain the aligned frequency domain feature map.

[0016] In some embodiments, feature vectors located at the same time position in the time-domain feature map and the aligned frequency-domain feature map are fused to obtain signal coding features, including:

[0017] The weights at different time positions in the temporal feature map and the aligned frequency domain feature map are assigned through the weight allocation subnetwork in the pre-defined fusion network to obtain the first weight and the second weight.

[0018] By using the computational subnetwork in the fusion network, weights are calculated based on the first weight and the time-domain feature map to obtain an intermediate time-domain feature map, and weights are calculated based on the second weight and the aligned frequency-domain feature map to obtain an intermediate frequency-domain feature map.

[0019] By splicing the feature vectors of the intermediate time-domain feature map and the intermediate frequency-domain feature map at the same time position through the splicing sub-network in the fusion network, the signal coding features are obtained.

[0020] In some embodiments, a neural network encoder is used to extract temporal features from the audio signal to be encoded to obtain a temporal feature map with a first time resolution, including:

[0021] The temporal feature extraction branch in the neural network encoder is used to extract temporal features from the audio signal to be encoded, so as to obtain a temporal feature map with the first time resolution. The temporal feature extraction branch includes N cascaded coding sub-networks, and each coding sub-network includes a downsampling layer, a receptive field enhancement layer and a feature residual enhancement layer.

[0022] In some embodiments, a neural network encoder is used to extract frequency domain features from the audio signal to obtain a frequency domain feature map with a second time resolution, including:

[0023] The frequency domain feature extraction branch in the neural network encoder is used to extract frequency domain features from the audio signal to obtain a frequency domain feature map with a second time resolution. The frequency domain feature extraction branch includes a multi-resolution Fourier transform layer, a logarithmic spectrum calculation layer, and a complex convolutional layer.

[0024] In some embodiments, the construction process of the alignment network and the fusion network includes:

[0025] Obtain the original time-domain feature map and the original frequency-domain feature map of the original audio signal;

[0026] The temporal resolution of the frequency domain feature map and the temporal resolution of the time domain feature map are aligned based on the initial alignment network to obtain the aligned original frequency domain feature map.

[0027] Based on the initial fusion network, the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map are fused to obtain the fused features;

[0028] The initial alignment network and the initial fusion network are trained based on the fusion features and the desired audio signal to obtain the trained alignment network and the trained fusion network.

[0029] In some embodiments, the initial alignment network and the initial fusion network are trained based on the fusion features and the desired audio signal to obtain the trained alignment network and the trained fusion network, including:

[0030] The fused features are compressed step by step using K cascaded residual vector quantizers, and the compressed features are input into the decoding network for signal reconstruction to obtain the desired audio signal.

[0031] Determine the target loss based on the original audio signal and the desired audio signal;

[0032] The parameters in the initial alignment network and the initial fusion network are adjusted according to the target loss to obtain the trained alignment network and the trained fusion network.

[0033] Secondly, this application also provides an audio signal encoding device, the device comprising:

[0034] The feature extraction module is used to extract time-domain features from the audio signal to be encoded using a neural network encoder to obtain a time-domain feature map with a first time resolution; and to extract frequency-domain features from the audio signal using a neural network encoder to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0035] The alignment module is used to align the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map to obtain an aligned frequency domain feature map, which has the first time resolution.

[0036] The fusion module is used to fuse the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain the signal coding features.

[0037] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0038] A neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and a neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0039] Align the second time resolution of the frequency domain feature map with the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map, which has the first time resolution.

[0040] The signal coding features are obtained by fusing the feature vectors at the same time position in the time domain feature map and the aligned frequency domain feature map.

[0041] Fourthly, this application also provides a chip, including a processor and a communication interface, wherein the processor is configured to cause the chip to perform:

[0042] A neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and a neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0043] Align the second time resolution of the frequency domain feature map with the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map, which has the first time resolution.

[0044] The signal coding features are obtained by fusing the feature vectors at the same time position in the time domain feature map and the aligned frequency domain feature map.

[0045] Fifthly, this application also provides a chip module, including a communication module, a power module, a storage module, and a chip, wherein:

[0046] The power module is used to provide power to the chip module;

[0047] Storage modules are used to store data and instructions;

[0048] The communication module is used for internal communication within the chip module, or for communication between the chip module and external devices;

[0049] The chip is used to perform the steps of the method provided in the first aspect above.

[0050] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0051] A neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and a neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0052] Align the second time resolution of the frequency domain feature map with the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map, which has the first time resolution.

[0053] The signal coding features are obtained by fusing the feature vectors at the same time position in the time domain feature map and the aligned frequency domain feature map.

[0054] In a seventh aspect, this application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0055] A neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and a neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0056] Align the second time resolution of the frequency domain feature map with the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map, which has the first time resolution.

[0057] The signal coding features are obtained by fusing the feature vectors at the same time position in the time domain feature map and the aligned frequency domain feature map.

[0058] The aforementioned audio signal encoding method, apparatus, and chip involve the following steps: First, a time-domain feature map with a first time resolution is obtained by using a neural network encoder to extract time-domain features from the audio signal to be encoded. Second, a frequency-domain feature map with a second time resolution is obtained by using the same neural network encoder to extract frequency-domain features from the audio signal. The second time resolution of the frequency-domain feature map and the first time resolution of the time-domain feature map are then aligned to obtain an aligned frequency-domain feature map. This aligned frequency-domain feature map retains the first time resolution. Finally, feature vectors at the same time position in the time-domain feature map and the aligned frequency-domain feature map are fused to obtain the signal encoding features. The first time resolution is higher than the second time resolution. In this method, by extracting a time-domain feature map with high time resolution and a frequency-domain feature map with low time resolution, aligning the frequency-domain feature map to the high time resolution, and then fusing it with feature vectors at the same time position in the time-domain feature map, the final signal encoding features simultaneously contain both time-domain and frequency-domain information of the audio signal. Since the first time resolution is higher than the second time resolution, the time-domain feature map retains more refined time details, while the frequency-domain feature map provides spectral distribution characteristics. The fused coding features have both the transient representation capability in the time domain and the spectral representation capability in the frequency domain, thereby enriching the feature dimensions of the encoder output and improving the completeness of feature expression. Attached Figure Description

[0059] Figure 1 These are schematic diagrams of the audio codec system in some embodiments;

[0060] Figure 2 This is one of the flowcharts illustrating the audio signal encoding method in some embodiments;

[0061] Figure 3 This is a second schematic flowchart of an audio signal encoding method in some embodiments;

[0062] Figure 4 This is the third flowchart illustrating the audio signal encoding method in some embodiments;

[0063] Figure 5 This is the fourth flowchart illustrating the audio signal encoding method in some embodiments;

[0064] Figure 6 This is the fifth flowchart illustrating the audio signal encoding method in some embodiments;

[0065] Figure 7 This is a structural block diagram of an audio signal encoding device in some embodiments;

[0066] Figure 8 These are internal structural diagrams of the computer device in some embodiments;

[0067] Figure 9 This is a schematic diagram of the chip module structure in some embodiments. Detailed Implementation

[0068] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0069] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0070] In the embodiments of this application, the term "at least one" means one or more. For example, at least one of A, B and C can represent six situations: A exists alone, B exists alone, C exists alone, A and B exist simultaneously, A and C exist simultaneously, B and C exist simultaneously, and A, B and C exist simultaneously.

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0072] With the rapid development of deep learning and digital signal processing technologies, neural network audio codecs have been widely used in fields such as speech synthesis, speech enhancement, audio compression, and real-time communication. Currently, the architecture of neural network audio codec systems mainly follows the paradigm of "encoder-latent space-decoder". The encoder extracts features from the input audio signal to obtain a latent space representation, which is then quantized and reconstructed into an audio waveform by the decoder.

[0073] Therefore, the feature extraction capability of the encoder in the encoding and decoding system directly affects the representation accuracy of the latent space representation, and thus affects the reconstruction quality of the audio signal by the system.

[0074] However, the encoder's ability to represent audio signals is limited in related technologies, which affects the overall reconstruction quality and speech intelligibility of the encoding and decoding system.

[0075] In view of this, embodiments of this application propose an audio signal encoding method, apparatus and chip, which improves the integrity of feature representation by aligning the frequency domain feature map to a high time resolution and then fusing it with the time domain feature map.

[0076] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0077] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0078] In some embodiments, the audio signal encoding method provided in this application can be applied to, for example... Figure 1 The audio encoding and decoding system shown includes a neural network encoder 101, a residual vector quantizer 102, and a decoder 103. The output of the neural network encoder 101 is connected to the input of the residual vector quantizer 102, and the output of the residual vector quantizer 102 is connected to the input of the decoder 103. The neural network encoder 101 extracts time-domain and frequency-domain features from the input audio signal, aligns and fuses the time-domain and frequency-domain features to obtain encoded features. The residual vector quantizer 102 quantizes the encoded features, and the decoder 103 decodes the quantized features to obtain the target audio signal. This audio encoding and decoding system is used in communication terminals or other computer equipment.

[0079] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the audio codec system to which the solution of this application is applied. Specifically, the audio codec system may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0080] In some embodiments, such as Figure 2 As shown, an audio signal encoding method is provided, which can be applied to... Figure 1 Taking the neural network encoder (hereinafter referred to as the encoder) as an example, the following steps are included:

[0081] S201, the neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and the neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0082] A neural network encoder is a deep learning-based neural network model used to extract, align, and fuse features from the audio signal to be encoded, thus achieving the signal encoding process. The audio signal to be encoded refers to the original digitized audio waveform data, typically represented as a one-dimensional time series. Where T represents the total number of sampling points of the signal on the time axis. The time-domain feature map is a feature tensor map formed from time-domain features. Time-domain features are used to represent the local waveform shape, transient changes, phase information, and fine fluctuation characteristics on the time axis of the audio signal. The frequency-domain feature map is a feature tensor map formed from frequency-domain features. Frequency-domain features are used to characterize the energy distribution, spectral envelope, and harmonic structure of the audio signal at different frequency components. Time resolution refers to the level of detail of the feature map on the time axis; high time resolution means that the time interval between two adjacent feature points is very small, enabling the capture of rapidly changing signal details; low time resolution means that the feature map is smoother and more abstract in time, focusing on long-term or steady-state characteristics.

[0083] In this embodiment, the encoder can receive audio signals input by the user or audio signals output by a connected upstream module, and use these audio signals as the audio signal to be encoded. The encoder can input the audio signal to be encoded into a pre-trained time-domain feature extraction network for time-domain feature extraction to obtain a time-domain feature map with a first time resolution. Simultaneously, it can input the audio signal into a pre-trained frequency-domain feature extraction network for frequency-domain feature extraction to obtain a frequency-domain feature map with a second time resolution. Optionally, the encoder can use wavelet transform or Fourier transform for frequency-domain feature extraction. Optionally, before performing feature extraction on the audio signal, the encoder can perform pre-filtering to remove noise.

[0084] S202, the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map are aligned to obtain an aligned frequency domain feature map, which has the first time resolution.

[0085] Among them, the aligned frequency domain feature map is a new frequency domain feature map obtained after alignment processing. Its time resolution has been improved from the original low second time resolution to the same first time resolution as the time domain feature map, thus making it feasible to fuse with the time domain feature map.

[0086] In this embodiment, the encoder can use a dynamic interpolation method to align the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map. Specifically, the encoder obtains the first time length of the time domain feature map and the second time length of the frequency domain feature map, and calculates the upsampling factor based on the ratio of the first time length and the second time length. Then, the encoder uses a linear interpolation algorithm to insert a target number of new time steps between every two adjacent time steps of the frequency domain feature map, based on the sampling factor. The target number is determined by the sampling factor, such as target number = sampling factor - 1, ultimately obtaining the aligned frequency domain feature map with the first time resolution.

[0087] S203, fuse the feature vectors at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain the signal coding features.

[0088] Among them, the signal coding features are the final fused output, which is the final encoded representation of the input audio signal after abstraction by the neural network encoder, containing complementary information from the time and frequency domains. The feature vectors in the time-domain feature map are determined by the time-domain features, and the feature vectors in the frequency-domain feature map are determined by the frequency-domain features.

[0089] In this embodiment, after obtaining the time-domain feature map and the aligned frequency-domain feature map, the encoder can fuse the feature vectors located at the same time position in the time-domain feature map and the aligned frequency-domain feature map using the aligned channel dimension to obtain the signal coding features. This fusion process can be represented by the following relationship: Here, `Conv` is the convolution operation, aligning the scale of the next input; `concat` is the channel concatenation operation. Specifically, the encoder acquires a temporal feature map `Ft` (number of channels `Ct`, temporal resolution `T`) and an aligned frequency domain feature map `Fs` (number of channels `Cs`, temporal resolution `T`). Then, for each identical time position, the encoder concatenates the temporal feature vector and the frequency domain feature vector end-to-end along the channel dimension, obtaining a concatenated feature vector with the number of channels `Ct+Cs`; the same operation is performed on all time positions to generate a concatenated feature map of dimension `(Ct+Cs)×T`. Next, the encoder inputs the concatenated feature map into a pre-constructed one-dimensional convolutional layer. This convolutional layer has a kernel size of 1 and a temporal resolution of `T`. In the one-dimensional convolutional layer, the convolutional kernel slides point-by-point along the time axis, independently performing linear combination and bias addition at each time position, outputting the fused feature vector at that position as the signal encoding feature.

[0090] The audio signal encoding method provided in this application involves extracting time-domain features from the audio signal to be encoded using a neural network encoder to obtain a time-domain feature map with a first time resolution, and extracting frequency-domain features from the audio signal using the same neural network encoder to obtain a frequency-domain feature map with a second time resolution. Then, the second time resolution of the frequency-domain feature map and the first time resolution of the time-domain feature map are aligned to obtain an aligned frequency-domain feature map, which retains the first time resolution. Finally, the feature vectors at the same time position in the time-domain feature map and the aligned frequency-domain feature map are fused to obtain the signal encoding features. The first time resolution is higher than the second time resolution. In this method, by extracting a time-domain feature map with high time resolution and a frequency-domain feature map with low time resolution, aligning the frequency-domain feature map to a high time resolution, and then fusing it with the feature vectors at the same time position in the time-domain feature map, the final signal encoding features simultaneously contain both time-domain and frequency-domain information of the audio signal. Since the first time resolution is higher than the second time resolution, the time-domain feature map retains more refined time details, while the frequency-domain feature map provides spectral distribution characteristics. The fused coding features have both the transient representation capability in the time domain and the spectral representation capability in the frequency domain, thereby enriching the feature dimensions of the encoder output and improving the completeness of feature expression.

[0091] In some embodiments, a specific implementation of time resolution alignment processing is also provided, such as... Figure 3 As shown, the "aligning the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map" in S202 above includes:

[0092] S301, through the prediction sub-network in the preset alignment network, analyzes the feature similarity between the time domain feature map and the frequency domain feature map within a local time window, and predicts the time offset of each time position in the frequency domain feature map.

[0093] The neural network encoder includes an alignment network for performing temporal-resolution alignment. This alignment network comprises a prediction subnetwork that predicts the time offset for each time position in the frequency domain feature map. A local time window is a continuous temporal neighborhood centered on the target time position, used to limit the context length for feature similarity analysis. The time offset characterizes how far the feature vector at a certain time position in the frequency domain feature map should be shifted forward or backward along the time axis to achieve optimal alignment with the corresponding local structure in the temporal feature map.

[0094] In this embodiment, the encoder can predict time offsets using a prediction sub-network in various ways. In one example, the prediction sub-network may include a convolutional layer with shared weights. This layer performs feature mapping on both the temporal and frequency domain feature maps, generating two intermediate feature maps with the same number of channels. The prediction sub-network then concatenates these two intermediate feature maps along the channel dimension and processes the concatenated feature map through one or more convolutional layers and activation function layers, ultimately outputting an offset sequence with the same time length as the frequency domain feature map. Each value in this sequence represents the time offset at the corresponding time position. In another example, the prediction sub-network may employ a method based on local cross-correlation calculation. Specifically, for each time position in the frequency domain feature map, the prediction sub-network extracts a feature vector within a local time window centered on that position, and simultaneously extracts a feature vector within the corresponding time neighborhood in the temporal feature map. By calculating the similarity between the two at different time lags, such as cosine similarity, the lag corresponding to the maximum similarity is taken as the time offset for that time position. In another example, the prediction subnetwork could be a recurrent neural network that takes the feature differences between the time-domain feature map and the frequency-domain feature map at corresponding time steps as input and predicts the time offset of each location step by step through sequence modeling.

[0095] S302, by aligning the mapping sub-network in the network, the time coordinates of each time position in the frequency domain feature map are mapped to the target time axis according to each time offset, so as to obtain the mapped coordinates of each time position in the frequency domain feature map on the target time axis.

[0096] The alignment network also includes a mapping sub-network for performing coordinate transformation operations based on the predicted time offset. The target time axis has a time axis corresponding to a time-domain feature map with a first time resolution (i.e., higher resolution), whose coordinate scale is denser than that of the original time axis of the frequency-domain feature map.

[0097] In this embodiment, the encoder can perform coordinate mapping through a mapping subnetwork in various ways. In one example, for each time position in the frequency domain feature map, after obtaining the time offset predicted by the prediction subnetwork for that time position, the mapping subnetwork can use the sum of the original time coordinates of the time position and the predicted time offset as the mapped coordinates of that time position on the target time axis. In another example, before performing coordinate mapping, the mapping subnetwork can first perform smoothing filtering on the predicted time offset sequence to remove abnormal jumps in the prediction-result, making the changes in the mapped coordinates smoother and more continuous.

[0098] S303, determine the aligned frequency domain feature map based on each mapping coordinate and the frequency domain feature map.

[0099] The aligned frequency domain feature map refers to the new frequency domain feature map obtained after time resolution alignment processing. Its time resolution is consistent with the target time axis, that is, it has the same first time resolution as the time domain feature map.

[0100] In this embodiment, for each target time position on the target time axis, the encoder can find the source time position whose mapped coordinates are closest to that target time position in the frequency domain feature map. Then, it directly uses the feature vector of that source time position as the feature value of the corresponding position in the aligned frequency domain feature map to obtain the aligned frequency domain feature map. Optionally, the encoder can copy the feature vector of each source time position in the frequency domain feature map to all target time positions within the target time axis interval covered by its mapped coordinates to obtain the aligned frequency domain feature map.

[0101] Optionally, the alignment network further includes an interpolation processing sub-network. Based on this, S303 includes: interpolating and sampling the frequency domain feature map according to each mapping coordinate through the interpolation processing sub-network to obtain the aligned frequency domain feature map.

[0102] The alignment network also includes an interpolation subnetwork, which is used to interpolate and resample the frequency domain feature map based on the mapped coordinates to generate a smooth, aligned frequency domain feature map with the target time resolution.

[0103] In one embodiment of this application, for any target time position p on the target time axis, the interpolation subnetwork finds the two nearest source time positions whose mapped coordinates are located on either side of p in the frequency domain feature map. Then, according to the distance of p from the two nearest source time positions, the two nearest source time positions are weighted and summed to obtain the feature vector at position p. After processing all time positions in the above manner, the aligned frequency domain feature map can be obtained. In another embodiment, the interpolation subnetwork utilizes the continuity of the higher-order derivatives of the feature vectors of multiple source time positions whose mapped coordinates are located near the target time position p in the frequency domain feature map to fit a cubic spline curve. Then, the value of this curve at p is calculated as the aligned feature vector. After processing all time positions in the above manner, the aligned frequency domain feature map can be obtained.

[0104] The method described in this application analyzes local similarity through a predictive subnetwork to predict time offsets, performs coordinate mapping through a mapping subnetwork, and performs interpolation sampling through a difference processing subnetwork. This achieves fine-grained, content-adaptive, non-uniform alignment between the frequency domain feature map and the time domain feature map. Compared with traditional fixed-rate upsampling alignment methods, this method can dynamically adjust the alignment strategy according to the local time-frequency structure of the audio signal. This allows the spectral information in the frequency domain feature map to more accurately match the transient detail positions in the time domain feature map, thereby achieving true time-frequency co-representation in the fusion stage and significantly improving the modeling accuracy and representation capability of the neural network encoder for complex audio signals.

[0105] In some embodiments, a specific implementation of feature fusion is also provided, such as Figure 4 As shown, the phrase "fusing the feature vectors located at the same time position in the time-domain feature map and the aligned frequency-domain feature map to obtain the signal coding features" in S203 above includes:

[0106] S401, through the weight allocation sub-network in the preset fusion network, the weights at different time positions in the time domain feature map and the aligned frequency domain feature map are allocated to obtain the first weight and the second weight.

[0107] The neural network encoder also includes a fusion network for performing feature fusion operations. The fusion network includes a weight allocation subnetwork, which adaptively assigns fusion weights to features at different time locations and in different domains based on the local characteristics of the temporal feature map and the aligned frequency domain feature map. The first weight corresponds to the weight in the temporal feature map, and the second weight corresponds to the weight in the aligned frequency domain feature map.

[0108] In this embodiment of the application, the encoder can input the temporal feature map and the aligned frequency domain feature map into a pre-trained weight allocation sub-network. The weight allocation sub-network adaptively assigns fusion weights to features at different time positions and in different domains, thereby obtaining the first weight α corresponding to the temporal feature map at each time position and the second weight β corresponding to the aligned frequency domain feature map at each time position.

[0109] It should be noted that the weight allocation subnetwork contains two learnable scalar parameters α and β, satisfying α + β = 1, α, β ∈ (0, 1). These two parameters serve as global weights, which are jointly optimized with other parts of the encoder during the training process at different time points, enabling the weight allocation subnetwork to adaptively learn the optimal contribution ratio of temporal and frequency domain information in the fusion process.

[0110] S402, through the computational sub-network in the fusion network, weight calculation is performed based on the first weight and the time-domain feature map to obtain an intermediate time-domain feature map, and weight calculation is performed based on the second weight and the aligned frequency-domain feature map to obtain an intermediate frequency-domain feature map.

[0111] The fusion network also includes a computational subnetwork, which applies the assigned weights to the corresponding feature maps to generate weighted intermediate feature representations. The intermediate time-domain feature map is the result of element-wise operations between the first weight and the time-domain feature map, and the intermediate frequency-domain feature map is the result of element-wise operations between the second weight and the aligned frequency-domain feature map.

[0112] In this embodiment, the encoder calculates weights using an element-wise weighted summation method through a computational subnetwork. Specifically, for the feature vector at each time position, the computational subnetwork multiplies the feature vector at that position in the time-domain feature map Ft by a first weight α to obtain an intermediate time-domain feature map; it then multiplies the feature vector at that position in the aligned frequency-domain feature map Fs by a second weight β to obtain an intermediate frequency-domain feature map. In other words, the intermediate time-domain feature map is... The intermediate frequency domain feature map is .

[0113] S403 uses a splicing sub-network in the fusion network to splice the feature vectors of the intermediate time-domain feature map and the intermediate frequency-domain feature map that are located at the same time position to obtain the signal coding features.

[0114] The fusion network also includes a splicing subnetwork, which merges the weighted time-domain features and weighted frequency-domain features along the channel dimension and performs cross-modal information fusion. The signal coding features are the final output of the fusion network and will serve as the input to the subsequent quantizer.

[0115] In this embodiment, the splicing sub-network performs element-wise addition or splicing operations on the intermediate time-domain feature map and the intermediate frequency-domain feature map, i.e., through the formula... The signal coding features are obtained. .

[0116] Furthermore, after obtaining the signal coding features, a 1×1 convolutional layer can be used to map the signal coding features to a channel dimension that matches the input expectations of subsequent modules of the encoder, thereby achieving deep fusion of cross-modal information.

[0117] The method described in this application adaptively assigns fusion weights to time-domain and frequency-domain features through a weight allocation subnetwork, performs weighted calculations through a computation subnetwork, and then merges features through a concatenation subnetwork, achieving flexible and efficient time-frequency feature fusion. Compared with the simple addition or concatenation in traditional methods, this method can dynamically adjust the contribution ratio of each domain according to the local time-frequency characteristics of the audio signal, making the final generated signal coding features more discriminative and information-complete, thereby significantly improving the modeling ability of the neural network encoder for complex audio signals.

[0118] In some embodiments, a specific implementation of temporal feature extraction is also provided. The "using a neural network encoder to extract temporal features from the audio signal to be encoded to obtain a temporal feature map with a first time resolution" in S201 above includes: using the temporal feature extraction branch in the neural network encoder to extract temporal features from the audio signal to be encoded to obtain a temporal feature map with a first time resolution.

[0119] The neural network encoder also includes a temporal feature extraction branch, used to extract temporal features from the original audio waveform. This branch comprises N cascaded coding sub-networks, each consisting of a downsampling layer, a receptive field enhancement layer, a residual layer, an activation layer, and a normalization layer. These N sub-networks are stacked in a cascaded manner to progressively reduce the temporal resolution and expand the channel dimension, thereby extracting audio structural features at different time scales. Specifically, the downsampling layer reduces the temporal resolution of the input feature sequence while proportionally expanding the channel dimension to achieve a temporal-channel dimension tradeoff; the receptive field enhancement layer expands the network's receptive field without further reducing the temporal resolution, enabling the network to capture contextual dependencies over a longer span; the residual layer alleviates the vanishing gradient problem in deep networks and enhances feature representation; the activation layer introduces nonlinear transformations to enhance the network's expressive power; and the normalization layer stabilizes the feature distribution and accelerates model convergence.

[0120] In this embodiment, the encoder can input the audio signal to be encoded into N cascaded coding sub-networks of the time-domain feature extraction branch for step-by-step feature extraction, thereby gradually reducing the time resolution and proportionally expanding the channel dimension to obtain a time-domain feature map with the first time resolution.

[0121] In each coding sub-network, the audio signal to be encoded (in the first coding sub-network) and the features output by the previous coding sub-network are input to the downsampling layer for downsampling and channel expansion to obtain a preliminary feature map. Then, the preliminary feature map is input to the receptive field enhancement layer for receptive field enhancement to obtain a local temporal feature map. Next, the audio signal to be encoded is input to the channel mapping layer to obtain a mapped temporal feature map. Then, the local temporal feature map and the mapped temporal feature map are input to the residual layer for residual connection to obtain a residual enhanced feature map. Finally, the residual enhanced feature map is input to the activation layer, and a nonlinear transformation is performed using the ReLU function to obtain a nonlinear feature map. Then, the nonlinear feature map is input to the normalization layer, and a temporal feature is obtained by normalization along the channel dimension. The temporal feature output by the last coding sub-network is determined as the temporal feature map of the first time resolution mentioned above.

[0122] For example, the temporal feature extraction branch includes N=4 cascaded coding sub-networks, and the receptive field enhancement layer uses a depthwise separable convolutional layer with a kernel size of k=7 and an inflation factor d of d=3^i according to the layer index i (i=0,…,N-1) to illustrate the above temporal feature extraction process:

[0123] The audio signal to be encoded The signal, with a shape of (1, T), is input into the first-level encoding sub-network. First, a downsampled convolution (Conv1D, kernel size = 3, stride = 2, output channels = 32) is performed on the signal to obtain a preliminary feature map with a shape of (32, T / 2). Then, the preliminary feature map (32, T / 2) obtained in the previous step is input into the receptive field enhancement layer (kernel size = 7, dilation factor d = 1, stride = 1) to obtain a local temporal feature map with a shape still of (32, T / 2). Finally, the audio signal to be encoded is... After passing through a 1x1 convolutional channel mapping layer, the number of channels is mapped from 1 to 32 to obtain a mapped temporal feature map. Then, the local temporal feature map and the mapped temporal feature map are input into the residual layer and added element-wise to obtain a residual enhancement feature map with a shape of (32, T / 2). Finally, the residual enhancement feature map is input into the activation layer and passed through the ReLU activation function to obtain a nonlinear transformation map with a shape of (32, T / 2). Then, the nonlinear feature map is input into the normalization layer and normalized along the channel dimension to obtain the first-level temporal feature output by the first-level encoding subnetwork, with a shape of (32, T / 2).

[0124] Then, the first-level temporal feature with shape (32, T / 2) is input into the second-level encoding sub-network and downsampled convolution is performed (Conv1D kernel 3, step 2, output channels = 64) to obtain a preliminary feature map with shape (64, T / 4). The process of the first-level encoding sub-network is repeated to obtain the second-level temporal feature with shapes (32, T / 2) and (64, T / 4). After iterating through 4 encoding sub-networks, the final temporal feature map with the first time resolution is obtained, with shape (256, T / 16).

[0125] The method described in this application embodiment achieves efficient hierarchical feature extraction of temporal waveforms by constructing a cascaded coding subnetwork comprising a downsampling layer, a receptive field enhancement layer, a residual layer, an activation layer, and a normalization layer. The downsampling layer progressively reduces the temporal resolution, enabling deeper networks to capture structural information at larger time scales. The receptive field enhancement layer employs exponentially increasing dilated convolutions, significantly expanding the network's receptive field while maintaining computational efficiency. The residual layer effectively alleviates the vanishing gradient problem through residual connections, allowing deeper networks to train stably and enhancing feature representation capabilities. The combination of the activation and normalization layers introduces nonlinearity and stabilizes the training process. The combined effect of these structures ensures that the temporal feature extraction branch generates high-quality temporal feature maps that retain fine transient information while possessing context-aware capabilities.

[0126] In some embodiments, a specific implementation of frequency domain feature extraction is also provided. The "using a neural network encoder to extract frequency domain features from the audio signal to obtain a frequency domain feature map with a second time resolution" in S201 above includes: using the frequency domain feature extraction branch in the neural network encoder to extract frequency domain features from the audio signal to obtain a frequency domain feature map with a second time resolution.

[0127] The neural network encoder also includes a frequency domain feature extraction branch, used to extract frequency domain features from the audio signal. This branch comprises a multi-resolution Fourier transform layer, a logarithmic spectrum calculation layer, and a complex convolutional layer. The multi-resolution Fourier transform layer generates complex spectrum matrices with different time-frequency resolutions, the logarithmic spectrum calculation layer performs a logarithmic transformation on the amplitude of the complex spectrum matrix, and the complex convolutional layer performs a complex convolution operation on the complex spectrum matrix.

[0128] In this embodiment, the encoder inputs the audio signal to a multi-resolution Fourier transform layer, performing short-time Fourier transforms on the audio signal using various window lengths to generate complex spectrum matrices with different time-frequency resolutions. Then, these complex spectrum matrices with different time-frequency resolutions are input to a logarithmic spectrum calculation layer to perform a logarithmic transform on the amplitude of the complex spectrum matrices, obtaining logarithmic power spectrum features. Next, the logarithmic power spectrum features are input to a complex convolution layer to perform a complex convolution operation on the complex spectrum matrix, simultaneously modeling the amplitude and phase information of the spectrum, resulting in multiple frequency domain feature maps with different resolutions. Finally, the multiple frequency domain feature maps with different resolutions are integrated to obtain a frequency domain feature map with a second time resolution.

[0129] For example, the encoder can use a Hann window function to perform a short-time Fourier transform (STFT), where the window length W = {64, 128, 256, 512, 1024} and the overlap ratio R = 75%. Then, the frame shifts are set to {16, 32, 64, 128, 256} to generate complex spectrum matrices with different time-frequency resolutions. Then, after processing through a logarithmic spectrum calculation layer and a complex convolutional layer, a one-dimensional convolution is used to integrate the features to obtain the frequency domain feature map at the second time resolution. Where C is the number of frequency channels, For time resolution.

[0130] The method described in this application, by constructing a frequency domain feature extraction branch comprising a multi-resolution Fourier transform layer, a logarithmic spectrum calculation layer, and a complex convolutional layer, achieves refined modeling of the spectral characteristics of audio signals. The multi-resolution Fourier transform layer generates spectra with different time-frequency resolutions through various window lengths, enabling the network to simultaneously capture both the broadband spectral envelope and the fine spectral structure. The logarithmic spectrum calculation layer transforms the amplitude to the logarithmic domain, which better aligns with the characteristics of human auditory perception. The complex convolutional layer overcomes the limitation of traditional real-number neural networks that can only process amplitude, enabling the network to simultaneously learn the joint features of amplitude and phase, thereby preserving more complete frequency domain information. The combined effect of these structures allows the frequency domain feature extraction branch to generate a frequency domain feature map that accurately characterizes the audio spectrum distribution.

[0131] In some embodiments, such as Figure 5 As shown, a method for constructing alignment networks and fusion networks is also provided, including:

[0132] S501, obtain the original time-domain feature map and the original frequency-domain feature map of the original audio signal.

[0133] The original time-domain feature map is obtained by the time-domain feature extraction branch after extracting time-domain features from the original audio signal, and has a first time resolution. The original frequency-domain feature map is obtained by the frequency-domain feature extraction branch after extracting frequency-domain features from the original audio signal, and has a second time resolution, which is lower than the first time resolution.

[0134] In this embodiment, the original audio signal is first acquired and input into pre-trained temporal feature extraction and frequency domain feature extraction branches. The temporal feature extraction branch outputs the original temporal feature map, and the frequency domain feature extraction branch outputs the original frequency domain feature map. The temporal and frequency domain feature extraction branches can be pre-trained feature extraction networks with fixed parameters, or they can be trainable networks jointly trained with alignment and fusion networks. In one example, the original audio signal is a clean, uncompressed, or undistorted reference audio signal used to supervise the training process.

[0135] S502, the temporal resolution of the frequency domain feature map and the temporal resolution of the time domain feature map are aligned according to the initial alignment network to obtain the aligned original frequency domain feature map.

[0136] The initial alignment network is the alignment network to be trained, and its parameters are initialized to random or pre-trained values. The alignment process is used to improve the second temporal resolution of the original frequency domain feature map to be the same as the first temporal resolution of the temporal domain feature map.

[0137] In this embodiment, the original time-domain feature map and the original frequency-domain feature map are input into an initial alignment network. The initial alignment network includes an initial prediction sub-network, an initial mapping sub-network, and an initial interpolation sub-network. First, the initial prediction sub-network analyzes the feature similarity between the time-domain feature map and the frequency-domain feature map within a local time window, predicting the time offset at each time position in the frequency-domain feature map. Then, the initial mapping sub-network maps the time coordinates of each time position in the frequency-domain feature map to a target time axis based on each time offset, obtaining mapped coordinates; the target time axis is the time axis corresponding to the first time resolution. Finally, the initial interpolation sub-network performs interpolation sampling on the frequency-domain feature map based on each mapped coordinate, obtaining the aligned original frequency-domain feature map. This aligned original frequency-domain feature map has the same first time resolution as the time-domain feature map.

[0138] S503, based on the initial fusion network, fuses the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain the fused feature.

[0139] The initial fusion network is the fusion network to be trained, and its parameters are initialized to random values ​​or pre-trained values. The fusion feature is a composite feature representation obtained by weighted fusion and concatenation of time-domain features and frequency-domain features.

[0140] In this embodiment, the original temporal feature map and the aligned original frequency domain feature map are input into the initial fusion network. The initial fusion network includes an initial weight allocation subnetwork, an initial computation subnetwork, and an initial splicing subnetwork. The initial weight allocation subnetwork assigns a first weight and a second weight to the temporal feature map and the aligned frequency domain feature map. Specifically, it includes two learnable scalar parameters α and β, where α + β = 1. The initial computation subnetwork performs weight calculation based on the first weight and the temporal feature map to obtain an intermediate temporal feature map, i.e., intermediate temporal feature map = α × temporal feature map; it also performs weight calculation based on the second weight and the aligned frequency domain feature map to obtain an intermediate frequency domain feature map, i.e., intermediate frequency domain feature map = β × aligned frequency domain feature map. The initial splicing subnetwork splices the feature vectors of the intermediate temporal feature map and the intermediate frequency domain feature map that are located at the same time position, and performs cross-modal fusion through 1×1 convolution to obtain fused features.

[0141] S504, the initial alignment network and the initial fusion network are trained based on the fusion features and the desired audio signal to obtain the trained alignment network and the trained fusion network.

[0142] The expected audio signal is a reference audio signal used as a supervision signal during training, which is used to calculate the loss function to guide the network parameter update.

[0143] In this embodiment, the reconstruction loss between the original audio signal and the desired audio signal is calculated. Then, the gradient is calculated based on the reconstruction loss, and the learnable parameters in the initial alignment network and the initial fusion network (including the convolution kernel parameters of the prediction subnetwork, the coordinate mapping related parameters in the mapping subnetwork, the convolution kernel parameters of the interpolation subnetwork, and the weight coefficients α, β, and 1×1 convolution kernel parameters in the fusion network) are updated using the backpropagation algorithm. Through multiple rounds of iterative training until the loss function converges, the trained alignment network and the trained fusion network are obtained.

[0144] Optionally, such as Figure 6 As shown, the above S704 includes:

[0145] S5041 compresses the fused features step by step using K cascaded residual vector quantizers, and inputs the compressed features into the decoding network for signal reconstruction to obtain the desired audio signal.

[0146] The residual vector quantizer (RVQ) is used to discretize and compress the fused features. K cascaded residual vector quantizers quantize the quantization residuals of the previous stage one by one to achieve compact representation at different bit rates. The decoding network is used to reconstruct the quantized features into an audio waveform.

[0147] In this embodiment, the fused features are input into K cascaded residual vector quantizers, where K ≥ 2, and the value of K is determined according to the bit rate requirements of the actual application. The first-stage quantizer quantizes the fused features to obtain the quantized features and residuals, the second-stage quantizer quantizes the residuals, and so on. After step-by-step compression, a compact discrete code is obtained. Then, this discrete code is input into a decoding network (the decoding network is basically symmetrical to the encoder structure, replacing the downsampling one-dimensional convolutional block with a transposed one-dimensional convolution for upsampling), and finally outputs the desired audio signal. Optionally, the hyperbolic tangent Tanh function can be used to limit the waveform range to [-1, 1] before the decoder output waveform.

[0148] S5042 determines the target loss based on the original audio signal and the desired audio signal.

[0149] The target loss measures the difference between the desired audio signal and the original audio signal and serves as the objective function guiding network parameter optimization. The target loss includes at least one of the following: time-domain waveform loss, multi-scale spectral loss, and multi-scale Mel frequency cepstral coefficient (MFCC) or Mel filter bank characteristic (FBANK) loss.

[0150] In this embodiment, at least one of the following is calculated between the original audio signal and the desired audio signal: multi-scale time-domain waveform loss, multi-scale spectral loss, and multi-scale MFCC or FBANK loss. Specifically, the mean square error of the waveform at different time scales is calculated using multi-scale time-domain waveform loss, the difference in spectral amplitude after short-time Fourier transform is calculated using multi-scale spectral loss, and the difference in perceptual domain features is calculated using multi-scale MFCC or FBANK loss. The target loss is obtained by weighted summation of the various losses.

[0151] S5043, adjust the parameters in the initial alignment network and the initial fusion network according to the target loss to obtain the trained alignment network and the trained fusion network.

[0152] In this embodiment, the gradient of the target loss with respect to the learnable parameters in the initial alignment network and the initial fusion network is calculated using the backpropagation algorithm, and the parameters are updated using an optimizer. S501 to S504 are repeated until the target loss converges or reaches a preset number of training epochs. At this point, the parameters of the alignment network and the fusion network are saved, resulting in the trained alignment network and the trained fusion network.

[0153] The method described in this application constructs a trainable architecture comprising an alignment network and a fusion network, and utilizes a residual vector quantizer and a decoding network for end-to-end joint optimization. This enables the alignment network to learn a content-adaptive non-uniform temporal alignment strategy, and the fusion network to learn the optimal time-frequency weight allocation ratio. Compared to training each module separately, the end-to-end training approach ensures that the alignment and fusion objectives directly serve the final audio reconstruction quality, thereby achieving superior overall encoding and decoding performance. Furthermore, by combining multi-scale temporal, frequency, and perceptual domain loss functions, the network learning can be supervised from different dimensions, improving both the subjective and objective quality of the reconstructed audio.

[0154] In summary, based on all the above embodiments, an audio signal encoding method is also provided, which includes training phases S601-S606 and application phases S607-S614:

[0155] S601, obtain the original time-domain feature map and the original frequency-domain feature map of the original audio signal.

[0156] S602, the temporal resolution of the frequency domain feature map and the temporal resolution of the time domain feature map are aligned according to the initial alignment network to obtain the aligned original frequency domain feature map.

[0157] S603, based on the initial fusion network, fuse the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain the fused feature.

[0158] S604 compresses the fused features step by step using K cascaded residual vector quantizers, and inputs the compressed features into the decoding network for signal reconstruction to obtain the desired audio signal.

[0159] S605 determines the target loss based on the original audio signal and the desired audio signal.

[0160] S606, adjust the parameters in the initial alignment network and the initial fusion network according to the target loss to obtain the trained alignment network and the trained fusion network, and form a neural network encoder from the trained alignment network and the trained fusion network.

[0161] S607 utilizes the temporal feature extraction branch in the neural network encoder to extract temporal features from the audio signal to be encoded, obtaining a temporal feature map with a first time resolution. The temporal feature extraction branch comprises N cascaded coding sub-networks, each including a downsampling layer, a receptive field enhancement layer, and a feature residual enhancement layer.

[0162] S608 utilizes the frequency domain feature extraction branch in the neural network encoder to extract frequency domain features from the audio signal, obtaining a frequency domain feature map with a second time resolution. The frequency domain feature extraction branch includes a multi-resolution Fourier transform layer, a logarithmic spectrum calculation layer, and a complex convolutional layer.

[0163] S609 analyzes the feature similarity between the time domain feature map and the frequency domain feature map within a local time window through the prediction sub-network in the preset alignment network, and predicts the time offset of each time position in the frequency domain feature map.

[0164] S610, through the mapping sub-network in the alignment network, maps the time coordinates of each time position in the frequency domain feature map to the target time axis according to each time offset, thus obtaining the mapped coordinates of each time position in the frequency domain feature map on the target time axis. The target time axis is the time axis corresponding to the first time resolution.

[0165] S611 uses an interpolation subnetwork to interpolate and sample the frequency domain feature map based on each mapped coordinate to obtain the aligned frequency domain feature map.

[0166] S612, through the weight allocation sub-network in the preset fusion network, allocates the weights at different time positions in the time domain feature map and the aligned frequency domain feature map to obtain the first weight and the second weight.

[0167] S613, through the computational subnetwork in the fusion network, weight calculation is performed based on the first weight and the time-domain feature map to obtain an intermediate time-domain feature map, and weight calculation is performed based on the second weight and the aligned frequency-domain feature map to obtain an intermediate frequency-domain feature map.

[0168] S614 uses a splicing sub-network in the fusion network to splice the feature vectors of the intermediate time-domain feature map and the intermediate frequency-domain feature map that are located at the same time position to obtain the signal coding features.

[0169] The methods described in each of the above steps have been described in the foregoing embodiments. For details, please refer to the foregoing descriptions. They will not be repeated here.

[0170] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0171] Based on the same inventive concept, this application also provides an audio signal encoding apparatus for implementing the audio signal encoding method described above. This apparatus can be applied to or integrated into a chip or chip module, for example. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more audio signal encoding apparatus embodiments provided below can be found in the limitations of the audio signal encoding method described above, and will not be repeated here.

[0172] In some embodiments, such as Figure 7 As shown, an audio signal encoding device is provided, comprising:

[0173] The feature extraction module 11 is used to extract time-domain features from the audio signal to be encoded using a neural network encoder to obtain a time-domain feature map with a first time resolution; and to extract frequency-domain features from the audio signal using a neural network encoder to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution.

[0174] Alignment module 12 is used to align the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map to obtain an aligned frequency domain feature map, which has the first time resolution.

[0175] The fusion module 13 is used to fuse the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain the signal coding features.

[0176] In some embodiments, the alignment module described above includes:

[0177] The prediction unit is used to analyze the feature similarity between the time domain feature map and the frequency domain feature map within a local time window through the prediction sub-network in the preset alignment network, and predict the time offset of each time position in the frequency domain feature map.

[0178] The mapping unit is used to map the time coordinates of each time position in the frequency domain feature map to the target time axis according to each time offset by the mapping sub-network in the alignment network, so as to obtain the mapped coordinates of each time position in the frequency domain feature map on the target time axis; the target time axis is the time axis corresponding to the first time resolution.

[0179] The determining unit is used to determine the aligned frequency domain feature map based on each mapping coordinate and the frequency domain feature map.

[0180] In some embodiments, the determining unit is specifically used to perform interpolation sampling on the frequency domain feature map according to each mapping coordinate through the interpolation processing sub-network to obtain the aligned frequency domain feature map.

[0181] In some embodiments, the fusion module includes:

[0182] The allocation unit is used to allocate weights at different time positions in the time-domain feature map and the aligned frequency-domain feature map through the weight allocation sub-network in the preset fusion network to obtain the first weight and the second weight.

[0183] The computational unit is used to perform weight calculation based on the first weight and the time-domain feature map through the computational sub-network in the fusion network to obtain an intermediate time-domain feature map, and to perform weight calculation based on the second weight and the aligned frequency-domain feature map to obtain an intermediate frequency-domain feature map.

[0184] The splicing unit is used to splice the feature vectors of the intermediate time-domain feature map and the intermediate frequency-domain feature map at the same time position through the splicing sub-network in the fusion network to obtain the signal coding features.

[0185] In some embodiments, the feature extraction module described above includes:

[0186] The first feature extraction unit is used to extract temporal features from the audio signal to be encoded using the temporal feature extraction branch in the neural network encoder, so as to obtain a temporal feature map with a first time resolution. The temporal feature extraction branch includes N cascaded coding sub-networks, and each coding sub-network includes a downsampling layer, a receptive field enhancement layer and a feature residual enhancement layer.

[0187] In some embodiments, the feature extraction module further includes:

[0188] The second feature extraction is used to extract frequency domain features from the audio signal using the frequency domain feature extraction branch in the neural network encoder, to obtain a frequency domain feature map with a second time resolution; the frequency domain feature extraction branch includes a multi-resolution Fourier transform layer, a logarithmic spectrum calculation layer, and a complex convolutional layer.

[0189] In some embodiments, the audio signal encoding device further includes:

[0190] The first training module is used to obtain the original time-domain feature map and the original frequency-domain feature map of the original audio signal;

[0191] The second training module is used to align the temporal resolution of the frequency domain feature map and the temporal resolution of the time domain feature map according to the initial alignment network, so as to obtain the aligned original frequency domain feature map.

[0192] The third training module is used to fuse the feature vectors at the same time position in the temporal feature map and the aligned frequency domain feature map according to the initial fusion network to obtain the fused features.

[0193] The fourth training module is used to train the initial alignment network and the initial fusion network based on the fusion features and the desired audio signal, so as to obtain the trained alignment network and the trained fusion network.

[0194] In some embodiments, the fourth training module is specifically used to compress the fused features step by step according to K cascaded residual vector quantizers, and input the compressed features into the decoding network for signal reconstruction to obtain the desired audio signal; determine the target loss according to the original audio signal and the desired audio signal; and adjust the parameters in the initial alignment network and the initial fusion network according to the target loss to obtain the trained alignment network and the trained fusion network.

[0195] Each module in the aforementioned audio signal encoding device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in a computer device, or stored in software within the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0196] Regarding the modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or a combination of both. For example, for various devices and products applied to or integrated into a chip, all of their modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits; for various devices and products applied to or integrated into a chip module, all of their modules / units can be implemented using hardware methods such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0197] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an audio signal encoding method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0198] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specifically, the computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0199] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the audio signal encoding method described in any of the above embodiments.

[0200] Based on the same inventive concept, this application also provides a chip, including a processor and a communication interface; the communication interface is used to receive or send data; the processor is configured to cause the chip to perform steps of the audio signal encoding method described in any of the above embodiments.

[0201] It is understood that the chip involved in the embodiments of this application may be a field-programmable gate array (FPGA), may be an application-specific integrated circuit (ASIC), may be a system on chip (SoC), may be a central processor unit (CPU), may be a network processor (NP), may be a digital signal processor (DSP), may be a microcontroller unit (MCU), may be a programmable logic device (PLD), or other integrated chips, etc.

[0202] Based on the same inventive concept, this application also provides a chip module, such as... Figure 9 As shown, the chip module includes a communication module, a power module, a storage module, and a chip. Among them:

[0203] The power module is used to provide power to the chip module; the storage module is used to store data and instructions; the communication module is used for internal communication within the chip module, or for communication between the chip module and external devices; this chip corresponds to the chip in the above chip embodiment.

[0204] The implementation of this chip module can be found in the relevant content of the above chip embodiment, and will not be repeated here.

[0205] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the audio signal encoding method described in any of the above embodiments.

[0206] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the audio signal encoding method described in any of the above embodiments.

[0207] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0208] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0209] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for encoding audio signals, characterized in that, The method includes: A neural network encoder is used to extract time-domain features from the audio signal to be encoded to obtain a time-domain feature map with a first time resolution; and the neural network encoder is used to extract frequency-domain features from the audio signal to obtain a frequency-domain feature map with a second time resolution; the first time resolution is higher than the second time resolution. Alignment processing is performed on the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map to obtain an aligned frequency domain feature map, wherein the aligned frequency domain feature map has the first time resolution; The feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map are fused to obtain the signal coding features.

2. The method according to claim 1, characterized in that, The step of aligning the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map to obtain the aligned frequency domain feature map includes: By using the prediction subnetwork in the preset alignment network, the feature similarity between the time domain feature map and the frequency domain feature map within a local time window is analyzed, and the time offset of each time position in the frequency domain feature map is predicted. Through the mapping sub-network in the alignment network, the time coordinates of each time position in the frequency domain feature map are mapped to the target time axis according to each time offset, so as to obtain the mapped coordinates of each time position in the frequency domain feature map on the target time axis; the target time axis is the time axis corresponding to the first time resolution; The aligned frequency domain feature map is determined based on the mapping coordinates and the frequency domain feature map.

3. The method according to claim 2, characterized in that, The alignment network further includes a difference processing sub-network, wherein determining the aligned frequency domain feature map based on each of the mapped coordinates and the frequency domain feature map includes: The frequency domain feature map is interpolated and sampled according to the mapping coordinates through the interpolation subnetwork to obtain the aligned frequency domain feature map.

4. The method according to any one of claims 1-3, characterized in that, The step of fusing the feature vectors located at the same time position in the time-domain feature map and the aligned frequency-domain feature map to obtain signal coding features includes: The weights at different time positions in the time-domain feature map and the aligned frequency-domain feature map are allocated through the weight allocation subnetwork in the preset fusion network to obtain the first weight and the second weight. Through the computational subnetwork in the fusion network, weight calculation is performed based on the first weight and the time-domain feature map to obtain an intermediate time-domain feature map, and weight calculation is performed based on the second weight and the aligned frequency-domain feature map to obtain an intermediate frequency-domain feature map. The signal coding features are obtained by splicing the feature vectors of the intermediate time-domain feature map and the intermediate frequency-domain feature map at the same time position through the splicing sub-network in the fusion network.

5. The method according to any one of claims 1-3, characterized in that, The step of using a neural network encoder to extract temporal features from the audio signal to be encoded to obtain a temporal feature map with a first time resolution includes: The temporal feature extraction branch in the neural network encoder is used to extract temporal features from the audio signal to be encoded to obtain a temporal feature map with a first time resolution. The temporal feature extraction branch includes N cascaded coding sub-networks, and each coding sub-network includes a downsampling layer, a receptive field enhancement layer, and a feature residual enhancement layer.

6. The method according to any one of claims 1-3, characterized in that, The step of extracting frequency domain features from the audio signal using the neural network encoder to obtain a frequency domain feature map with a second time resolution includes: The frequency domain feature extraction branch in the neural network encoder is used to extract frequency domain features from the audio signal to obtain a frequency domain feature map with a second time resolution; the frequency domain feature extraction branch includes a multi-resolution Fourier transform layer, a logarithmic spectrum calculation layer, and a complex convolutional layer.

7. The method according to any one of claims 1-3, characterized in that, The construction process of aligned networks and fused networks includes: Obtain the original time-domain feature map and the original frequency-domain feature map of the original audio signal; The temporal resolution of the frequency domain feature map and the temporal resolution of the time domain feature map are aligned according to the initial alignment network to obtain the aligned original frequency domain feature map. The initial fusion network is used to fuse the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain the fused feature. The initial alignment network and the initial fusion network are trained based on the fusion features and the desired audio signal to obtain the trained alignment network and the trained fusion network.

8. The method according to claim 7, characterized in that, The step of training the initial alignment network and the initial fusion network based on the fusion features and the desired audio signal to obtain the trained alignment network and the trained fusion network includes: The fused features are compressed step by step using K cascaded residual vector quantizers, and the compressed features are input into the decoding network for signal reconstruction to obtain the desired audio signal. The target loss is determined based on the original audio signal and the desired audio signal; The parameters in the initial alignment network and the initial fusion network are adjusted according to the target loss to obtain the trained alignment network and the trained fusion network.

9. An audio signal encoding device, characterized in that, The device includes: The feature extraction module is used to extract time-domain features from the audio signal to be encoded using a neural network encoder to obtain a time-domain feature map with a first time resolution; and to extract frequency-domain features from the audio signal using the neural network encoder to obtain a frequency-domain feature map with a second time resolution; wherein the first time resolution is higher than the second time resolution. An alignment module is used to align the second time resolution of the frequency domain feature map and the first time resolution of the time domain feature map to obtain an aligned frequency domain feature map, wherein the aligned frequency domain feature map has the first time resolution. The fusion module is used to fuse the feature vectors located at the same time position in the time domain feature map and the aligned frequency domain feature map to obtain signal coding features.

10. A chip, characterized in that, It includes a processor and a communication interface, the processor being configured to cause the chip to perform the steps of the method described in any one of claims 1-8.

Images