Neural network audio encoding and decoding method for low-delay Bluetooth headset

Abstract

The invention relates to the technical field of neural network audio coding and decoding methods, in particular to a neural network audio coding and decoding method of a low-delay Bluetooth headset, which comprises the following steps of: audio signal coding: taking an audio signal as coding input, designing a double-flow network, extracting frequency characteristics and audio characteristics, and decoding the audio signal; selecting a corresponding coding and decoding network, and taking the coded audio signal as decoder input; the state parameters comprise an audio signal input frame number and an average amplitude variation; the decoder adopts a knowledge distillation mode, carries out decoding through a method of combining shallow layer features and deep layer features, combines a signal output by the decoder with an audio signal input by an original signal, inputs the combined signal into a loudspeaker, can realize decoding of the audio signal, and realizes decoding of the audio signal through ingenious bidirectional processing and a gating mechanism. And more context information is captured while low delay is kept, so that the audio processing quality and efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network audio encoding and decoding methods, and specifically to a neural network audio encoding and decoding method for a low-latency Bluetooth headset. Background Art

[0002] In recent years, with the rapid development of wireless audio technology, Bluetooth headsets have become an indispensable electronic device in people's daily lives. However, with the continuous improvement of users' requirements for sound quality and user experience, the limitations of traditional Bluetooth headsets in audio codec technology have become increasingly prominent.

[0003] The current mainstream Bluetooth headset audio codec method usually adopts a fixed codec strategy, which often makes it difficult to balance sound quality and latency when processing different types of audio signals. For example, when processing complex music signals, in order to ensure sound quality, it often leads to higher latency; and when processing voice signals, although low latency can be achieved, the sound quality may be affected. This "one-size-fits-all" processing method is difficult to meet the diverse needs of users and complex and changing usage scenarios.

[0004] In addition, existing technologies often ignore the dynamic characteristics and contextual information of audio when processing audio signals. This makes it difficult to accurately capture and reproduce the subtle changes and layering of audio when processing certain audio containing rich dynamic changes, such as classical music or multi-person conversations. At the same time, due to the lack of full use of contextual information, when processing continuous audio streams, problems such as incoherent sound quality or loss of details often occur.

[0005] In addition, existing codec methods also face challenges in terms of computational efficiency. Although many high-quality codec algorithms can provide excellent sound quality, their computational complexity is often high, which is a serious problem for Bluetooth headsets with limited battery capacity. Excessive computational complexity will not only increase the power consumption of the device and shorten the battery life, but may also cause the device to heat up severely, affecting the user experience.

[0006] In view of the above problems, the industry urgently needs a new audio codec method that can strike a balance between sound quality, latency and computational efficiency. This method should be able to dynamically adjust the codec strategy according to different audio features and usage scenarios, while making full use of the dynamic characteristics and contextual information of the audio, and minimize latency and computational complexity while ensuring high sound quality. Summary of the invention

[0007] The present invention proposes a neural network audio codec method for low-latency Bluetooth headsets in response to the above technical problems. By introducing a dynamic codec strategy based on a neural network, the present invention realizes intelligent processing of different types of audio signals. Combined with the innovative dual-stream network structure and hybrid feature attention mechanism, the method can effectively capture the dynamic characteristics and contextual information of audio signals, thereby maintaining excellent performance in various complex scenarios.

[0008] The present invention proposes a neural network audio encoding and decoding method for a low-latency Bluetooth headset, comprising the following steps: audio signal encoding: taking an audio signal as encoding input, designing a dual-stream network, extracting frequency features and audio features, and sending them to the encoding network to realize encoding of the input audio signal; the dual-stream network adopts a method of combining dynamic audio features and static audio features to extract static audio features through a convolutional layer; in the encoder, a hybrid feature attention unit is used to enhance the dynamic audio features and static audio features extracted by the dual-stream network; a hole convolution is introduced in the decoder part to enrich the receptive field; the encoder and the decoder are connected in series, and the decoded audio signal is the encoded audio signal;

[0009] Audio signal decoding: According to the state parameters, the corresponding codec network is selected, and the encoded audio signal is used as the decoder input; the state parameters include the number of audio signal input frames and the average amplitude change; the decoder adopts the knowledge distillation method, and decodes by combining shallow features with deep features. The signal output by the decoder is merged with the audio signal of the original signal input and input into the speaker to realize the decoding of the audio signal.

[0010] Preferably, the dual-flow network is specifically constructed as follows:

[0011] A one-dimensional convolution with a convolution kernel size of 3x3x3 and 32 channels is used to obtain the static features of the audio signal, and a one-dimensional group convolution with a convolution kernel size of 3x3x5 and 32 channels is used to enhance the dynamic frequency features. At the concatenation of the residuals, the features obtained after grouping are mixed across groups, and the cross features are rearranged in shape through a one-dimensional convolution with a convolution kernel size of 3x3x1 and 32 channels, and then all mixed feature channels are summed up and activated by the RELU function. After that, the mixed audio signal features are concatenated and flattened as the input of the next fully connected layer. In the encoder, the encoding result is output, and a mixed feature attention unit is designed to extract different time domain features. In the decoder part, a hole convolution is introduced to enrich the receptive field and improve the model's extraction effect on time domain features.

[0012] Preferably, the dynamic implementation mode of the corresponding codec network is selected according to the state parameter:

[0013] 1) When inputting for the first time, set it as the default network; set the threshold TH of the average amplitude change. If the value of the average amplitude change is greater than TH, use a high-quality codec network; if the value of the average amplitude change is not greater than TH, determine the codec network based on the number of audio signal input frames T. The number of audio signal input frames T satisfies:

[0014] When aT is greater than or equal to 2, select a low-latency codec network;

[0015] When bT is equal to 1, a low-rate codec network is selected;

[0016] 2) After the first input, when the state parameter changes, select the corresponding codec network: set the audio signal input frame number T and the audio input frame number threshold TH; when T is less than or equal to TH, select a low-rate codec network; when T is greater than TH, determine the codec network based on the average amplitude change; set the average amplitude change threshold TH1, when the average amplitude change is greater than TH1, use a high-quality codec network, otherwise use a low-latency codec network.

[0017] Preferably, the low-rate codec network, the low-latency codec network, and the high-quality codec network are specifically constructed as follows:

[0018] A one-dimensional residual full convolutional network is used to linearly map the input audio signal from a high dimension to a low dimension. The dimension of the mapped audio signal is 1 / 4 of the original input, that is, the input audio signal is reduced in dimension. Then, the mapped audio signal is input into a DBRNN network. The DBRNN network includes N layers of bidirectional GRU networks, each of which is composed of two GRU sublayers connected, and GRU joint training is performed between sublayers. The subpath is input between the 1st step and the Nth step, and the forward and reverse paths are jointly input between the N+1th step and the i-th step. The input of the lth step is x l and h l-1 , and a gate mechanism is used to control the signal flow. The gate mechanism is calculated as: g l =σ(W g *[x l ,h l-1 ]+b g ), where g l represents the gate value of the lth step, σ is the sigmoid activation function, W g is the gating weight matrix, x l is the input of step l, h l -1 is the hidden state of the previous step, b g is the bias term;

[0019] From step 1 to step N, subpath (traditional convolution) training is adopted. In layers N+1 to i, both forward and reverse signals are used and controlled by gate mechanism. From step N+1 to i, the output is from the input layer, and the GRU network passes the two input signals to the output layer. A hybrid attention unit is adopted to decide whether to pass the signals through the input and gate and the output and gate. Finally, convolution, layer normalization, ReLU activation function and normalized exponential function are used to output the final reconstructed audio signal.

[0020] Preferably, the input of the subpath is adopted, and the input function of the subpath is:

[0021] h l =f(W h *x l +U h *h l-1 +b h )

[0022] Among them, h l represents the output of step l, f is the activation function, W h is the input weight matrix, x l is the input of step l, U h is the cyclic weight matrix, h l -1 is the hidden state of the previous step, b h is the bias term;

[0023] The joint input of the forward and reverse paths is adopted, and the joint input function of the forward and reverse paths is:

[0024] h l =f(W f *x l +U f *h l-1 +W b *x i-l+1 +U b *h i-l+2 +b h )

[0025] Among them, h l represents the output of step l, and f is the activation function W f and W b are the forward and reverse input weight matrices, x l and x i-l+1 The forward and reverse inputs U f and U b are the forward and reverse cyclic weight matrices, h l-1 and h i-l+2 are the previous hidden states of the forward and reverse directions, respectively, and b h is the bias term, and i is the total number of steps.

[0026] Preferably, the low-latency codec network adopts a gate mechanism to jointly train the input signal, divides the input signal into an upstream signal and an output signal, introduces a gate mechanism to jointly control the signal flow, and the gate mechanism is controlled as follows: the first layer to the Nth layer are jointly trained under the guidance of the sub-path signal, the N+1 to the i-th layer adopts bidirectional input, and the signal flow is controlled by the gate mechanism, and finally at the output, a mixed attention unit is used to decide whether to increase the output signal.

[0027] Preferably, the low-latency codec network is specifically constructed as follows:

[0028] A one-dimensional residual fully convolutional network is used to linearly map the input audio signal from a high dimension to a low dimension. The mapped audio signal is passed to the output layer through the GRU network for reconstruction. The reconstructed audio signal is y t ; Whether to transfer is controlled by the input and output AND gates. The input AND gate is calculated as:

[0029] i t =σ(W i *[x t ,h t-1 ]+b i )

[0030] Among them, i t represents the value of the input gate, σ is the sigmoid activation function, W i is the weight matrix of the input gate, x t is the input h of the current time step t-1 is the hidden state of the previous time step, b i is the bias term;

[0031] The output AND gate is calculated as:

[0032] o t =σ(W o *[y t ,h t ]+b o )

[0033] Among them, t represents the value of the output gate, σ is the sigmoid activation function, W o is the weight matrix of the output gate, y t is the output of the current time step, h t is the hidden state of the current time step, b o is the bias term.

[0034] Preferably, the input audio processing and the output audio processing are implemented by the input gate and the output gate respectively; the input gate determines whether to increase the input of the previous time step, and its calculation is:

[0035] i t =σ(W i *x t +U i *h t-1 +b i )

[0036] Among them, i t represents the value of the input gate, σ is the sigmoid activation function, W i is the weight matrix of the input gate, x t is the input U of the current time step i is the cyclic weight matrix, h t-1 is the hidden state of the previous time step, b i is the bias term;

[0037] The output gate determines whether to pass the time step signal to the next time step. The calculation of the output gate is:

[0038] o t =σ(W o *y t +U o *h t +b o )

[0039] Among them, t represents the value of the output gate, σ is the sigmoid activation function, W o is the weight matrix of the output gate, y t is the output of the current time step, U o is the cyclic weight matrix, h t is the hidden state of the current time step, b o is the bias term.

[0040] Preferably, in the low-latency audio signal processing, the feedforward activation module is composed of N layers of neural networks, and the output of the first step is h l , calculated as:

[0041] h l =f(W l *x l +b l )

[0042] Among them, h l represents the output of step l, f is the activation function, W l is the weight matrix, x l is the input of step l, b l is the bias term;

[0043] From step N+1 to step i, a mixed feature attention module is used, and the output signal mixture is expressed as:

[0044] h l =f(W l *(i l ⊙x l )+b l )

[0045] Among them, h l represents the output of step l, f is the activation function, W l is the weight matrix, i l represents the input gate function, ⊙ represents the Hadamard product (element-wise multiplication), x l represents the mixed feature input signal, b l is the bias term.

[0046] Preferably, the audio signal decoding is specifically implemented as follows:

[0047] 1) Initialization: When playing for the first time, set to a high-quality codec; after the first play, decide whether to perform codec network transformation based on the number of audio signal input frames and the average amplitude change, and then decide whether to transform based on the average amplitude change; when the state parameters change during the first play and after the first play, transform the codec network based on the state parameters;

[0048] 2) When the state parameter changes: define a threshold TH of the number of audio signal input frames; the number of audio signal input frames meets the following conditions:

[0049] a. When the number of audio signal input frames T is greater than or equal to TH, it is transformed into the low-latency codec network,

[0050] b. When the number of audio signal input frames T is less than or equal to TH, the low-rate codec network is selected;

[0051] When the number of audio signal input frames meets the above condition a, determine the current decoder state and set the average amplitude fluctuation thresholds TH1 and TH2; when the average amplitude fluctuation is greater than TH1 or less than TH2, use the high-quality codec network for decoding, otherwise select the low-latency codec network for decoding, convert the signal dimension through one-dimensional convolution, convert it to a data size of 1 / 4, and then input it into the DBRNN network; the DBRNN network includes N bidirectional layers, each bidirectional layer is composed of two GRU sublayers stacked, and GRUs are jointly trained between sublayers; after the convolution is transformed into a dimensionality reduction, a subpath input is performed between steps 1 to N, and a forward and reverse input is performed between steps N+1 to i; a gate mechanism is used to control the flow of the signal, and its calculation is as follows: The subpath input is calculated as: hl =f(W s *x l +U s *h l-1 +b s ), where h l represents the output of step l, f is the activation function, W s is the weight matrix of the subpath, x l is the input of step l, U s is the cyclic weight matrix of the subpath, h l-1 is the hidden state of the previous step, b s is the bias term of the subpath.

[0052] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0053] The core of the present invention lies in its dynamic selection mechanism and improved DBRNN structure. The dynamic selection mechanism allows the system to adjust the encoding and decoding strategy in real time according to the audio characteristics and network conditions, effectively solving the problem that the traditional fixed encoding and decoding method is difficult to adapt to different audio types. The improved DBRNN structure captures more contextual information while maintaining low latency through clever two-way processing and gating mechanisms, effectively improving the quality and efficiency of audio processing.

[0054] This innovative approach brings significant improvements in many aspects. First, it greatly improves audio quality, providing clearer and more natural sound effects for both speech and music. Second, it significantly reduces the latency of audio processing, improving real-time performance, which is especially important in latency-sensitive scenarios such as online games or video calls. Finally, while improving sound quality and reducing latency, this method also optimizes computational efficiency, which is of great significance for resource-constrained Bluetooth headsets and can effectively extend the battery life of devices.

[0055] In general, the present invention provides a new solution for audio processing of low-latency Bluetooth headsets. It not only achieves comprehensive improvements in technical indicators, but more importantly, these improvements are directly related to the optimization of user experience. By balancing audio quality, latency and computing efficiency, the present invention points the way for the development of next-generation Bluetooth audio devices, and is expected to bring users an unprecedented high-quality listening experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a logic block diagram of the method of the present invention.

[0057] Figure 2 A logical block diagram selected for the codec network of the present invention.

[0058] Figure 3This is a logic block diagram of the processing of the DBRNN network of the present invention. DETAILED DESCRIPTION

[0059] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0060] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0061] Example 1

[0062] See also Figure 1-3 The present invention relates to a neural network audio codec method for low-latency Bluetooth headsets. In Bluetooth headset applications, the codec quality and delay of audio signals are crucial to user experience. The method proposed in the present invention aims to balance audio quality and delay to provide users with a better listening experience.

[0063] First, the core of the present invention lies in its audio signal encoding and decoding process. In the encoding stage, we designed an innovative two-stream network structure. This two-stream network can not only extract frequency features, but also capture the dynamic features of audio. Specifically, the network extracts static audio features through convolutional layers and uses dynamic audio features to comprehensively characterize the input audio signal.

[0064] In the encoder, we introduced a hybrid feature attention unit. The role of this unit is to enhance the dynamic and static features extracted by the two-stream network. In this way, we can highlight important audio features and improve the efficiency and accuracy of encoding. For example, when processing an audio containing human voice and background music, the hybrid feature attention unit can help us better capture the subtle changes in the human voice while not ignoring the overall structure of the background music.

[0065] Secondly, in the decoder part, we introduced dilated convolution. The main advantage of dilated convolution is that it can increase the receptive field without increasing the number of parameters or computational complexity. In the application scenario of Bluetooth headsets, this means that we can capture a wider range of audio context information without significantly increasing latency. For example, when processing a symphony, dilated convolution can help the decoder better understand the relationship between different instruments, thereby reconstructing a more natural and richer audio effect.

[0066] Again, a key feature of the present invention is the dynamic selection of the codec network based on state parameters. These state parameters include the number of audio signal input frames and the average amplitude change. Through this dynamic selection mechanism, we can select the most appropriate codec strategy based on different audio features and network conditions. For example, when it is detected that the average amplitude change of the audio signal is large, the system will automatically switch to a high-quality codec network to ensure high-fidelity reconstruction of complex audio clips. On the contrary, when the audio signal is relatively stable, the system may select a low-latency codec network to reduce delays and improve real-time performance.

[0067] In one embodiment of the present invention, we design the structure of the two-stream network in detail. The network uses convolution kernels of different sizes to extract static and dynamic features respectively. Specifically, we use a 3x3x3 convolution kernel to extract static features, and a 3x3x5 one-dimensional group convolution to enhance dynamic frequency features. This design allows the network to capture audio features at different time scales, thereby more comprehensively understanding the structure of the audio signal.

[0068] Preferably, we introduce cross-group mixing at the residual connection. This mixing strategy rearranges the feature shape through a 3x3x1 one-dimensional convolution, and then sums all the mixed feature channels. The purpose of this step is to enhance the interaction between different features, thereby improving the richness of feature representation. Finally, we use the ReLU activation function to introduce nonlinearity, which helps the network learn more complex feature mappings.

[0069] Next, let's discuss the selection strategy of the codec network in detail. When the first input is received, the system defaults to using a high-quality codec network. This is because in the absence of prior information, it is a safe choice to prioritize sound quality. After that, the system will dynamically adjust the strategy based on the characteristics of the audio signal.

[0070] Specifically, we set two key thresholds: the average amplitude change threshold TH and the audio signal input frame number threshold TH. When the average amplitude change is greater than TH, the system will select a high-quality codec network to ensure accurate reconstruction of complex audio. When the average amplitude change is not greater than TH, the system will further decide based on the number of audio signal input frames T. If T is greater than or equal to 2, the system selects a low-latency codec network; if T is equal to 1, a low-rate codec network is selected.

[0071] The advantage of this strategy is that it can make the best choice based on the real-time characteristics of the audio signal and the network conditions. For example, when processing a fast-paced piece of music, if frequent high-amplitude changes are detected, the system will tend to use a high-quality codec network to ensure that the dynamic range and details of the music are accurately reproduced. On the contrary, when processing a smooth voice signal, the system may choose a low-latency or low-rate network to reduce delays and improve the real-time nature of the call.

[0072] In another embodiment of the present invention, we describe in detail the construction of low-rate, low-latency and high-quality encoding and decoding networks. These networks all use the basic structure of a one-dimensional residual full convolutional network, but they are different in specific implementation.

[0073] First, the role of the one-dimensional residual fully convolutional network is to map the input high-dimensional audio signal to a low-dimensional space. In our implementation, the dimension of the mapped audio signal is 1 / 4 of the original input. This dimensionality reduction operation can not only reduce the amount of computation for subsequent processing, but also help the network extract more abstract and representative features.

[0074] Secondly, we introduced the DBRNN (Deep Bidirectional Recurrent Neural Network) network. The DBRNN network contains N layers of bidirectional GRU (Gated Recurrent Unit) networks, each of which is composed of two GRU sub-layers connected. This structure allows the network to consider both past and future contextual information when processing audio sequences, thereby making more accurate predictions.

[0075] In the DBRNN network, we adopt an innovative training strategy. Between step 1 and step N, we input the subpath, and between step N+1 and step i, we input the joint forward and backward paths. The advantage of this strategy is that it allows the network to focus on different information at different processing stages. For example, in the early stages, the network may pay more attention to local features, while in the later stages, it can comprehensively consider broader contextual information.

[0076] In order to control the information flow, we introduced a gate mechanism. The calculation formula of the gate mechanism is as follows:

[0077] g l =σ(W g *[x l ,h l-1 ]+b g )

[0078] Among them, g l represents the gate value of the lth step, σ is the sigmoid activation function, W g is the gating weight matrix, xl is the input of step l, h l -1 is the hidden state of the previous step, b g This gate mechanism allows the network to dynamically decide which information should be retained and which should be ignored, thereby improving the network's ability to process long sequence data.

[0079] In a preferred embodiment of the present invention, we describe in detail the calculation method of the sub-path input and the forward and reverse path combined input. For the sub-path input, the calculation formula is:

[0080] h l =f(W h *x l +U h *h l-1 +b h )

[0081] Among them, h l represents the output of step l, f is the activation function, W h is the input weight matrix, x l is the input of step l, U h is the cyclic weight matrix, h l -1 is the hidden state of the previous step, b h is the bias term.

[0082] For the combined input of the forward and reverse paths, the calculation formula is:

[0083] h l =f(W f *x l +U f *h l-1 +W b *x i-l+1 +U b *h i-l+2 +b h )

[0084] Among them, h l represents the output of step l, and f is the activation function W f and W b are the forward and reverse input weight matrices, x l and x i-l+1 The forward and reverse inputs U f and U b are the forward and reverse cyclic weight matrices, h l-1 and h i-l+2 are the previous hidden states of the forward and reverse directions, respectively, and b h is the bias term, and i is the total number of steps.

[0085] This bidirectional input design allows the network to consider both past and future information when processing an audio sequence, thereby making more accurate predictions. For example, when processing a piece of speech, this structure can help the network better understand the context and more accurately predict the next phoneme or word.

[0086] In the low-latency codec network, we use a special gate mechanism to jointly train the input signal. Specifically, we divide the input signal into the previous step signal and the output signal, and introduce a gate mechanism to control the signal flow. From the first layer to the Nth layer, the network is jointly trained under the guidance of the subpath signal. From the N+1th layer to the i-th layer, the network uses bidirectional input and controls the signal flow through the gate mechanism.

[0087] The advantage of this design is that it can make full use of contextual information while maintaining low latency. For example, when processing real-time voice calls, this structure can help the network respond quickly to current input while also using short-term historical information to improve sound quality.

[0088] To better understand how the low-latency codec network works, let's take a closer look at the key computational steps. First, the network uses a one-dimensional residual full convolution to map the input audio signal from a high dimension to a low dimension. Then, the mapped audio signal is passed to the output layer for reconstruction through the GRU network. The reconstructed audio signal is represented by y_t.

[0089] In this process, the network uses input gates and output gates to control the transmission of signals. The calculation formula of the input gate is:

[0090] i t =σ(W i *[x t ,h t-1 ]+b i )

[0091] Among them, i t represents the value of the input gate, σ is the sigmoid activation function, W i is the weight matrix of the input gate, x t is the input h of the current time step t-1 is the hidden state of the previous time step, b i is the bias term.

[0092] The calculation formula of the output gate is:

[0093] o t =σ(W o *[y t ,h t ]+b o )

[0094] Among them,t represents the value of the output gate, σ is the sigmoid activation function, W o is the weight matrix of the output gate, y t is the output of the current time step, h t is the hidden state of the current time step, b o is the bias term.

[0095] This gating mechanism allows the network to dynamically decide which information should be retained or updated, thereby capturing important contextual information while maintaining low latency.

[0096] In one embodiment of the present invention, we also describe in detail the feedforward activation module in low-latency audio signal processing. This module consists of N layers of neural networks, and the output calculation formula in step l is:

[0097] h l =f(W l *x l +b l )

[0098] Among them, h l represents the output of step l, f is the activation function, W l is the weight matrix, x l is the input of step l, b l is the bias term.

[0099] From step N+1 to step i, we introduced a mixed feature attention module, and the output signal mixture is expressed as:

[0100] h l =f(W l *(i l ⊙x l )+b l )

[0101] Among them, h l represents the output of step l, f is the activation function, W l is the weight matrix, i l represents the input gate function, ⊙ represents the Hadamard product (element-wise multiplication), x l represents the mixed feature input signal, b l is the bias term.

[0102] The advantage of this design is that it can focus on different features at different processing stages. For example, in the early stages, the network may pay more attention to local features, while in the later stages, it can comprehensively consider broader contextual information. This is particularly helpful for processing complex audio signals such as music or multi-person conversations.

[0103] Finally, let's discuss the specific implementation of audio signal decoding in detail. During the initialization phase, the system is set to a high-quality codec when it first plays. This is to ensure that the sound quality is prioritized in the absence of prior information. After that, the system will decide whether to perform a codec network transformation based on the number of audio signal input frames and the average amplitude change.

[0104] When the state parameters change, the system will select the appropriate codec network according to the following rules:

[0105] First, we define the threshold TH of the number of audio signal input frames. When the number of audio signal input frames T is greater than or equal to TH, the system will switch to a low-latency codec network. This is because more input frames usually mean that there is enough information for low-latency processing. On the contrary, when T is less than or equal to TH, the system will choose a low-rate codec network. This choice is suitable for situations where there is less input information, which can save bandwidth while ensuring basic sound quality.

[0106] Secondly, when the number of audio signal input frames meets the conditions for switching to a low-latency codec network, the system will further determine the current decoder state. We set two average amplitude fluctuation thresholds TH1 and TH2. When the average amplitude fluctuation is greater than TH1 or less than TH2, the system will use a high-quality codec network for decoding. This is because larger amplitude fluctuations usually mean an increase in the complexity of the audio content, requiring higher-quality codecs to ensure sound quality. When the amplitude fluctuation is between TH1 and TH2, the system will choose a low-latency codec network to balance sound quality and latency.

[0107] For example, suppose we set TH = 5, TH1 = 0.8, and TH2 = 0.2. When 5 or more consecutive frames of audio input are detected, the system will consider switching to low-latency mode. Then, if the average amplitude change of this audio is greater than 0.8 or less than 0.2, such as when processing a piece of music with a large dynamic range, the system will choose a high-quality codec network. On the contrary, if the average amplitude change is between 0.2 and 0.8, such as when processing normal voice conversations, the system will choose a low-latency codec network.

[0108] After selecting the codec network, the system will convert the signal dimension to 1 / 4 of the original size through one-dimensional convolution, and then input it into the DBRNN network. This dimensionality reduction process can not only reduce the amount of calculation for subsequent processing, but also help the network extract more abstract and representative features.

[0109] The structure of the DBRNN network is very unique. It contains N layers of bidirectional layers, each of which is composed of two GRU sublayers stacked together. This structure allows the network to consider both past and future contextual information when processing audio sequences, thereby making more accurate predictions. For example, when processing speech, this bidirectional structure can help the network better understand the context of the speech and thus more accurately predict the next phoneme or word.

[0110] In the DBRNN network, we adopt an innovative training strategy. Subpath input is performed between steps 1 to N, and forward and reverse input is performed between steps N+1 to i. The advantage of this strategy is that it allows the network to focus on different information at different processing stages.

[0111] The calculation formula for subpath input is:

[0112] h l =f(W s *x l +U s *h l-1 +b s ), where h l represents the output of step l, f is the activation function, W s is the weight matrix of the subpath, x l is the input of step l, U s is the cyclic weight matrix of the subpath, h l-1 is the hidden state of the previous step, b s is the bias term of the subpath.

[0113] The joint input of the forward and backward paths allows the network to focus more on local features in the early stages, while taking into account broader contextual information in the later stages. For example, when processing a piece of music, the early stages may focus more on the features of a single note, while the later stages can consider the structure of the entire phrase.

[0114] In order to further improve the performance of the network, we introduced input gates and output gates from step N+1 to step i to control the signal flow.

[0115] This gating mechanism allows the network to dynamically decide which information should be retained or updated, thereby capturing important contextual information while maintaining low latency. For example, when processing a piece of speech, the network may choose to retain important speech features while ignoring background noise.

[0116] It is worth noting that we pass the output values ​​of the input gate and output gate as the initial values ​​to the input and output gates of the next layer. This design allows information to be passed more efficiently between different layers of the network, thereby improving overall performance.

[0117] Finally, after the decoder output, we use a series of post-processing steps to further optimize the output signal. This includes convolution, layer normalization, ReLU activation function, and normalized exponential function. The purpose of these steps is to ensure the quality and stability of the output signal. For example, layer normalization can help reduce the differences between different audio samples, while the ReLU activation function can introduce nonlinearity, allowing the network to learn more complex feature maps.

[0118] After processing, the system merges the output signal with the original audio signal and then inputs it into the speakers. This merging step can help preserve certain characteristics of the original audio while using the signal processed by the neural network to improve the overall sound quality.

[0119] In general, the low-latency neural network audio codec method for Bluetooth headsets proposed in the present invention has multiple advantages. First, it can dynamically select the most appropriate codec strategy based on the characteristics of the audio signal and the network conditions, thereby maintaining good performance in different scenarios. Second, by using innovative network structures such as DBRNN and gating mechanisms, this method can capture important contextual information while maintaining low latency, thereby improving audio quality. Finally, the design of this method takes into account the needs of actual application scenarios, such as reducing the amount of computation by reducing dimensionality, which makes it particularly suitable for resource-constrained devices such as Bluetooth headsets.

[0120] In future research, we can consider further optimizing the network structure, such as exploring a more efficient attention mechanism or introducing a new neural network architecture. At the same time, we can also consider extending this method to other audio processing tasks, such as noise removal or audio enhancement. In general, this invention provides a new direction for low-latency audio processing, which is expected to bring significant performance improvements in practical applications.

[0121] When verifying the superiority of our invention, we designed a series of comprehensive tests, focusing on three core indicators: audio quality, latency, and computational complexity. These indicators not only directly reflect our innovations, but are also closely related to user experience.

[0122] First, we selected multiple types of audio samples, including speech, music, and environmental sounds, to ensure the comprehensiveness and representativeness of the test results. In order to objectively evaluate the audio quality, we adopted the PESQ (Perceptual Evaluation of Speech Quality) scoring system widely recognized by the industry. The PESQ score ranges from -0.5 to 4.5, and the higher the score, the better the sound quality. In terms of latency, we use a high-precision timer to measure the entire process time from audio input to output. For computational complexity, we recorded the number of audio samples processed per second, which directly reflects the real-time performance of the algorithm.

[0123] In the best embodiment, we compared the method of the present invention with the traditional fixed encoding and decoding method. The test results are shown in the following table:

[0124]

[0125]

[0126] These data vividly demonstrate the superiority of the present invention. In terms of audio quality, both speech and music, our method has achieved significant improvements. The improvement in PESQ score means that users will experience clearer and more natural sound. Especially in music processing, the 11.1% improvement shows that our method can better preserve the details and dynamic range of music.

[0127] What’s even more exciting is the significant reduction in latency. Compared to traditional methods, we reduced the average latency from 25 milliseconds to 15 milliseconds, a full 40% reduction. This improvement is of great significance in practical applications. For example, in online games or video calls, lower latency can significantly improve the user experience and reduce the unnatural feeling in communication.

[0128] At the same time, our method also shows obvious advantages in computational efficiency. It can process 2,000 samples per second, which is 33.3% higher than the traditional method. This means that under the same hardware conditions, our algorithm can process more audio data, or in other words, it can achieve the same performance on lighter hardware. This is especially important for resource-constrained Bluetooth headsets because it can extend battery life while maintaining excellent audio processing capabilities.

[0129] It is worth mentioning that these excellent results are mainly due to our innovative dynamic codec network selection mechanism and improved DBRNN structure. The dynamic selection mechanism allows the system to adjust the strategy in real time according to audio characteristics and network conditions, maintaining optimal performance in different scenarios. The improved DBRNN structure captures more contextual information while maintaining low latency through clever bidirectional processing and gating mechanisms.

[0130] These test results not only verify the technical superiority of our method, but more importantly, they directly reflect the actual improvements that users can perceive. Higher PESQ scores mean clearer call quality and more immersive music experience. Lower latency can bring a more natural interaction experience. And improved computational efficiency may translate into longer battery life or smaller device design.

[0131] Overall, these test results fully demonstrate that our method has made a significant breakthrough in balancing audio quality, latency, and computational efficiency. It not only surpasses traditional methods in terms of technical indicators, but more importantly, these improvements are directly related to the improvement of user experience. We believe that this comprehensive performance improvement will bring revolutionary changes to the next generation of Bluetooth audio devices and bring users an unprecedented high-quality listening experience.

[0132] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural network audio encoding and decoding method for low-latency Bluetooth headsets, characterized in that : Includes the following steps: Audio signal encoding: The audio signal is used as the encoding input, and a dual-stream network is designed to extract frequency features and audio features, which are sent to the encoding network to realize the encoding of the input audio signal; the dual-stream network adopts a method of combining dynamic audio features and static audio features to extract static audio features through a convolutional layer; in the encoder, a hybrid feature attention unit is used to enhance the dynamic audio features and static audio features extracted by the dual-stream network; a hole convolution is introduced in the decoder part to enrich the receptive field; the encoder and the decoder are connected in series, and the decoded audio signal is the encoded audio signal; Audio signal decoding: According to the state parameters, the corresponding codec network is selected, and the encoded audio signal is used as the decoder input; the state parameters include the number of audio signal input frames and the average amplitude change; the decoder adopts the knowledge distillation method, and decodes by combining shallow features with deep features. The signal output by the decoder is merged with the audio signal of the original signal input and input into the speaker to realize the decoding of the audio signal.

2. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 1, characterized in that The specific construction method of the dual-flow network is as follows: A one-dimensional convolution with a convolution kernel size of 3x3x3 and 32 channels is used to obtain the static features of the audio signal, and a one-dimensional group convolution with a convolution kernel size of 3x3x5 and 32 channels is used to enhance the dynamic frequency features. At the concatenation of the residuals, the features obtained after grouping are mixed across groups, and the cross features are rearranged in shape through a one-dimensional convolution with a convolution kernel size of 3x3x1 and 32 channels, and then all mixed feature channels are summed up and activated by the RELU function. After that, the mixed audio signal features are concatenated and flattened as the input of the next fully connected layer. In the encoder, the encoding result is output, and a mixed feature attention unit is designed to extract different time domain features. In the decoder part, a hole convolution is introduced to enrich the receptive field and improve the model's extraction effect on time domain features.

3. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 1, characterized in that : Select the dynamic implementation method of the corresponding codec network according to the state parameters: 1) When inputting for the first time, set it as the default network; set the threshold TH of the average amplitude change. If the value of the average amplitude change is greater than TH, use a high-quality codec network; if the value of the average amplitude change is not greater than TH, determine the codec network based on the number of audio signal input frames T. The number of audio signal input frames T satisfies: When aT is greater than or equal to 2, select a low-latency codec network; When bT is equal to 1, a low-rate codec network is selected; 2) After the first input, when the state parameter changes, select the corresponding codec network: set the audio signal input frame number T and the audio input frame number threshold TH; when T is less than or equal to TH, select a low-rate codec network; when T is greater than TH, determine the codec network based on the average amplitude change; set the average amplitude change threshold TH1, when the average amplitude change is greater than TH1, use a high-quality codec network, otherwise use a low-latency codec network.

4. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 3, characterized in that : The low-rate codec network, the low-latency codec network, and the high-quality codec network are specifically constructed as follows: A one-dimensional residual full convolutional network is used to linearly map the input audio signal from a high dimension to a low dimension. The dimension of the mapped audio signal is 1 / 4 of the original input, that is, the input audio signal is reduced in dimension. Then, the mapped audio signal is input into a DBRNN network. The DBRNN network includes N layers of bidirectional GRU networks, each of which is composed of two GRU sublayers connected, and GRU joint training is performed between sublayers. The subpath is input between the 1st step and the Nth step, and the forward and reverse paths are jointly input between the N+1th step and the i-th step. The input of the lth step is x l and h l-1 , and a gate mechanism is used to control the signal flow. The gate mechanism is calculated as: g l =σ(W g *[x l ,h l -1]+b g ), where g l represents the gate value of the lth step, σ is the sigmoid activation function, W g is the gating weight matrix, x l is the input of step l, h l -1 is the hidden state of the previous step, b g is the bias term; From step 1 to step N, subpath (traditional convolution) training is adopted. In layers N+1 to i, both forward and reverse signals are used and controlled by gate mechanism. From step N+1 to i, the output is from the input layer, and the GRU network passes the two input signals to the output layer. A hybrid attention unit is adopted to decide whether to pass the signals through the input and gate and the output and gate. Finally, convolution, layer normalization, ReLU activation function and normalized exponential function are used to output the final reconstructed audio signal.

5. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 4, characterized in that : Using the input of the subpath, the input function of the subpath is: h l =f(W h *x l +U h *h l-1 +b h ) Among them, h l represents the output of step l, f is the activation function, W h is the input weight matrix, x l is the input of step l, U h is the cyclic weight matrix, h l -1 is the hidden state of the previous step, b h is the bias term; The joint input of the forward and reverse paths is adopted, and the joint input function of the forward and reverse paths is: h l =f(W f *x l +U f *h l-1 +W b *x i-l+1 +U b *h i-l+2 +b h ) Among them, h l represents the output of step l, and f is the activation function W f and W b are the forward and reverse input weight matrices, x l and x i-l+1 The forward and reverse inputs U f and U b are the forward and reverse cyclic weight matrices, h l-1 and h i-l+2 are the previous hidden states of the forward and reverse directions, respectively, and b h is the bias term, and i is the total number of steps.

6. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 4, characterized in that The low-latency codec network adopts a gate mechanism to jointly train the input signal, divides the input signal into an upstream signal and an output signal, and introduces a gate mechanism to jointly control the signal flow. The gate mechanism is controlled as follows: the first layer to the Nth layer are jointly trained under the guidance of the sub-path signal, and the N+1 to the i-th layer adopts bidirectional input and controls the signal flow through the gate mechanism. Finally, at the output, a mixed attention unit is used to decide whether to increase the output signal.

7. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 6, characterized in that : The low-latency codec network is specifically constructed as follows: A one-dimensional residual fully convolutional network is used to linearly map the input audio signal from a high dimension to a low dimension. The mapped audio signal is passed to the output layer through the GRU network for reconstruction. The reconstructed audio signal is y t ; Whether to transfer is controlled by the input and output AND gates. The input AND gate is calculated as: i t =σ(W i *[x t ,h t-1 ]+b i ) Among them, i t represents the value of the input gate, σ is the sigmoid activation function, W i is the weight matrix of the input gate, x t is the input h of the current time step t-1 is the hidden state of the previous time step, b i is the bias term; The output AND gate is calculated as: the t =σ(W o *[y t ,h t ]+b o ) Among them, t represents the value of the output gate, σ is the sigmoid activation function, W o is the weight matrix of the output gate, y t is the output of the current time step, h t is the hidden state of the current time step, b o is the bias term.

8. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in any one of claims 4 or 6, characterized in that : Input audio processing and output audio processing are implemented by the input gate and output gate respectively; the input gate determines whether to increase the input of the previous time step, and its calculation is: i t =σ(W i *x t +U i *h t-1 +b i ) Among them, i t represents the value of the input gate, σ is the sigmoid activation function, W i is the weight matrix of the input gate, x t is the input U of the current time step i is the cyclic weight matrix, h t-1 is the hidden state of the previous time step, b i is the bias term; The output gate determines whether to pass the time step signal to the next time step. The calculation of the output gate is: o t =σ(W o *y t +U o *h t +b o ) Among them, t represents the value of the output gate, σ is the sigmoid activation function, W o is the weight matrix of the output gate, y t is the output of the current time step, U o is the cyclic weight matrix, h t is the hidden state of the current time step, b o is the bias term.

9. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 4, characterized in that In the low-latency audio signal processing, the feedforward activation module consists of N layers of neural networks, and the output of the first step is h l , calculated as: h l =f(W l *x l +b l ) Among them, h l represents the output of step l, f is the activation function, W l is the weight matrix, x l is the input of step l, b l is the bias term; From step N+1 to step i, a mixed feature attention module is used, and the output signal mixture is expressed as: h l =f(W l *(i l ⊙x l )+b l ) Among them, h l represents the output of step l, f is the activation function, W l is the weight matrix, i l represents the input gate function, ⊙ represents the Hadamard product (element-wise multiplication), x l represents the mixed feature input signal, b l is the bias term.

10. A neural network audio encoding and decoding method for a low-latency Bluetooth headset as described in claim 1, characterized in that : The audio signal decoding is specifically implemented as follows: 1) Initialization: When playing for the first time, set to a high-quality codec; after the first play, decide whether to perform codec network transformation based on the number of audio signal input frames and the average amplitude change, and then decide whether to transform based on the average amplitude change; when the state parameters change during the first play and after the first play, transform the codec network based on the state parameters; 2) When the state parameter changes: define a threshold TH of the number of audio signal input frames; the number of audio signal input frames meets the following conditions: a. When the number of audio signal input frames T is greater than or equal to TH, it is transformed into the low-latency codec network, b. When the number of audio signal input frames T is less than or equal to TH, the low-rate codec network is selected; When the number of audio signal input frames meets the above condition a, determine the current decoder state and set the average amplitude fluctuation thresholds TH1 and TH2; when the average amplitude fluctuation is greater than TH1 or less than TH2, use the high-quality codec network for decoding, otherwise select the low-latency codec network for decoding, convert the signal dimension through one-dimensional convolution, convert it to a data size of 1 / 4, and then input it into the DBRNN network; the DBRNN network includes N bidirectional layers, each bidirectional layer is composed of two GRU sublayers stacked, and GRUs are jointly trained between sublayers; after the convolution is transformed into a dimensionality reduction, a subpath input is performed between steps 1 to N, and a forward and reverse input is performed between steps N+1 to i; a gate mechanism is used to control the flow of the signal, and its calculation is as follows: The subpath input is calculated as: h l =f(W s *x l +U s *h l-1 +b s ), where h l represents the output of step l, f is the activation function, W s is the weight matrix of the subpath, x l is the input of step l, U s is the cyclic weight matrix of the subpath, h l-1 is the hidden state of the previous step, b s is the bias term of the subpath.

Images