Space audio positioning system and open space immersive audio system

Abstract

The invention provides a spatial audio positioning system and an open space immersive audio system. The spatial audio positioning system comprises at least one audio receiving device and at least one processor. The audio receiving device is used for receiving voice information. The processor executes a normalization module and a dense connection network after being loaded into at least one storage device, and the normalization module performs normalization processing on the voice information to obtain normalized voice data after receiving the voice information; and the dense connection network receives the normalized voice data so as to map to spatial information corresponding to the voice information source position based on the normalized voice data. According to the invention, the voice information source of the user is determined, and the customized sound effect experience is provided according to the voice instruction according to the area where the user is located, so that the specific requirements of individual users are better met or different environments and conditions are adapted.

Description

Technical Field

[0001] The present invention relates to a positioning system and an open space immersive audio system, and in particular to a spatial audio positioning system and an open space immersive audio system. Background Art

[0002] General open space environments, such as homes, offices, public areas or transportation hubs, are often susceptible to high levels of background noise, which can have a negative impact on personal comfort, concentration and work efficiency. In order to solve these problems, noise reduction systems have been born. There are two main types of existing noise reduction systems, namely passive noise reduction technology and active noise reduction systems.

[0003] Passive noise reduction techniques, such as sound-absorbing materials or physical barriers, can help reduce noise levels in open spaces. However, these techniques have limited effectiveness and may not be suitable for all environments or situations. Active noise reduction systems work by actively generating cancelling sound waves to eliminate unwanted noise. While these active noise reduction systems may be more effective than passive noise reduction techniques, they generally require the user to wear headphones or other personal audio devices, which may not be convenient or comfortable for all users.

[0004] Furthermore, existing noise reduction systems are not able to optimally meet the specific needs of individual users or adapt to different environments and situations. For the reasons stated above, there is a need for an improved noise reduction system that addresses the limitations of the prior art by providing a more versatile and user-friendly solution to reduce noise in open spaces. Summary of the invention

[0005] The main purpose of the present invention is to provide a spatial audio positioning system, comprising at least one audio receiving device, at least one processor and at least one storage device. The at least one audio receiving device is used to receive a voice message. The at least one processor executes a normalization module and a densely connected network after loading the at least one storage device, wherein: after receiving the voice message, the normalization module normalizes the voice message to obtain normalized voice data; and the densely connected network receives the normalized voice data to map the normalized voice data to the spatial information corresponding to the source position of the voice message based on the normalized voice data.

[0006] Optionally, the at least one processor executes a coordinate mapping module after loading the at least one storage device, and the coordinate mapping module maps the input spatial information to a spatial coordinate.

[0007] Optionally, the densely connected network model architecture from input to output includes:

[0008] An input layer, used for receiving the normalized speech data and inputting the normalized speech data into the model;

[0009] A convolutional layer is used to extract the features of the output data of the previous layer and extract the features;

[0010] A batch normalization layer performs batch normalization on the output data of the previous layer to adjust the distribution of multiple features;

[0011] A scaling factor layer that scales the features of the output data of the previous layer and restores the representation ability of the data by introducing a learnable scaling factor;

[0012] A ReLU layer applies the ReLU activation function to the output data of the previous layer, adjusting negative inputs to 0 and retaining positive inputs;

[0013] A pooling layer, which reduces the spatial size of the output data of the previous layer through pooling operations;

[0014] a first stacked dense block group layer connected to the output of the pooling layer, the first stacked dense block group layer comprising a plurality of first convolutional layers and a first batch normalization layer, the input of each first convolutional layer being connected to the outputs of all other preceding first convolutional layers;

[0015] a first transition layer connected to the output of the first stacked dense block group layer, the first transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map;

[0016] a second stacked dense block group layer connected to the output of the first transition layer, the second stacked dense block group layer comprising a plurality of second convolutional layers and a second batch normalization layer, the input of each second convolutional layer being connected to the outputs of all other previous second convolutional layers;

[0017] a second transition layer connected to the second stacked dense block group layer, the second transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map;

[0018] a third stacked dense block group layer connected to the second stacked dense block group layer, the third stacked dense block group layer comprising a plurality of third convolutional layers and a third batch normalization layer, the input of each third convolutional layer being connected to the outputs of all other preceding third convolutional layers;

[0019] a third transition layer connected to the third stacked dense block group layer, the third transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map;

[0020] a fourth stacked dense block group layer connected to the third transition layer, the fourth stacked dense block group layer comprising a plurality of fourth convolutional layers and a fourth batch normalization layer, the input of each fourth convolutional layer being connected to the outputs of all other preceding fourth convolutional layers; and

[0021] A classification layer is connected to the fourth stacked dense block group layer, and maps the output of the fourth stacked dense block group layer to the spatial information.

[0022] Another object of the present invention is to provide an open space immersive audio system based on spatial positioning, comprising at least one audio receiving device, at least one audio output device, at least one reverse audio output device, at least one processor and at least one storage device. The at least one processor executes a sound elimination module, an intelligent speech recognition module, an audio processing module, a normalization module, a densely connected network, a coordinate mapping module and a processing module after loading the at least one storage device, wherein: the sound elimination module receives the error sound in the non-immersive area through any one of the audio receiving devices, and then generates a reverse audio signal based on the error sound and outputs it to the audio output device to produce a reverse cancellation sound; the audio processing module outputs a playback audio signal to the at least one audio output device to play the sound effect in the immersive area; the intelligent speech recognition module receives voice information from any one of the audio receiving devices, and obtains a voice command by analyzing the voice information. The intelligent speech recognition module includes a waveform neural network or a parallel waveform neural network for generating a real speech signal based on the environmental sound, and a waveform neural network for generating a real speech signal based on the real The invention discloses an MFCC unit for extracting audio features from a speech signal and a speech recognition module for mapping the audio features output by the MFCC unit to the speech command; the normalization module receives speech information from any one of the audio receiving devices, and after receiving the speech information, normalizes the speech information to obtain normalized speech data; the densely connected network receives the normalized speech data from the normalization module to map the normalized speech data to spatial information corresponding to the source position of the speech information; the coordinate mapping module obtains the spatial information from the output of the densely connected network, and maps the input spatial information to a spatial coordinate; the processing module generates a corresponding control instruction based on the user's speech command and the spatial coordinate, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

[0023] Optionally, the speech recognition module is a convolutional neural network model.

[0024] Optionally, the intelligent speech recognition module includes a CTC training unit, which is used to calculate the distance between the output sequence and the label sequence through a concatenated temporal classification loss function, and update the model parameters of the speech recognition module through a back-propagation algorithm.

[0025] Optionally, the densely connected network is trained according to the following process:

[0026] Collecting a training data set containing the voice information and the location of the source of the voice information;

[0027] The voice information is normalized in advance to ensure that the data scale of the voice information is consistent;

[0028] Perform forward propagation through this densely connected network to map the input data to spatial information and calculate the difference between the predicted result and the true label;

[0029] Through the back propagation algorithm, the model parameters of the densely connected network are adjusted according to the loss function so that the prediction results of the densely connected network are closer to the true label;

[0030] updating the model parameters of the densely connected network according to the optimizer to minimize the loss function; and

[0031] Repeat forward propagation, backpropagation, and parameter update operations until the model converges or reaches a predetermined number of training times.

[0032] Optionally, the intelligent voice recognition module triggers a program for parsing the voice information by specifying the voice information or a human-machine interface instruction when in standby mode.

[0033] Optionally, the densely connected network model architecture from input to output includes:

[0034] An input layer, used for receiving the normalized speech data and inputting the normalized speech data into the model;

[0035] A convolutional layer, used to extract features of the output data of the previous layer;

[0036] A batch normalization layer performs batch normalization on the output data of the previous layer to adjust the distribution of multiple features;

[0037] A scaling factor layer that scales the features of the output data of the previous layer and restores the representation ability of the data by introducing a learnable scaling factor;

[0038] A ReLU layer applies the ReLU activation function to the output data of the previous layer, adjusting negative inputs to 0 and retaining positive inputs;

[0039] a first stacked dense block group layer connected to the output of the ReLU layer, the first stacked dense block group layer comprising a plurality of first convolutional layers and a first batch normalization layer, the input of each first convolutional layer being connected to the outputs of all other preceding first convolutional layers;

[0040] a first transition layer connected to the output of the first stacked dense block group layer, the first transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map;

[0041] a second stacked dense block group layer connected to the output of the first transition layer, the second stacked dense block group layer comprising a plurality of second convolutional layers and a second batch normalization layer, the input of each second convolutional layer being connected to the outputs of all other previous second convolutional layers;

[0042] a second transition layer connected to the second stacked dense block group layer, the second transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map;

[0043] a third stacked dense block group layer connected to the second stacked dense block group layer, the third stacked dense block group layer comprising a plurality of third convolutional layers and a third batch normalization layer, the input of each third convolutional layer being connected to the outputs of all other preceding third convolutional layers;

[0044] a third transition layer connected to the third stacked dense block group layer, the third transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map;

[0045] a fourth stacked dense block group layer connected to the third transition layer, the fourth stacked dense block group layer comprising a plurality of fourth convolutional layers and a fourth batch normalization layer, the input of each fourth convolutional layer being connected to the outputs of all other preceding fourth convolutional layers; and

[0046] A classification layer is connected to the fourth stacked dense block group layer, and the output of the fourth stacked dense block group layer is mapped to the spatial information. Therefore, the present invention provides a customized sound experience according to the voice command based on the user's area by determining the source of the user's voice information, thereby better meeting the specific needs of individual users or adapting to different environments and situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 FIG. 1 is a block diagram of an open space immersive audio system based on spatial positioning in the present invention (I).

[0048] Figure 2 FIG2 is a block diagram of an open space immersive audio system based on spatial positioning in the present invention (II).

[0049] Figure 3 FIG. 1 is a block diagram of the intelligent audio adjustment system of the present invention. FIG.

[0050] Figure 4 FIG. 2 is a block diagram of the intelligent audio adjustment system of the present invention. FIG.

[0051] Figure 5 Schematic diagram of the overall architecture of the intelligent speech recognition module in the present invention.

[0052] Figure 6 Schematic diagram of the architecture of the waveform neural network algorithm in the present invention.

[0053] Figure 7 It is a flowchart diagram of the waveform neural network algorithm in the present invention.

[0054] Figure 8 It is a schematic diagram of the architecture of a single node of the waveform neural network in the present invention.

[0055] Fig. 9 Schematic diagram of the architecture of the autoregressive flow model.

[0056] Fig.10 Schematic diagram of the architecture of the inverse autoregressive flow model.

[0057] Fig.11 Schematic diagram of the architecture of the parallel waveform neural network in the present invention.

[0058] Fig.12 It is a training flow chart of the parallel waveform neural network in the present invention.

[0059] Fig.13 Schematic diagram of the calculation process of the MFCC algorithm in the present invention.

[0060] Fig.14 Schematic diagram of the algorithm flow of the Mel filter bank in the present invention.

[0061] Fig.15 It is a schematic diagram of the discrete cosine transform process in the present invention.

[0062] Fig.16 Schematic diagram of the convolutional neural network model architecture used in the speech recognition module of the present invention.

[0063] Fig.17 Schematic diagram of the training process of the CTC trainer in the present invention.

[0064] Fig.18 It is a block diagram of the position detection mechanism in the present invention.

[0065] Fig.19 It is a schematic diagram of the architecture of the densely connected network in the present invention.

[0066] Fig. 20 This is a training flow chart of a densely connected network in the present invention.

[0067] Fig.21 Schematic diagram of the overall operation flow of the open space immersive audio system of the present invention.

[0068] Fig. 22 FIG. 1 is a schematic diagram of the configuration of the immersive audio system of the present invention (I).

[0069] Fig.23 FIG. 2 is a schematic diagram of the configuration of the immersive audio system of the present invention. FIG.

[0070] Fig.24 FIG. 3 is a schematic diagram of the configuration of the immersive audio system in the present invention. FIG.

[0071] Fig.25 Schematic diagram (IV) of the configuration of the immersive audio system in the present invention.

[0072] The following are marked in the figure: 100 Open Space Immersive Audio System

[0073] 10 Audio receiving device

[0074] 11 Microphone

[0075] 12 Preamplifier

[0076] 13 Anti-aliasing filter

[0077] 14 Analog-to-digital converter

[0078] 20 Audio output device

[0079] 21 Speakers

[0080] 22 Power Amplifier

[0081] 23 Reconstruction filter

[0082] 24-bit digital-to-analog converter

[0083] 30 Reverse audio output device

[0084] 31 Speakers

[0085] 32 Power Amplifier

[0086] 33 Reconstruction filter

[0087] 34 Digital-to-analog converter

[0088] 40 Intelligent audio adjustment system

[0089] 41 Processor

[0090] 42 Storage Devices

[0091] M1 Sound Cancellation Module

[0092] M2 Audio Processing Module

[0093] M3 Intelligent Voice Recognition Module

[0094] M31 Real Voice Signal Conversion Module

[0095] B1 Causal Convolutional Layer

[0096] B2 dilated convolutional layer

[0097] B3 tanh function

[0098] B4 sigmoid function

[0099] B5 Multiplication Unit

[0100] B6 The first convolutional layer

[0101] B7 Addition Unit

[0102] B8 Addition Unit

[0103] B9 ReLU unit

[0104] B10 Second convolutional layer

[0105] B11 ReLU unit

[0106] B12 The third convolutional layer

[0107] B13 Softmax

[0108] M311 Student Waveform Neural Network

[0109] M312 Mentor Waveform Neural Network

[0110] M32 MFCC Unit

[0111] M33 Voice Recognition Module

[0112] LA1 convolutional layer

[0113] LA2 Pooling Layer

[0114] LA3 fully connected layer

[0115] M34 CTC Trainer

[0116] M4 Normalization Module

[0117] M5 Densely Connected Network

[0118] M501 Input Layer

[0119] M502 convolutional layer

[0120] M503 Batch Normalization Layer

[0121] M504 Scaling Factor Layer

[0122] M505 ReLU layer

[0123] M506 Pooling Layer

[0124] M507 The first stacked dense block group layer

[0125] M508 First transition layer

[0126] M509 Second stacked dense block group layer

[0127] M510 Second transition layer

[0128] M511 Third stacked dense block group layer

[0129] M512 Third transition layer

[0130] M513 Fourth stacked dense block group layer

[0131] M514 Classification Layer

[0132] M6 Coordinate Mapping Module

[0133] M7 Processing Module

[0134] AP Audio Device

[0135] A Immersive Space

[0136] A1 Immersive Space

[0137] A2 Immersive Space

[0138] B Non-immersive space

[0139] D Non-immersive space

[0140] L1 reverses the direction of the audio output

[0141] L2 reverses the direction of the audio output

[0142] PH Virtual Face

[0143] J1 Autoregressive Flow Model

[0144] J2 Inverse Autoregressive Flow Model DETAILED DESCRIPTION

[0145] The detailed description and technical content of the present invention are described as follows with reference to the accompanying drawings. The "one side" mentioned in the present invention can be the upper side, lower side, left side, right side, front side, rear side of the corresponding object, or be arranged at any adjacent position of the object, or be directly or indirectly connected to the object, etc., which is not limited in the present invention.

[0146] The "device", "module", "model", "unit" and the corresponding functions described in the present invention can be performed by a single chip or a combination of multiple chips, and the number of such chip configurations is not within the scope of the present invention. In addition, the chip can be, but is not limited to, a combination of processors (Processor), central processing unit (CPU), microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) and other devices, which are not limited in the present invention.

[0147] The "immersive space" described in the present invention refers to the target area of ​​the sound effect in the playback environment, that is, the area where the sound effect is expected; the "non-immersive space" described in the present invention refers to the area in the environment where the sound is not expected to be transmitted, which can be any area other than the "immersive space". In another embodiment, the "non-immersive space" can also be a specific direction where the volume is expected to be reduced in order to improve the directivity of the playback sound, and the stereo effect in the playback environment is improved by reducing the volume in the specific direction.

[0148] The immersive audio system disclosed in the present invention can be applied to audio devices such as televisions, laptops, mobile devices, soundbars, portable devices, or car head units. The immersive audio system controls the sound output direction or range of the audio devices, thereby reducing the impact of spillover sound on non-target areas.

[0149] The following is a description of one embodiment of the present invention. Figure 1 and Figure 2 , which are the block diagram (1) and block diagram (2) of the open space immersive audio system based on spatial positioning in the present invention, as shown in the figure: the open space immersive audio system 100 based on spatial positioning of this embodiment mainly includes an audio receiving device 10, an audio output device 20, a reverse audio output device 30 and an intelligent audio adjustment system 40.

[0150] The audio receiving device 10 is used to receive ambient sound. The audio receiving device 10 is used to receive error sound in a non-immersive space and convert it into error audio; in another embodiment, another audio receiving device may be further included to specifically receive ambient sound in an immersive space, which is not limited in the present invention. In another embodiment, another audio receiving device may be further included to specifically receive voice commands in an immersive space or a non-immersive space, which is not limited in the present invention. In one embodiment, the audio receiving device 10 includes a microphone 11, a preamplifier 12 connected to the rear end of the microphone 11, an anti-aliasing filter 13 connected to the rear end of the preamplifier 12, and an analog-to-digital converter 14 connected to the rear end of the anti-aliasing filter 13. These devices can be increased or decreased according to practical needs, which is not limited in the present invention.

[0151] The audio output device 20 outputs the playback sound according to the audio source signal. The "audio source signal" mentioned herein refers to the original sound effect signal provided by the intelligent audio adjustment system 40, that is, the original file of the sound effect. The "playback sound" is the sound effect broadcast by the audio output device 20 in the real field environment (the space included in the immersive area), which is first explained here. In one embodiment, the audio output device 20 includes a speaker 21, a power amplifier 22 connected to the front end of the speaker 21, a reconstruction filter 23 connected to the front end of the power amplifier 22, and a digital-to-analog converter 24 connected to the front end of the reconstruction filter 23. These devices can be increased or decreased according to practical needs, and are not limited in the present invention.

[0152] The reverse audio output device 30 outputs reverse cancellation sound according to the reverse audio. The "reverse audio" mentioned here refers to the original signal provided by the intelligent audio adjustment system 40 for canceling the sound played in the non-immersive area. The "reverse cancellation sound" is the sound effect broadcast by the reverse audio output device 30 in the real field environment (the space included in the non-immersive area), which is first explained here. In one embodiment, the reverse audio output device 30 includes a speaker 31, a power amplifier 32 connected to the front end of the speaker 31, a reconstruction filter 33 connected to the front end of the power amplifier 32, and a digital-to-analog converter 34 connected to the front end of the reconstruction filter 33. These devices can be increased or decreased according to practical needs, and are not limited in the present invention.

[0153] The intelligent audio adjustment system 40 is connected to the audio receiving device 10, the audio output device 20 and the reverse audio output device 30 respectively, and controls the output of the audio output device 20 and the reverse audio output device 30 through the error sound and voice command received by the audio receiving device 10. The intelligent audio adjustment system 40 includes a processor 41 and a storage device 42. The processor 41 executes the corresponding program after loading the program code stored in the storage device 42.

[0154] For the work performed by the intelligent audio adjustment system 40, please refer to the following Figure 3 and Figure 4 , which are the block diagram (I) and block diagram (II) of the intelligent audio adjustment system of the present invention, as shown in the figure: the intelligent audio adjustment system 40 of the present invention includes a sound elimination module M1, an audio processing module M2, an intelligent voice recognition module M3, a normalization module M4, a dense connection network M5, a coordinate mapping module M6 and a processing module M7 according to different executed programs. The normalization module M4, the dense connection network M5, and the coordinate mapping module M6 executed by the audio receiving device 10 and the processor 41 and the storage device 42 together constitute a spatial audio positioning system.

[0155] After the sound cancellation module M1 receives the error sound in the non-immersive area via the audio receiving device 10, it generates an inverted audio (Inverted Audio) based on the error sound and outputs it to the audio output device 20 to produce an inverted cancellation sound. In one embodiment, the reference audio may include, for example, a sound source signal or a playback sound received by the audio receiving device 10 in the immersive area, which is not limited in the present invention. The sound cancellation module M1 may directly receive the playback audio from the microphone 11, and adjust the phase of the playback audio to generate an inverted audio with the same amplitude but opposite phase relative to the playback sound. Specifically, the sound cancellation module M1 may be, for example, but not limited to, completed by phase inversion, fast Fourier transform (FFT), time domain convolution, audio decoding and encoding, neural network (such as machine learning network, deep learning network or recurrent neural network, etc.) or other similar methods, which are not limited in the present invention.

[0156] The audio processing module M2 is used to output the played audio to the audio output device 20, so as to play the sound in the immersive area. Specifically, the audio processing module M2 outputs the audio source signal to the audio output device 20 after reading the long-term data or temporary data of any storage device, so as to play the sound through the audio output device 20. In one embodiment, the audio processing module M2 can be, for example, any sound card, onboard sound card or sound processing chip, etc. Such sound cards include, for example, decoders, digital to analog converters (Digital to Analog Converter, DAC), analog to digital converters (Analog to Digital Converter), amplifiers, filters, sound synthesizers, etc., which are not limited in the present invention.

[0157] The intelligent voice recognition module M3 is used to obtain the user's voice information from the ambient sound after the audio receiving device 10 receives the ambient sound, and obtain the voice command by analyzing the voice information. For the architecture of the intelligent voice recognition module M3, please refer to Figure 5 , which is a schematic diagram of the overall architecture of the intelligent speech recognition module in the present invention, as shown in the figure: the intelligent speech recognition module M3 in this embodiment mainly includes a real speech signal conversion module M31, an MFCC unit M32, a speech recognition module M33 and a CTC trainer M34.

[0158] The real speech signal conversion module M31 is used to generate a real speech signal based on the environmental sound, the MFCC unit M32 is used to extract audio features from the real speech signal, the speech recognition module M33 is used to map the audio features output by the MFCC unit M32 to speech commands, and the CTC training unit M34 is used to calculate the distance between the output sequence and the label sequence through the connection time classification loss function (CTC loss), and update the model parameters of the speech recognition module model M33 through the back propagation algorithm. Among them, the real speech signal conversion module M31 can be a waveform neural network (WaveNet) or a parallel waveform neural network (Parallel WaveNet).

[0159] Regarding the embodiment of the real speech signal conversion module M31 as a waveform neural network, the architecture of the waveform neural network can be found in Figure 6, is a schematic diagram of the architecture of the waveform neural network algorithm in the present invention, as shown in the figure: the waveform neural network uses dilated causal convolution, which can expand the observation range, but also takes into account the causal relationship of the signal. As shown in the figure, the dilation rate used in the waveform neural network is 1, 2, 4, 8, ..., when the dilation rate is 512 (only 8 is shown in the figure), it will continue to cycle from 1, and a group of 1 to 512 requires 1024 inputs. Taking the signal with a sampling frequency of 16kHz as an example, it can focus on about 64ms of information, which can be quite helpful for the accuracy of the speech signal model. The waveform neural network makes full use of causal and dilated convolution, and uses a series of small (length = 2) convolution filters with exponentially increasing dilation factors. Therefore, the receptive field of the network increases exponentially with the increase of the network depth. The causal relationship is achieved by asymmetric padding proportional to the dilation factor, which prevents the activation from propagating back in time.

[0160] For the calculation steps of the waveform neural network, please refer to Figure 7 , which is a flow chart of the waveform neural network algorithm in the present invention, as shown in the figure: The waveform neural network (WaveNet) in the present invention is a speech generation model that uses a deep convolutional neural network to process audio features extracted from environmental sounds to generate more realistic real speech signals.

[0161] Step S01: Initialize the Wavenet model. In this step, relevant parameters are set for the Wavenet model to prepare for subsequent network structure and calculation.

[0162] The following are some of the main parameters.

[0163] Batch size: During the training process, dividing the data into multiple small batches for training helps speed up the training process and improve the generalization ability of the model. The batch size refers to the number of samples contained in each batch.

[0164] Filter width: Filter width refers to the size of the convolution kernel in the time dimension. In the Wavenet model, the filter width is usually set to 2, which means that the convolution kernel can cover two consecutive input samples in the time dimension.

[0165] Hidden size: The hidden size refers to the number of neurons in each layer of the network. This parameter determines the network's representation ability and computational effort.

[0166] Number of layers: In the WaveNet model, the number of layers determines the depth of the network. Each layer contains a causal dilation convolution layer. The increase in the number of layers can increase the receptive field of the network, allowing the model to capture longer time series dependencies.

[0167] Dilation rate: The dilation rate in the dilated convolution affects the step size of the convolution kernel in the time dimension. In the waveform neural network model, the dilation rate of each layer is usually set to an exponential power of 2, which can achieve an exponential expansion of the receptive field.

[0168] Step S02: Create and initialize bias variables. For example, the receptive field size bias, etc. In this step, some bias variables are created for the waveform neural network model. The bias variables play a translation role in the neural network. Even if all inputs are zero, the bias variables can activate neurons.

[0169] Here are some of the main variance variables.

[0170] Convolution bias: These bias variables are added to the output of the convolutional layer to increase the activation of neurons. Each convolutional layer needs to create a bias variable with the same number of output channels as the layer.

[0171] Receptive field size bias: In the waveform neural network model, the receptive field will expand as the number of layers increases. The dilation rate of each causal dilation convolution layer can be regarded as the determining factor of the receptive field size of that layer.

[0172] Global conditioning bias: If the waveform neural network model uses global conditioning, a bias variable needs to be created for the global conditioning. This bias variable is added to the global conditioning to help the network generate outputs based on global information.

[0173] When creating bias variables, you need to make sure that their shapes match the corresponding network structure. Usually, the bias variables are initialized to zero or a small value close to zero. This ensures that the activation of the network is not too large or too small at the beginning of training, which helps improve the stability of training.

[0174] Step S03: Create variables required by the network, such as filter variables, dilation variables, etc. In this step, we need to create some variables for the waveform neural network model to store and adjust the weights required for the convolution layer, dilation layer, etc.

[0175] Here are some of the main variables.

[0176] Filter variables: Each convolutional layer and dilation layer needs to create a filter variable to store its weights. The shape of the weight matrix depends on the size of the convolution kernel, the number of input channels, and the number of output channels. The weight matrix is ​​usually continuously updated during training to minimize the loss.

[0177] Dilation variables: The dilated convolution layer requires a dilation factor to control the span of the dilated convolution kernel on the input. The dilation factor for each layer may be different and is an integer power of 2. These dilation variables are usually set at model initialization and remain unchanged throughout the training process.

[0178] Gate variable: For expansion layers that use gate control units, you need to create a gate variable to store the weights of the gate unit. The gate unit is used to control which information in the output of the convolutional layer can flow to the next layer. The gate variable is similar to the filter variable, has the corresponding shape and is updated during training.

[0179] Global Condition Variable: If the waveform neural network model uses global conditions, you need to create a variable for the global condition. This variable is used to learn the mapping from global information to output, and its shape depends on the dimension of the global condition and the number of output channels.

[0180] When creating these variables, you need to make sure their shapes match the corresponding network structures. These variables are usually initialized with random variables, such as Gaussian distribution or uniform distribution, to ensure that the network has enough diversity at the beginning of training.

[0181] Step S04: Create a causal convolution layer. The causal convolution layer is one of the basic components of the waveform neural network model, ensuring that only past information is used when processing time series data. In the causal convolution layer, the convolution operation will not go beyond the current moment, thus avoiding the use of future data.

[0182] Following are the main steps to create a causal convolutional layer.

[0183] Define the convolution kernel size: Choose an appropriate convolution kernel size to capture local features in the input data. The size of the convolution kernel affects the receptive field and number of parameters of the model.

[0184] Define the number of input and output channels: Set the number of input channels (matching the number of features of the input data) and the number of output channels (used to adjust the representation capacity of the model) for the causal convolution layer.

[0185] Initialize convolution weights and biases: Initialize the weights and biases of the convolution layer according to the convolution kernel size, the number of input channels, and the number of output channels. These weights and biases are continuously updated during training to minimize the loss.

[0186] Perform causal convolution operation: Perform causal convolution operation on the input data, which means that only the current and previous data are used during the convolution process. To implement causal convolution, an appropriate amount of zero padding can be added to the beginning of the input data so that each output element is calculated from the corresponding input element and the previous element.

[0187] Activation function: Apply an activation function such as ReLU, tanh, or other nonlinear functions to the output of the convolution operation to increase the representation power of the model.

[0188] After completing the above steps, a causal convolution layer is created. In the waveform neural network model, multiple causal convolution layers and causal dilated convolution layers are used to capture features at different time scales.

[0189] Step S05: Create a causal dilated convolution layer. The causal dilated convolution layer includes the establishment of filter unit layers and gate control units to achieve the purpose of increasing the receptive field. The causal dilated convolution layer is the core component of the waveform neural network model, which is used to capture long-range temporal dependencies. In this step, create a causal dilated convolution layer for the model.

[0190] Here are the main steps.

[0191] Definition of dilation rate: The dilation rate is an integer that represents the distance between two adjacent convolution kernels during the convolution process. The larger the dilation rate, the wider the receptive field, and the longer range of temporal dependencies can be captured.

[0192] Initialize the dilated convolution weights and biases: Initialize the weights and biases of the dilated convolution layer according to the convolution kernel size, the number of input channels, the number of output channels, and the dilation rate. These weights and biases will be continuously updated during training to minimize the loss.

[0193] Create a filter layer and a gate layer: These two layers are responsible for capturing different features in the input data. The filter layer is responsible for performing convolution operations on the input data to extract features, while the gate layer is responsible for controlling the speed at which information flows through the network.

[0194] Perform causal dilated convolution: Perform a causal dilated convolution on the input data, ensuring that only the current and previous data are used. To implement causal dilated convolution, an appropriate amount of zero padding can be added to the beginning of the input data. The amount of zero padding depends on the kernel size and the dilation rate.

[0195] Activation functions for filter and gate units: Activation functions such as tanh and sigmoid are applied to the outputs of filter and gate units respectively. These non-linear activation functions help enhance the representation capabilities of the model.

[0196] Element-wise multiplication and addition: The outputs of the filter unit and the gate unit are multiplied element-wise and the result is added to the original input to form a residual connection. The residual connection helps in the training and gradient flow of the network.

[0197] Finally, a causal dilated convolution layer is created in the waveform neural network model. The main function of this layer is to capture long-range temporal dependencies and extract and control different features in the input data through filtering units and gating units. This layer also includes residual connections, which helps with model training and gradient flow. In practical applications, multiple causal dilated convolution layers can be stacked as needed to improve the representation ability of the model.

[0198] The following are the sub-steps for one-hot encoding of waveform amplitudes. Figure 8 , is a schematic diagram of the architecture of a single node of the waveform neural network in the present invention, as shown in the figure: Since the original audio is usually stored as a string of 16-bit integer values ​​(one for each time period), a softmax layer needs to output 65,536 probabilities in each time period to simulate all possible values. In order to make this problem easier to solve, the data is first subjected to μ-law compression conversion (ITU-T, 1988) and then quantized into 256 possible values. The input signal is a waveform of a sound signal, which will oscillate back and forth between +1 and -1. Due to the desired output form, the waveform of a sound signal will also oscillate back and forth between +1 and -1. If a function such as ReLU is used whose value range does not include 0 to -1, the output will be truncated in the part <0. Therefore, it is hoped to find an Activation function with a value range of -1 to +1, preferably with positive and negative symmetry. That is, an odd function, and tanh is used here.

[0199] About Figure 8 The functions of each block are described as follows:

[0200] Causal convolution layer B1: The causal convolution layer B1 ensures that the model is causal in time, and the input layer maps the audio signal into features.

[0201] Dilated convolution layer B2: Use multiple dilated convolution layers B2 to extract features, obtain features at different resolutions and enhance the receptive field of the model.

[0202] tanh function B3: The output range of tanh function B3 is between -1 and +1. It is used to capture activation features in audio data. The role of tanh function B3 is to provide richer expression capabilities in capturing the amplitude changes and waveform details of the signal, because its output range covers negative and positive values, and it can better represent the oscillation of the waveform between positive and negative amplitudes.

[0203] Sigmoid function B4 (σ): The output of the sigmoid function B4 ranges from 0 to 1 and is used in the gating mechanism to learn how to combine features between different layers. In the context of the WaveNet model, the role of the sigmoid function B4 is to pass or suppress certain features as needed to implement an adaptive gating mechanism.

[0204] Multiplication unit B5: Multiplication unit B5 multiplies the outputs of tanh function B3 and sigmoid function B4 element by element, so as to realize the adaptive gating mechanism between network layers.

[0205] The first convolution layer B6 (1×1): The output after multiplication passes through a 1×1 convolution layer. This layer can realize nonlinear transformation of features. In the residual block, the output of the tanh function B3 and the sigmoid function B4 is element-wise multiplied and then passes through the first convolution layer B6. The purpose of this convolution layer is to perform nonlinear transformation on the output to obtain a new feature representation. This process is performed inside the residual block and is used to learn linear and nonlinear combinations between features.

[0206] Addition unit B7 (+): The addition operation can help the network learn the identity mapping, which is helpful for training deep networks. The output of the 1×1 convolution is added to the original input to form a residual output. This addition operation can help the network learn the identity mapping, which is helpful for training deep networks.

[0207] Addition unit B8 (+) and ReLU unit B9: The outputs of all residual blocks are added together, and then activated by the ReLU unit B9. This step can achieve nonlinear combination of features.

[0208] The second convolutional layer B10 (1×1): The output of the ReLU unit B9 passes through a 1×1 convolutional layer. The purpose of this convolution is to reduce the dimension of the features so that the features can be more easily combined in the subsequent stacking process.

[0209] ReLU unit B11: The output of the second convolutional layer B10 is passed through the ReLU activation function. This step introduces nonlinearity again, which helps to extract higher-order features.

[0210] The third convolutional layer B12 (1×1): The output of the ReLU unit B11 is passed through another 1×1 convolutional layer, which maps the features to the same dimension as the target output (e.g., the probability distribution of the audio sample) so that the probability distribution can be calculated before the final Softmax layer.

[0211] Softmax output B13: The output of the 1×1 convolutional layer is passed through the Softmax function to convert it into a probability distribution. This gives a probability for each possible audio sample value to generate a new audio signal.

[0212] The following expression describes the operation process of the gate control unit:

[0213] Z = tanh(w f *x)⊙σ(w g *x);

[0214] Among them, x is the input signal of the network, Z is the final output signal after the filter unit layer (filter) and the gate unit (gate) operation, and w f is the weight matrix of the filter, w g is the weight matrix of the gate, * is the convolution operator, ⊙ is the element-level multiplication operator, which is used to multiply the output of the filter by the output of the gate element by element, tanh(·): the hyperbolic tangent activation function, which is used to compress the output of the filter to the range between -1 and 1, and σ() is the sigmoid activation function, which is used to compress the output of the gate to the range between 0 and 1.

[0215] Step S06: Perform convolution of a single convolutional layer. In this step, the past weights are multiplied by the past inputs, and the results are combined and output. The main purpose of the convolution operation is to extract local features of the signal in the time domain.

[0216] Specifically, the convolution operation includes the following steps.

[0217] Step SS1: Align the input signal with the convolution kernel (filter), usually starting from the beginning of the signal.

[0218] Step SS2: perform dot multiplication of the aligned signal and the convolution kernel (multiply the corresponding elements and then sum them).

[0219] Step SS3: Store the dot product result as an element of the output signal.

[0220] Step SS4: Move the convolution kernel along the input signal and repeat steps SS2 and SS3 until the end of the signal is reached.

[0221] Through this process, local features of the input signal can be extracted into the output signal. This is especially important in audio processing because it helps capture the frequency content and time domain features in the sound. In the waveform neural network model, these convolution operations are performed in the causal convolution layer and the causal dilated convolution layer.

[0222] Step S07: Build the waveform neural network model. For example, create a list for the output, add a defined expansion layer, etc. In this step, the waveform neural network model is stacked by multiple causal expansion convolutional layers. In order to build the model, the following sub-steps will be performed.

[0223] Create a list for outputs: This list will store the outputs obtained from each expansion layer. Each element of the list will be the output of an expansion layer.

[0224] Add defined dilation layers: Add the previously created causal dilated convolutional layers to the model. Each dilation layer will capture features of different time ranges in the input data. The number of dilation layers and the dilation rate can be adjusted according to the specific application scenario.

[0225] Residual and skip connections: For each expanded layer, its output is added to the output list and its residual connection is added to the output of the previous layer. Skip connections connect the output of each expanded layer directly to the last layer of the model, which helps the gradient flow better throughout the network.

[0226] Add all elements in the output list: Add all elements in the output list item by item to get the final waveform neural network output. This output will contain the time domain features captured by the entire network.

[0227] In summary, in step S07, the entire waveform neural network model is constructed, which consists of multiple causal expansion convolutional layers and uses residual and skip connections to promote better gradient circulation in the network. This model can capture the long-range time domain features in the input signal and encode them into the output.

[0228] Step S08: Construct an efficient incremental generator. It is used to quickly generate output samples when new input samples arrive. In this step, an efficient incremental generator is constructed to generate new audio samples one by one. Since the waveform neural network is an autoregressive model, it can generate one sample at a time and use the generated sample as part of future input. However, this process can be very time-consuming, especially for generating long time series. Therefore, it is necessary to develop an efficient method to generate audio samples without recalculating the entire model every time.

[0229] To achieve this goal, an incremental generator can be constructed in the following sub-steps.

[0230] Cache intermediate calculations: During the generation process, for each input sample, we can cache the results of the intermediate calculations. In this way, in the next generation, we can directly use these cached results without recalculating the entire model.

[0231] Efficiently update the cache: Every time a new sample is generated, we need to update the cached results so that the newly generated samples are included in subsequent calculations. We can achieve this by efficiently updating the cache, thereby reducing the computational cost.

[0232] Fast generation of new samples: By leveraging cached intermediate results and updating the cache efficiently, we can quickly generate new audio samples without running the entire model every time.

[0233] In summary, in step S08, an efficient incremental generator is constructed, which quickly generates new audio samples by caching intermediate calculation results and effectively updating the cache.

[0234] Step S09: Encode the waveform amplitude using one-hot encoding. One-hot encoding is a method of converting a categorical variable into a binary vector. In WaveNet, audio samples are usually represented by quantized values, which are integers representing the amplitude level of the audio waveform at a specific time point.

[0235] The purpose of one-hot encoding the amplitude is to represent the audio waveform as a vector with fixed dimensions so that it can be input into the neural network. This makes it easier for the neural network to learn the characteristics of the waveform and improve the quality of the generated audio.

[0236] Audio quantization: First, the continuous amplitude values ​​of the audio waveform are mapped to a finite set of integers. This can be achieved by rounding the amplitude value to the nearest integer, or using more complex quantization methods such as μ-law encoding. In one embodiment, quantization can be performed using the following expression:

[0237]

[0238] -1<x t <1, μ=255;

[0239] x t is the continuous amplitude value of the signal at a specific time point t, μ is the parameter of μ-law encoding, which controls the number of quantization levels. u=255, which means that the audio amplitude value is quantized into 256 discrete levels, sign(x t ) is x t The sign of x t If it is positive, it is 1; if it is negative, it is -1.

[0240] Create a one-hot encoded vector: For each quantized value, create an all-zero vector with a length equal to the total number of quantized values. Then, set the vector element corresponding to that quantized value to 1.

[0241] Replace the original audio waveform: Replace each quantized value in the original audio waveform with the created one-hot encoded vector.

[0242] After the above steps, the amplitude of the audio waveform is converted into a one-hot encoding vector for input into the waveform neural network model. This makes it easier for the model to learn audio features and improves the quality of generated audio.

[0243] Step S10: Return the embedding of the global condition. In the waveform neural network, the global condition can be some features related to the entire audio segment, such as the category of music, speaker identity, etc. The global condition can help the model generate audio of different styles based on these features.

[0244] In this step, the embedding vector of the global condition is calculated, which is a vector that maps the global condition features to a low-dimensional space. The embedding vector can convert the categorical features into continuous values ​​for better use with the neural network model.

[0245] Following are the sub-steps to compute the global conditional embedding.

[0246] Define global conditions: First, determine the features to use as global conditions, such as music category, speaker identity, etc.

[0247] Encode global conditions: Encode the global conditions, usually using one-hot encoding or integer encoding. This produces an encoding vector that represents the global conditions.

[0248] Create an embedding matrix: Initialize an embedding matrix with dimension (number of global conditional categories x embedding vector dimension).

[0249] Find the globally conditioned embedding: Use the globally conditioned encoding vector to find the corresponding embedding vector in the embedding matrix. For example, if one-hot encoding is used, the embedding vector can be obtained by multiplying the encoding vector with the embedding matrix.

[0250] Adding global conditional embeddings: Global conditional embedding vectors are added to the appropriate locations of the waveform neural network model to take global conditions into account during the generation process. The global conditional embedding vectors can be combined with local features (such as the one-hot encoding of the audio waveform) and propagated through the layers of the neural network.

[0251] After the above steps, the global conditions are embedded into the waveform neural network model to generate audio with specific style or characteristics.

[0252] Step S11: Calculate the probability distribution of the next sample based on all samples of the input waveform. In the waveform neural network model, the goal is to predict the probability distribution of the next sample based on all samples of the input waveform. These probabilities can then be used to generate new audio samples. The following are the sub-steps to do this.

[0253] Pass the input waveform through the waveform neural network model: Pass the input waveform (one-dimensional time series data) through the constructed waveform neural network model to obtain the output of the last layer.

[0254] Calculate the probability distribution: Pass the output of the last layer through the softmax activation function to convert it into a probability distribution. The softmax function can convert the output value into a probability value so that the sum of all probabilities is 1.

[0255] Select the most likely sample: Based on the calculated probability distribution, select the sample with the highest probability as the audio sample for the next moment. Other sampling methods can also be used, such as random sampling based on probability distribution.

[0256] By executing the above sub-steps, the probability distribution of the next audio sample can be generated based on all samples of the input waveform. This helps the model generate audio signals with consistency and continuity.

[0257] Step S12: Based on a single sample and all the samples that have passed before, the probability distribution of the next sample is calculated step by step. In this step, the audio sequence is generated step by step using the waveform neural network model. This means predicting the probability distribution of the next sample based on the current sample and all the previous samples. The following are the sub-steps to do this.

[0258] Step SA1, initialize the audio sequence: create an empty audio sequence to store the samples generated by the model.

[0259] Step SA2, generate initial samples: take the current audio sequence as input and use the WaveNet model to generate the probability distribution of the next sample (as described in step S11).

[0260] Step SA3, select a sample: select a sample according to the generated probability distribution (for example, select the sample with the highest probability or randomly select a sample according to the probability distribution).

[0261] Step SA4, updating the audio sequence: adding the selected samples to the audio sequence.

[0262] Step SA5, repeat the generation process: repeat the above steps SA2 to SA4 until an audio sequence of the required length is generated.

[0263] In this way, the waveform neural network model can gradually generate an audio sequence, gradually calculating the probability distribution of the next sample based on a single sample and all the samples that have passed before. This makes the generated audio more consistent and continuous.

[0264] In this step, the probability distribution of the next audio sample is generated according to the following expression:

[0265]

[0266] Where x = {x 1 ,x 2 ,…,x T} is the waveform sampling point, P(x|λ) is the probability of the entire sequence x given the model parameter λ, λ is the model parameter, T is the total time step of the sequence x, x t The sample at time step t, x 1 ,...,x t-1 is the first t-1 samples of the sequence x.

[0267] Step S13: Create a waveform neural network and return the autoencoder loss. The autoencoder loss is used to evaluate the quality of the generated waveform and optimize the model during the training process. In this step, a waveform neural network model is created and the autoencoder loss is calculated during the training process. The autoencoder loss is used to measure the performance of the model in reconstructing the input signal, usually using cross entropy loss or mean squared error as the evaluation criteria. The following are the sub-steps to do this.

[0268] Constructing a waveform neural network model: According to the previous steps S01 to S08, a complete waveform neural network model is constructed.

[0269] Training data preparation: Split the raw audio data into input and target output. The typical setting is to use the input as the previous time step of the target output.

[0270] Prediction: Use the constructed waveform neural network model to predict the input data and obtain the predicted probability distribution.

[0271] Calculate loss: Calculate the autoencoding loss based on the predicted probability distribution and the target output. Common loss functions include cross entropy loss and mean squared error.

[0272] Backpropagation and optimization: Based on the calculated loss, backpropagate and optimize the parameters in the waveform neural network model to minimize the loss.

[0273] Repeat the training process: Through multiple iterations of the training process, the model parameters are gradually optimized until convergence or the specified stopping condition is reached.

[0274] Through these sub-steps, a waveform neural network model can be created and the performance of the model during the training process can be measured by calculating the auto-encoding loss. The training process aims to minimize the loss so that when generating audio sequences, it can better reconstruct the input signal and ultimately generate more realistic speech signals.

[0275] Regarding the embodiment of the real speech signal conversion module M31 as a parallel waveform neural network, the architecture of the parallel waveform neural network can be found in Fig. 9 , Fig.10 and Fig.11 , which are schematic diagrams of the architecture of the autoregressive flow model, the inverse autoregressive flow model and the parallel waveform neural network in the present invention, as shown in the figure: First, when the parallel waveform neural network receives the environmental sound, it selects the input features and prepares the speech data set. Specifically, the parallel waveform neural network converts the original speech signal into a form that is more suitable for model learning, such as using Mel-spectrogram features to extract features. These features can capture the key information of the speech signal, and the data volume is smaller than the original speech signal, which is more beneficial to the training of the model.

[0276] Furthermore, before inputting into the parallel waveform neural network, the converted environmental sound data is normalized, which can improve the learning effect of the model and prevent the model from being numerically unstable during the training process. In one embodiment, Z-score normalization is used, and the formula is as follows:

[0277]

[0278] Among them, Z is the value after Z-score normalization; x is the original data value; u is the mean or expected value of the data set; σ is the standard deviation of the data set.

[0279] Through the above-mentioned preprocessing and normalization procedures, the environmental sound is converted into a data form suitable for parallel waveform neural network learning.

[0280] Before explaining the parallel waveform neural network, we must first briefly explain the autoregressive flow model architecture and the inverse autoregressive flow model architecture.

[0281] like Fig. 9 As shown in the figure, the autoregressive flow model J1 (AF) is a generative model based on autoregression, which generates data by transforming samples from a simple probability distribution (such as a standard Gaussian distribution) to a target complex distribution. The basic idea of ​​the autoregressive flow model J1 is to generate each dimension of the sample in sequence, and the generation of each dimension depends on the previously generated dimension. The computational overhead of the autoregressive flow model J1 when generating data is relatively large because the generation of each dimension needs to be performed in sequence.

[0282] The expression of the autoregressive flow model J1 is as follows:

[0283] y i =v i exp a i +v i ;

[0284]

[0285]

[0286]

[0287] Among them, y i is the state variable output at time step i; v i The intermediate variable at time step i; a i is another intermediate variable at time step i; A mapping function from previously generated dimensions to intermediate variables; It is another mapping function from previously generated dimensions to intermediate variables. Autoregressive flow model training can be parallelized to fully utilize the advantages of GPU, but when sampling and generating samples, they must be generated step by step in sequence, so the speed is very slow.

[0288] like Fig.10As shown in the figure, compared with the autoregressive flow model J1, the inverse autoregressive flow model J2 (IAF) is a variation of the autoregressive flow model J1. Its core idea is to reverse the generation process of the autoregressive flow, so as to be more efficient in generating data. When generating data, the inverse autoregressive flow model J2 can use vectorized operations (such as matrix multiplication) to generate all dimensions simultaneously, instead of sequentially like the autoregressive flow model J1. This makes the inverse autoregressive flow model J2 generate data faster, which is particularly suitable for scenarios of real-time or large-scale data generation.

[0289] The inverse autoregressive flow model J2 is expressed as follows:

[0290] y i =u i exp b i +u i ;

[0291]

[0292]

[0293]

[0294] Among them, y i is the state variable output at time step i; u i The intermediate variable at time step i; b i is another intermediate variable at time step i; A mapping function from the previously input dimensions to the intermediate variables; is another mapping function from the previously input dimension to the intermediate variable. As can be seen from the attached figure, the autoregressive machine of the autoregressive flow model J1 is placed at the output end, resulting in the inability to generate in parallel. The inverse autoregressive flow model J2 reverses the structure of the autoregressive flow model J1 and places the regression machine at the input end. i No longer directly dependent on y 1 ,y 2 …y i-1 , but x 1 , x 2 …x i-1 , if x i If it has been calculated in advance, y can be generated in parallel i .

[0295] Since the results are reversed, the characteristics of the inverse autoregressive flow model J2 and the autoregressive flow model J1 are exactly opposite. The autoregressive flow model J1 can be parallel during training but can only be serial during regeneration, while the inverse autoregressive flow model J2 only puts the autoregressive mechanism at the input end and still cannot be parallel during training.

[0296] The method to complement these two advantages is probability density distillation (PDD), which uses the autoregressive flow model J1 to quickly train a model in parallel as the mentor waveform neural network M312, and uses the inverse autoregressive flow model J2 as a student waveform neural network M311 to learn the output of the mentor waveform neural network M312. Specifically, during the training process of the parallel waveform neural network, a trained waveform neural network model is used as the mentor waveform neural network M312 to guide the learning of the parallel waveform neural network, and the features of the same speech segment are input into the mentor waveform neural network M312 and the student waveform neural network M311, and the student waveform neural network M311 is allowed to try to learn the output of the mentor waveform neural network M312. The architecture of the parallel waveform neural network is shown in the figure.

[0297] Similar to the generative adversarial network, the student waveform neural network M311 plays the role of the generator, while the mentor waveform neural network M312 plays the role of the discriminator; unlike the generative adversarial network, the student waveform neural network M311 does not try to fool the mentor waveform neural network M312 in an adversarial manner; it cooperates by trying to match the probability of the mentor waveform neural network M312. In addition, the mentor waveform neural network M312 remains unchanged and is not trained together with the student waveform neural network M311, and after the training of the student waveform neural network M311 is completed, the mentor waveform neural network M312 is no longer used, and the student waveform neural network M311 is used to perform the generation task.

[0298] The student waveform neural network M311 and the mentor waveform neural network M312 no longer output discrete speech signals, but the parameters of the mixed logic distribution (Mixture of Logistics, MoL) of the speech signal; the mixed logic distribution output by the student waveform neural network M311 is sampled and sent to the mentor waveform neural network M312, and the KL divergence (Kullback-Leibler divergence) of the mixed logic distribution of the mentor waveform neural network M312 can characterize the gap between the student waveform neural network M311 and the real sample. This is used as the objective function to train the student waveform neural network M311. The KL divergence expression of the mixed logic distribution of the mentor waveform neural network M312 is as follows:

[0299] DKL (P S ||P T )=H(P s , P T )-H(P S );

[0300] Where D KL (P S ||P T ) represents the KL divergence between the student waveform neural network M311 and the tutor waveform neural network M312; P s represents the probability distribution of the student waveform neural network M311; P T represents the probability distribution of the mentor waveform neural network M312; H(P s , P T ) represents the cross-entropy between the student waveform neural network M311 and the tutor waveform neural network M312; H(P S ) represents the entropy of the student waveform neural network M311.

[0301] The smaller the value of KL divergence, the closer the distribution between the student waveform neural network M311 and the tutor waveform neural network M312 is, and the better the distillation effect is. Therefore, in probability density distillation, KL divergence is usually minimized so that the output distribution of the student waveform neural network M311 is as close as possible to the output distribution of the tutor waveform neural network M312.

[0302] Because the sound quality synthesized by the student waveform neural network M311 is often poor when KL divergence is used as the objective function alone, power loss is added to the objective function. The expression of power loss is as follows:

[0303]

[0304] Wherein, B is the number of short-time Fourier transform frequencies (in one embodiment, B=1025); STFT(x) represents the result of Fourier transforming the input signal x; x n represents the actual sample; TFT(x n ) means that the actual sample x n The result of Fourier transform. By minimizing the power loss, the STFT coefficients output by the model can be made closer to the STFT coefficients of the actual sample, thereby improving the performance of tasks such as speech enhancement and speech separation.

[0305] Since the probability density distillation training alone may not be sufficient to constrain the student waveform neural network M311 to generate high-quality audio streams, the present invention introduces an additional loss function to guide the student waveform neural network M311 to distribute to the desired output space.

[0306] The first additional loss is the power loss, which ensures that the power used in different frequency bands of speech is averaged as much as human speech. The energy loss helps to avoid the collapse of the student waveform neural network M311 to the high-entropy WaveNet-mode. The expression of the power loss is as follows:

[0307]

[0308] Among them, g(z, c) represents the output of the model, z is the input of the model, c is the condition information of the model; y represents the input of the training sample, that is, the reference signal.

[0309] Since the power loss can be averaged over time before taking the Euclidean distance, the effect is not much different, which means that the average power of different frequencies is important. Therefore, perceptual loss, feature reconstruction loss (Euclidean distance between feature maps in the classifier) ​​and style loss (Euclidean distance between Gram matrices) are adopted.

[0310] Regarding perceptual loss, the expression of perceptual loss is as follows:

[0311]

[0312] j represents the jth layer of the network; C j H represents the number of convolution kernels in the jth layer; j and W j Respectively represent the height and width of the j-th convolution kernel; represents the output of the conversion network, that is, the feature map of the generated image at the jth layer; Φ j (y) represents the feature map of the bull's-eye diagram at layer j; represents the perceptual loss at the jth layer.

[0313] Regarding style loss, the expression of style loss is as follows:

[0314] Ψ is Φj(x);

[0315] Where Ψ is Φj(x) reshaped into (C j , H j , W j ) matrix; It is Gram-Matrix;

[0316] j represents the jth layer of the network; C j H represents the number of convolution kernels in the jth layer; j and W j Respectively represent the height and width of the j-th convolution kernel; represents the feature map of the bull's-eye graph at layer j; G j (y) represents the output of the conversion network, that is, the feature map of the generated image at layer j; represents the Gram matrix of the bull's eye diagram image x at the jth layer, where c and c′ represent the numbers of the convolution kernels respectively; represents the style loss at the jth layer.

[0317] The Gram matrix can actually be regarded as the covariance matrix between features. In the feature map, each number comes from the convolution of a specific filter at a specific position, so each number represents the strength of a feature, and what Gram actually calculates is the correlation between features, which two features appear at the same time, which two are mutually exclusive, and so on. At the same time, the diagonal elements of Gram also reflect the amount of each feature appearing in the audio. Therefore, Gram helps to grasp the general style of the entire audio.

[0318] Regarding the contrastive loss, the expression of the contrastive loss is as follows:

[0319] D KL (P S (c 1 )||P T (c 1 ))-γD KL (P S (c 1 )||P T (c 2 ));

[0320] γ is 0.3; c 1 and c 2 are constants used to determine the loss of positive and negative sample pairs; P S Represents the similarity score of the positive sample pair; P T Represents the similarity score of the negative sample pair.

[0321] When the instructor waveform neural network M312 and the student waveform neural network M311 are under the same information c (e.g., language features, speaker ID, etc.), the difference in KL between them is minimized. 1 =c2 , the difference will still be maximized. In order to achieve this loss, make x=g(z, c 1 ) and evaluate the waveform twice under the guidance of the mentor waveform neural network M312: once using the same conditioning reflex P T (x|c 1 ), once using a randomly sampled conditioned input P T (x|c 2 ).

[0322] Through these steps, the created student waveform neural network M311 can better reconstruct the input signal when generating audio sequences, and ultimately generate more realistic speech signals.

[0323] For the training process of parallel waveform neural network, please refer to Fig.12 , is a training flow chart of the parallel waveform neural network in the present invention, as shown in the figure.

[0324] Step P11: First, read the training data from the database. Specifically, the training engine reads the training data set from the database. The data may include audio files of speech segments and corresponding text labels.

[0325] Step P12, then, initialize the network structure of the tutor waveform neural network M312 and the student waveform neural network M311. In this step, initialize the network structure of the tutor waveform neural network M312 and the student waveform neural network M311, which includes setting the number of layers of the neural network, the number of neurons in each layer, and the selection of the activation function, as shown below:

[0326] Number of Layers: Select the number of layers between 3 and 8;

[0327] Number of Neurons: Set the number of neurons in each layer to 128, 256, or 512;

[0328] Activation Function: For the hidden layer, ReLU (Rectified Linear Unit) is used as the activation function. For the output layer, a linear activation function is used;

[0329] Output Layer: Since audio waveform is generated here, the activation function of the output layer can be a linear function;

[0330] Model Size: Choose between approximately 1,000,000 and 5,000,000 parameters.

[0331] Step P13, then, initialize the parameters of the instructor waveform neural network M312 and the student waveform neural network M311. For initialization, choose to use Xavier initialization, which initializes the weights of each layer to ensure that the initialized weights have an appropriate range. The Xavier initialization formula is as follows:

[0332]

[0333] Among them, n in is the number of neurons in the input layer; n out is the number of neurons in the output layer; u is a random value sampled from a uniform distribution.

[0334] Step P14, then, establish the student waveform neural network M311: Establishing the student waveform neural network M311 includes the following steps.

[0335] Step P14-1: Obtain input features; specifically, obtain input features from training data, such as voice features or speaker ID, etc. These features will serve as inputs to the student waveform neural network M311.

[0336] Step P14-2: Generate white noise and perform right shift conversion: Generate white noise and perform right shift conversion. This conversion is to introduce randomness and diversity to increase the generalization ability of the model.

[0337] Step P14-3: Update white noise in each loop layer. In each loop layer, update white noise to change the input of the model. This helps the student waveform neural network M311 better adapt to different input conditions.

[0338] Step P14-4: Update the z value by calculating the average value and ratio of the output. Specifically, perform the following operation on each element in the output vector, assuming x is the output vector and N is the dimension of the vector. Calculate the output average value u and ratio σ as follows:

[0339]

[0340]

[0341] Here, xi represents the i-th element in the vector.

[0342] Update the z value as follows:

[0343] z=(x-μ) / (σ+∈);

[0344] ∈ is a small positive number (e.g. 10 -8 ) to avoid division by zero.

[0345] This formula describes the process of updating the z value by calculating the mean and scale of the output vector and then standardizing the output vector. The standardized value is used as the new z value. The z value is calculated by subtracting the output vector from its mean and then dividing it by the standard deviation. This will make the numerical distribution of the output vector closer to the standard normal distribution, which will help improve the stability and convergence of the model.

[0346] Step P14-5: Adjust the mean value and proportion of white noise to obtain a new logistic distribution, and sample sample_x from it. Generate a new logistic distribution based on the adjusted mean value and proportion of white noise. Through this distribution, sample a sample sample_x as the generation result of the student waveform neural network M311.

[0347] Step P14-6: Calculate Power_loss and store the value of this loss function for adjustment in the subsequent model training process. Power_loss is a loss function used to measure the difference between the sample sample_x generated by the student waveform neural network M311 and the real data. By calculating the value of this loss function, it can be stored for adjustment and optimization in the subsequent model training process.

[0348] Step P15, then, train the mentor waveform neural network M312: The mentor waveform neural network M312 training process includes the following sub-steps:

[0349] Step P15-1: Generate white noise and perform right shift conversion; Generate white noise and perform right shift conversion, similar to step P14-2 in the training process of the student waveform neural network M311.

[0350] Step P15-2: Update white noise in each layer of the loop; Update white noise in each layer of the loop to make the tutor waveform neural network M312 better adapt to different input conditions, similar to step P14-3 in the training process of the student waveform neural network M311.

[0351] Step P15-3: Predict the sound signal using the output of the student waveform neural network M311 and the current parameters: In this step, the sound signal is predicted using the output of the student waveform neural network M311 and the current parameters. The prediction result will be used in the subsequent step P15-4.

[0352] Step P15-4: Calculate H(Ps,Pt)_loss to measure the difference between the "predict_x" predicted by the student waveform neural network M311 and the actual sample "sample_x". This loss function is used to guide the output of the student waveform neural network M311 to approach the output of the mentor waveform neural network M312.

[0353] Here we calculate H(Ps,Pt)_loss, which is a loss function used to measure the difference between "predict_x" predicted by the student waveform neural network M311 and the actual sample "sample_x". Specifically, the mean square error (Mean Squared Error) or cross-entropy loss (Cross-Entropy Loss) can be used as the loss function. By calculating the value of this loss function, we can measure the prediction ability of the student waveform neural network M311 and guide the model training process to a more accurate direction.

[0354] Step P16: Update model parameters using optimization algorithms such as gradient descent; In this step, this embodiment uses optimization algorithms such as gradient descent and the Adam algorithm to update the parameters of the model. By calculating the gradient of the loss function, the weights and biases in the model can be updated according to the direction and size of the gradient to minimize the value of the loss function.

[0355] Step P17: Finally, repeat steps P14 to P16 until a predetermined number of training rounds or a stop condition is reached; in this step, repeat steps P14 to P16 until a predetermined number of training rounds is reached or a stop condition is met. In this way, multiple rounds of model training can be performed to improve the performance and accuracy of the model. Through the above steps, the training of the parallel waveform neural network is completed, and a more realistic speech signal is generated through multiple trainings.

[0356] The MFCC unit M32 is used to perform the MFCC method. MFCCs (Mel Frequency Cepstral Coefficients) is an algorithm for extracting audio features, which is used to convert the original audio into feature information for subsequent processing. In the present invention, MFCC learns the human voice features of the real speech signal received by the parallel waveform neural network.

[0357] Please refer to the MFCC calculation steps. Fig.13 , is a schematic diagram of the calculation process of the MFCC algorithm in the present invention, as shown in the figure: The following is a detailed description of the main steps of MFCC:

[0358] Step S21, audio file reading: In this step, an audio file (eg, .wav) of a real voice signal is read. After reading the audio file, a digital signal containing audio data can be obtained.

[0359] Step S22, pre-emphasis: In order to balance the spectrum in the audio signal, we perform pre-emphasis in this step. Usually a first-order high-pass filter is used to process the audio signal, which helps to strengthen the high-frequency components and reduce the resonance phenomenon in the signal. The expression of pre-emphasis is as follows:

[0360] s′ n =s n -as n-1 ;

[0361] Among them, s′ n is the output, s n is the input, s n-1 is the input of the previous stage, and a is any parameter between 0.9 and 1.

[0362] Step S23, framing: dividing the audio signal into multiple frames of a certain length. Since the speech signal is time-varying, the framing process can help us extract features of the signal within a local time range. In one embodiment, the length of each frame can be, for example, but not limited to, 20-30 milliseconds, and there is a certain overlap between adjacent frames, which is not limited in the present invention.

[0363] The expression for setting the number of frames is as follows:

[0364] F n =f s ×F L ;

[0365] Among them, F n is the number of frames, f s is the sampling frequency, F L is the frame length.

[0366] The expression of the detection and translation setting is as follows:

[0367]

[0368] Among them, F S is the frame translation, f s is the sampling frequency, F L is the frame length.

[0369] Step S24, windowing: In order to reduce signal interference at the frame boundary, we need to apply a window function (such as a Hamming window) to each frame. The window function can make the signal at the frame boundary transition smoothly and reduce the frequency domain fluctuation caused by the truncation of the time domain signal. The expression of the Hamming window is as follows:

[0370]

[0371] 1≤n≤N;

[0372] Among them, Ham(N) is a window function with a length of N, n represents the index of the point of the window function (between 1 and N), and N represents the length of the Hamming window function.

[0373] Step S25, Fourier transform: Perform Fourier transform on the windowed signal, convert it from the time domain to the frequency domain, and obtain the spectrum of the signal. When the transform size is 2 to the power of N, the fast Fourier transform (FFT) can be used to speed up the calculation. The expression of Fourier transform is as follows:

[0374]

[0375] 0≤k≤N;

[0376] Among them, x a (k) is the value (complex number) of the kth point in the frequency domain (representing the component of the signal at frequency k / N), x(n) is the value of the nth point in the time domain, N is the number of points of the Fourier transform (used to determine the accuracy of the conversion), k is the index in the frequency domain (0 to N-1), and n is the index in the time domain (0 to N-1). Using the above expression, the time domain signal x(n) can be converted to the frequency domain signal x a (k).

[0377] Step S26, Mel filter bank: The spectrum obtained after Fourier transform is passed through a set of Mel filters to simulate the human ear's perception of sound. The Mel scale is related to the frequency resolution ability of the human ear, with a higher resolution for low frequencies and a lower resolution for high frequencies. The Mel filter bank usually contains multiple triangular filters, which divide the spectrum into different frequency bands and calculate the energy of each frequency band. This can capture important information in the speech signal while reducing irrelevant details. For the algorithm executed by the Mel filter bank, please refer to Fig.14 , is a schematic diagram of the algorithm flow of the Mel filter bank in the present invention, as shown in the figure:

[0378] Step S31, determine the Mel scale range: select a suitable Mel scale range, usually select frequencies within the human hearing range, such as 20Hz to 20kHz. Then convert the frequencies in this range into Mel scale. The conversion expression of Mel scale is as follows:

[0379]

[0380] Where Mel(f) is the Mel scale and f is the original frequency.

[0381] Step S32, define the number of Mel filters: select an appropriate number n of Mel filters according to application requirements and computing resources. Usually, the number of Mel filters is 20-40.

[0382] Step S33, inserting n positions within the Mel scale range: uniformly inserting n positions within the selected Mel scale range, and these positions will serve as the center points of the Mel filter.

[0383] Step S34, convert n+2 Mel frequencies into frequencies: convert the n Mel scale center points obtained in the previous step back into frequencies, add the start and end frequencies, and obtain a total of n+2 frequency values. The expression of the inverse Mel function is as follows:

[0384] f=700·(10 m / 2595 -1);

[0385] Where m is the Mel frequency and f is the corresponding original frequency in Hertz.

[0386] Step S35, calculate n+2 index numbers k: map n+2 frequency values ​​to the spectrum index after Fourier transformation to obtain n+2 index numbers k. These indexes will be used to calculate the frequency response of the Mel filter. The expression of index number k is as follows:

[0387]

[0388] Where k is the index number; N is the number of FFT points; f s is the sampling frequency; f m is the value after converting the Mel scale back to frequency.

[0389] Step S36, Hm matrix: According to the index number k obtained in the above steps, construct an Hm matrix of n rows. Each row represents the frequency response of a Mel filter. The filter usually has a triangular shape, with the center point as the maximum value and the two sides gradually decaying to zero. Multiplying the spectrum with the Hm matrix, the spectrum characteristics after the Mel filter bank processing can be obtained. These characteristics have an energy distribution related to human hearing. When the frequency is very small, the filter width is very narrow. As its frequency increases, the filter width will gradually increase, but its amplitude will also gradually decrease.

[0390]

[0391] Among them, Hm (k) is the converted Hm matrix, m is the filter number, and k is the horizontal coordinate (e.g., 0-2048).

[0392] Step S37, Discrete Cosine Transform (DCT): Next, perform discrete cosine transform on the output of the Mel filter bank. DCT is a method for converting signals from the frequency domain to the time domain, and has good energy concentration. The purpose of DCT is to convert the Mel filter bank energy coefficients with high correlation into cepstral coefficients with low correlation. Usually only a portion of the low-frequency cepstral coefficients (such as the first 12 or the first 13) are retained because they contain the main features of the speech signal. For the process of discrete cosine transform, please refer to Fig.15 , is a schematic diagram of the discrete cosine transform process in the present invention, as shown in the figure:

[0393] Step S41, natural logarithm operation: after the signal passes through the Mel filter bank, the natural logarithm of the output energy of each filter is taken. This step helps to simulate the perception characteristics of the human ear to sound energy.

[0394]

[0395] The H matrix in the expression is the energy spectral density E and the Mel filter bank matrix H m The energy spectral density E is the energy distribution of the signal at different frequencies, and the expression is as follows:

[0396]

[0397] It is the instantaneous power of the signal at a certain frequency. The energy spectral density E is combined with the Mel filter bank matrix H m After multiplying the transpose of , the output energy of each Mel filter can be obtained.

[0398] Step S42, discrete cosine transform: perform discrete cosine transform on the energy value after taking the natural logarithm. This step converts the speech signal from the Mel frequency to the cepstrum domain and helps to identify the basic features and details in the speech. The expression of discrete cosine transform of natural logarithm is as follows:

[0399]

[0400] Where mfcc(i,n) represents the nth MFCC parameter of the i-th frame data, represents the nth MFCC parameter of the i-th frame data, i is the frame data, and H is H m matrix, M is the number of Mel filters, n is the nth column of the i-th frame, and H(i, m) represents the energy value between the i-th frame data and the m-th filter in the energy matrix H after the Mel filter group.

[0401] Step S43, Cepstral operation: In this step, a cepstral operation is performed to convert the time domain signal into a frequency domain signal, usually retaining the first few coefficients in the DCT output. These retained coefficients capture the main information of the speech signal while ignoring unimportant high-frequency details. In one embodiment, 12 to 22 coefficients may be retained, depending on the application requirements. The expression of the cepstral operation is as follows:

[0402]

[0403] k(i) represents the Mel cepstral coefficient of the i-th frame data, L is the cepstral parameter (the value is 22 in this embodiment), and i is the frame data. Because most of the signal data is generally concentrated in the low-frequency area after transformation, only the first 13 data are taken.

[0404] Step S44, MFCC basic parameters Feat: The retained cepstral coefficients are used as the basic parameters of MFCC features. These features can be used for applications such as speech recognition, speech synthesis, and speaker recognition. Sometimes, the MFCC features can be further processed, such as adding derivatives, second-order derivatives, etc., to improve the expressiveness of the features. The final MFCC basic parameter matrix feat is expressed as follows:

[0405] feat(i,j)=J(i,j)·K(j);

[0406] i=1,2,3,…,301;j=1,2,…,13;

[0407] Among them, feat(i, j) is the jth MFCC basic parameter of the i-th frame data, J(i, j) represents the i-th frame data and the j-th column in the MFCC parameter matrix, and K(j) represents the j-th Mel cepstral coefficient.

[0408] Furthermore, the second set of parameters is a first-order differential operation under the existing basic parameters, and the third set of parameters is a first-order differential operation under the second set of parameters. The expression is as follows:

[0409] dtf eat [i][j]=feat[i+1][j]-feat[i-1][j]+2·feat[[i+2][j]-

[0410] 2feat[i-2][j];

[0411] i=1,2,3,…,301;j=1,2,…,13;

[0412] dtf eat[i][j] is the first-order differential value of the j-th MFCC parameter of the i-th frame data, f eat [i][j] represents the jth MFCC basic parameters of the i-th frame data. In this expression, by eat Each element in the matrix is ​​subjected to a first-order differential operation to calculate the first-order differential parameter dtf eat Similarly, you can do the same for dtf eat Perform first-order differential operation to obtain second-order differential parameters.

[0413] Step S27, finally, when comparing the MFCC feature values ​​between different samples, the scale differences between them are eliminated, so as to better perform model training and feature selection. For the i-th MFCC feature value, the normalization process is as follows:

[0414]

[0415] x i is the original i-th MFCC feature value; mean i is the average value of the i-th MFCC feature value in all samples; s tdi is the standard deviation of the ith MFCC feature value in all samples; F(x i ) is the standardized ith MFCC eigenvalue. Standardizing the original MFCC eigenvalues ​​can eliminate distribution differences, making the MFCC eigenvalues ​​between different samples more comparable, which is beneficial to model training and evaluation.

[0416] On the other hand, the original energy amplitude value is converted to a standardized energy amplitude value. This helps to eliminate the scale differences in the data so that different data can be compared. The standardized formula is as follows:

[0417]

[0418] P(x i ) is the standardized energy amplitude value, which indicates the standardized value of the energy amplitude of the i-th sample; energy is the original energy amplitude value, which indicates the original energy amplitude of the i-th sample; mean_energy is the average value of the energy amplitude values ​​in all samples, which is used to center the energy amplitude value; s tdenergy It is the standard deviation of the energy amplitude values ​​in all samples and is used to scale the energy amplitude values.

[0419] Finally, these two input feature groups (audio feature vector and audio energy amplitude) are combined into a feature vector as the input of the CNN model. For example, suppose 12 features and 1 energy amplitude value are extracted and they are concatenated into a 13-dimensional feature vector.

[0420] The speech recognition module M33 is used to map the audio features output by the MFCC unit M32 to voice commands. In one embodiment, the speech recognition module M33 may be, for example, but not limited to, a convolutional neural network model (CNN); in an embodiment using a convolutional neural network model, the convolutional neural network model may be used to extract features of speech signals, extracting short-term spectral features, such as MFCC features, from the original time series.

[0421] Please also read Fig.16 , which is a schematic diagram of the convolutional neural network model architecture used in the speech recognition module of the present invention, as shown in the figure: the convolutional neural network model used in the speech recognition module of the present invention includes a convolutional layer LA1 (Convolutional Layer), a pooling layer LA2 (Pooling Layer), and a fully connected layer LA3 (Fully Connected Layer).

[0422] Convolution layer LA1 is mainly used to extract features. Specifically, convolution layer LA1 is the core of the convolutional neural network model. It extracts features from the input image through a certain number of convolution kernels. Each convolution kernel contains a set of weights, which are convolved with the input image to generate a new feature map. The size and step size of the convolution kernel are set by hyperparameters.

[0423] The pooling layer LA2 is used to reduce the dimension of features. Specifically, the pooling layer LA2 is mainly used for feature dimensionality reduction, reducing the number of parameters and the amount of calculation. Through certain pooling operations (such as maximum pooling, average pooling, etc.), the feature map is reduced in dimension to obtain a new feature map. The main hyperparameters of the pooling layer are the size and step size of the pooling kernel.

[0424] The fully connected layer LA3 is used to generate the final output. Specifically, the fully connected layer LA3 is usually used to convert the feature map into the corresponding output and perform tasks such as classification or regression. It expands the feature map into a vector, performs linear transformation and nonlinear transformation through multiple fully connected layers LA3, and finally obtains the output result. In the present invention, the output result is directly mapped to the voice instruction, or mapped to the code or value related to the voice instruction, and the corresponding instruction is found from the database through the code or value, which is not limited in the present invention.

[0425] The CTC trainer M34 is used to execute the CTC Loss algorithm (Connectionist Temporal Classification Loss), which is used as a loss function in sequence learning to solve the sequence alignment problem. In sequence learning tasks such as speech recognition and text recognition, there is often a situation where the input sequence and the output sequence are not completely consistent or aligned, which requires the use of CTC loss to train the convolutional neural network model.

[0426] More specifically, in speech recognition, the data set is the audio file and its corresponding text data. However, it is difficult to align the audio file and the text data in units of words. If the model is trained directly without alignment, it will be difficult for the model to converge due to the different speech speeds of people or the different distances between characters. CTC Trainer M34 can avoid manual alignment of input and output, and is suitable for speech recognition.

[0427] The following expression represents the probability of observing a specific alignment path π given an input feature sequence x. The expression is as follows:

[0428]

[0429] L′=L∪{blank};

[0430] Where p(π|x) is the probability of observing a specific alignment path π given an input feature sequence x. x is the input feature sequence, usually the data of the speech signal after feature extraction. y is the output label sequence, that is, the expected text sequence. T is the length of the input feature sequence x. L′ is the value space, including the output vocab (L) and an additional blank label. L is the output vocab, that is, all possible output characters. πt is the observation probability, which represents the probability of observing a certain character at time t.

[0431] The following expression represents the sum of the probabilities of selecting the target sequence I among all possible pathsπ (in the mapping set β) given an input x.

[0432]

[0433] Where x is the input sequence, usually a speech signal or text, I is the target sequence, that is, the text sequence to be recognized or generated, and β is the mapping set, also known as the "alphabet", containing all possible text sequences, ∈β -1(I) is the set of all extended sequences of the target sequence I in the mapping set β, π is a path in the mapping set β, indicating a path selected from all possible paths, P(π|x) is the conditional probability from the input sequence x to pathπ, indicating the probability of mapping x to π, and P(I|x) is the conditional probability from the input sequence x to the target sequence I, indicating the probability of mapping x to I.

[0434] For the CTC Loss process, please refer to Fig.17 , is a schematic diagram of the training process of the CTC trainer in the present invention, as shown in the figure: The process of CTC Loss is as follows:

[0435] Step S51: Initialization Before performing CTC loss calculation, some variables need to be initialized, for example, the probability calculation group (used to calculate forward and reverse probabilities) and the gradient calculation group (used to calculate gradients).

[0436] Probability calculation group: Create two two-dimensional matrices (usually called alpha and beta) to store the calculation results of forward probability and reverse probability. The size of these matrices is usually T×L, where T represents the length of the input sequence and L represents the length of the target sequence (after expansion, including blank symbols).

[0437] Gradient calculation group: Create a gradient calculation group to store the gradient of the loss function with respect to the model parameters. This will be used during the backpropagation process in order to update the model parameters.

[0438] Other variables: Some other variables need to be initialized, such as loss values, regularization terms, etc.

[0439] Before doing CTC loss calculation, make sure all variables are initialized correctly to avoid calculation errors or inconsistencies.

[0440] Step S52: Function import. The steps include 1. Generate two matrices to calculate forward and reverse probabilities: these matrices will be used to store the calculation results of forward probability and reverse probability; 2. Expand labels with blank symbols: insert blank symbols in the target sequence for alignment in CTC loss calculation.

[0441] Generate two matrices to calculate forward and backward probabilities: Here, two two-dimensional matrices (usually called alpha and beta) will be created, which will be used to store the results of the forward and backward probabilities. The size of these matrices is usually T×L, where T represents the length of the input sequence and L represents the length of the target sequence (after expansion, including blank symbols). Each element in the matrix represents the probability at a given time t and position l in the target sequence.

[0442] Augment labels with blank symbols: For CTC calculation, a special blank symbol needs to be inserted between each character of the target sequence. In addition, blank symbols need to be added at the beginning and end of the sequence. For example, if the target sequence is "AB", the augmented sequence will be "-AB-", where "-" represents the blank symbol. The purpose of this is to allow the model to predict the same target character at different input time steps t, making the alignment problem simpler.

[0443] After completing these operations, you can proceed to calculate the forward probability and reverse probability.

[0444] Step S53: Perform forward probability calculation, and calculate the forward probability based on the given input sequence and target sequence. This is the first part of the CTC loss calculation, which represents the probability of observing the input sequence given the target sequence. In this process, given an input sequence and a label sequence, CTC Forward calculates the alignment result of the input sequence and the label sequence, and calculates the loss value between the predicted result and the legal alignment result of the label sequence.

[0445] The expression for forward probability calculation is as follows:

[0446]

[0447] This expression means that the calculation of α t The value of (s) needs to be summed over all possible alignment paths π, where π is a sequence of length T, representing the correspondence between each time step of the input feature sequence x and the output sequence S, and B represents a set that contains all possible paths π that match the first s characters of the current label sequence 1 after t time steps. Therefore, B(π 1:t )=l 1:s Represents the path π 1:t After t time steps, it matches the first s characters of label sequence 1.

[0448] Update rules The expression is as follows:

[0449]

[0450] This update rule is used to calculate the new Forward variable value of the sequence S at time step t. It means that to calculate the new The forward variable α of the previous time step t-1 needs to be t-1 (s) and α t-1(s-1) is added. This update rule is used to recursively calculate α in the Forward process of the CTC algorithm. t (s), thereby estimating the probability of the output label sequence y given the input feature sequence x.

[0451] Among them, α t (s) is the forward variable, which indicates the probability of generating sequence S at time step t; t is the current time step, and its value range is [1, T]; where T is the length of the input feature sequence x, s is the current position of the sequence S, and its value range is [1, |S|], where |S| indicates the length of the sequence S; π is the alignment path, which indicates the correspondence between each time step of the input feature sequence x and the output sequence S, and π is a sequence of length T; N T (1:s) is the set of all possible alignment paths, where each path π satisfies B(π 1:t )=l 1:s ; B is the operation of removing duplicate labels and blank labels, converting the path π into a label sequence. For example, if the path π is "aa--b-bc", then B(π) = "abc"; l 1:s are the first s characters of the target output sequence; t′ is the variable of time step, which is used to calculate the product of the observation probability from time step 1 to time step t, and the value range of t′ is [1, t]; At time step t′, the input feature sequence x generates label π t′ For example, if at a certain time step, the probability of the input feature sequence x generating the character "a" is 0.8, then y("a") = 0.8.

[0452] Step S54: Execute reverse probability calculation. Similarly, calculate reverse probability. This step is to calculate the correspondence between the target sequence and the speech signal so as to calculate the loss function in the subsequent steps. In this step, the reverse probability of each time step is calculated, which represents the sum of the probabilities of all possible paths from the current time step to the end of the target sequence.

[0453] The expression for reverse probability calculation is as follows:

[0454]

[0455] This expression means that the calculation of β t The value of (s) needs to be summed over all possible alignment paths π, where π is a sequence of length T, representing the correspondence between each time step of the input feature sequence x and the output sequence S.

[0456] Update rules The expression is as follows:

[0457]

[0458] This update rule is used to calculate the new Backward variable value of the sequence S at time step t. It means that to calculate the new It is necessary to change the Backward variable β of the next time step t+1 t+1 (s) and β t+1 (s+1) is added. This update rule is used to recursively calculate β in the Backward process of the CTC algorithm. t (s), thereby estimating the probability of the output label sequence y given the input feature sequence x.

[0459] Among them, β t (s) is the backward variable, which indicates the probability of generating sequence S from time step t to the last time step T; t is the current time step, and its value range is [1, T], where T is the length of the input feature sequence x; s is the position of the current sequence S, and its value range is [1, |S|], where |S| indicates the length of the sequence S; π is the alignment path, which indicates the correspondence between each time step of the input feature sequence x and the output sequence S, and π is a sequence of length T; N T (s:|l|) is the set of all possible alignment paths, where each path π satisfies (π t:T )=l s:|l| ; B removes repeated labels and blank labels and converts the path π into a label sequence. For example, if the path π is "aa--b-bc", then B(π) = "abc"; l s:|l| is the subsequence from s to the last character of the target output sequence; t′ is the variable of time step, which is used to calculate the product of the observation probability from time step t to time step T, and the value range of t′ is [t,T]; is the probability that the input feature sequence x generates the label π(t') at time step t'. For example, if the probability that the input feature sequence x generates the character "a" at a certain time step is 0.8, then y("a") = 0.8.

[0460] Through these two formulas, the Backward variable β can be calculated in the Backward process of the CTC algorithm. t (s) to estimate the probability of outputting label sequence y given input feature sequence x. Combined with the forward variable α t (s) and Backward variable β t (s), the CTC loss function can be calculated and the parameters of the model can be optimized.

[0461] Step S55: Calculate the loss function. Based on the calculation results of the forward probability and the reverse probability, calculate the CTC loss. The CTC loss represents the alignment error between the given input sequence and the target sequence. By minimizing this loss, the model will learn to better align the input sequence, thereby improving the recognition effect. In this step, the forward probability α is used t (s) and the reverse probability β t (s), calculate the gradient of the CTC loss function. These gradient values ​​are used to optimize the parameters of the model, thereby improving the accuracy of the model in predicting the output label sequence y given the input feature sequence x. The expression of CTC loss is as follows:

[0462]

[0463] is the CTC loss function with respect to the startup value The gradient of , which needs to be calculated to optimize the model parameters; is the starting value of the model output (such as the output of a neural network) at time step t, corresponding to label t; is the probability of generating label k for input feature sequence x at time step t; p(l|x) is the probability of outputting label sequence y given input feature sequence x, which can be calculated by CTC Forward variable α t (s) and Backward variable β t (s) is obtained; l is the target output sequence; x is the input feature sequence; α t (s) is the CTC Forward variable, which is the probability of generating sequence s at time step t given the input feature sequence x; β t (s) is the CTC Backward variable, which is the probability of generating sequence s from time step t to the last time step T given the input feature sequence x; α t (s) and β t (s) is the normalized CTC Forward variable and Backward variable; lab(l, k) is the set of all sequences s containing label k; |l′| represents the length of the output sequence after expansion; At time step t, the input feature sequence x generates the label l' in the expanded output sequence l' s probability.

[0464] This formula defines the calculation of the CTC loss function with respect to the startup value The gradient is used to optimize the model parameters to improve the accuracy of the model in predicting the output label sequence y given the input feature sequence x. In this formula, the gradient Used to update the parameters of the speech recognition module M33. Through the above steps, the CTC training unit M34 uses the connection time series classification loss function (CTC loss) to calculate the distance between the output sequence and the label sequence and updates the model parameters of the speech recognition module M33 through forward probability calculation and reverse probability calculation loss functions.

[0465] Through the architecture of real speech signal conversion module M31, MFCC unit M32, speech recognition module M33 and CTC trainer M34, the present invention has the effects of high efficiency (by using MFCC features and WAVENET model, the system can realize efficient speech recognition function), high accuracy (by using CTC LOSS, the system can realize higher recognition accuracy rate when processing speech information with noise) and high real-time performance (the system can process speech information in real time and realize real-time speech recognition).

[0466] In the present invention, the normalization module M4, the dense connection network M5 and the coordinate mapping module M6 are used to obtain the spatial coordinates corresponding to the source of the voice information through the voice information. The following is a detailed description of the functions and architectures of the normalization module M4, the dense connection network M5 and the coordinate mapping module M6. Please refer to Fig.18 and Fig.19 , which is a block diagram of the position detection mechanism in the present invention and a schematic diagram of the architecture of the densely connected network in the present invention, as shown in the figure: the normalization module M4 is used to normalize the voice information after receiving the voice information to obtain normalized voice data. Specifically, before the voice information is input into the densely connected network M5, the normalization module M4 first converts the voice information into a normalized time-frequency spectrum (spectrogram) through spectral analysis (spectrogram analysis), so that the densely connected network M5 can recognize the voice information.

[0467] Specifically, the sound signal can be converted into a time-frequency spectrum in the following manner: first, resample and convert the sound signal into stereo. Then, segment the sound signal, usually using a window function (such as a Hamming window) and an overlapping technique (such as a 50% overlap) to divide it into multiple short time periods, and translate the audio signal through the audio augmentation time axis (Audio Augmentation TimeShift) to increase the generalization ability of the model, thereby completing the regularization. Then, for the sound signal in each short time period, use Fourier transform (FFT) to convert it into a spectrum signal. Since the spectrum signal obtained by Fourier transform is a continuous frequency signal from 0Hz to the sampling frequency, it needs to be divided into several discrete frequency bands for subsequent processing. This is usually implemented using short-time Fourier transform (STFT), which divides the spectrum signal of each short time period into a series of discrete frequency bands to form a two-dimensional matrix, where the horizontal axis is time and the vertical axis is frequency. Finally, for each frequency band in a short time period, its energy or intensity can be calculated and drawn into a color image or a grayscale image to represent the sound intensity distribution at different frequencies and times, that is, to obtain a time-frequency spectrum. Finally, Spectrogram Augmentation (SpecAugment) is performed to enhance the audio data through time masking, frequency masking, time shifting and frequency shifting to improve the generalization ability and robustness of the model.

[0468] After the speech information is converted into a time-frequency spectrum (i.e., normalized speech data), it is input into the densely connected network M5, which maps the time-frequency spectrum to the spatial information corresponding to the location of the speech information source. Fig.19 As shown, the model architecture of the densely connected network M5 from input to output includes an input layer M501, a convolutional layer M502, a batch normalization layer M503, a scaling factor layer M504, a ReLU layer M505, a pooling layer M506, a first stacked dense block group layer M507, a first transition layer M508, a second stacked dense block group layer M509, a second transition layer M510, a third stacked dense block group layer M511, a third transition layer M512, a fourth stacked dense block group layer M513 and a classification layer M514. The functions of each layer are described below.

[0469] The input layer M501 is used to accept normalized speech data and input the normalized speech data into the model; specifically, this embodiment accepts the audio features of the normalized speech data as input, with a size of N×F, where N represents the number of time points and F represents the feature dimension.

[0470] The convolution layer M502 is used to extract the features of the output data of the previous layer. The batch normalization layer M503 performs batch normalization on the output data of the previous layer to adjust the distribution of multiple features. The scaling factor layer M504 performs scaling on the features of the output data of the previous layer, and restores the representation ability of the data by introducing a learnable scaling factor. The ReLU layer M505 applies the ReLU activation function to the output data of the previous layer, adjusts the negative input to 0 and retains the positive input; specifically, the architecture in this embodiment uses 64 7×7 filters with a stride of 2 and a padding of 3. This convolution layer performs a convolution operation on the input, and after the batch normalization layer M503 (BatchNorm), the scaling factor layer M504 (Scale) and the ReLU layer M505, the output is That is, 64 feature maps.

[0471] The pooling layer M506 reduces the spatial size of the output data of the previous layer through pooling operation; specifically, the pooling layer M506 in this embodiment performs a maximum pooling operation, the filter is 3×3, the stride is 2, and the padding is 1. The pooling layer M506 performs a pooling operation on the output of the convolution layer to reduce the size of the feature map, and the output is There are still 64 feature maps.

[0472] The first stacked dense block group layer M507 is connected to the output of the pooling layer M506. The first stacked dense block group layer M507 includes multiple first convolutional layers and first batch normalization layers. The input of each first convolutional layer is connected to the output of all other previous first convolutional layers. Specifically, the overall output of the first stacked dense block group layer M506 in this embodiment is 256 feature maps. The first stacked dense block group layer M507 includes 6 consecutive stacked dense blocks. Each stacked dense block contains two convolutional layers, and the convolution kernel sizes are 1×1 and 3×3 respectively. Batch normalization, scaling factor and ReLU operations are also performed before convolution. The output feature maps of the first and second convolutional layers are 128 and 32, respectively. Because k=32, the number of output feature maps of each stacked dense block will be 32 more than the previous stacked dense block. In consecutive stacked dense blocks, the output feature map of each stacked dense block will be concatenated with the output feature maps of all previous stacked dense blocks, and these stacked dense blocks will be connected in series by dimension.

[0473] The first transition layer M508 is connected to the output of the first stacked dense block group layer M507. The first transition layer includes a convolution layer and a pooling layer for reducing the size of the feature map. The output of the first transition layer M508 is There are 128 feature maps; specifically, the first transition layer M508 of this embodiment includes batch normalization, scaling factor ReLU operation, and convolution kernel 1×1 convolution layer operation before convolution, and its output is And an average pooling operation with a filter of 2×2 and a stride of 2, the output is

[0474] The second stacked dense block group layer M509 is connected to the output of the first transition layer M508. The second stacked dense block group layer M509 includes multiple second convolutional layers and second batch normalization layers. The input of each second convolutional layer is connected to the output of all other previous second convolutional layers. Specifically, the overall output of the second stacked dense block group layer M509 in this embodiment is 512 feature maps. The second stacked dense block group layer M509 includes 12 consecutive stacked dense blocks. Each stacked dense block contains two convolutional layers, and the convolution kernel sizes are 1×1 and 3×3 respectively. Batch normalization, scaling factor and ReLU operations are also performed before convolution. The output feature maps of the first and second convolutional layers are 128 and 32, respectively. Because k=32, the number of output feature maps of each stacked dense block will be 32 more than the previous stacked dense block. In consecutive stacked dense blocks, the output feature map of each stacked dense block will be concatenated with the output feature maps of all previous stacked dense blocks, and these stacked dense blocks will be connected in series by dimension.

[0475] The second transition layer M510 is connected to the second stacked dense block group layer M509, and the second transition layer M510 includes a convolution layer and a pooling layer for reducing the size of the feature map; the output of the second transition layer M510 is There are 256 feature maps; specifically, the second transition layer M510 of this embodiment includes batch normalization, scaling factor ReLU operation, and convolution kernel 1×1 convolution layer operation before convolution, and its output is And an average pooling operation with a filter of 2×2 and a stride of 2, the output is

[0476] The third stacked dense block group layer M511 is connected to the second transition layer M510, and the third stacked dense block group layer M511 includes multiple third convolutional layers and third batch normalization layers, and the input of each third convolutional layer is connected to the output of all other third convolutional layers; specifically, the overall output of the third stacked dense block group layer M511 in this embodiment is 1024 feature maps. The third stacked dense block group layer M511 includes 24 consecutive stacked dense blocks. Each stacked dense block contains two convolutional layers, and the convolution kernel sizes are 1×1 and 3×3 respectively. Batch normalization, scaling factor and ReLU operations are also performed before convolution. The output feature maps of the first and second convolutional layers are 128 and 32, respectively. Because k=32, the number of output feature maps of each stacked dense block will be 32 more than the previous stacked dense block. In consecutive stacked dense blocks, the output feature map of each stacked dense block will be concatenated with the output feature maps of all previous stacked dense blocks, and these stacked dense blocks will be connected in series by dimension.

[0477] The third transition layer M512 is connected to the third stacked dense block group layer M511, and the third transition layer M512 includes a convolution layer and a pooling layer for reducing the size of the feature map; the output of the third transition layer M512 is There are 512 feature maps; specifically, the third transition layer M512 of this embodiment includes batch normalization, scaling factor ReLU operation, and convolution kernel 1×1 convolution layer operation before convolution, and its output is And an average pooling operation with a filter of 2×2 and a stride of 2, the output is

[0478] The fourth stacked dense block group layer M513 is connected to the third transition layer M512, and the fourth stacked dense block group layer M513 includes multiple fourth convolutional layers and fourth batch normalization layers, and the input of each fourth convolutional layer is connected to the output of all other previous fourth convolutional layers; specifically, the overall output of the fourth stacked dense block group layer M513 in this embodiment is 1024 feature maps. The fourth stacked dense block group layer M513 includes 16 consecutive stacked dense blocks. Each stacked dense block contains two convolutional layers, and the convolution kernel sizes are 1×1 and 3×3 respectively. Batch normalization, scaling factor and ReLU operations are also performed before convolution. The output feature maps of the first and second convolutional layers are 128 and 32, respectively. Because k=32, the number of output feature maps of each stacked dense block will be 32 more than the previous stacked dense block. In consecutive stacked dense blocks, the output feature map of each stacked dense block will be concatenated with the output feature maps of all previous stacked dense blocks, and these stacked dense blocks will be connected in series by dimension.

[0479] The classification layer M514 is connected to the fourth stacked dense block group layer M513, and the output of the fourth stacked dense block is mapped to spatial information; specifically, the classification layer M514 in the present invention includes an average pooling layer, a convolution layer, and a Softmax layer; the average pooling layer performs global average pooling on the output feature map, and the filter is The stride is 1, the padding is 0, and the output size is 1×1×1024, 1024 feature maps. Then there is a convolution layer with a convolution kernel of 1×1, which outputs M feature maps, where M=5832, indicating the number of categories of audio classification. Finally, classification is performed through the Softmax layer, and the output feature map is mapped to one of the M possible categories, namely the spatial ID (spatial information). The 5832 sets of outputs described are only one embodiment of the present invention. According to the actual requirements for accuracy, the 5832 sets of outputs can also be increased or decreased according to the requirements, and are not limited in the present invention.

[0480] For more information about the training process of the densely connected network M5, please refer to Fig. 20 , which is a training flow chart of the densely connected network in the present invention, as shown in the figure: first, a training data set with speech information and the location of the source of the speech information is collected (step S61); then, the speech information is pre-normalized to ensure that the data scale of the speech information is consistent (step S62); forward propagation is performed through the densely connected network M5 to map the input data to spatial information, and the difference between the predicted result and the true label is calculated (step S63); through the back propagation algorithm, the model parameters of the densely connected network M5 are adjusted according to the loss function to make the predicted result of the densely connected network M5 closer to the true label (step S64); the model parameters of the densely connected network M5 are updated according to the optimizer to minimize the loss function (step S65); and the forward propagation, back propagation and parameter update operations are repeated until the model converges or the predetermined number of training times is reached (step S66).

[0481] In one embodiment, the source position of the voice information in step S61 is a space ID (space information) in real space. In one embodiment, it can be but not limited to cutting the three-dimensional space into grids of fixed size, such as 18×18×18=5832 block coordinates, equivalent to (0,0,0)~(17,17,17). In this embodiment, each block coordinate is identified by a unique ID, starting from 0, and a number is assigned to each block coordinate. The ID of each block coordinate has a direct correspondence with its position in the three-dimensional space.

[0482] In addition, the normalization process described in step S62 is performed by the normalization module M4, which converts the speech information into a normalized time-frequency spectrum (spectrogram) through spectral analysis.

[0483] In the model training of steps S63 to S65, the softmax function can convert the output of the neuron into a probability distribution, and its formula is:

[0484]

[0485] Among them, S(y i ) represents the output of the Softmax function, which can also be interpreted as the input feature y i The probability of y i represents the i-th element in the input feature vector, that is, the output of the i-th neuron, y j is a general variable used to represent the output of all neurons, j represents the index of the neuron, e is the base of the natural logarithm, represents the sum of the exponential functions of all neuron outputs, where j represents the index of the neuron.

[0486] In the loss function, the cross entropy loss function is used to train the neural network. The cross entropy can measure the difference between the actual distribution and the predicted distribution. Its formula is:

[0487] H(y,y′)=-∑(y′ i ×logy i );

[0488] Among them, H(y, y′) is the cross entropy loss, y i is the i-th element predicted by the model, y′ i is the ith element of the true label (represented by one-hot encoding). The model is trained in the above way.

[0489] The coordinate mapping module M6 maps the input space ID (space information) to space coordinates. For example, in the embodiment of mapping to 18×18×18=5832 block coordinates ((0,0,0) to (17,17,17)), 5832 space IDs are output, and the space ID mapping relationship can be mapped to coordinates using the following formula.

[0490] For example, the block coordinate ID is i, and its x, y, z coordinate range can be calculated as:

[0491] x=i / / (18*18)y=(i / / 18)%18z=i%18;

[0492] The “ / / ” in the expression is integer division, and “%” is the remainder.

[0493] For example, assuming the coordinates are (17,17,17): ID = x×(18×18)+y×18+z, substitute x=17, y=17, z=17 into the above formula, and we get: ID = 17×(18×18)+17×18+17=5831. Therefore, when the coordinates are (17,17,17), the corresponding ID is 5831. This mapping method ensures that each grid has a unique ID, and any ID can be uniquely mapped back to the coordinates of a grid.

[0494] The processing module M7 is used to generate corresponding control instructions based on the user's voice instructions and spatial coordinates, and switch the audio output mode of the audio processing module M2 or the reverse audio output mode of the sound elimination module M1 according to the control instructions.

[0495] Specifically, the processing module M7 can confirm the user's position based on the spatial coordinates, and change the form of the control command based on the user's position and the content of the voice command. In one embodiment, the processing module M7 can first confirm whether the user is in the immersive area or the non-immersive area based on the spatial coordinates, and determine what control command to generate through the voice command. Specifically, the following implementation scenarios can be included:

[0496] Scenario 1: When the processing module M7 receives a voice command from the immersive area and receives a voice command of "turn up the volume", the processing module M7 can respond by turning up the volume of the immersive area (i.e., increasing the gain of the audio output device 20) through the audio processing module M2;

[0497] Scenario 2: When the processing module M7 receives a voice command from the immersive area and receives a voice command of "lower the volume", the processing module M7 can respond by lowering the volume of the immersive area (i.e., lowering the gain of the audio output device 20) through the audio processing module M2;

[0498] Scenario 3: When the processing module M7 receives a voice command from a non-immersive area and receives a voice command of "turn up the volume", the processing module M7 can respond by turning up the volume of the non-immersive area (i.e., reducing the gain of the reverse cancellation sound) through the sound cancellation module M1;

[0499] Scenario 4: When the processing module M7 receives a voice command from a non-immersive area and receives a voice command of "lower the volume", the processing module M7 can respond by lowering the volume of the non-immersive area (i.e., increasing the gain of the reverse cancelling sound) through the sound cancellation module M1;

[0500] Scenario 5: When the processing module M7 receives a voice command from the immersive area and receives a voice command of "improving the immersive effect", the processing module M7 can respond by eliminating the noise from the non-immersive area through the sound elimination module M1;

[0501] Scenario 6: When the processing module M7 receives a voice command from a non-immersive area and receives a voice command to "enhance the immersive effect", the processing module M7 can respond by increasing the gain of the reverse cancelling sound;

[0502] Scenario seven: When the processing module M7 receives a voice command from the immersive area and receives a voice command to "reduce immersion effect", the processing module M7 can respond by reducing the stereo output through the audio processing module M2.

[0503] In other embodiments, the processing module M7 may directly determine the control command according to the spatial coordinates of the voice information and the voice command. Specifically, the following implementation scenarios may be included:

[0504] Scenario 8: When the processing module M7 confirms the spatial coordinates of the voice command and receives a voice command of "turn up the volume", the processing module M7 can respond by turning up the directivity of the loudspeakers 21 and 31 to increase the volume individually corresponding to the user's location;

[0505] Scenario 9: When the processing module M7 confirms the spatial coordinates of the voice command and receives the voice command of "lower the volume", the processing module M7 can reduce the volume at the user's location by reducing the directivity of the speakers 21 and 31 (or directly reducing the gain of the audio processing module M2);

[0506] Scenario 10: When the processing module M7 confirms the spatial coordinates of the voice command and receives the voice command of "turn up the volume", the processing module M7 can respond by turning up the volume at the user's location (i.e., reducing the directivity of the reverse cancellation sound to the user's location or reducing the gain of the reverse cancellation sound) through the sound cancellation module M1;

[0507] Scenario 11: The processing module M7 confirms the spatial coordinates of the voice command. When receiving the voice command of "lower the volume", the processing module M7 can respond by lowering the volume at the user's location through the sound cancellation module M1 (i.e., increasing the directivity of the reverse cancellation sound to the user's location or increasing the gain of the reverse cancellation sound);

[0508] Scenario 12: When the processing module M7 receives the spatial coordinates of the voice command and receives the voice command of "changing the immersive effect to the current position", the processing module M7 can reallocate the range of the immersive area and the non-immersive area by confirming the spatial coordinates of the voice command, and adjust the directionality of the reverse cancellation sound and the played sound to align the axis position of the immersive area to the user position (i.e., the spatial coordinates of the voice command), and generate a non-immersive area by directing the reverse cancellation sound to other positions.

[0509] It should be noted that the above scenario descriptions are merely examples of several embodiments of the present invention, and these embodiments are not intended to limit the scope of protection of the present invention.

[0510] In summary, please refer to Fig.21 , which is a schematic diagram of the overall operation flow of the open space immersive audio system of the present invention, as shown in the figure: first, the user submits a voice message (step S71); the audio receiving device 10 receives the voice message including the voice command (or the ambient sound including the voice) through the microphone 11 (step S72), and the received voice message will be processed by the processor 41; after receiving the voice message, the processor 41 analyzes the voice message by running the intelligent voice recognition module M3 to obtain the voice command, and finds the text data corresponding to the voice command through the voice recognition module M33 (step S73); at the same time as step S73 (it can also be before or after), the processor 41 analyzes the user's spatial coordinates through the voice information (step S74); after confirming the text data of the voice command and the user's spatial coordinates, the processor 41 generates a corresponding control instruction based on the user's text data and the spatial coordinates by running the processing module M7, and switches the audio output mode of the audio processing module M2 or the reverse audio output mode of the sound elimination module M1 according to the control instruction (step S75). In one embodiment, before step S71, the intelligent voice recognition module M3 of the processor 41 can be triggered by specific designated voice information (such as short voice), and after being triggered, it switches from standby mode to working mode, thereby reducing the consumption caused by calculation in the normal state.

[0511] The following is a more detailed description of the embodiments of the specific configuration positions in the immersive audio system of the present invention. Figure 22 to Figure 25 , which are configuration schematic diagram (1), configuration schematic diagram (2), configuration schematic diagram (3) and configuration schematic diagram (4) of the immersive audio system of the present invention.

[0512] For the first configuration method, please refer to Fig. 22In this embodiment, the reverse audio output device 30 can be configured at any position of the audio device AP to eliminate the audio played in the non-immersive space, where the immersive space is marked as A and the non-immersive space is marked as B. The position can be, for example, configured at the back side of the audio device AP (such as (a1)) to eliminate the audio played at the back of the audio device AP when the audio output device 20 plays the audio in the front direction; configured at the left side of the audio device AP (such as (a2)) to eliminate the audio played on the left side of the audio device AP; configured at the right side of the audio device AP (such as (a3)) to eliminate the audio played on the right side of the audio device AP; configured at the top side of the audio device AP (such as (a4)) to eliminate the audio played on the upper side of the audio device AP; configured at the bottom side of the audio device AP (such as (a5)) to eliminate the audio played on the bottom side of the audio device AP; or a combination of multiple and configured at any of the above positions, which is not limited in the present invention. In another embodiment, although not shown in the figure, the reverse audio output device 30 can also be set at the corner position or the corner position of the audio device AP, which is explained here first.

[0513] For the second configuration method, see Fig.23 : On the audio device AP, the reverse audio output device 30 can be configured on both sides of the audio output device 20. By eliminating the playback audio on both sides of the audio output device 20, the directional effect of the audio output device 20 is increased, and the sound field disorder caused by the sound with too wide a propagation range is avoided. It can be seen from the accompanying drawings that the reverse audio output devices 30 on both sides will increase the range of the non-immersive space D1 on both sides, making the transmission range of the immersive space A1 narrower.

[0514] The third configuration mode is shown in the following figure. Fig.24 : In one embodiment, the angle θ1 between the reverse cancellation audio output direction L1 of the reverse audio output device 30 on the left and the virtual plane PH centered on the audio device AP and the immersive space A2 is 20 degrees to 45 degrees, and the angle θ2 between the reverse cancellation audio output direction L2 of the reverse audio output device 30 on the right and the virtual plane PH centered on the audio device AP and the immersive space A2 is 20 degrees to 45 degrees.

[0515] For the fourth configuration method, see Fig.25: The reverse audio output device 30 can be distributed in an array on the front, back, left, right, top and bottom sides of the audio device AP. For example, in (b1), the reverse audio output device 30 is set on the surface positions of the front, back, left, right, top and bottom sides of the audio device AP; or, for example, in (b2), the reverse audio output device 30 is set on the edge position at each corner of the audio device AP; or, for example, in (b3), the reverse audio output device 30 is set on the vertex position at each corner of the audio device AP; or any combination of the above positions; multiple reverse audio output devices 30 are set at the same surface position, edge position or vertex position to form an array, which is not limited in the present invention. In the described embodiment, the non-immersive space where the audio is to be eliminated can be selected according to the factory configuration or according to the setting, so as to configure the best sound field effect in accordance with the corresponding environment. Although the above is represented by dots in the drawings, the dots can be understood as the reverse audio output device 30 being a plurality of arrays distributed on the surfaces, edges, and corners, thereby more accurately adjusting the range of the non-immersive space or the direction of the sound output through the array.

[0516] In summary, the present invention provides a customized sound experience based on the user's area and voice information by determining whether the user is located in an immersive area or a non-immersive area, thereby better meeting the specific needs of individual users or adapting to different environments and situations.

[0517] The present invention has been described in detail above. The above is only one preferred embodiment of the present invention and should not be used to limit the scope of the present invention. That is, all equivalent changes and modifications made according to the scope of the patent application of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A spatial audio positioning system, characterized in that: include: At least one audio receiving device for receiving a voice message; as well as At least one processor executes a normalization module and a densely connected network after loading at least one storage device, wherein: After receiving the voice information, the normalization module performs normalization processing on the voice information to obtain normalized voice data; as well as The densely connected network receives the normalized speech data to map the normalized speech data to spatial information corresponding to the source location of the speech information.

2. The spatial audio positioning system according to claim 1, characterized in that: The at least one processor executes a coordinate mapping module after loading the at least one storage device, and the coordinate mapping module maps the input spatial information to a spatial coordinate.

3. The spatial audio positioning system according to claim 1, characterized in that: The model architecture of this densely connected network from input to output includes: An input layer, used for receiving the normalized speech data and inputting the normalized speech data into the model; A convolutional layer is used to extract the features of the output data of the previous layer and extract the features; A batch normalization layer performs batch normalization on the output data of the previous layer to adjust the distribution of multiple features; A scaling factor layer that scales the features of the output data of the previous layer and restores the representation ability of the data by introducing a learnable scaling factor; A ReLU layer applies the ReLU activation function to the output data of the previous layer, adjusting negative inputs to 0 and retaining positive inputs; A pooling layer, which reduces the spatial size of the output data of the previous layer through pooling operations; a first stacked dense block group layer connected to the output of the pooling layer, the first stacked dense block group layer comprising a plurality of first convolutional layers and a first batch normalization layer, the input of each first convolutional layer being connected to the outputs of all other preceding first convolutional layers; a first transition layer connected to the output of the first stacked dense block group layer, the first transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map; a second stacked dense block group layer connected to the output of the first transition layer, the second stacked dense block group layer comprising a plurality of second convolutional layers and a second batch normalization layer, the input of each second convolutional layer being connected to the outputs of all other previous second convolutional layers; a second transition layer connected to the second stacked dense block group layer, the second transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map; a third stacked dense block group layer connected to the second stacked dense block group layer, the third stacked dense block group layer comprising a plurality of third convolutional layers and a third batch normalization layer, the input of each third convolutional layer being connected to the outputs of all other preceding third convolutional layers; a third transition layer connected to the third stacked dense block group layer, the third transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map; a fourth stacked dense block group layer connected to the third transition layer, the fourth stacked dense block group layer comprising a plurality of fourth convolutional layers and a fourth batch normalization layer, the input of each fourth convolutional layer being connected to the outputs of all other preceding fourth convolutional layers; and A classification layer is connected to the fourth stacked dense block group layer, and maps the output of the fourth stacked dense block group layer to the spatial information.

4. An open space immersive audio system based on spatial positioning, characterized in that: include: At least one audio receiving device; at least one audio output device; At least one reverse audio output device; as well as At least one processor executes a sound elimination module, an intelligent speech recognition module, an audio processing module, a normalization module, a densely connected network, a coordinate mapping module and a processing module after loading at least one storage device, wherein: After receiving the error sound in the non-immersive area via any of the audio receiving devices, the sound cancellation module generates a reverse audio signal based on the error sound and outputs it to the audio output device to generate a reverse cancellation sound; The audio processing module outputs a play audio to the at least one audio output device, so as to play the sound effect in the immersive area; The intelligent speech recognition module receives speech information from any of the audio receiving devices and obtains a speech command by analyzing the speech information. The intelligent speech recognition module includes a waveform neural network or a parallel waveform neural network for generating a real speech signal based on the environmental sound, an MFCC unit for extracting audio features from the real speech signal, and a speech recognition module for mapping the audio features output by the MFCC unit to the speech command; The normalization module receives voice information from any of the audio receiving devices, and after receiving the voice information, performs normalization processing on the voice information to obtain normalized voice data; The densely connected network receives the normalized speech data from the normalization module to map the normalized speech data to spatial information corresponding to the source location of the speech information; The coordinate mapping module obtains the spatial information from the output of the densely connected network and maps the input spatial information to a spatial coordinate; The processing module generates a corresponding control instruction based on the voice instruction of the user and the spatial coordinates, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

5. The open space immersive audio system based on spatial positioning according to claim 4, characterized in that: The speech recognition module is a convolutional neural network model.

6. The open space immersive audio system based on spatial positioning according to claim 5, characterized in that: The intelligent speech recognition module includes a CTC training unit, which is used to calculate the distance between the output sequence and the label sequence through a connection time series classification loss function, and to update the model parameters of the speech recognition module through a back propagation algorithm.

7. The open space immersive audio system based on spatial positioning according to claim 6, characterized in that: The densely connected network is trained according to the following process: Collecting a training data set containing the voice information and the location of the source of the voice information; The voice information is normalized in advance to ensure that the data scale of the voice information is consistent; Perform forward propagation through this densely connected network to map the input data to spatial information and calculate the difference between the predicted result and the true label; Through the back propagation algorithm, the model parameters of the densely connected network are adjusted according to the loss function so that the prediction results of the densely connected network are closer to the true label; The model parameters of the densely connected network are updated according to the optimizer to minimize the loss function; as well as Repeat forward propagation, backpropagation, and parameter update operations until the model converges or reaches a predetermined number of training times.

8. The open space immersive audio system based on spatial positioning according to claim 4, characterized in that: The intelligent voice recognition module triggers a program for analyzing the voice information by specifying the voice information or a human-machine interface instruction when in a standby state.

9. The open space immersive audio system based on spatial positioning according to claim 4, characterized in that: The model architecture of this densely connected network from input to output includes: An input layer, used for receiving the normalized speech data and inputting the normalized speech data into the model; A convolutional layer, used to extract features of the output data of the previous layer; A batch normalization layer performs batch normalization on the output data of the previous layer to adjust the distribution of multiple features; A scaling factor layer that scales the features of the output data of the previous layer and restores the representation ability of the data by introducing a learnable scaling factor; A ReLU layer applies the ReLU activation function to the output data of the previous layer, adjusting negative inputs to 0 and retaining positive inputs; a first stacked dense block group layer connected to the output of the ReLU layer, the first stacked dense block group layer comprising a plurality of first convolutional layers and a first batch normalization layer, the input of each first convolutional layer being connected to the outputs of all other preceding first convolutional layers; a first transition layer connected to the output of the first stacked dense block group layer, the first transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map; a second stacked dense block group layer connected to the output of the first transition layer, the second stacked dense block group layer comprising a plurality of second convolutional layers and a second batch normalization layer, the input of each second convolutional layer being connected to the outputs of all other previous second convolutional layers; a second transition layer connected to the second stacked dense block group layer, the second transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map; a third stacked dense block group layer connected to the second stacked dense block group layer, the third stacked dense block group layer comprising a plurality of third convolutional layers and a third batch normalization layer, the input of each third convolutional layer being connected to the outputs of all other preceding third convolutional layers; a third transition layer connected to the third stacked dense block group layer, the third transition layer comprising a convolutional layer and a pooling layer for reducing the size of the feature map; a fourth stacked dense block group layer connected to the third transition layer, the fourth stacked dense block group layer comprising a plurality of fourth convolutional layers and a fourth batch normalization layer, the input of each fourth convolutional layer being connected to the outputs of all other preceding fourth convolutional layers; and A classification layer is connected to the fourth stacked dense block group layer, and maps the output of the fourth stacked dense block group layer to the spatial information.

Images