Dynamic intelligent voice open space immersive audio system and method thereof

Abstract

The invention relates to an open space immersive audio system of dynamic intelligent voice and a method thereof. The open space immersive audio system of dynamic intelligent voice comprises an audio receiving device, an audio output device, a reverse audio output device, a processor and at least one storage device. The processor runs the sound elimination module, the intelligent voice recognition module, the position detection module, the audio processing module and the processing module. The sound elimination module receives the error sound and then generates a reverse counteracting sound. The intelligent voice recognition module obtains user voice information from environment sound and analyzes the user voice information into a voice instruction. The position detection module analyzes user position information to determine an immersive or non-immersive area. And the processing module generates a corresponding control instruction according to the voice instruction and the position information. According to the method and the device, the user is determined to be located in the immersive area or the non-immersive area, and customized sound effect experience is provided according to the area where the user is located and the voice information, so that the specific requirements of individual users are better met or different environments and conditions are adapted.

Description

Technical Field

[0001] The present invention relates to an open space immersive audio system with dynamic intelligent voice and a method thereof, and in particular to an open space immersive audio system with dynamic intelligent voice that generates control instructions based on user position and voice instructions and a method thereof. Background Art

[0002] General open space environments, such as homes, offices, public areas or transportation hubs, are often susceptible to high levels of background noise, which can have a negative impact on personal comfort, concentration and work efficiency. In order to solve these problems, noise reduction systems have been born. There are two main types of existing noise reduction systems, namely passive noise reduction technology and active noise reduction systems.

[0003] Passive noise reduction techniques, such as sound-absorbing materials or physical barriers, can help reduce noise levels in open spaces. However, these techniques have limited effectiveness and may not be suitable for all environments or situations. Active noise reduction systems work by actively generating cancelling sound waves to eliminate unwanted noise. While these active noise reduction systems may be more effective than passive noise reduction techniques, they generally require the user to wear headphones or other personal audio devices, which may not be convenient or comfortable for all users.

[0004] Furthermore, existing noise reduction systems are not able to optimally meet the specific needs of individual users or adapt to different environments and situations. For the reasons stated above, there is a need for an improved noise reduction system that addresses the limitations of the prior art by providing a more versatile and user-friendly solution to reduce noise in open spaces. Summary of the invention

[0005] The main purpose of the present invention is to provide an open space immersive audio system with dynamic intelligent voice, including an audio receiving device, an audio output device, a reverse audio output device, at least one processor and at least one storage device. The at least one processor executes a sound elimination module, an intelligent voice recognition module, a position detection module, an audio processing module and a processing module after loading the at least one storage device, wherein: the sound elimination module receives the error sound in the non-immersive area through the audio receiving device, generates a reverse audio based on the error sound and outputs it to the audio output device to produce a reverse cancellation sound; the audio processing module outputs a playback audio to the audio output device to play the sound effect in the immersive area; the intelligent voice recognition module receives the ambient sound from the audio receiving device, obtains the user's voice information from the ambient sound, and obtains a voice command by parsing the voice information; the position detection module analyzes the user's position information from the ambient sound to confirm that the user is located in the immersive area or the non-immersive area; the processing module generates a corresponding control command based on the user's voice command and the position information, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control command.

[0006] Optionally, the intelligent speech recognition module includes a waveform neural network for generating a real speech signal based on the environmental sound, an MFCC unit for extracting audio features from the real speech signal, and a speech recognition module for mapping the audio features output by the MFCC unit to the voice command.

[0007] Optionally, the speech recognition module is a convolutional neural network model.

[0008] Optionally, the intelligent speech recognition module includes a CTC training unit, which is used to calculate the distance between the output sequence and the label sequence through a concatenated temporal classification loss function, and update the model parameters of the speech recognition module through a back-propagation algorithm.

[0009] Optionally, the position detection module includes a convolutional neural network, and the convolutional neural network is trained to map the input environmental sound to the voice command, and the source of the voice command is from the immersive area or the non-immersive area.

[0010] Optionally, the convolutional neural network includes a plurality of convolutional layers for extracting features of the ambient sound and a fully connected layer for mapping the features to one of the classification results of the immersive area or the non-immersive area according to weights.

[0011] Optionally, the convolutional neural network is trained according to the following process:

[0012] Forward propagation: forward propagation is performed through the CNN model to map the input data to one of the classification results of the immersive area or the non-immersive area, and the difference between the predicted result and the true label is calculated;

[0013] Back propagation: Through the back propagation algorithm, the model parameters of the convolutional neural network are adjusted according to the loss function to make the model's prediction results closer to the true label;

[0014] Parameter update: updating the model parameters of the convolutional neural network according to the optimizer to minimize the loss function; and

[0015] Repeated training: Repeat forward propagation, backpropagation, and parameter update operations until the model converges or reaches a predetermined number of training times.

[0016] Optionally, the intelligent voice recognition module is a program for parsing the voice information triggered by a specified voice information or a human-machine interface command when in standby mode. Another object of the present invention is to provide an open space immersive audio method of dynamic intelligent voice, comprising: after an audio receiving device receives ambient sound, the user's voice information is obtained from the ambient sound, and a voice command is obtained by parsing the voice information; the user's location information is analyzed from the ambient sound to confirm that the user is located in an immersive area or a non-immersive area; and a corresponding control command is generated based on the user's voice command and the location information, and the audio output mode of the audio output device or the reverse audio output mode of the reverse audio output device is switched according to the control command.

[0017] Optionally, the open space immersive audio method of the dynamic intelligent voice is a program for analyzing the voice information triggered by a specified voice information or a human-computer interface instruction.

[0018] Therefore, the present invention provides a customized sound experience according to the user's area based on voice information by determining whether the user is located in an immersive area or a non-immersive area, thereby better meeting the specific needs of individual users or adapting to different environments and situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 FIG1 is a block diagram of an open space immersive audio system with dynamic intelligent voice in the present invention (I).

[0020] Figure 2 Schematic diagram of the block diagram of the open space immersive audio system of dynamic intelligent voice in the present invention (II).

[0021] Figure 3 FIG. 1 is a block diagram of the intelligent audio adjustment system of the present invention. FIG.

[0022] Figure 4FIG. 2 is a block diagram of the intelligent audio adjustment system of the present invention. FIG.

[0023] Figure 5 Schematic diagram of the overall architecture of the intelligent speech recognition module in the present invention.

[0024] Figure 6 Schematic diagram of the architecture of the waveform neural network algorithm in the present invention.

[0025] Figure 7 It is a flowchart diagram of the waveform neural network algorithm in the present invention.

[0026] Figure 8 It is a schematic diagram of the architecture of a single node of the waveform neural network in the present invention.

[0027] Fig. 9 Schematic diagram of the calculation process of the MFCC algorithm in the present invention.

[0028] Fig.10 Schematic diagram of the algorithm flow of the Mel filter bank in the present invention.

[0029] Fig.11 It is a schematic diagram of the discrete cosine transform process in the present invention.

[0030] Fig.12 Schematic diagram of the convolutional neural network model architecture used in the speech recognition module of the present invention.

[0031] Fig.13 Schematic diagram of the training process of the CTC trainer in the present invention.

[0032] Fig.14 FIG. 1 is a schematic diagram of the structure of the position detection module in the present invention (I).

[0033] Fig.15 FIG. 2 is a schematic diagram of the structure of the position detection module in the present invention. FIG.

[0034] Fig.16 Schematic diagram of the training process of the convolutional neural network classifier in the present invention.

[0035] Fig.17 Schematic diagram of the overall operation flow of the open space immersive audio system of the present invention.

[0036] Fig.18 FIG. 1 is a schematic diagram of the configuration of the immersive audio system of the present invention (I).

[0037] Fig.19 FIG. 2 is a schematic diagram of the configuration of the immersive audio system of the present invention. FIG.

[0038] Fig. 20 FIG. 3 is a schematic diagram of the configuration of the immersive audio system in the present invention. FIG.

[0039] Fig.21 Schematic diagram (IV) of the configuration of the immersive audio system in the present invention.

[0040] The following are marked in the figure:

[0041] 100 Immersive Audio System

[0042] 10 Audio receiving device

[0043] 11 Microphone

[0044] 12 Preamplifier

[0045] 13 Anti-aliasing filter

[0046] 14 Analog-to-digital converter

[0047] 20 Audio output device

[0048] 21 Speakers

[0049] 22 Power Amplifier

[0050] 23 Reconstruction filter

[0051] 24-bit digital-to-analog converter

[0052] 30 Reverse audio output device

[0053] 31 Speakers

[0054] 32 Power Amplifier

[0055] 33 Reconstruction filter

[0056] 34 Digital-to-analog converter

[0057] 40 Intelligent audio adjustment system

[0058] 41 Processor

[0059] 42 Storage Devices

[0060] M1 Sound Cancellation Module

[0061] M2 Audio Processing Module

[0062] M3 Intelligent Voice Recognition Module

[0063] LA1 convolutional layer

[0064] LA2 Pooling Layer

[0065] LA3 fully connected layer

[0066] M31 Waveform Neural Network

[0067] M32 MFCC Unit

[0068] M33 Voice Recognition Module

[0069] M34 CTC Trainer

[0070] M4 Position Detection Module

[0071] M41 Convolutional Neural Network Classifier

[0072] AW Ambient Audio

[0073] SP Time spectrum

[0074] CA Convolutional Neural Network

[0075] FM feature map

[0076] FC fully connected layer

[0077] M5 Processing Module

[0078] AP Audio Device

[0079] A Immersive Space

[0080] A1 Immersive Space

[0081] B Non-immersive space

[0082] BB1 Non-immersive space

[0083] L1 reverses the direction of the audio output L2 reverses the direction of the audio output

[0084] PH Virtual Face DETAILED DESCRIPTION

[0085] The detailed description and technical content of the present invention are described as follows with reference to the accompanying drawings. The "one side" mentioned in the present invention can be the upper side, lower side, left side, right side, front side, rear side of the corresponding object, or be arranged at any adjacent position of the object, or be directly or indirectly connected to the object, etc., which is not limited in the present invention.

[0086] The "device", "module", "model", "unit" and the corresponding functions described in the present invention can be performed by a single chip or a combination of multiple chips, and the number of such chip configurations is not within the scope of the present invention. In addition, the chip can be, but is not limited to, a combination of processors (Processor), central processing unit (CPU), microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) and other devices, which are not limited in the present invention.

[0087] The "immersive space" described in the present invention refers to the target area of ​​sound effects in the playback environment, that is, the area where the sound is expected to act; the "non-immersive space" described in the present invention refers to the area in the environment where sound is not desired to be transmitted, and can be any area other than the "immersive space". In another embodiment, the "non-immersive space" can also be a specific direction where the volume is desired to be reduced in order to enhance the directionality of the playback sound, thereby enhancing the stereo effect in the playback environment by reducing the volume in the specific direction.

[0088] The immersive system disclosed in the present invention can be applied to audio devices such as televisions, laptops, mobile devices, soundbars, portable devices or car head units. The sound output direction or range of the audio devices can be controlled by the immersive audio system to reduce the impact of spillover sound on non-target areas.

[0089] The following is a description of one embodiment of the present invention. Figure 1 and Figure 2 , which are the block diagram (1) and block diagram (2) of the open space immersive audio system with dynamic intelligent voice in the present invention, as shown in the figure: the open space immersive audio system 100 of this embodiment mainly includes an audio receiving device 10, an audio output device 20, a reverse audio output device 30 and an intelligent audio adjustment system 40.

[0090] The audio receiving device 10 is used to receive ambient sound. The audio receiving device 10 is used to receive error sound in a non-immersive space and convert it into error audio; in another embodiment, another audio receiving device may be further included to specifically receive ambient sound in an immersive space, which is not limited in the present invention. In another embodiment, another audio receiving device may be further included to specifically receive voice commands in an immersive space or a non-immersive space, which is not limited in the present invention. In one embodiment, the audio receiving device 10 includes a microphone 11, a preamplifier 12 connected to the rear end of the microphone 11, an anti-aliasing filter 13 connected to the rear end of the preamplifier 12, and an analog-to-digital converter 14 connected to the rear end of the anti-aliasing filter 13. These devices can be increased or decreased according to practical needs, which is not limited in the present invention.

[0091] The audio output device 20 outputs the playback sound according to the audio source signal. The "audio source signal" mentioned here refers to the original sound effect signal provided by the intelligent audio adjustment system 40, that is, the original file of the sound effect. "Playback sound" is the sound effect broadcast by the audio output device 20 in the real field environment (the space included in the immersive area), which is first explained here. In one embodiment, the audio output device 20 includes a speaker 21, a power amplifier 22 connected to the front end of the speaker 21, a reconstruction filter 23 connected to the front end of the power amplifier 22, and a digital-to-analog converter 24 connected to the front end of the reconstruction filter 23. These devices can be increased or decreased according to practical needs, and are not limited in the present invention.

[0092] The reverse audio output device 30 outputs reverse cancellation sound according to the reverse audio. The "reverse audio" mentioned here refers to the original signal provided by the intelligent audio adjustment system 40 for canceling the sound played in the non-immersive area. The "reverse cancellation sound" is the sound effect broadcast by the reverse audio output device 30 in the real field environment (the space included in the non-immersive area), which is first explained here. In one embodiment, the reverse audio output device 30 includes a speaker 31, a power amplifier 32 connected to the front end of the speaker 31, a reconstruction filter 33 connected to the front end of the power amplifier 32, and a digital-to-analog converter 34 connected to the front end of the reconstruction filter 33. These devices can be increased or decreased according to practical needs, and are not limited in the present invention.

[0093] The intelligent audio adjustment system 40 is connected to the audio receiving device 10, the audio output device 20 and the reverse audio output device 30, and controls the output of the audio output device 20 and the reverse audio output device 30 through the error sound and voice command received by the audio receiving device 10. The intelligent audio adjustment system 40 includes a processor 41 and a storage device 42. The processor 41 executes the corresponding program after loading the program code stored in the storage device 42.

[0094] For the work performed by the intelligent audio adjustment system 40, please refer to the following Figure 3 and Figure 4 , which are the block diagram (1) and block diagram (2) of the intelligent audio adjustment system in the present invention, as shown in the figure: the intelligent audio adjustment system 40 in the present invention includes a sound elimination module M1, an audio processing module M2, an intelligent voice recognition module M3, a position detection module M4 and a processing module M5 according to different executed programs.

[0095] After the sound cancellation module M1 receives the error sound in the non-immersive area through the audio receiving device 10, it generates an inverted audio (Inverted Audio) based on the error sound and outputs it to the audio output device 20 to produce an inverted cancellation sound. In one embodiment, the reference audio may include, for example, a sound source signal or a playback sound received by the audio receiving device 10 in the immersive area, which is not limited in the present invention. The sound cancellation module M1 may directly receive the playback audio from the microphone 11, and adjust the phase of the playback audio to generate an inverted audio with the same amplitude but opposite phase relative to the playback sound. Specifically, the sound cancellation module M1 may be, for example, but not limited to, completed by phase inversion, fast Fourier transform (FFT), time domain convolution, audio decoding and encoding, neural network (such as machine learning network, deep learning network or recurrent neural network, etc.) or other similar methods, which are not limited in the present invention.

[0096] The audio processing module M2 is used to output the played audio to the audio output device 20, so as to play the sound effects in the immersive area. Specifically, the audio processing module M2 outputs the audio source signal to the audio output device 20 after reading the long-term data or temporary data of any storage device, so as to play the sound through the audio output device 20. In one embodiment, the audio processing module M2 can be, for example, any sound card, onboard sound card or sound processing chip, etc. Such sound cards include, for example, decoders, digital to analog converters (Digital to Analog Converter, DAC), analog to digital converters (Analog to Digital Converter), amplifiers, filters, sound synthesizers, etc., which are not limited in the present invention.

[0097] The intelligent voice recognition module M3 is used to obtain the user's voice information from the ambient sound after the audio receiving device 10 receives the ambient sound, and obtain the voice command by analyzing the voice information. For the architecture of the intelligent voice recognition module M3, please refer to Figure 5 , which is a schematic diagram of the overall architecture of the intelligent speech recognition module in the present invention, as shown in the figure: the intelligent speech recognition module M3 in this embodiment mainly includes a waveform neural network M31 (Wavenet), an MFCC unit M32, a speech recognition module M33 and a CTC trainer M34.

[0098] The waveform neural network M31 is used to generate a real speech signal based on the environmental sound, the MFCC unit M32 is used to extract audio features from the real speech signal, the speech recognition module M33 is used to map the audio features output by the MFCC unit M32 to voice commands, and the CTC training unit M34 is used to calculate the distance between the output sequence and the label sequence through the connection time classification loss function (CTC loss), and update the model parameters of the speech recognition module M33 through the back propagation algorithm.

[0099] Please refer to the architecture of the waveform neural network M31. Figure 6 , is a schematic diagram of the architecture of the waveform neural network algorithm in the present invention, as shown in the figure: the waveform neural network M31 uses dilated causal convolution, which can expand the observation range, but also takes into account the causal relationship of the signal. As shown in the figure, the dilation rate used in the waveform neural network M31 is 1, 2, 4, 8, ..., when the dilation rate is 512 (only 8 is shown in the figure), it will continue to cycle from 1, and a group of 1 to 512 requires 1024 inputs. Taking the signal with a sampling frequency of 16kHz as an example, it can focus on about 64ms of information, which can be quite helpful for the accuracy of the speech signal model. The waveform neural network M31 makes full use of causal and dilated convolution, and uses a series of small (length = 2) convolution filters with exponentially increasing dilation factors. Therefore, the receptive field of the network increases exponentially with the increase of the network depth. The causal relationship is achieved by asymmetric padding proportional to the dilation factor, which prevents the activation from being propagated back in time.

[0100] For the calculation steps of the waveform neural network M31, please refer to Figure 7 , which is a flow chart of the waveform neural network algorithm in the present invention, as shown in the figure: The waveform neural network (WaveNet) M31 in the present invention is a speech generation model that uses a deep convolutional neural network to process audio features extracted from environmental sounds to generate a more realistic real speech signal.

[0101] Step S01: Initialize the Wavenet model. In this step, relevant parameters are set for the Wavenet model to prepare for subsequent network structure and calculation.

[0102] The following are some of the main parameters.

[0103] Batch size: During the training process, dividing the data into multiple small batches for training helps speed up the training process and improve the generalization ability of the model. The batch size refers to the number of samples contained in each batch.

[0104] Filter width: Filter width refers to the size of the convolution kernel in the time dimension. In the Wavenet model, the filter width is usually set to 2, which means that the convolution kernel can cover two consecutive input samples in the time dimension.

[0105] Hidden size: The hidden size refers to the number of neurons in each layer of the network. This parameter determines the network's representation ability and computational effort.

[0106] Number of layers: In the WaveNet model, the number of layers determines the depth of the network. Each layer contains a causal dilation convolution layer. The increase in the number of layers can increase the receptive field of the network, allowing the model to capture longer time series dependencies.

[0107] Dilation rate: The dilation rate in the dilated convolution affects the step size of the convolution kernel in the time dimension. In the waveform neural network model, the dilation rate of each layer is usually set to an exponential power of 2, which can achieve an exponential expansion of the receptive field.

[0108] Step S02: Create and initialize bias variables. For example, the receptive field size bias, etc. In this step, some bias variables are created for the waveform neural network model. The bias variables play a translation role in the neural network. Even if all inputs are zero, the bias variables can activate neurons.

[0109] Here are some of the main variance variables.

[0110] Convolution bias: These bias variables are added to the output of the convolutional layer to increase the activation of neurons. Each convolutional layer needs to create a bias variable with the same number of output channels as the layer.

[0111] Receptive field size bias: In the waveform neural network model, the receptive field will expand as the number of layers increases. The dilation rate of each causal dilation convolution layer can be regarded as the determining factor of the receptive field size of that layer.

[0112] Global conditioning bias: If the waveform neural network model uses global conditioning, a bias variable needs to be created for the global conditioning. This bias variable is added to the global conditioning to help the network generate outputs based on global information.

[0113] When creating bias variables, you need to make sure that their shapes match the corresponding network structure. Usually, the bias variables are initialized to zero or a small value close to zero. This ensures that the activation of the network is not too large or too small at the beginning of training, which helps improve the stability of training.

[0114] Step S03: Create variables required by the network, such as filter variables, dilation variables, etc. In this step, we need to create some variables for the waveform neural network model to store and adjust the weights required for the convolution layer, dilation layer, etc.

[0115] Here are some of the main variables.

[0116] Filter variables: Each convolutional layer and dilation layer needs to create a filter variable to store its weights. The shape of the weight matrix depends on the size of the convolution kernel, the number of input channels, and the number of output channels. The weight matrix is ​​usually continuously updated during training to minimize the loss.

[0117] Dilation variables: The dilated convolution layer requires a dilation factor to control the span of the dilated convolution kernel on the input. The dilation factor for each layer may be different and is an integer power of 2. These dilation variables are usually set at model initialization and remain unchanged throughout the training process.

[0118] Gate variable: For expansion layers that use gate control units, you need to create a gate variable to store the weights of the gate unit. The gate unit is used to control which information in the output of the convolutional layer can flow to the next layer. The gate variable is similar to the filter variable, has the corresponding shape and is updated during training.

[0119] Global Condition Variable: If the waveform neural network model uses global conditions, you need to create a variable for the global condition. This variable is used to learn the mapping from global information to output, and its shape depends on the dimension of the global condition and the number of output channels.

[0120] When creating these variables, you need to make sure their shapes match the corresponding network structures. These variables are usually initialized with random variables, such as Gaussian distribution or uniform distribution, to ensure that the network has enough diversity at the beginning of training.

[0121] Step S04: Create a causal convolution layer. The causal convolution layer is one of the basic components of the waveform neural network model, ensuring that only past information is used when processing time series data. In the causal convolution layer, the convolution operation will not go beyond the current moment, thus avoiding the use of future data.

[0122] Following are the main steps to create a causal convolutional layer.

[0123] Define the convolution kernel size: Choose an appropriate convolution kernel size to capture local features in the input data. The size of the convolution kernel affects the receptive field and number of parameters of the model.

[0124] Define the number of input and output channels: Set the number of input channels (matching the number of features of the input data) and the number of output channels (used to adjust the representation capacity of the model) for the causal convolution layer.

[0125] Initialize convolution weights and biases: Initialize the weights and biases of the convolution layer according to the convolution kernel size, the number of input channels, and the number of output channels. These weights and biases are continuously updated during training to minimize the loss.

[0126] Perform causal convolution operation: Perform causal convolution operation on the input data, which means that only the current and previous data are used during the convolution process. To implement causal convolution, an appropriate amount of zero padding can be added to the beginning of the input data so that each output element is calculated from the corresponding input element and the previous element.

[0127] Activation function: Apply an activation function such as ReLU, tanh function, or other nonlinear functions to the output of the convolution operation to increase the representation power of the model.

[0128] After completing the above steps, a causal convolution layer is created. In the waveform neural network model, multiple causal convolution layers and causal dilated convolution layers are used to capture features at different time scales.

[0129] Step S05: Create a causal dilated convolution layer. The causal dilated convolution layer includes the establishment of filter unit layers and gate control units to achieve the purpose of increasing the receptive field. The causal dilated convolution layer is the core component of the waveform neural network model, which is used to capture long-range temporal dependencies. In this step, create a causal dilated convolution layer for the model.

[0130] Here are the main steps.

[0131] Definition of dilation rate: The dilation rate is an integer that represents the distance between two adjacent convolution kernels during the convolution process. The larger the dilation rate, the wider the receptive field, and the longer range of temporal dependencies can be captured.

[0132] Initialize the dilated convolution weights and biases: Initialize the weights and biases of the dilated convolution layer according to the convolution kernel size, the number of input channels, the number of output channels, and the dilation rate. These weights and biases will be continuously updated during training to minimize the loss.

[0133] Create a filter layer and a gate layer: These two layers are responsible for capturing different features in the input data. The filter layer is responsible for performing convolution operations on the input data to extract features, while the gate layer is responsible for controlling the speed at which information flows through the network.

[0134] Perform causal dilated convolution: Perform a causal dilated convolution on the input data, ensuring that only the current and previous data are used. To implement causal dilated convolution, an appropriate amount of zero padding can be added to the beginning of the input data. The amount of zero padding depends on the kernel size and the dilation rate.

[0135] Activation functions of filter units and gate units: Activation functions such as tanh and sigmoid are applied to the outputs of filter units and gate units respectively. These nonlinear activation functions help enhance the representation ability of the model.

[0136] Element-wise multiplication and addition: The outputs of the filter unit and the gate unit are multiplied element-wise and the result is added to the original input to form a residual connection. The residual connection helps in the training and gradient flow of the network.

[0137] Finally, a causal dilated convolution layer is created in the waveform neural network model. The main function of this layer is to capture long-range temporal dependencies and extract and control different features in the input data through filtering units and gating units. This layer also includes residual connections, which helps with model training and gradient flow. In practical applications, multiple causal dilated convolution layers can be stacked as needed to improve the representation ability of the model.

[0138] The following are the sub-steps for one-hot encoding of waveform amplitudes. Figure 8 , is a schematic diagram of the architecture of a single node of the waveform neural network in the present invention, as shown in the figure: Since the original audio is usually stored as a string of 16-bit integer values ​​(one for each time period), a softmax layer needs to output 65,536 probabilities in each time period to simulate all possible values. In order to make this problem easier to solve, the data is first subjected to μ-law compression conversion (ITU-T, 1988) and then quantized into 256 possible values. The input signal is a waveform of a sound signal, which will oscillate back and forth between +1 and -1. Due to the desired output form, the waveform of a sound signal will also oscillate back and forth between +1 and -1. If a function such as ReLU whose value range does not include 0 to -1 is used, the output will be truncated in the part <0. Therefore, it is hoped to find an activation function (Activation function) with a value range of -1 to +1, preferably with positive and negative symmetry. That is, an odd function, and the tanh function is used here.

[0139] About Figure 8 The functions of each block are described as follows:

[0140] Causal convolution layer B1: Causal convolution layer B1 ensures that the model is causal in time, and the input layer maps the audio signal into features;

[0141] Dilated convolution layer B2: Use multiple dilated convolution layers B2 to extract features, obtain features at different resolutions and enhance the receptive field of the model;

[0142] tanh function B3: The output range of tanh function B3 is between -1 and +1. It is used to capture activation features in audio data. The role of tanh function B3 is to provide richer expression capabilities in capturing the amplitude changes and waveform details of the signal, because its output range covers negative and positive values, and it can better represent the oscillation of the waveform between positive and negative amplitudes;

[0143] Sigmoid function B4 (σ): The output of the sigmoid function B4 ranges from 0 to 1 and is used in the gating mechanism to learn how to combine features between different layers. In the context of the WaveNet model, the role of the sigmoid function B4 is to pass or suppress certain features as needed to implement an adaptive gating mechanism.

[0144] Multiplication unit B5: Multiplication unit B5 multiplies the output of tanh function B3 and sigmoid function B4 element by element, so as to realize the adaptive gating mechanism between network layers;

[0145] The first convolution layer B6 (1×1): The output after multiplication passes through a 1×1 convolution layer. This layer can realize nonlinear transformation of features. In the residual block, the output of the tanh function B3 and the sigmoid function B4 is element-wise multiplied and then passes through the first convolution layer B6. The purpose of this convolution layer is to perform nonlinear transformation on the output to obtain a new feature representation. This process is performed inside the residual block and is used to learn linear and nonlinear combinations between features.

[0146] Addition unit B7 (+): The addition operation can help the network learn the identity mapping, which is helpful for training deep networks. The output of the 1x1 convolution is added to the original input to form a residual output. This addition operation can help the network learn the identity mapping, which is helpful for training deep networks;

[0147] Addition unit B8 (+) and ReLU unit B9: The outputs of all residual blocks are added together, and then activated by the ReLU unit B9. This step can achieve nonlinear combination of features.

[0148] The second convolutional layer B10 (1×1): The output of the ReLU unit B9 is passed through a 1×1 convolutional layer. The purpose of this convolution is to reduce the dimension of the features so that they can be more easily combined in the subsequent stacking process.

[0149] ReLU unit B11: The output of the second convolutional layer B10 is passed through the ReLU activation function. This step introduces nonlinearity again, which helps to extract higher-order features;

[0150] The third convolutional layer B12 (1×1): The output of the ReLU unit B11 is passed through another 1x1 convolutional layer. This layer maps the features to the same dimension as the target output (e.g., the probability distribution of the audio sample) so that the probability distribution can be calculated before the final Softmax layer.

[0151] Softmax output B13: The output of the 1x1 convolutional layer is passed through the Softmax function to convert it into a probability distribution. This gives a probability for each possible audio sample value to generate a new audio signal.

[0152] The following expression describes the operation process of the gate control unit:

[0153] Z = tanh(w f *x)⊙σ(wg*x);

[0154] Where x is the input signal of the network, Z is the final output signal after the filter layer and the gate operation, wf is the weight matrix of the filter, wg is the weight matrix of the gate, * is the convolution operator, ⊙ is the element-level multiplication operator, which is used to multiply the output of the filter by the output of the gate element by element, tanh(·): hyperbolic tangent activation function, which is used to compress the output of the filter to the range between -1 and 1, and σ9) is the sigmoid activation function, which is used to compress the output of the gate to the range between 0 and 1.

[0155] Step S06: Perform convolution of a single convolutional layer. In this step, the past weights are multiplied by the past inputs, and the results are combined and output. The main purpose of the convolution operation is to extract local features of the signal in the time domain.

[0156] Specifically, the convolution operation includes the following steps.

[0157] Step SS1: Align the input signal with the convolution kernel (filter), usually starting from the beginning of the signal.

[0158] Step SS2: perform dot multiplication of the aligned signal and the convolution kernel (multiply the corresponding elements and then sum them).

[0159] Step SS3: Store the dot product result as an element of the output signal.

[0160] Step SS4: Move the convolution kernel along the input signal and repeat steps SS2 and SS3 until the end of the signal is reached.

[0161] Through this process, local features of the input signal can be extracted into the output signal. This is especially important in audio processing because it helps capture the frequency content and time domain features in the sound. In the waveform neural network model, these convolution operations are performed in the causal convolution layer and the causal dilated convolution layer.

[0162] Step S07: Build the waveform neural network model. For example, create a list for the output, add a defined dilation layer, etc. In this step, the waveform neural network model is stacked with multiple causal dilation convolutional layers. To build the model, the following sub-steps will be performed:

[0163] Create a list for outputs: This list will store the outputs obtained from each expansion layer. Each element of the list will be the output of an expansion layer.

[0164] Add defined dilation layers: Add the previously created causal dilated convolutional layers to the model. Each dilation layer will capture features of different time ranges in the input data. The number of dilation layers and the dilation rate can be adjusted according to the specific application scenario.

[0165] Residual and skip connections: For each expanded layer, its output is added to the output list and its residual connection is added to the output of the previous layer. Skip connections connect the output of each expanded layer directly to the last layer of the model, which helps the gradient flow better throughout the network.

[0166] Add all elements in the output list: Add all elements in the output list item by item to get the final waveform neural network output. This output will contain the time domain features captured by the entire network.

[0167] In summary, in step S07, the entire waveform neural network model is constructed, which consists of multiple causal expansion convolutional layers and uses residual and skip connections to promote better gradient circulation in the network. This model can capture the long-range time domain features in the input signal and encode them into the output.

[0168] Step S08: Construct an efficient incremental generator. It is used to quickly generate output samples when new input samples arrive. In this step, an efficient incremental generator is constructed to generate new audio samples one by one. Since the waveform neural network is an autoregressive model, it can generate one sample at a time and use the generated sample as part of future input. However, this process can be very time-consuming, especially for generating long time series. Therefore, it is necessary to develop an efficient method to generate audio samples without recalculating the entire model every time.

[0169] To achieve this goal, an incremental generator can be constructed in the following sub-steps.

[0170] Cache intermediate calculations: During the generation process, for each input sample, we can cache the results of the intermediate calculations. In this way, in the next generation, we can directly use these cached results without recalculating the entire model.

[0171] Efficiently update the cache: Every time a new sample is generated, we need to update the cached results so that the newly generated samples are included in subsequent calculations. We can achieve this by efficiently updating the cache, thereby reducing the computational cost.

[0172] Fast generation of new samples: By leveraging cached intermediate results and updating the cache efficiently, we can quickly generate new audio samples without running the entire model every time.

[0173] In summary, in step S08, an efficient incremental generator is constructed, which quickly generates new audio samples by caching intermediate calculation results and effectively updating the cache.

[0174] Step S09: Encode the waveform amplitude using one-hot encoding. One-hot encoding is a method of converting a categorical variable into a binary vector. In WaveNet, audio samples are usually represented by quantized values, which are integers representing the amplitude level of the audio waveform at a specific time point.

[0175] The purpose of one-hot encoding the amplitude is to represent the audio waveform as a vector with fixed dimensions so that it can be input into the neural network. This makes it easier for the neural network to learn the characteristics of the waveform and improve the quality of the generated audio.

[0176] Audio quantization: First, the continuous amplitude values ​​of the audio waveform are mapped to a finite set of integers. This can be achieved by rounding the amplitude value to the nearest integer, or using more complex quantization methods such as μ-law encoding. In one embodiment, quantization can be performed using the following expression:

[0177]

[0178] -1 <x t <1,μ=255;

[0179] x t is the continuous amplitude value of the signal at a specific time point t, μ is the parameter of μ-law encoding, which controls the number of quantization levels. u=255, which means that the audio amplitude value is quantized into 256 discrete levels, sign(x t ) is x t The sign of x t If it is positive, it is 1; if it is negative, it is -1.

[0180] Create a one-hot encoded vector: For each quantized value, create an all-zero vector with a length equal to the total number of quantized values. Then, set the vector element corresponding to that quantized value to 1.

[0181] Replace the original audio waveform: Replace each quantized value in the original audio waveform with the created one-hot encoded vector.

[0182] After the above steps, we convert the amplitude of the audio waveform into a one-hot encoding vector for input into the waveform neural network model. This makes it easier for the model to learn audio features and improves the quality of generated audio.

[0183] Step S10: Return the embedding of the global condition. In the waveform neural network M31, the global condition may be some features related to the entire audio segment, such as the category of music, speaker identity, etc. The global condition can help the model generate audios of different styles based on these features.

[0184] In this step, the embedding vector of the global condition is calculated, which is a vector that maps the global condition features to a low-dimensional space. The embedding vector can convert the categorical features into continuous values ​​for better use with the neural network model.

[0185] Following are the sub-steps to compute the global conditional embedding.

[0186] Define global conditions: First, determine the features to use as global conditions, such as music category, speaker identity, etc.

[0187] Encode global conditions: Encode the global conditions, usually using one-hot encoding or integer encoding. This produces an encoding vector representing the global conditions.

[0188] Create an embedding matrix: Initialize an embedding matrix with dimension (number of global conditional categories x embedding vector dimension).

[0189] Find the globally conditioned embedding: Use the globally conditioned encoding vector to find the corresponding embedding vector in the embedding matrix. For example, if one-hot encoding is used, the embedding vector can be obtained by multiplying the encoding vector with the embedding matrix.

[0190] Adding global conditional embeddings: Global conditional embedding vectors are added to the appropriate locations of the waveform neural network model to take global conditions into account during the generation process. The global conditional embedding vectors can be combined with local features (such as the one-hot encoding of the audio waveform) and propagated through the layers of the neural network.

[0191] After the above steps, the global conditions are embedded into the waveform neural network model to generate audio with specific style or characteristics.

[0192] Step S11: Calculate the probability distribution of the next sample based on all samples of the input waveform. In the waveform neural network model, the goal is to predict the probability distribution of the next sample based on all samples of the input waveform. These probabilities can then be used to generate new audio samples. The following are the sub-steps to do this.

[0193] Pass the input waveform through the waveform neural network model: Pass the input waveform (one-dimensional time series data) through the constructed waveform neural network model to obtain the output of the last layer.

[0194] Calculate the probability distribution: Pass the output of the last layer through the softmax activation function to convert it into a probability distribution. The softmax function can convert the output value into a probability value so that the sum of all probabilities is 1.

[0195] Select the most likely sample: Based on the calculated probability distribution, select the sample with the highest probability as the audio sample for the next moment. Other sampling methods can also be used, such as random sampling based on probability distribution.

[0196] By executing the above sub-steps, the probability distribution of the next audio sample can be generated based on all samples of the input waveform. This helps the model generate audio signals with consistency and continuity.

[0197] Step S12: Based on a single sample and all the samples that have passed before, the probability distribution of the next sample is calculated step by step. In this step, the audio sequence is generated step by step using the waveform neural network model. This means predicting the probability distribution of the next sample based on the current sample and all the previous samples. The following are the sub-steps to do this.

[0198] Step SA1, initialize the audio sequence: create an empty audio sequence to store the samples generated by the model.

[0199] Step SA2, generate initial samples: take the current audio sequence as input and use the WaveNet model to generate the probability distribution of the next sample (as described in step S11).

[0200] Step SA3, select a sample: select a sample according to the generated probability distribution (for example, select the sample with the highest probability or randomly select a sample according to the probability distribution).

[0201] Step SA4, updating the audio sequence: adding the selected samples to the audio sequence.

[0202] Step SA5, repeat the generation process: repeat the above steps SA2 to SA4 until an audio sequence of the required length is generated.

[0203] In this way, the waveform neural network model can gradually generate an audio sequence, gradually calculating the probability distribution of the next sample based on a single sample and all the samples that have passed before. This makes the generated audio more consistent and continuous.

[0204] In this step, the probability distribution of the next audio sample is generated according to the following expression:

[0205]

[0206] Where x={x1,x2,…,x T} is the waveform sampling point, P(x|λ) is the probability of the entire sequence x given the model parameter λ, λ is the model parameter, T is the total time step of the sequence x, x t The samples at time step t, x1,...,x t-1 is the first t-1 samples of the sequence x.

[0207] Step S13: Create a waveform neural network and return the autoencoder loss. The autoencoder loss is used to evaluate the quality of the generated waveform and optimize the model during the training process. In this step, a waveform neural network model is created and the autoencoder loss is calculated during the training process. The autoencoder loss is used to measure the performance of the model in reconstructing the input signal, usually using cross entropy loss or mean squared error as the evaluation criteria. The following are the sub-steps to do this.

[0208] Constructing a waveform neural network model: According to the previous steps S01 to S08, a complete waveform neural network model is constructed.

[0209] Training data preparation: Split the raw audio data into input and target output. The typical setting is to use the input as the previous time step of the target output.

[0210] Prediction: Use the constructed waveform neural network model to predict the input data and obtain the predicted probability distribution.

[0211] Calculate loss: Calculate the autoencoding loss based on the predicted probability distribution and the target output. Common loss functions include cross entropy loss and mean squared error.

[0212] Backpropagation and optimization: Based on the calculated loss, backpropagate and optimize the parameters in the waveform neural network model to minimize the loss.

[0213] Repeat the training process: Through multiple iterations of the training process, the model parameters are gradually optimized until convergence or the specified stopping condition is reached.

[0214] Through these sub-steps, a waveform neural network model can be created and the performance of the model during the training process can be measured by calculating the auto-encoding loss. The training process aims to minimize the loss so that when generating audio sequences, it can better reconstruct the input signal and ultimately generate more realistic speech signals.

[0215] The MFCC unit M32 is used to perform the MFCC method. MFCCs (Mel Frequency Cepstral Coefficients) is an algorithm for extracting audio features, which is used to convert the original audio into feature information for subsequent processing. In the present invention, the MFCC learning waveform neural network model receives the human voice features of the real speech signal.

[0216] For the calculation steps of MFCC, please refer to Fig. 9 , is a schematic diagram of the calculation process of the MFCC algorithm in the present invention, as shown in the figure: The following is a detailed description of the main steps of MFCC:

[0217] Step S21, audio file reading: In this step, an audio file (eg, .wav) of a real voice signal is read. After reading the audio file, a digital signal containing audio data can be obtained.

[0218] Step S22, pre-emphasis: In order to balance the spectrum in the audio signal, we perform pre-emphasis in this step. Usually a first-order high-pass filter is used to process the audio signal, which helps to strengthen the high-frequency components and reduce the resonance phenomenon in the signal. The expression of pre-emphasis is as follows:

[0219] s′ n =s n -as n-1 ;

[0220] Among them, s′ n is the output, s n is the input, s n-1 is the input of the previous stage, and a is any parameter between 0.9 and 1.

[0221] Step S23, framing: dividing the audio signal into multiple frames of a certain length. Since the speech signal is time-varying, the framing process can help us extract features of the signal within a local time range. In one embodiment, the length of each frame can be, for example, but not limited to, 20-30 milliseconds, and there is a certain overlap between adjacent frames, which is not limited in the present invention.

[0222] The expression for setting the number of frames is as follows:

[0223] F n =f s ×F L ;

[0224] Among them, F n is the number of frames, f s is the sampling frequency, F L is the frame length.

[0225] The expression of the detection and translation setting is as follows:

[0226]

[0227] Among them, F S is the frame translation, f s is the sampling frequency, F L is the frame length.

[0228] Step S24, windowing: In order to reduce signal interference at the frame boundary, we need to apply a window function (such as a Hamming window) to each frame. The window function can make the signal at the frame boundary transition smoothly and reduce the frequency domain fluctuation caused by the truncation of the time domain signal. The expression of the Hamming window is as follows:

[0229]

[0230] 1≤n≤N;

[0231] Among them, Ham(N) is a window function with a length of N, n represents the index of the point of the window function (between 1 and N), and N represents the length of the Hamming window function.

[0232] Step S25, Fourier transform: Perform Fourier transform on the windowed signal, convert it from the time domain to the frequency domain, and obtain the spectrum of the signal. When the transform size is 2 to the power of N, the fast Fourier transform (FFT) can be used to speed up the calculation. The expression of Fourier transform is as follows:

[0233]

[0234] 0≤k≤N;

[0235] Among them, x a (k) is the value (complex number) of the kth point in the frequency domain (representing the component of the signal at frequency k / N), x(n) is the value of the nth point in the time domain, N is the number of points of the Fourier transform (used to determine the accuracy of the conversion), k is the index in the frequency domain (0 to N-1), and n is the index in the time domain (0 to N-1). Using the above expression, the time domain signal x(n) can be converted to the frequency domain signal x a (k).

[0236] Step S26, Mel filter bank: The spectrum obtained after Fourier transform is passed through a set of Mel filters to simulate the human ear's perception of sound. The Mel scale is related to the frequency resolution ability of the human ear, with a higher resolution for low frequencies and a lower resolution for high frequencies. The Mel filter bank usually contains multiple triangular filters, which divide the spectrum into different frequency bands and calculate the energy of each frequency band. This can capture important information in the speech signal while reducing irrelevant details. For the algorithm executed by the Mel filter bank, please refer to Fig.10 , is a schematic diagram of the algorithm flow of the Mel filter bank in the present invention, as shown in the figure:

[0237] Step S31, determine the Mel scale range: select a suitable Mel scale range, usually select frequencies within the human hearing range, such as 20Hz to 20kHz. Then convert the frequencies in this range into Mel scale. The conversion expression of Mel scale is as follows:

[0238]

[0239] Where Mel(f) is the Mel scale and f is the original frequency.

[0240] Step S32, define the number of Mel filters: select an appropriate number n of Mel filters according to application requirements and computing resources. Usually, the number of Mel filters is 20-40.

[0241] Step S33, inserting n positions within the Mel scale range: uniformly inserting n positions within the selected Mel scale range, and these positions will serve as the center points of the Mel filter.

[0242] Step S34, convert n+2 Mel frequencies into frequencies: convert the n Mel scale center points obtained in the previous step back into frequencies, add the start and end frequencies, and obtain a total of n+2 frequency values. The expression of the inverse Mel function is as follows:

[0243] f=700·(10 m / 2595 -1);

[0244] Where m is the Mel frequency and f is the corresponding original frequency in Hertz.

[0245] Step S35, calculate n+2 index numbers k: map n+2 frequency values ​​to the spectrum index after Fourier transformation to obtain n+2 index numbers k. These indexes will be used to calculate the frequency response of the Mel filter. The expression of index number k is as follows:

[0246]

[0247] Where k is the index number; N is the number of FFT points; fs is the sampling frequency; f m is the value after converting the Mel scale back to frequency.

[0248] Step S36, Hm matrix: According to the index number k obtained in the above steps, construct an Hm matrix of n rows. Each row represents the frequency response of a Mel filter. The filter usually has a triangular shape, with the center point as the maximum value and the two sides gradually decaying to zero. Multiplying the spectrum with the Hm matrix, the spectrum characteristics after the Mel filter bank processing can be obtained. These characteristics have an energy distribution related to human hearing. When the frequency is very small, the filter width is very narrow. As its frequency increases, the filter width will gradually increase, but its amplitude will also gradually decrease.

[0249]

[0250] Among them, H m (k) is the converted Hm matrix, m is the filter number, and k is the horizontal coordinate (e.g., 0-2048).

[0251] Step S37, Discrete Cosine Transform (DCT): Next, perform discrete cosine transform on the output of the Mel filter bank. DCT is a method for converting signals from the frequency domain to the time domain, and has good energy concentration. The purpose of DCT is to convert the Mel filter bank energy coefficients with high correlation into cepstral coefficients with low correlation. Usually only a portion of the low-frequency cepstral coefficients (such as the first 12 or the first 13) are retained because they contain the main features of the speech signal. For the process of discrete cosine transform, please refer to Fig.11 , is a schematic diagram of the discrete cosine transform process in the present invention, as shown in the figure:

[0252] Step S41, natural logarithm operation: after the signal passes through the Mel filter bank, the natural logarithm of the output energy of each filter is taken. This step helps to simulate the perception characteristics of the human ear to sound energy.

[0253]

[0254] The H matrix in the expression is the energy spectral density E and the Mel filter bank matrix H m The energy spectral density E is the energy distribution of the signal at different frequencies, and the expression is as follows:

[0255]

[0256] It is the instantaneous power of the signal at a certain frequency. The energy spectral density E is combined with the Mel filter bank matrix H m After multiplying the transpose of , the output energy of each Mel filter can be obtained.

[0257] Step S42, discrete cosine transform: perform discrete cosine transform on the energy value after taking the natural logarithm. This step converts the speech signal from the Mel frequency to the cepstrum domain and helps to identify the basic features and details in the speech. The expression of discrete cosine transform of natural logarithm is as follows:

[0258]

[0259] Among them, mfcc(i,n) represents the nth MFCC parameter of the i-th frame data, represents the nth MFCC parameter of the i-th frame data, i is the frame data, and H is H m matrix, M is the number of Mel filters, n is the nth column of the i-th frame, and H(i,m) represents the energy value between the i-th frame data and the m-th filter in the energy matrix H after the Mel filter bank.

[0260] Step S43, Cepstral operation: In this step, a cepstral operation is performed to convert the time domain signal into a frequency domain signal, usually retaining the first few coefficients in the DCT output. These retained coefficients capture the main information of the speech signal while ignoring unimportant high-frequency details. In one embodiment, 12 to 22 coefficients may be retained, depending on the application requirements. The expression of the cepstral operation is as follows:

[0261]

[0262] k(i) represents the Mel cepstral coefficient of the i-th frame data, L is the cepstral parameter (the value is 22 in this embodiment), and i is the frame data. Because most of the signal data is generally concentrated in the low-frequency area after transformation, only the first 13 data are taken.

[0263] Step S44, MFCC basic parameters Feat: The retained cepstral coefficients are used as the basic parameters of MFCC features. These features can be used for applications such as speech recognition, speech synthesis, and speaker recognition. Sometimes, the MFCC features can be further processed, such as adding derivatives, second-order derivatives, etc., to improve the expressiveness of the features. The final MFCC basic parameter matrix feat is expressed as follows:

[0264] feat(i,j)=J(i,j)·K(j);

[0265] i=1,2,3,…,301; j=1,2,…,13;

[0266] Among them, feat(i,j) is the jth MFCC basic parameter of the i-th frame data, J(i,j) represents the i-th frame data and the j-th column in the MFCC parameter matrix, and K(j) represents the j-th Mel cepstral coefficient.

[0267] Furthermore, the second set of parameters is a first-order differential operation under the existing basic parameters, and the third set of parameters is a first-order differential operation under the second set of parameters. The expression is as follows:

[0268] dtf eat [i][j]=feat[i+1][j]-feat[i-1][j]+2·feat[i+2][j]-2feat[i-2][j];

[0269] i=1,2,3,…,301; j=1,2,…,13;

[0270] dtf eat [i][j] is the first-order differential value of the j-th MFCC parameter of the i-th frame data, f eat [i][j] represents the jth MFCC basic parameters of the i-th frame data. In this expression, by eat Each element in the matrix is ​​subjected to a first-order differential operation to calculate the first-order differential parameter dtf eat Similarly, you can do the same for dtf eat Perform first-order differential operation to obtain second-order differential parameters.

[0271] Step S27, finally, when comparing the MFCC feature values ​​between different samples, the scale differences between them are eliminated, so as to better perform model training and feature selection. For the i-th MFCC feature value, the normalization process is as follows:

[0272]

[0273] x i is the original i-th MFCC feature value; mean i is the average value of the i-th MFCC feature value in all samples; s tdi is the standard deviation of the ith MFCC feature value in all samples; F(x i ) is the standardized ith MFCC eigenvalue. Standardizing the original MFCC eigenvalues ​​can eliminate distribution differences, making the MFCC eigenvalues ​​between different samples more comparable, which is beneficial to model training and evaluation.

[0274] On the other hand, the original energy amplitude value is converted to a standardized energy amplitude value. This helps to eliminate the scale differences in the data so that different data can be compared. The standardized formula is as follows:

[0275]

[0276] P(x i ) is the standardized energy amplitude value, which indicates the standardized value of the energy amplitude of the i-th sample; energy is the original energy amplitude value, which indicates the original energy amplitude of the i-th sample; mean_energy is the average value of the energy amplitude values ​​in all samples, which is used to center the energy amplitude value; s tdenergy It is the standard deviation of the energy amplitude values ​​in all samples and is used to scale the energy amplitude values.

[0277] Finally, these two input feature groups (audio feature vector and audio energy amplitude) are combined into a feature vector as the input of the CNN model. For example, suppose 12 features and 1 energy amplitude value are extracted and they are concatenated into a 13-dimensional feature vector.

[0278] The speech recognition module M33 is used to map the audio features output by the MFCC unit M32 to voice commands. In one embodiment, the speech recognition module M33 may be, for example, but not limited to, a convolutional neural network model (CNN); in an embodiment using a convolutional neural network model, the convolutional neural network model may be used to extract features of speech signals, extracting short-term spectral features, such as MFCC features, from the original time series.

[0279] Please also read Fig.12 , which is a schematic diagram of the convolutional neural network model architecture used in the speech recognition module of the present invention, as shown in the figure: the convolutional neural network model used in the speech recognition module of the present invention includes a convolutional layer LA1 (Convolutional Layer), a pooling layer LA2 (Pooling Layer), and a fully connected layer LA3 (Fully Connected Layer).

[0280] Convolution layer LA1 is mainly used to extract features. Specifically, convolution layer LA1 is the core of the convolutional neural network model. It extracts features from the input image through a certain number of convolution kernels. Each convolution kernel contains a set of weights, which are convolved with the input image to generate a new feature map. The size and step size of the convolution kernel are set by hyperparameters.

[0281] The pooling layer LA2 is used to reduce the dimension of features. Specifically, the pooling layer LA2 is mainly used for feature dimensionality reduction, reducing the number of parameters and the amount of calculation. Through certain pooling operations (such as maximum pooling, average pooling, etc.), the feature map is reduced in dimension to obtain a new feature map. The main hyperparameters of the pooling layer are the size and step size of the pooling kernel.

[0282] The fully connected layer LA3 is used to generate the final output. Specifically, the fully connected layer LA3 is usually used to convert the feature map into the corresponding output and perform tasks such as classification or regression. It expands the feature map into a vector, performs linear transformation and nonlinear transformation through multiple fully connected layers LA3, and finally obtains the output result. In the present invention, the output result is directly mapped to the voice instruction, or mapped to the code or value related to the voice instruction, and the corresponding instruction is found from the database through the code or value, which is not limited in the present invention.

[0283] The CTC trainer M34 is used to execute the CTC Loss algorithm (Connectionist Temporal Classification Loss), which is used as a loss function in sequence learning to solve the sequence alignment problem. In sequence learning tasks such as speech recognition and text recognition, there is often a situation where the input sequence and the output sequence are not completely consistent or aligned, which requires the use of CTC loss to train the convolutional neural network model.

[0284] More specifically, in speech recognition, the data set is the audio file and its corresponding text data. However, it is difficult to align the audio file and the text data in units of words. If the model is trained directly without alignment, it will be difficult for the model to converge due to the different speech speeds of people or the different distances between characters. CTC Trainer M34 can avoid manual alignment of input and output, and is suitable for speech recognition.

[0285] The following expression represents the probability of observing a specific alignment path π given an input feature sequence x. The expression is as follows:

[0286]

[0287] L′=L∪{blank};

[0288] Where p(π|x) is the probability of observing a specific alignment path π given an input feature sequence x. x is the input feature sequence, usually the data of the speech signal after feature extraction. y is the output label sequence, that is, the expected text sequence. T is the length of the input feature sequence x. L′ is the value space, including the output vocab (L) and an additional blank label. L is the output vocab, that is, all possible output characters. πt is the observation probability, which represents the probability of observing a certain character at time t.

[0289] The following expression represents the sum of the probabilities of selecting the target sequence I among all possible pathsπ (in the mapping set β) given an input x.

[0290]

[0291] Where x is the input sequence, usually a speech signal or text, I is the target sequence, that is, the text sequence to be recognized or generated, and β is the mapping set, also known as the "alphabet", containing all possible text sequences, ∈β -1 (I) is the set of all extended sequences of the target sequence I in the mapping set β, π is a path in the mapping set β, indicating a path selected from all possible paths, P(π|x) is the conditional probability from the input sequence x to pathπ, indicating the probability of mapping x to π, and P(I|x) is the conditional probability from the input sequence x to the target sequence I, indicating the probability of mapping x to I.

[0292] For the CTC Loss process, please refer to Fig.13 , is a schematic diagram of the training process of the CTC trainer in the present invention, as shown in the figure: The process of CTC Loss is as follows:

[0293] Step S51: Initialization Before performing CTC loss calculation, some variables need to be initialized, for example, the probability calculation group (used to calculate forward and reverse probabilities) and the gradient calculation group (used to calculate gradients).

[0294] Probability calculation group: Create two two-dimensional matrices (usually called alpha and beta) to store the calculation results of forward probability and reverse probability. The size of these matrices is usually T×L, where T represents the length of the input sequence and L represents the length of the target sequence (after expansion, including blank symbols).

[0295] Gradient calculation group: Create a gradient calculation group to store the gradient of the loss function with respect to the model parameters. This will be used during the backpropagation process in order to update the model parameters.

[0296] Other variables: Some other variables need to be initialized, such as loss values, regularization terms, etc.

[0297] Before doing CTC loss calculation, make sure all variables are initialized correctly to avoid calculation errors or inconsistencies.

[0298] Step S52: Function import. The steps include 1. Generate two matrices to calculate forward and reverse probabilities: these matrices will be used to store the calculation results of forward probability and reverse probability; 2. Expand labels with blank symbols: insert blank symbols in the target sequence for alignment in CTC loss calculation.

[0299] Generate two matrices to calculate forward and backward probabilities: Here, two two-dimensional matrices (usually called alpha and beta) will be created, which will be used to store the results of the forward and backward probabilities. The size of these matrices is usually T×L, where T represents the length of the input sequence and L represents the length of the target sequence (after expansion, including blank symbols). Each element in the matrix represents the probability at a given time t and position l in the target sequence.

[0300] Augment labels with blank symbols: For CTC calculation, a special blank symbol needs to be inserted between each character of the target sequence. In addition, blank symbols need to be added at the beginning and end of the sequence. For example, if the target sequence is "AB", the augmented sequence will be "-AB-", where "-" represents the blank symbol. The purpose of this is to allow the model to predict the same target character at different input time steps t, making the alignment problem simpler.

[0301] After completing these operations, you can proceed to calculate the forward probability and reverse probability.

[0302] Step S53: Perform forward probability calculation, and calculate the forward probability based on the given input sequence and target sequence. This is the first part of the CTC loss calculation, which represents the probability of observing the input sequence given the target sequence. In this process, given an input sequence and a label sequence, CTC Forward calculates the alignment result of the input sequence and the label sequence, and calculates the loss value between the predicted result and the legal alignment result of the label sequence.

[0303] The expression for forward probability calculation is as follows:

[0304]

[0305] This expression means that the calculation of α tThe value of (s) needs to be summed over all possible alignment paths π, where π is a sequence of length T, representing the correspondence between each time step of the input feature sequence x and the output sequence S, and B represents a set that contains all possible paths π that match the first s characters of the current label sequence 1 after t time steps. Therefore, B(π 1：t )=l 1：s Represents the path π 1：t After t time steps, it matches the first s characters of label sequence 1.

[0306] Update rules The expression is as follows:

[0307]

[0308] This update rule is used to calculate the new Forward variable value of the sequence S at time step t. It means that to calculate the new The forward variable α of the previous time step t-1 needs to be t-1 (s) and α t-1 (s-1) is added. This update rule is used to recursively calculate α in the Forward process of the CTC algorithm. t (s), thereby estimating the probability of the output label sequence y given the input feature sequence x.

[0309] Among them, α t (s) is the forward variable, which indicates the probability of generating sequence S at time step t; t is the current time step, and its value range is [1, T]; where T is the length of the input feature sequence x, s is the current position of the sequence S, and its value range is [1, |S|], where |S| indicates the length of the sequence S; π is the alignment path, which indicates the correspondence between each time step of the input feature sequence x and the output sequence S, and π is a sequence of length T; N T (1: s) is the set of all possible alignment paths, where each path π satisfies B(π 1：t )=l 1：s ; B is the operation of removing duplicate labels and blank labels, converting the path π into a label sequence. For example, if the path π is "aa--b-bc", then B(π) = "abc"; l 1：s are the first s characters of the target output sequence; t′ is the variable of time step, which is used to calculate the product of the observation probability from time step 1 to time step t, and the value range of t′ is [1, t]; At time step t′, the input feature sequence x generates label π t′For example, if at a certain time step, the probability of the input feature sequence x generating the character "a" is 0.8, then at this time y("a") = 0.8.

[0310] Step S54: Execute reverse probability calculation. Similarly, calculate reverse probability. This step is to calculate the correspondence between the target sequence and the speech signal so as to calculate the loss function in the subsequent steps. In this step, the reverse probability of each time step is calculated, which represents the sum of the probabilities of all possible paths from the current time step to the end of the target sequence.

[0311] The expression for reverse probability calculation is as follows:

[0312]

[0313] This expression means that the calculation of β t The value of (s) needs to be summed over all possible alignment paths π, where π is a sequence of length T, representing the correspondence between each time step of the input feature sequence x and the output sequence S.

[0314] Update rules The expression is as follows:

[0315]

[0316] This update rule is used to calculate the new Backward variable value of the sequence S at time step t. It means that to calculate the new It is necessary to change the Backward variable β of the next time step t+1 t+1 (s) and β t+1 (s+1) is added. This update rule is used to recursively calculate p in the Backward process of the CTC algorithm. t (s), thereby estimating the probability of the output label sequence y given the input feature sequence x.

[0317] Among them, p t (s) is the backward variable, which indicates the probability of generating sequence S from time step t to the last time step T; t is the current time step, and its value range is [1, T], where T is the length of the input feature sequence x; s is the position of the current sequence S, and its value range is [1, |S|], where |S| indicates the length of the sequence S; π is the alignment path, which indicates the correspondence between each time step of the input feature sequence x and the output sequence S, and π is a sequence of length T; N T (s:|1|) is the set of all possible alignment paths, where each path π satisfies (π t：T )=l s：|l|; B removes repeated labels and blank labels and converts the path π into a label sequence. For example, if the path π is "aa--b-bc", then B(π) = "abc"; l s：|l| is the subsequence from s to the last character of the target output sequence; t′ is the variable of time step, which is used to calculate the product of the observation probability from time step t to time step T, and the value range of t′ is [t, T]; is the probability that the input feature sequence x generates the label π(t′) at time step t′. For example, if the probability that the input feature sequence x generates the character “a” is 0.8 at a certain time step, then y(“a”) = 0.8.

[0318] Through these two formulas, the Backward variable β can be calculated in the Backward process of the CTC algorithm. t (s) to estimate the probability of outputting label sequence y given input feature sequence x. Combined with the forward variable α t (s) and Backward variable β t (s), the CTC loss function can be calculated and the parameters of the model can be optimized.

[0319] Step S55: Calculate the loss function. Based on the calculation results of the forward probability and the reverse probability, calculate the CTC loss. The CTC loss represents the alignment error between the given input sequence and the target sequence. By minimizing this loss, the model will learn to better align the input sequence, thereby improving the recognition effect. In this step, the forward probability α is used t (s) and the reverse probability β t (s), calculate the gradient of the CTC loss function. These gradient values ​​are used to optimize the parameters of the model, thereby improving the accuracy of the model in predicting the output label sequence y given the input feature sequence x. The expression of CTC loss is as follows:

[0320]

[0321] is the CTC loss function with respect to the startup value The gradient of , which needs to be calculated to optimize the model parameters; is the starting value of the model output (such as the output of a neural network) at time step t, corresponding to label t; is the probability of generating label k for input feature sequence x at time step t; p(l|x) is the probability of outputting label sequence y given input feature sequence x, which can be calculated by CTC Forward variable α t (s) and Backward variable β t (s) is obtained; l is the target output sequence; x is the input feature sequence; αt (s) is the CTC Forward variable, which is the probability of generating sequence s at time step t given the input feature sequence x; β t (s) is the CTC Backward variable, which is the probability of generating sequence s from time step t to the last time step T given the input feature sequence x; α t (s) and p t (s) is the normalized CTC Forward variable and Backward variable; lab(l, k) is the set of all sequences s containing label k; |l′| represents the length of the output sequence after expansion; At time step t, the input feature sequence x generates the label l' in the expanded output sequence l' s probability.

[0322] This formula defines the calculation of the CTC loss function with respect to the startup value The gradient is used to optimize the model parameters to improve the accuracy of the model in predicting the output label sequence y given the input feature sequence x. In this formula, the gradient Used to update the parameters of the speech recognition module M34. Through the above steps, the CTC trainer M34 uses the connection time series classification loss function (CTC loss) to calculate the distance between the output sequence and the label sequence, and updates the model parameters of the speech recognition module M33 through forward probability calculation and reverse probability calculation loss functions.

[0323] Through the architecture of waveform neural network M31, MFCC unit M32, speech recognition module M33, and CTC trainer M34, the present invention has the effects of high efficiency (by using MFCC features and WAVENET model, the system can realize efficient speech recognition function), high accuracy (by using CTC LOSS, the system can achieve higher recognition accuracy when processing speech information with noise) and high real-time performance (the system can process speech information in real time and realize real-time speech recognition).

[0324] The position detection module M4 analyzes the user's position information based on the ambient sound to confirm whether the user is located in the immersive area or the non-immersive area.

[0325] For the functions and architecture of the location detection module M4, please refer to Fig.14 and Fig.15, which are the architecture diagram (1) and architecture diagram (2) of the position detection module in the present invention, as shown in the figure: In one embodiment, the position detection module M4 includes a convolutional neural network classifier (CNN classification) M41, and the convolutional neural network classifier M41 is trained to map the input environmental sound into a voice command, and the source of the voice command comes from one of the two results of the immersive area or the non-immersive area.

[0326] The convolutional neural network classifier M41 classifies information based on the source of the generated audio information of the received ambient sound. Specifically, the convolutional neural network classifier M41 refers to a sound classifier that uses convolutional neural networks (CNNs) as the main algorithm. In the CNN classification task, the network is trained with labeled sounds to classify the received ambient sounds according to the characteristics of the ambient sounds.

[0327] Before the ambient audio is input to the convolutional neural network classifier M41, the processor first converts the ambient audio AW into a normalized time-frequency spectrum SP (spectrogram) through spectral analysis so that the convolutional neural network classifier M41 can identify the ambient audio AW.

[0328] Specifically, the sound signal can be converted into a time-frequency spectrum SP in the following manner: first, resample and convert the sound signal into stereo. Then, segment the sound signal, usually using a window function (such as a Hamming window) and an overlapping technique (such as 50% overlap) to divide it into multiple short time periods, and translate the audio signal through the audio augmentation time axis (Audio Augmentation TimeShift) to increase the generalization ability of the model, thereby completing the normalization. Then, for the sound signal in each short time period, use Fourier transform (FFT) to convert it into a spectrum signal. Since the spectrum signal obtained by Fourier transform is a continuous frequency signal from 0Hz to the sampling frequency, it needs to be divided into multiple discrete frequency bands for subsequent processing. This is usually implemented using short-time Fourier transform (STFT), which divides the spectrum signal of each short time period into a series of discrete frequency bands to form a two-dimensional matrix, where the horizontal axis is time and the vertical axis is frequency. Finally, for each frequency band in a short time period, its energy or intensity can be calculated and drawn into a color image or a grayscale image to represent the sound intensity distribution at different frequencies and times, that is, to obtain the time-frequency spectrum SP. Finally, Spectrogram Augmentation (SpecAugment) is performed to enhance the audio data through time masking, frequency masking, time shifting and frequency shifting to improve the generalization ability and robustness of the model.

[0329] After the ambient audio AW is converted into a time-frequency spectrum SP, it is input into a convolutional neural network CA. The convolutional neural network CA consists of a series of convolutional layers and pooling layers. The convolutional layer is used to detect features such as edges, shapes, and patterns. The pooling layer is used to reduce the dimension of the input data and finally establish the feature map (Features Maps) FM of the input time-frequency spectrum SP. The feature map FM contains the local information and global structure of the time-frequency spectrum SP and can be used for subsequent audio classification tasks.

[0330] After obtaining the feature map FM, the feature map FM is input into the fully connected layer FC. The fully connected layer FC is used for actual classification, mapping the extracted features to the corresponding category labels to obtain audio classification information. Specifically, the function of the fully connected layer FC is to flatten the extracted feature map FM, convert it into a one-dimensional vector, and then perform matrix multiplication with the weight matrix to obtain the final output result. Each neuron in the fully connected layer FC is connected to all neurons in the previous layer, and each connection has a weight. In this way, all the features of the previous layer can be taken into account for more accurate classification or prediction.

[0331] The audio classification information may be one of the two results of being located in the immersive area or being located in the non-immersive area. In another embodiment, the audio classification information may be a weight value of the immersive area or the non-immersive area, for example, expressing the possibility or relative amount of the ambient sound coming from the immersive area or the non-immersive area in percentage or fraction, for example, the result of the audio classification information may be 60% for the immersive area, 40% for the non-immersive area, or 6 points for the immersive area, 4 points for the non-immersive area, etc., which is not limited in the present invention.

[0332] For more information on how to train a convolutional neural network classifier, please refer to Fig.16, is a schematic diagram of the training process of the convolutional neural network classifier in the present invention, as shown in the figure: first, prepare multiple sets of original audio files for training (step S61). The original audio files can be, for example, but not limited to, MP3 audio files (MPEG-1 Audio Layer 3), WAV audio files (Waveform Audio File Format), AIFF audio files (Audio Interchange File Format), FLAC audio files (Free Lossless Audio Codec), AAC audio files (Advanced Audio Coding), OGG audio files (Ogg Vorbis), MIDI audio files (Musical Instrument Digital Interface), WMA audio files (Windows Media Audio) or other similar audio files, which are not limited in the present invention. In the step, a large amount of environmental audio can be recorded and divided into short time segments, each segment corresponding to a sample.

[0333] Next, use MFCC to extract time-frequency features for all samples (step S62); in this step, the following sub-steps are included: step SF1, perform short-time Fourier transform on all samples to obtain their spectral features; step SF2, based on the human ear's higher sensitivity to the low-frequency region, use the Mel scale to normalize the short-time Fourier transform spectrum to obtain the Mel cepstrum; step SF3, discretize the Mel cepstrum, and use the discretized Filter Bank to obtain MFCC features. These features are more in line with the nonlinear frequency perception of the human ear; step SF4, construct a two-dimensional time-frequency feature map for each sample based on the MFCC features; step SF5, construct an enhanced data set based on the MFCC feature time-frequency map to provide richer feature information for model training.

[0334] Next, define a convolutional neural network classifier (step S63); in this embodiment, the CNN model includes 1. using 5 convolutional layers, the convolution kernel size decreases from 11×11 to 3×3, and the activation function is ReLU; 2. using 3 maximum pooling layers, the pooling kernel size is 3×3, and the step size is 2; 3. using 3 fully connected layers, the number of neurons in the 1st and 2nd layers is 1024, the number of neurons in the 3rd output layer is 2, representing the two categories of binary classification tasks, and adding a dropout regularization of 0.5; 4. selecting the ADAM optimizer, and the learning rate is set to 0.001; 5. ReLU is selected as the activation function of each layer of CNN, and the ADAM optimizer's β1=0.9, β2=0.999.

[0335] Next, the convolutional neural network classifier is pre-trained (step S64); in this step, a large number of labeled samples are used to train the convolutional neural network classifier so that it can learn the characteristic representation of the environmental sound; some samples are manually labeled as immersive areas or non-immersive areas to form a labeled sample set. It should be noted that since audio may produce different sound field information in different environments, in order to improve reliability, the environmental sound samples for training can be obtained on-site or in a general environment, which is not limited in the present invention.

[0336] Next, the convolutional neural network classifier is trained and adjusted (step S65). In this step, the convolutional neural network classifier is fine-tuned using the labeled sample set so that in addition to learning the audio features, it also learns the features that distinguish between immersive areas and non-immersive areas.

[0337] The convolutional neural network classifier is trained according to the following process: Step SG1, forward propagation: forward propagation is performed through the CNN model to map the input data to one of the classification results of the immersive area or the non-immersive area, and the difference between the predicted result and the true label is calculated; Step SG2, back propagation: through the back propagation algorithm, the model parameters of the convolutional neural network are adjusted according to the loss function to make the model's prediction result closer to the true label; Step SG3, parameter update: the model parameters of the convolutional neural network are updated according to the optimizer to minimize the loss function; and Step SG4, repeated training: repeated forward propagation, back propagation and parameter update operations until the model converges or reaches a predetermined number of training times.

[0338] Finally, the threshold logic is determined (step S66); in this step, based on a large number of manually labeled samples, the threshold or judgment logic for determining whether a new sample is an immersive area or a non-immersive area is determined. In principle, after the ambient sound enters the convolutional neural network classifier, the convolutional neural network classifier will calculate the probability weight of an immersive area. When this probability weight is greater than a threshold, it is determined that the ambient sound is from the immersive area, otherwise it is from the non-immersive area.

[0339] The processing module M5 generates corresponding control instructions based on the user's voice instructions and location information, and switches the audio output mode of the audio processing module M2 or the reverse audio output mode of the sound elimination module M1 according to the control instructions. Specifically, the following implementation scenarios may be included:

[0340] Scenario 1: The processing module M5 receives a voice command from the immersive area. When receiving the voice command of "turn up the volume", the processing module M5 can respond by turning up the volume of the immersive area (i.e., increasing the gain of the audio output device 20) through the audio processing module M2;

[0341] Scenario 2: The processing module M5 receives a voice command from the immersive area. When receiving the voice command of "lower the volume", the processing module M5 can respond by lowering the volume of the immersive area (i.e., lowering the gain of the audio output device 20) through the audio processing module M2;

[0342] Scenario 3: When the processing module M5 receives a voice command from a non-immersive area and receives a voice command of "turn up the volume", the processing module M5 can respond by turning up the volume of the non-immersive area (i.e., reducing the gain of the reverse cancellation sound) through the sound cancellation module M1;

[0343] Scenario 4: When the processing module M5 receives a voice command from a non-immersive area and receives a voice command of "lower the volume", the processing module M5 can respond by lowering the volume of the non-immersive area through the sound cancellation module M1 (i.e., increasing the gain of the reverse cancellation sound);

[0344] Scenario 5: When the processing module M5 receives a voice command from the immersive area and receives a voice command of "enhancing the immersive effect", the processing module M5 can respond by eliminating the noise from the non-immersive area through the sound elimination module;

[0345] Scenario 6: When the processing module M5 receives a voice command from a non-immersive area and receives a voice command to "enhance the immersive effect", the processing module M5 can respond by increasing the gain of the reverse offset sound;

[0346] Scenario 7: When the processing module M5 receives a voice command from the immersive area and receives a voice command to "reduce immersion effect", the processing module M5 can respond by reducing the stereo output through the audio processing module M2.

[0347] It should be noted that the above scenario descriptions are merely examples of several embodiments of the present invention, and these embodiments are not intended to limit the scope of protection of the present invention.

[0348] In summary, please refer to Fig.17, which is a schematic diagram of the overall operation flow of the open space immersive audio system of the present invention, as shown in the figure: first, the user submits a voice message in the immersive space or the non-immersive space (step S71); the audio receiving device 10 receives the ambient sound including the voice command through the microphone 11 (step S72), and the received ambient sound is processed by the processor 41; after receiving the ambient sound, the processor 41 analyzes the voice information by running the intelligent voice recognition module M3 to obtain a voice command, and finds the text data corresponding to the voice command through the voice recognition module M33 (step S73); and the step At the same time as step S73 (it can be before or after), the processor 41 runs the position detection module M4 to analyze the user's position information through the ambient sound to confirm that the user is in the immersive area or the non-immersive area (step S74); after confirming the text data of the voice command and the user's position information (in the immersive area or in the non-immersive area), the processor 41 generates corresponding control instructions based on the user's text data and position information by running the processing module M5, and switches the audio output mode of the audio processing module M2 or the reverse audio output mode of the sound elimination module M1 according to the control instruction (step S75). In one embodiment, before step S71, the intelligent voice recognition module M3 of the processor 41 can be triggered by specific designated voice information (such as short voice), and after being triggered, it switches from standby mode to working mode, thereby reducing the consumption caused by calculation in the normal state.

[0349] The following is a more detailed description of the embodiments of the specific configuration positions in the immersive audio system of the present invention. Figures 18 to 21 , which are configuration schematic diagram (1), configuration schematic diagram (2), configuration schematic diagram (3) and configuration schematic diagram (4) of the immersive audio system of the present invention.

[0350] For the first configuration method, please refer to Fig.18In this embodiment, the reverse audio output device 30 can be configured at any position of the audio device AP to eliminate the audio played in the non-immersive space, where the immersive space is marked as A and the non-immersive space is marked as B. The position can be, for example, configured at the back side of the audio device AP (such as (a1)) to eliminate the audio played at the back of the audio device AP when the audio output device 20 plays the audio in the front direction; configured at the left side of the audio device AP (such as (a2)) to eliminate the audio played on the left side of the audio device AP; configured at the right side of the audio device AP (such as (a3)) to eliminate the audio played on the right side of the audio device AP; configured at the top side of the audio device AP (such as (a4)) to eliminate the audio played on the upper side of the audio device AP; configured at the bottom side of the audio device AP (such as (a5)) to eliminate the audio played on the bottom side of the audio device AP; or a combination of multiple and configured at any of the above positions, which is not limited in the present invention. In another embodiment, although not shown in the figure, the reverse audio output device 30 can also be set at the corner position or the corner position of the audio device AP, which is explained here first.

[0351] For the second configuration method, see Fig.19 : On the audio device AP, the reverse audio output device 30 can be configured on both sides of the audio output device 20. By eliminating the playback audio on both sides of the audio output device 20, the directional effect of the audio output device 20 is increased, and the sound field disorder caused by the sound with too wide a propagation range is avoided. It can be seen from the accompanying drawings that the reverse audio output devices 30 on both sides will increase the range of the non-immersive space BB1 ​​on both sides, making the transmission range of the immersive space A1 narrower.

[0352] The third configuration mode is shown in the following figure. Fig. 20 : In one embodiment, the angle θ1 between the reverse cancellation audio output direction L1 of the reverse audio output device 30 on the left and the virtual plane PH centered on the audio device AP and the immersive space A2 is 20 degrees to 45 degrees, and the angle θ2 between the reverse cancellation audio output direction L2 of the reverse audio output device 30 on the right and the virtual plane PH centered on the audio device AP and the immersive space A2 is 20 degrees to 45 degrees.

[0353] For the fourth configuration method, see Fig.21: The reverse audio output device 30 can be distributed in an array on the front, back, left, right, top and bottom sides of the audio device AP. For example, in (b1), the reverse audio output device 30 is set on the surface positions of the front, back, left, right, top and bottom sides of the audio device AP; or, for example, in (b2), the reverse audio output device 30 is set on the edge position at each corner of the audio device AP; or, for example, in (b3), the reverse audio output device 30 is set on the vertex position at each corner of the audio device AP; or any combination of the above positions. The invention does not limit the simultaneous arrangement of multiple reverse audio output devices 30 at the same surface position, edge position or vertex position to form an array. In the described embodiment, the non-immersive space where the audio is to be eliminated can be selected according to the factory configuration or according to the setting, so as to configure the best sound field effect in accordance with the corresponding environment. Although the above is represented by dots in the drawings, the dots can be understood as the reverse audio output device 30 being a plurality of arrays distributed on the surfaces, edges, and corners, thereby more accurately adjusting the range of the non-immersive space or the direction of the sound output through the array.

[0354] In summary, the present invention provides a customized sound experience based on the user's area and voice information by determining whether the user is located in an immersive area or a non-immersive area, thereby better meeting the specific needs of individual users or adapting to different environments and situations.

[0355] The present invention has been described in detail above. The above is only one preferred embodiment of the present invention and should not be used to limit the scope of the present invention. That is, all equivalent changes and modifications made according to the scope of the patent application of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. An open space immersive audio system with dynamic intelligent voice, characterized in that: include: an audio receiving device; an audio output device; A reverse audio output device; as well as At least one processor executes a sound elimination module, an intelligent voice recognition module, a position detection module, an audio processing module and a processing module after loading at least one storage device, wherein: After receiving the error sound in the non-immersive area through the audio receiving device, the sound cancellation module generates a reverse audio signal based on the error sound and outputs it to the audio output device to generate a reverse cancellation sound; The audio processing module outputs a play audio to the audio output device, so as to play the sound effect in the immersive area; After the intelligent voice recognition module receives the ambient sound from the audio receiving device, it obtains the user's voice information from the ambient sound and obtains a voice command by analyzing the voice information; The position detection module analyzes the user's position information from the ambient sound to confirm that the user is located in the immersive area or the non-immersive area; The processing module generates a corresponding control instruction based on the voice instruction of the user and the position information, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

2. The open space immersive audio system of dynamic intelligent voice according to claim 1, characterized in that: The intelligent speech recognition module includes a waveform neural network for generating a real speech signal based on the environmental sound, an MFCC unit for extracting audio features from the real speech signal, and a speech recognition module for mapping the audio features output by the MFCC unit to the speech command.

3. The open space immersive audio system of dynamic intelligent voice according to claim 2, characterized in that: The speech recognition module is a convolutional neural network model.

4. The open space immersive audio system of dynamic intelligent voice according to claim 3, characterized in that: The intelligent speech recognition module includes a CTC training unit, which is used to calculate the distance between the output sequence and the label sequence through a connection time series classification loss function, and to update the model parameters of the speech recognition module through a back propagation algorithm.

5. The open space immersive audio system of dynamic intelligent voice according to claim 1, characterized in that: The position detection module includes a convolutional neural network, and the convolutional neural network is trained to map the input environmental sound into the voice command, and the source of the voice command comes from the immersive area or the non-immersive area.

6. The open space immersive audio system of dynamic intelligent voice according to claim 5, characterized in that: The convolutional neural network includes a plurality of convolutional layers for extracting features of the environmental sound and a fully connected layer for mapping the features to a classification result of the immersive area or the non-immersive area according to weights.

7. The open space immersive audio system of dynamic intelligent voice according to claim 6, characterized in that: The convolutional neural network is trained according to the following process: Forward propagation: forward propagation is performed through the CNN model to map the input data to one of the classification results of the immersive area or the non-immersive area, and the difference between the predicted result and the true label is calculated; Back propagation: Through the back propagation algorithm, the model parameters of the convolutional neural network are adjusted according to the loss function to make the model's prediction results closer to the true label; Parameter update: updating the model parameters of the convolutional neural network according to the optimizer to minimize the loss function; and Repeated training: Repeat forward propagation, backpropagation, and parameter update operations until the model converges or reaches a predetermined number of training times.

8. The open space immersive audio system of dynamic intelligent voice according to claim 1, characterized in that: The intelligent voice recognition module is in a standby state and is a program for analyzing the voice information triggered by a designated voice information or a human-machine interface instruction.

9. An open space immersive audio method for dynamic intelligent voice, characterized in that: include: After the audio receiving device receives the ambient sound, the user's voice information is obtained from the ambient sound, and a voice command is obtained by analyzing the voice information; Analyzing the user's location information from the ambient sound to confirm whether the user is located in an immersive area or a non-immersive area; and A corresponding control instruction is generated based on the voice instruction of the user and the position information, and the audio output mode of the audio output device or the reverse audio output mode of the reverse audio output device is switched according to the control instruction.

10. The open space immersive audio method of dynamic intelligent voice according to claim 9, characterized in that: The open space immersive audio method of dynamic intelligent voice is a program for analyzing the voice information triggered by designated voice information or human-machine interface instructions.

Images