Open space immersive system with high-speed dynamic intelligent voice

Abstract

The invention provides an open space immersive system with high-speed dynamic intelligent voice. The open space immersive system comprises an audio receiving device, an audio output device and at least one processor. The processor executes the sound elimination module, the intelligent voice recognition module, the audio processing module and the processing module. The intelligent voice recognition module comprises a parallel waveform neural network for generating a real voice signal based on environment sound, an MFCC unit for extracting audio characteristics and a voice recognition module for mapping the audio characteristics to a voice instruction. The audio processing module plays the sound effect in the immersive area. The sound elimination module receives the error sound and generates a reverse audio output to counteract the sound. And the processing module generates a corresponding control instruction according to the voice instruction, switches an audio output mode of the audio processing module or a reverse audio output mode of the sound elimination module, immediately determines an instruction of a user through high-speed voice recognition, and immediately feeds back a signal according to the voice instruction of the user so as to control audio output experience and a sound elimination effect.

Description

Technical Field

[0001] The present invention relates to an open space immersive system with high-speed dynamic intelligent voice, and in particular to an open space immersive system with high-speed dynamic intelligent voice that generates control instructions by comparing voice instructions. Background Art

[0002] General open space environments, such as homes, offices, public areas or transportation hubs, are often susceptible to high levels of background noise, which can have a negative impact on personal comfort, concentration and work efficiency. In order to solve these problems, noise reduction systems have been born. There are two main types of existing noise reduction systems, namely passive noise reduction technology and active noise reduction systems.

[0003] Passive noise reduction techniques, such as sound-absorbing materials or physical barriers, can help reduce noise levels in open spaces. However, these techniques have limited effectiveness and may not be suitable for all environments or situations. Active noise cancellation systems work by actively generating cancelling sound waves to eliminate unwanted noise. While these active noise cancellation systems may be more effective than passive noise reduction techniques, they generally require the user to wear headphones or other personal audio devices, which may not be convenient or comfortable for all users. Summary of the invention

[0004] The main purpose of the present invention is to provide an open space immersive system with high-speed dynamic intelligent voice, including an audio receiving device, an audio output device and at least one processor. The processor executes a sound elimination module, an intelligent voice recognition module, an audio processing module and a processing module after loading at least one storage device, wherein the sound elimination module receives the error sound of the non-immersive area through the audio receiving device, generates a reverse audio based on the error sound and outputs it to the audio output device to generate a reverse cancellation sound. The audio processing module outputs a playback audio to the at least one audio output device to play the sound effect in the immersive area. After the intelligent voice recognition module receives the ambient sound from the audio receiving device, it obtains the user's voice information from the ambient sound, and obtains a voice command by parsing the voice information. The intelligent voice recognition module includes a parallel waveform neural network that generates a real voice signal based on the ambient sound, an MFCC unit for extracting audio features from the real voice signal, and a voice recognition module that maps the audio features output by the MFCC unit to the voice command. The processing module generates a corresponding control instruction based on the voice instruction of the user, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

[0005] Optionally, the processor executes a position detection module after loading the at least one storage device, and the position detection module analyzes the user's position information from the ambient sound to confirm that the user is located in the immersive area or the non-immersive area. The processing module generates corresponding control instructions based on the user's voice command and the position information, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

[0006] Optionally, the speech recognition module is a convolutional neural network model.

[0007] Optionally, the intelligent speech recognition module includes a CTC training unit, which is used to calculate the distance between the output sequence and the label sequence through a concatenated temporal classification loss function, and update the model parameters of the speech recognition module model through a back propagation algorithm.

[0008] Optionally, the position detection module includes a convolutional neural network, and the convolutional neural network is trained to map the input environmental sound to the voice command, and the source of the voice command is from the immersive area or the non-immersive area.

[0009] Optionally, the convolutional neural network includes multiple convolutional layers for extracting the environmental sound features and a fully connected layer for mapping the multiple features to one of the classification results of the immersive area or the non-immersive area according to weights.

[0010] Optionally, the convolutional neural network is trained according to the following process:

[0011] Forward propagation: forward propagation is performed through the CNN model to map the input data to one of the classification results of the immersive area or the non-immersive area, and the difference between the predicted result and the true label is calculated;

[0012] Back propagation: Through the back propagation algorithm, the model parameters of the convolutional neural network are adjusted according to the loss function to make the model's prediction results closer to the true label;

[0013] Parameter update: updating the model parameters of the convolutional neural network according to the optimizer to minimize the loss function; and

[0014] Repeated training: Repeat forward propagation, backpropagation, and parameter update operations until the model converges or reaches a predetermined number of training times.

[0015] Optionally, the intelligent voice recognition module triggers a program to analyze the voice information through a designated voice information or a human-machine interface command in the standby state. Therefore, the present invention instantly determines the user's command through high-speed voice recognition, and instantly feeds back a signal based on the user's voice command to control the audio output experience and the sound elimination effect, thereby instantly meeting the specific needs of individual users. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram of an open space immersive system with high-speed dynamic intelligent voice in the present invention (I).

[0017] Figure 2 This is a block diagram of an open space immersive system with high-speed dynamic intelligent voice in the present invention (II).

[0018] Figure 3 FIG. 1 is a block diagram of the intelligent audio adjustment system of the present invention. FIG.

[0019] Figure 4 FIG. 2 is a block diagram of the intelligent audio adjustment system of the present invention. FIG.

[0020] Figure 5 Schematic diagram of the overall architecture of the intelligent speech recognition module in the present invention.

[0021] Figure 6 Schematic diagram of the architecture of the autoregressive flow model.

[0022] Figure 7 Schematic diagram of the architecture of the inverse autoregressive flow model.

[0023] Figure 8 Schematic diagram of the architecture of the parallel waveform neural network in the present invention.

[0024] Fig. 9 It is a training flow chart of the parallel waveform neural network in the present invention.

[0025] Fig.10 Schematic diagram of the calculation process of the MFCC algorithm in the present invention.

[0026] Fig.11 Schematic diagram of the algorithm flow of the Mel filter bank in the present invention.

[0027] Fig.12 It is a schematic diagram of the discrete cosine transform process in the present invention.

[0028] Fig.13 Schematic diagram of the convolutional neural network model architecture used in the speech recognition module of the present invention.

[0029] Fig.14 Schematic diagram of the training process of the CTC trainer in the present invention.

[0030] Fig.15 FIG. 1 is a schematic diagram of the structure of the position detection module in the present invention (I).

[0031] Fig.16 FIG. 2 is a schematic diagram of the structure of the position detection module in the present invention. FIG.

[0032] Fig.17 Schematic diagram of the training process of the convolutional neural network classifier in the present invention.

[0033] Fig.18 It is a schematic diagram of the overall operation flow of the open space immersive system of the present invention.

[0034] Fig.19 FIG. 1 is a schematic diagram of the configuration of the open space immersive audio system of the present invention (I).

[0035] Fig. 20 FIG. 2 is a schematic diagram of the configuration of the open space immersive audio system of the present invention. FIG.

[0036] Fig.21 Schematic diagram of the configuration of the open space immersive audio system of the present invention (III).

[0037] Fig. 22 Schematic diagram of the configuration of the open space immersive audio system of the present invention (IV). The following are marked in the figure: 100 Open Space Immersive Audio System 10 Audio receiving device

[0038] 11 Microphone

[0039] 12 Preamplifier

[0040] 13 Anti-aliasing filter

[0041] 14 Analog-to-digital converter

[0042] 20 Audio output device

[0043] 21 Speakers

[0044] 22 Power Amplifier

[0045] 23 Reconstruction filter

[0046] 24-bit digital-to-analog converter

[0047] 30 Reverse audio output device

[0048] 31 Speakers

[0049] 32 Power Amplifier

[0050] 33 Reconstruction filter

[0051] 34 Digital-to-analog converter

[0052] 40 Intelligent audio adjustment system

[0053] 41 Processor

[0054] 42 Storage Devices

[0055] M1 Sound Cancellation Module

[0056] M2 Audio Processing Module

[0057] M3 Intelligent Voice Recognition Module

[0058] J1 Autoregressive Flow Model

[0059] J2 Inverse Autoregressive Flow Model

[0060] M31 Parallel Waveform Neural Network

[0061] M311 Student Waveform Neural Network

[0062] M312 Mentor Waveform Neural Network

[0063] M32 MFCC Unit

[0064] M33 Voice Recognition Module

[0065] LA1 convolutional layer

[0066] LA2 Pooling Layer

[0067] LA3 fully connected layer

[0068] M34 CTC Trainer

[0069] M4 Position Detection Module

[0070] M41 Convolutional Neural Network Classifier

[0071] AW Ambient Audio

[0072] SP Time spectrum

[0073] CA Convolutional Neural Network

[0074] FM feature map

[0075] FC fully connected layer

[0076] M5 Processing Module

[0077] AP Audio Device

[0078] A Immersive Space

[0079] A1 Immersive Space

[0080] A2 Immersive Space

[0081] B Non-immersive space

[0082] B1 Non-immersive space

[0083] L1 reverses the direction of the audio output

[0084] L2 reverses the direction of the audio output

[0085] PH Virtual Face DETAILED DESCRIPTION

[0086] The detailed description and technical content of the present invention are described as follows with reference to the accompanying drawings. The "one side" mentioned in the present invention can be the upper side, lower side, left side, right side, front side, rear side of the corresponding object, or be arranged at any adjacent position of the object, or be directly or indirectly connected to the object, etc., which is not limited in the present invention.

[0087] The "device", "module", "model", "unit" and the corresponding functions described in the present invention can be performed by a single chip or a combination of multiple chips, and the number of such chip configurations is not within the scope of the present invention. In addition, the chip can be, but is not limited to, a combination of processors (Processor), central processing unit (CPU), microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) and other devices, which are not limited in the present invention.

[0088] The "immersive space" described in the present invention refers to the target area of ​​sound effects in the playback environment, that is, the area where the sound is expected to act; the "non-immersive space" described in the present invention refers to the area in the environment where sound is not desired to be transmitted, and can be any area other than the "immersive space". In another embodiment, the "non-immersive space" can also be a specific direction where the volume is desired to be reduced in order to enhance the directionality of the playback sound, thereby enhancing the stereo effect in the playback environment by reducing the volume in the specific direction.

[0089] The immersive system disclosed in the present invention can be applied to audio devices such as televisions, laptops, mobile devices, soundbars, portable devices or car head units. The sound output direction or range of the audio devices can be controlled by the immersive audio system to reduce the impact of spillover sound on non-target areas.

[0090] The following is a description of one embodiment of the present invention. Figure 1 and Figure 2 , which are the block diagram (1) and block diagram (2) of the open space immersive system with high-speed dynamic intelligent voice in the present invention, as shown in the figure: the open space immersive system 100 with high-speed dynamic intelligent voice in this embodiment mainly includes an audio receiving device 10, an audio output device 20, a reverse audio output device 30, and an intelligent audio adjustment system 40.

[0091] The audio receiving device 10 is used to receive ambient sound. The audio receiving device 10 is used to receive error sound in a non-immersive space and convert it into error audio; in another embodiment, another audio receiving device may be further included to specifically receive ambient sound in an immersive space, which is not limited in the present invention. In another embodiment, another audio receiving device may be further included to specifically receive voice commands in an immersive space or a non-immersive space, which is not limited in the present invention. In one embodiment, the audio receiving device 10 includes a microphone 11, a preamplifier 12 connected to the rear end of the microphone 11, an anti-aliasing filter 13 connected to the rear end of the preamplifier 12, and an analog-to-digital converter 14 connected to the rear end of the anti-aliasing filter 13. These devices can be increased or decreased according to practical needs, which is not limited in the present invention.

[0092] The audio output device 20 outputs the playback sound according to the audio source signal. The "audio source signal" mentioned here refers to the original sound effect signal provided by the intelligent audio adjustment system 40, that is, the original file of the sound effect. "Playback sound" is the sound effect broadcast by the audio output device 20 in the real field environment (the space included in the immersive area), which is described in advance. In one embodiment, the audio output device 20 includes a speaker 21, a power amplifier 22 connected to the front end of the speaker 21, a reconstruction filter 23 connected to the front end of the power amplifier 22, and a digital-to-analog converter 24 connected to the front end of the reconstruction filter 23. These devices can be increased or decreased according to practical needs, and are not limited in the present invention.

[0093] The reverse audio output device 30 outputs reverse cancellation sound according to the reverse audio. The "reverse audio" mentioned here refers to the original signal provided by the intelligent audio adjustment system 40 for canceling the sound played in the non-immersive area. The "reverse cancellation sound" is the sound effect broadcast by the reverse audio output device 30 in the real field environment (the space included in the non-immersive area), which is first explained here. In one embodiment, the reverse audio output device 30 includes a speaker 31, a power amplifier 32 connected to the front end of the speaker 31, a reconstruction filter 33 connected to the front end of the power amplifier 32, and a digital-to-analog converter 34 connected to the front end of the reconstruction filter 33. These devices can be increased or decreased according to practical needs, and are not limited in the present invention.

[0094] The intelligent audio adjustment system 40 is connected to the audio receiving device 10, the audio output device 20 and the reverse audio output device 30, and controls the output of the audio output device 20 and the reverse audio output device 30 through the error sound and voice command received by the audio receiving device 10. The intelligent audio adjustment system 40 includes a processor 41 and a storage device 42. The processor 41 executes the corresponding program after loading the program code stored in the storage device 42.

[0095] For the work performed by the intelligent audio adjustment system 40, please refer to the following Figure 3 and Figure 4 , which are the block diagram (1) and block diagram (2) of the intelligent audio adjustment system in the present invention, as shown in the figure: the intelligent audio adjustment system 40 in the present invention includes a sound elimination module M1, an audio processing module M2, an intelligent voice recognition module M3, a position detection module M4, and a processing module M5 according to different executed programs.

[0096] After the sound elimination module M1 receives the error sound in the non-immersive area via the audio receiving device 10, it generates an inverted audio (Inverted Audio) based on the error sound and outputs it to the audio output device 20 to produce an inverted cancellation sound. In one embodiment, the reference audio may include, for example, an audio source signal or a playback sound received by the audio receiving device 10 in the immersive area, which is not limited in the present invention. The sound elimination module M1 may directly receive the playback audio from the audio processor, and adjust the phase of the playback audio to generate an inverted audio with the same amplitude but opposite phase relative to the playback sound. Specifically, the sound elimination module M1 may be, for example but not limited to, completed by phase inversion, fast Fourier transform (FFT), time domain convolution, audio decoding and encoding, neural network (such as machine learning network, deep learning network or recurrent neural network, etc.) or other similar methods, which is not limited in the present invention.

[0097] The audio processing module M2 is used to output the played audio to the audio output device 20 so as to play the sound in the immersive area. Specifically, the audio processing module M2 outputs the audio source signal to the audio output device 20 after reading the long-term data or temporary data of any storage device, so as to play the sound through the audio output device 20. In one embodiment, the audio processing module M2 can be, for example, any sound card, onboard sound card or sound processing chip, etc. Such sound cards include, for example, decoders, digital to analog converters (Digital to Analog Converter, DAC), analog to digital converters (Analog to Digital Converter), amplifiers, filters, sound synthesizers, etc., which are not limited in the present invention.

[0098] The intelligent voice recognition module M3 is used to obtain the user's voice information from the ambient sound after the audio receiving device 10 receives the ambient sound, and obtain the voice command by analyzing the voice information. For the architecture of the intelligent voice recognition module M3, please refer to Figure 5 , which is a schematic diagram of the overall architecture of the intelligent speech recognition module in the present invention, as shown in the figure: the intelligent speech recognition module M3 in this embodiment mainly includes a parallel waveform neural network M31 (Parallel Wavenet), an MFCC unit M32, a speech recognition module M33, and a CTC trainer M34.

[0099] The parallel waveform neural network M31 is used to generate a real speech signal based on the environmental sound, the MFCC unit M32 is used to extract audio features from the real speech signal, the speech recognition module M33 is used to map the audio features output by the MFCC unit M32 to voice commands, and the CTC training unit M34 is used to calculate the distance between the output sequence and the label sequence through the connection temporal classification loss function (CTC loss), and update the model parameters of the speech recognition module M33 through the back propagation algorithm.

[0100] Please refer to the architecture of the parallel waveform neural network M31. Figure 6 , Figure 7 and Figure 8 , which are schematic diagrams of the architecture of the autoregressive flow model, the inverse autoregressive flow model and the parallel waveform neural network in the present invention, as shown in the figure: First, when the parallel waveform neural network M31 receives the environmental sound, it selects the input features and prepares the speech data set. These features can capture the key information of the speech signal, and the data volume is smaller than the original speech signal, which is more beneficial to the training of the model.

[0101] Furthermore, before inputting into the parallel waveform neural network M31, the converted environmental sound data is normalized, which can improve the learning effect of the model and prevent the model from having numerical instability during the training process. In one embodiment, Z-score normalization is adopted, and the formula is as follows:

[0102]

[0103] Among them, Z is the value after Z-score normalization; x is the original data value; u is the mean or expected value of the data set; σ is the standard deviation of the data set.

[0104] Through the above-mentioned preprocessing and normalization procedures, the environmental sound is converted into a data form suitable for learning of the parallel waveform neural network M31.

[0105] Before explaining the parallel waveform neural network M31, a brief description of the autoregressive flow model architecture and the inverse autoregressive flow model architecture is required.

[0106] like Figure 6 As shown in the figure, the autoregressive flow model J1 (AF) is a generative model based on autoregression, which generates data by transforming samples from a simple probability distribution (such as a standard Gaussian distribution) to a target complex distribution. The basic idea of ​​the autoregressive flow model J1 is to generate each dimension of the sample in sequence, and the generation of each dimension depends on the previously generated dimension. The computational overhead of the autoregressive flow model J1 when generating data is relatively large because the generation of each dimension needs to be performed in sequence.

[0107] The expression of the autoregressive flow model J1 is as follows:

[0108] y i =v i exp a i +v i ;

[0109]

[0110]

[0111]

[0112] Among them, y i is the state variable output at time step i; v i The intermediate variable at time step i; a i is another intermediate variable at time step i; A mapping function from previously generated dimensions to intermediate variables; It is another mapping function from previously generated dimensions to intermediate variables. Autoregressive flow model training can be parallelized to fully utilize the advantages of GPU, but when sampling and generating samples, they must be generated step by step in sequence, so the speed is very slow.

[0113] like Figure 7 As shown in the figure, compared with the autoregressive flow model J1, the inverse autoregressive flow model J2 (IAF) is a variation of the autoregressive flow model J1. Its core idea is to reverse the generation process of the autoregressive flow, so as to be more efficient in generating data. When generating data, the inverse autoregressive flow model J2 can use vectorized operations (such as matrix multiplication) to generate all dimensions simultaneously, instead of sequentially like the autoregressive flow model J1. This makes the inverse autoregressive flow model J2 generate data faster, which is particularly suitable for scenarios of real-time or large-scale data generation.

[0114] The inverse autoregressive flow model J2 is expressed as follows:

[0115] y i =u i exp b i +u i ;

[0116]

[0117]

[0118]

[0119] Among them, y i is the state variable output at time step i; u i The intermediate variable at time step i; b i is another intermediate variable at time step i; A mapping function from the previously input dimensions to the intermediate variables; is another mapping function from the previously input dimension to the intermediate variable. As can be seen from the attached figure, the autoregressive machine of the autoregressive flow model J1 is placed at the output end, resulting in the inability to generate in parallel. The inverse autoregressive flow model J2 reverses the structure of the autoregressive flow model J1 and places the regression machine at the input end. i No longer directly dependent on y1, y2…y i-1 , but x1, x2…x i-1 , if x i If it has been calculated in advance, y can be generated in parallel i .

[0120] Since the results are reversed, the characteristics of the inverse autoregressive flow model J2 and the autoregressive flow model J1 are exactly opposite. The autoregressive flow model J1 can be parallel during training but can only be serial during regeneration, while the inverse autoregressive flow model J2 only puts the autoregressive mechanism at the input end and still cannot be parallel during training.

[0121] The method to complement these two advantages is probability density distillation (PDD), which uses the autoregressive flow model J1 to quickly train a model in parallel as the mentor waveform neural network M312, and uses the inverse autoregressive flow model J2 as a student waveform neural network M311 to learn the output of the mentor waveform neural network M312. Specifically, during the training process of the parallel waveform neural network M31, a trained waveform neural network model is used as the mentor waveform neural network M312 to guide the learning of the parallel waveform neural network M311, and the features of the same speech segment are input into the mentor waveform neural network M312 and the student waveform neural network M311, and the student waveform neural network M311 is allowed to try to learn the output of the mentor waveform neural network M312. The architecture of the parallel waveform neural network M31 is shown in the figure.

[0122] Similar to the generative adversarial network, the student waveform neural network M311 plays the role of the generator, while the mentor waveform neural network M312 plays the role of the discriminator; unlike the generative adversarial network, the student waveform neural network M311 does not try to fool the mentor waveform neural network M312 in an adversarial manner; it cooperates by trying to match the probability of the mentor waveform neural network M312. In addition, the mentor waveform neural network M312 remains unchanged and is not trained together with the student waveform neural network M311, and after the training of the student waveform neural network M311 is completed, the mentor waveform neural network M312 is no longer used, and the student waveform neural network M311 is used to perform the generation task.

[0123] The student waveform neural network M311 and the mentor waveform neural network M312 no longer output discrete speech signals, but the parameters of the mixed logic distribution (Mixture of Logistics, MoL) of the speech signal; the mixed logic distribution output by the student waveform neural network M311 is sampled and sent to the mentor waveform neural network M312, and the KL divergence (Kullback-Leibler divergence) of the mixed logic distribution of the mentor waveform neural network M312 can characterize the gap between the student waveform neural network M311 and the real sample, so that the objective function can train the student waveform neural network M311. The KL divergence expression of the mixed logic distribution of the mentor waveform neural network M312 is as follows:

[0124] D KL (P S ||P T )=H(P s ,P T )-H(P S );

[0125] Where D KL (P S ||P T ) represents the KL divergence between the student waveform neural network M311 and the tutor waveform neural network M312; P s represents the probability distribution of the student waveform neural network M311; P T represents the probability distribution of the mentor waveform neural network M312; H(P s ,P T ) represents the cross-entropy between the student waveform neural network M311 and the tutor waveform neural network M312; H(P S ) represents the entropy of the student waveform neural network M311.

[0126] The smaller the value of KL divergence, the closer the distribution between the student waveform neural network M311 and the tutor waveform neural network M312 is, and the better the distillation effect is. Therefore, in probability density distillation, KL divergence is usually minimized so that the output distribution of the student waveform neural network M311 is as close as possible to the output distribution of the tutor waveform neural network M312.

[0127] Because the sound quality synthesized by the student waveform neural network M311 is often poor when KL divergence is used as the objective function alone, power loss is added to the objective function. The expression of power loss is as follows:

[0128]

[0129] Wherein, B is the number of short-time Fourier transform frequencies (in one embodiment, B=1025); STFT(x) represents the result of Fourier transforming the input signal x; x n represents the actual sample; TFT(x n ) means that the actual sample x n The result of Fourier transform. By minimizing the power loss, the STFT coefficients output by the model can be made closer to the STFT coefficients of the actual sample, thereby improving the performance of tasks such as speech enhancement and speech separation.

[0130] Since the probability density distillation training alone may not be sufficient to constrain the student waveform neural network M311 to generate high-quality audio streams, the present invention introduces an additional loss function to guide the student waveform neural network M311 to distribute to the desired output space.

[0131] The first additional loss is the power loss, which ensures that the power used in different frequency bands of speech is averaged as much as human speech. The energy loss helps to avoid the collapse of the student waveform neural network M311 to the high-entropy WaveNet-mode. The expression of the power loss is as follows:

[0132]

[0133] Among them, g(z,c) represents the output of the model, z is the input of the model, c is the condition information of the model; y represents the input of the training sample, that is, the reference signal.

[0134] Since the power loss can be averaged over time before taking the Euclidean distance, the effect is not much different, which means that the average power of different frequencies is important. Therefore, perceptual loss, style loss (Euclidean distance between Gram matrices) and contrastive distillation loss are adopted.

[0135] Regarding perceptual loss, the expression of perceptual loss is as follows:

[0136]

[0137] j represents the jth layer of the network; C j H represents the number of convolution kernels in the jth layer; j and W j Respectively represent the height and width of the j-th convolution kernel; represents the output of the conversion network, that is, the feature map of the generated image at the jth layer; Φ j (y) represents the feature map of the bull's-eye diagram at layer j; represents the perceptual loss at the jth layer.

[0138] Regarding style loss, the expression of style loss is as follows:

[0139]

[0140] Where Ψ is Φj(x) reshaped into (C j ,H j ,W j ) matrix; It is Gram-Matrix;

[0141] j represents the jth layer of the network; C j H represents the number of convolution kernels in the jth layer; j and W j Respectively represent the height and width of the j-th convolution kernel; represents the feature map of the bull's-eye graph at layer j; G j (y) represents the output of the conversion network, that is, the feature map of the generated image at layer j; represents the Gram matrix of the bull's eye diagram image x at the jth layer, where c and c′ represent the numbers of the convolution kernels respectively; represents the style loss at the jth layer.

[0142] The Gram matrix can actually be seen as a covariance matrix between features. In the feature map, each number comes from the convolution of a specific filter at a specific position, so each number represents the strength of a feature. What Gram actually calculates is the correlation between features, which two features appear at the same time, which two are mutually exclusive, and so on. At the same time, the diagonal elements of Gram also reflect the amount of each feature appearing in the audio. Therefore, Gram helps to grasp the general style of the entire audio.

[0143] Regarding the contrastive loss, the expression of the contrastive loss is as follows:

[0144] D KL (P S (c1)||P T (c1))-γD KL (P S (c1)||P T (c2));

[0145] γ is 0.3; c1 and c2 are constants used to determine the loss of positive and negative sample pairs; P S represents the similarity score of the positive sample pair; P T Represents the similarity score of the negative sample pair.

[0146] When the tutor waveform neural network M312 and the student waveform neural network M311 are under the condition of the same information c1 (e.g., language features, speaker ID, etc.), the difference in KL between them is minimized. However, for different conditions c1=c2, the difference is still maximized. In order to achieve this loss, the output of x=g(z,c1) in the student waveform neural network M311 is made to evaluate the waveform twice under the guidance of the tutor waveform neural network M312: once using the same conditional reflex P T(x|c1), once using the randomly sampled conditioned input P T (x|c2).

[0147] For the training process of parallel waveform neural network, please refer to Fig. 9 , is a training flow chart of the parallel waveform neural network in the present invention, as shown in the figure.

[0148] Step S11: First, read the training data from the database. Specifically, the training engine reads the training data set from the database. The data may include the audio file of the speech segment and the corresponding text label.

[0149] Step S12, then, initialize the network structure of the tutor waveform neural network M312 and the student waveform neural network M311. In this step, initialize the network structure of the tutor waveform neural network M312 and the student waveform neural network M311, which includes setting the number of layers of the neural network, the number of neurons in each layer, and the selection of the activation function, as shown below:

[0150] Number of Layers: Select the number of layers between 3 and 8;

[0151] Number of Neurons: Set the number of neurons in each layer to 128, 256, or 512;

[0152] Activation Function: For the hidden layer, ReLU (Rectified Linear Unit) is used as the activation function. For the output layer, a linear activation function is used;

[0153] Output Layer: Since audio waveform is generated here, the activation function of the output layer can be a linear function;

[0154] Model Size: Choose between approximately 1,000,000 and 5,000,000 parameters.

[0155] Step S13, then, initialize the parameters of the tutor waveform neural network M312 and the student waveform neural network M311. For initialization, choose to use Xavier initialization, which initializes the weights of each layer to ensure that the initialized weights have an appropriate range. The Xavier initialization formula is as follows:

[0156]

[0157] Among them, n in is the number of neurons in the input layer; nout is the number of neurons in the output layer; u is a random value sampled from a uniform distribution.

[0158] Step S14: Next, establish the student waveform neural network M311: The steps of establishing the student waveform neural network M311 include:

[0159] Step S14-1: Obtain input features: Specifically, obtain input features from training data, such as voice features or speaker ID, etc. These features will serve as inputs to the student waveform neural network M311.

[0160] Step S14-2: Generate white noise and perform right shift conversion: Generate white noise and perform right shift conversion on it. This conversion is to introduce randomness and diversity to increase the generalization ability of the model.

[0161] Step S14-3: Update white noise in each loop layer. In each loop layer, update white noise to change the input of the model. This helps the student waveform neural network M311 to better adapt to different input conditions.

[0162] Step S14-4: Update the z value by calculating the average value and ratio of the output. Specifically, perform the following operation on each element in the output vector, assuming x is the output vector and N is the dimension of the vector. Calculate the output average value u and ratio σ as follows:

[0163]

[0164]

[0165] Here, xi represents the i-th element in the vector.

[0166] Update the z value as follows:

[0167] z=(x-μ) / (σ+∈);

[0168] ∈ is a small positive number (e.g. 10 -8 ) to avoid division by zero.

[0169] This formula describes the process of updating the z value by calculating the mean and scale of the output vector and then standardizing the output vector. The standardized value is used as the new z value. The z value is calculated by subtracting the output vector from its mean and then dividing it by the standard deviation. This will make the numerical distribution of the output vector closer to the standard normal distribution, which will help improve the stability and convergence of the model.

[0170] Step S14-5: Adjust the mean value and proportion of the white noise to obtain a new logistic distribution, and sample sample_x from it. Generate a new logistic distribution based on the adjusted mean value and proportion of the white noise. Through this distribution, sample a sample sample_x as the generation result of the student waveform neural network M311.

[0171] Step S14-6: Calculate Power_loss and store the value of this loss function for adjustment in the subsequent model training process. Power_loss is a loss function used to measure the difference between the sample sample_x generated by the student waveform neural network M311 and the real data. By calculating the value of this loss function, it can be stored for adjustment and optimization in the subsequent model training process.

[0172] Step S15, then, train the mentor waveform neural network M312: The mentor waveform neural network M312 training process includes the following sub-steps:

[0173] Step S15-1: Generate white noise and perform right shift conversion; Generate white noise and perform right shift conversion, similar to step 14-2 in the training process of the student waveform neural network M311.

[0174] Step S15-2: updating white noise in each layer of the loop; updating white noise in each layer of the loop so that the tutor waveform neural network M312 can better adapt to different input conditions, similar to step 14-3 in the training process of the student waveform neural network M311.

[0175] Step S15-3: Predict the sound signal by using the output of the student waveform neural network M311 and the current parameters: In this step, the sound signal is predicted using the output of the student waveform neural network M311 and the current parameters. The prediction result will be used in the subsequent step 15-4.

[0176] Step S15-4: Calculate H(Ps,Pt)_loss to measure the difference between the "predict_x" predicted by the student waveform neural network M311 and the actual sample "sample_x". This loss function is used to guide the output of the student waveform neural network M311 to approach the output of the mentor waveform neural network M312.

[0177] Here we calculate H(Ps,Pt)_loss, which is a loss function used to measure the difference between "predict_x" predicted by the student waveform neural network M311 and the actual sample "sample_x". Specifically, the mean square error (Mean Squared Error) or cross-entropy loss (Cross-Entropy Loss) can be used as the loss function. By calculating the value of this loss function, we can measure the prediction ability of the student waveform neural network M311 and guide the model training process to a more accurate direction.

[0178] Step S16: Update model parameters using optimization algorithms such as gradient descent; In this step, this embodiment uses optimization algorithms such as gradient descent and the Adam algorithm to update the parameters of the model. By calculating the gradient of the loss function, the weights and biases in the model can be updated according to the direction and size of the gradient to minimize the value of the loss function.

[0179] Step S17: Finally, repeat steps S14 to S16 until a predetermined number of training rounds or a stop condition is reached; in this step, steps S14 to S16 are repeated until a predetermined number of training rounds is reached or a stop condition is met. In this way, multiple rounds of model training can be performed to improve the performance and accuracy of the model. Through the above steps, the training of the parallel waveform neural network M31 is completed.

[0180] The MFCC unit M32 is used to perform the MFCC method. MFCCs (Mel Frequency Cepstral Coefficients) is an algorithm for extracting audio features and is used to convert raw audio into feature information for subsequent processing. In the present invention, MFCC learns the human voice features of the real speech signal received by the parallel waveform neural network model M31.

[0181] Please refer to the MFCC calculation steps. Fig.10 , is a schematic diagram of the calculation process of the MFCC algorithm in the present invention, as shown in the figure: The following is a detailed description of the main steps of MFCC:

[0182] Step S21, audio file reading: In this step, an audio file (eg, .wav) of a real voice signal is read. After reading the audio file, a digital signal containing audio data can be obtained.

[0183] Step S22, pre-emphasis: In order to balance the spectrum in the audio signal, we perform pre-emphasis in this step. Usually a first-order high-pass filter is used to process the audio signal, which helps to strengthen the high-frequency components and reduce the resonance phenomenon in the signal. The expression of pre-emphasis is as follows:

[0184] s′ n =s n -as n-1 ;

[0185] Among them, s′ n is the output, s n is the input, s n-1 is the input of the previous stage, and a is any parameter between 0.9 and 1.

[0186] Step S23, framing: dividing the audio signal into multiple frames of a certain length. Since the speech signal is time-varying, the framing process can help us extract features of the signal within a local time range. In one embodiment, the length of each frame can be, for example, but not limited to, 20-30 milliseconds, and there is a certain overlap between adjacent frames, which is not limited in the present invention.

[0187] The expression for setting the number of frames is as follows:

[0188] F n =f s ×F L ;

[0189] Among them, F n is the number of frames, f s is the sampling frequency, F L is the frame length.

[0190] The expression of the detection and translation setting is as follows:

[0191]

[0192] Among them, F S is the frame translation, f s is the sampling frequency, F L is the frame length.

[0193] Step S24, windowing: In order to reduce signal interference at the frame boundary, we need to apply a window function (such as a Hamming window) to each frame. The window function can make the signal at the frame boundary transition smoothly and reduce the frequency domain fluctuation caused by the truncation of the time domain signal. The expression of the Hamming window is as follows:

[0194]

[0195] 1≤n≤N;

[0196] Wherein, Ham(N) is a window function of length N, n represents the index of the point of the window function (between 1 and N), and N represents the length of the Hamming window function.

[0197] Step S25, Fourier transform: Perform Fourier transform on the windowed signal, convert it from the time domain to the frequency domain, and obtain the spectrum of the signal. When the transform size is 2 to the power of N, the fast Fourier transform (FFT) can be used to speed up the calculation. The expression of Fourier transform is as follows:

[0198]

[0199] 0≤k≤N;

[0200] Among them, x a (k) is the value (complex number) of the kth point in the frequency domain (representing the component of the signal at frequency k / N), x(n) is the value of the nth point in the time domain, N is the number of points of the Fourier transform (used to determine the accuracy of the conversion), k is the index in the frequency domain (0 to N-1), and n is the index in the time domain (0 to N-1). Using the above expression, the time domain signal x(n) can be converted to the frequency domain signal x a (k).

[0201] Step S26, Mel filter bank: The spectrum obtained after Fourier transform is passed through a set of Mel filters to simulate the human ear's perception of sound. The Mel scale is related to the frequency resolution ability of the human ear, with a higher resolution for low frequencies and a lower resolution for high frequencies. The Mel filter bank usually contains multiple triangular filters, which divide the spectrum into different frequency bands and calculate the energy of each frequency band. This can capture important information in the speech signal while reducing irrelevant details. For the algorithm executed by the Mel filter bank, please refer to Fig.11 , is a schematic diagram of the algorithm flow of the Mel filter bank in the present invention, as shown in the figure:

[0202] Step S31, determine the Mel scale range: select a suitable Mel scale range, usually select frequencies within the human hearing range, such as 20Hz to 20kHz. Then convert the frequencies in this range into Mel scale. The conversion expression of Mel scale is as follows:

[0203]

[0204] Where Mel(f) is the Mel scale and f is the original frequency.

[0205] Step S32, define the number of Mel filters: select an appropriate number n of Mel filters according to application requirements and computing resources. Usually, the number of Mel filters is 20-40.

[0206] Step S33, inserting n positions within the Mel scale range: uniformly inserting n positions within the selected Mel scale range, and these positions will serve as the center points of the Mel filter.

[0207] Step S34, convert n+2 Mel frequencies into frequencies: convert the n Mel scale center points obtained in the previous step back into frequencies, add the start and end frequencies, and obtain a total of n+2 frequency values. The expression of the inverse Mel function is as follows:

[0208] f=700·(10 m / 2595 -1);

[0209] Where m is the Mel frequency and f is the corresponding original frequency in Hertz.

[0210] Step S35, calculate n+2 index numbers k: map n+2 frequency values ​​to the spectrum index after Fourier transformation to obtain n+2 index numbers k. These indexes will be used to calculate the frequency response of the Mel filter. The expression of index number k is as follows:

[0211]

[0212] Where k is the index number; N is the number of FFT points; f s is the sampling frequency; f m is the value after converting the Mel scale back to frequency.

[0213] Step S36, Hm matrix: According to the index number k obtained in the above steps, construct an Hm matrix of n rows. Each row represents the frequency response of a Mel filter. The filter usually has a triangular shape, with the center point as the maximum value and the two sides gradually decaying to zero. Multiplying the spectrum with the Hm matrix, the spectrum characteristics after the Mel filter bank processing can be obtained. These characteristics have an energy distribution related to human hearing. When the frequency is very small, the filter width is very narrow. As its frequency increases, the filter width will gradually increase, but its amplitude will also gradually decrease.

[0214]

[0215] Among them, H m (k) is the converted Hm matrix, m is the filter number, and k is the horizontal coordinate (e.g., 0-2048).

[0216] Step S37, Discrete Cosine Transform (DCT): Next, perform discrete cosine transform on the output of the Mel filter bank. DCT is a method for converting signals from the frequency domain to the time domain, and has good energy concentration. The purpose of DCT is to convert the Mel filter bank energy coefficients with high correlation into cepstral coefficients with low correlation. Usually only a portion of the low-frequency cepstral coefficients (such as the first 12 or the first 13) are retained because they contain the main features of the speech signal. For the process of discrete cosine transform, please refer to Fig.12 , is a schematic diagram of the discrete cosine transform process in the present invention, as shown in the figure:

[0217] Step S41, natural logarithm operation: after the signal passes through the Mel filter bank, the natural logarithm of the output energy of each filter is taken. This step helps to simulate the perception characteristics of the human ear to sound energy.

[0218]

[0219] The H matrix in the expression is the energy spectral density E and the Mel filter bank matrix H m The energy spectral density E is the energy distribution of the signal at different frequencies, and the expression is as follows:

[0220]

[0221] It is the instantaneous power of the signal at a certain frequency. The energy spectral density E is combined with the Mel filter bank matrix H m After multiplying the transpose of , the output energy of each Mel filter can be obtained.

[0222] Step S42, discrete cosine transform: perform discrete cosine transform on the energy value after taking the natural logarithm. This step converts the speech signal from the Mel frequency to the cepstrum domain and helps to identify the basic features and details in the speech. The expression of discrete cosine transform of natural logarithm is as follows:

[0223]

[0224] Among them, mfcc(i,n) represents the nth MFCC parameter of the i-th frame data, represents the nth MFCC parameter of the i-th frame data, i is the frame data, and H is H m matrix, M is the number of Mel filters, n is the nth column of the i-th frame, and H(i,m) represents the energy value between the i-th frame data and the m-th filter in the energy matrix H after the Mel filter bank.

[0225] Step S43, Cepstral operation: In this step, a cepstral operation is performed to convert the time domain signal into a frequency domain signal, usually retaining the first few coefficients in the DCT output. These retained coefficients capture the main information of the speech signal while ignoring unimportant high-frequency details. In one embodiment, 12 to 22 coefficients may be retained, depending on the application requirements. The expression of the cepstral operation is as follows:

[0226]

[0227] k(i) represents the Mel cepstral coefficient of the i-th frame data, L is the cepstral parameter (the value is 22 in this embodiment), and i is the frame data. Because most of the signal data is generally concentrated in the low-frequency area after transformation, only the first 13 data are taken.

[0228] Step S44, MFCC basic parameters Feat: The retained cepstral coefficients are used as the basic parameters of MFCC features. These features can be used for applications such as speech recognition, speech synthesis, and speaker recognition. Sometimes, the MFCC features can be further processed, such as adding derivatives, second-order derivatives, etc., to improve the expressiveness of the features. The final MFCC basic parameter matrix feat is expressed as follows:

[0229] feat(i,j)=J(i,j)·K(j);

[0230] i=1,2,3,…,301; j=1,2,…,13;

[0231] Among them, feat(i,j) is the jth MFCC basic parameter of the i-th frame data, J(i,j) represents the i-th frame data and the j-th column in the MFCC parameter matrix, and K(j) represents the j-th Mel cepstral coefficient.

[0232] Furthermore, the second set of parameters is a first-order differential operation under the existing basic parameters, and the third set of parameters is a first-order differential operation under the second set of parameters. The expression is as follows:

[0233] dtf eat [i][j]=feat[i+1][j]-feat[i-1][j]+2·feat[i+2][j]-

[0234] 2feat[i-2][j];

[0235] i=1,2,3,…,301; j=1,2,…,13;

[0236] dtf eat [i][j] is the first-order differential value of the j-th MFCC parameter of the i-th frame data, f eat [i][j] represents the jth MFCC basic parameters of the i-th frame data. In this expression, by eat Each element in the matrix is ​​subjected to a first-order differential operation to calculate the first-order differential parameter dtf eat Similarly, you can do the same for dtf eat Perform first-order differential operation to obtain second-order differential parameters.

[0237] Step S27, finally, when comparing the MFCC feature values ​​between different samples, the scale differences between them are eliminated, so as to better perform model training and feature selection. For the i-th MFCC feature value, the normalization process is as follows:

[0238]

[0239] x i is the original i-th MFCC feature value; mean i is the average value of the i-th MFCC feature value in all samples; s tdi is the standard deviation of the ith MFCC feature value in all samples; F(x i ) is the standardized ith MFCC eigenvalue. Standardizing the original MFCC eigenvalues ​​can eliminate distribution differences, making the MFCC eigenvalues ​​between different samples more comparable, which is beneficial to model training and evaluation.

[0240] On the other hand, the original energy amplitude value is converted to a standardized energy amplitude value. This helps to eliminate the scale differences in the data so that different data can be compared. The standardized formula is as follows:

[0241]

[0242] P(x i ) is the standardized energy amplitude value, which indicates the standardized value of the energy amplitude of the i-th sample; energy is the original energy amplitude value, which indicates the original energy amplitude of the i-th sample; mean_energy is the average value of the energy amplitude values ​​in all samples, which is used to center the energy amplitude value; s tdenergy It is the standard deviation of the energy amplitude values ​​in all samples and is used to scale the energy amplitude values.

[0243] Finally, these two input feature groups (audio feature vector and audio energy amplitude) are combined into a feature vector as the input of the CNN model. For example, suppose 12 features and 1 energy amplitude value are extracted and they are concatenated into a 13-dimensional feature vector.

[0244] The speech recognition module M33 is used to map the audio features output by the MFCC unit M32 to voice commands. In one embodiment, the speech recognition module M33 may be, for example, but not limited to, a convolutional neural network model (CNN); in an embodiment using a convolutional neural network model, the convolutional neural network model may be used to extract features of speech signals, extracting short-term spectral features, such as MFCC features, from the original time series.

[0245] Please also read Fig.13, which is a schematic diagram of the convolutional neural network model architecture used in the speech recognition module of the present invention, as shown in the figure: the convolutional neural network model used in the speech recognition module of the present invention includes a convolutional layer LA1 (Convolutional Layer), a pooling layer LA2 (Pooling Layer), and a fully connected layer LA3 (Fully Connected Layer).

[0246] Convolution layer LA1 is mainly used to extract features. Specifically, convolution layer LA1 is the core of the convolutional neural network model. It extracts features from the input image through a certain number of convolution kernels. Each convolution kernel contains a set of weights, which are convolved with the input image to generate a new feature map. The size and step size of the convolution kernel are set by hyperparameters.

[0247] The pooling layer LA2 is used to reduce the dimension of features. Specifically, the pooling layer LA2 is mainly used for feature dimensionality reduction, reducing the number of parameters and the amount of calculation. Through certain pooling operations (such as maximum pooling, average pooling, etc.), the feature map is reduced in dimension to obtain a new feature map. The main hyperparameters of the pooling layer are the size and step size of the pooling kernel.

[0248] The fully connected layer LA3 is used to generate the final output. Specifically, the fully connected layer is usually used to convert the feature map into the corresponding output and perform tasks such as classification or regression. It expands the feature map into a vector, performs linear transformation and nonlinear transformation through multiple fully connected layers, and finally obtains the output result. In the present invention, the output result is directly mapped to the voice instruction, or mapped to the code or value related to the voice instruction, and the corresponding instruction is found from the database through the code or value, which is not limited in the present invention.

[0249] The CTC trainer M34 is used to execute the CTC Loss algorithm (Connectionist Temporal Classification Loss), which is used as a loss function in sequence learning to solve the sequence alignment problem. In sequence learning tasks such as speech recognition and text recognition, there is often a situation where the input sequence and the output sequence are not completely consistent or aligned, which requires the use of CTC loss to train the convolutional neural network model.

[0250] More specifically, in speech recognition, the data set is the audio file and its corresponding text data. However, it is difficult to align the audio file and the text data in units of words. If the model is trained directly without alignment, it will be difficult for the model to converge due to the different speech speeds of people or the different distances between characters. CTC Trainer M34 can avoid manual alignment of input and output, and is suitable for speech recognition.

[0251] The following expression represents the probability of observing a specific alignment path π given an input feature sequence x. The expression is as follows:

[0252]

[0253] L′=L∪{blank};

[0254] Where p(π|x) is the probability of observing a specific alignment path π given an input feature sequence x. x is the input feature sequence, usually the data of the speech signal after feature extraction. y is the output label sequence, that is, the expected text sequence. T is the length of the input feature sequence x. L′ is the value space, including the output vocab (L) and an additional blank label. L is the output vocab, that is, all possible output characters. πt is the observation probability, which represents the probability of observing a certain character at time t.

[0255] The following expression represents the sum of the probabilities of selecting the target sequence I among all possible pathsπ (in the mapping set β) given an input x.

[0256]

[0257] Where x is the input sequence, usually a speech signal or text, I is the target sequence, that is, the text sequence to be recognized or generated, and β is the mapping set, also known as the "alphabet", containing all possible text sequences, ∈β -1 (I) is the set of all extended sequences of the target sequence I in the mapping set β, π is a path in the mapping set β, indicating a path selected from all possible paths, P(π|x) is the conditional probability from the input sequence x to pathπ, indicating the probability of mapping x to π, and P(I|x) is the conditional probability from the input sequence x to the target sequence I, indicating the probability of mapping x to I.

[0258] For the CTC Loss process, please refer to Fig.14 , is a schematic diagram of the training process of the CTC trainer in the present invention, as shown in the figure: The process of CTC Loss is as follows:

[0259] Step S51: Initialization Before performing CTC loss calculation, some variables need to be initialized, for example, the probability calculation group (used to calculate forward and reverse probabilities) and the gradient calculation group (used to calculate gradients).

[0260] Probability calculation group: Create two two-dimensional matrices (usually called alpha and beta) to store the calculation results of forward probability and reverse probability. The size of these matrices is usually T×L, where T represents the length of the input sequence and L represents the length of the target sequence (after expansion, including blank symbols).

[0261] Gradient calculation group: Create a gradient calculation group to store the gradient of the loss function with respect to the model parameters. This will be used during the backpropagation process in order to update the model parameters.

[0262] Other variables: Some other variables need to be initialized, such as loss values, regularization terms, etc.

[0263] Before doing CTC loss calculation, make sure all variables are initialized correctly to avoid calculation errors or inconsistencies.

[0264] Step S52: Function import. The steps include 1. Generate two matrices to calculate forward and reverse probabilities: these matrices will be used to store the calculation results of forward probability and reverse probability; 2. Expand labels with blank symbols: insert blank symbols in the target sequence for alignment in CTC loss calculation.

[0265] Generate two matrices to calculate forward and backward probabilities: Here, two two-dimensional matrices (usually called alpha and beta) will be created, which will be used to store the results of the forward and backward probabilities. The size of these matrices is usually T×L, where T represents the length of the input sequence and L represents the length of the target sequence (after expansion, including blank symbols). Each element in the matrix represents the probability at a given time t and position l in the target sequence.

[0266] Augment labels with blank symbols: For CTC calculation, a special blank symbol needs to be inserted between each character of the target sequence. In addition, blank symbols need to be added at the beginning and end of the sequence. For example, if the target sequence is "AB", the augmented sequence will be "-AB-", where "-" represents the blank symbol. The purpose of this is to allow the model to predict the same target character at different input time steps t, making the alignment problem simpler.

[0267] After completing these operations, you can proceed to calculate the forward probability and reverse probability.

[0268] Step S53: Perform forward probability calculation, and calculate the forward probability based on the given input sequence and target sequence. This is the first part of the CTC loss calculation, which represents the probability of observing the input sequence given the target sequence. In this process, given an input sequence and a label sequence, CTC Forward calculates the alignment result of the input sequence and the label sequence, and calculates the loss value between the predicted result and the legal alignment result of the label sequence.

[0269] The expression for forward probability calculation is as follows:

[0270]

[0271] This expression means that the calculation of α t The value of (s) needs to be summed over all possible alignment paths π, where π is a sequence of length T, representing the correspondence between each time step of the input feature sequence x and the output sequence S, and B represents a set that contains all possible paths π that match the first s characters of the current label sequence 1 after t time steps. Therefore, B(π 1:t )=l 1:s Represents the path π 1:t After t time steps, it matches the first s characters of label sequence 1.

[0272] Update rules The expression is as follows:

[0273]

[0274] This update rule is used to calculate the new Forward variable value of the sequence S at time step t. It means that to calculate the new The forward variable α of the previous time step t-1 needs to be t-1 (s) and α t-1 (s-1) is added. This update rule is used to recursively calculate α in the Forward process of the CTC algorithm. t (s), thereby estimating the probability of the output label sequence y given the input feature sequence x.

[0275] Among them, α t (s) is the forward variable, which indicates the probability of generating sequence S at time step t; t is the current time step, and its value range is [1, T]; where T is the length of the input feature sequence x, s is the current position of the sequence S, and its value range is [1, |S|], where |S| indicates the length of the sequence S; π is the alignment path, which indicates the correspondence between each time step of the input feature sequence x and the output sequence S, and π is a sequence of length T; N T(1:s) is the set of all possible alignment paths, where each path π satisfies B(π 1:t )=l 1:s ; B is the operation of removing duplicate labels and blank labels, converting the path π into a label sequence. For example, if the path π is "aa--b-bc", then B(π) = "abc"; l 1:s are the first s characters of the target output sequence; t′ is the variable of time step, which is used to calculate the product of the observation probability from time step 1 to time step t, and the value range of t′ is [1, t]; At time step t′, the input feature sequence x generates label π t′ For example, if at a certain time step, the probability of the input feature sequence x generating the character "a" is 0.8, then y("a") = 0.8.

[0276] Step S54: Execute reverse probability calculation. Similarly, calculate reverse probability. This step is to calculate the correspondence between the target sequence and the speech signal so as to calculate the loss function in the subsequent steps. In this step, the reverse probability of each time step is calculated, which represents the sum of the probabilities of all possible paths from the current time step to the end of the target sequence.

[0277] The expression for reverse probability calculation is as follows:

[0278]

[0279] This expression means that the calculation of β t The value of (s) needs to be summed over all possible alignment paths π, where π is a sequence of length T, representing the correspondence between each time step of the input feature sequence x and the output sequence S.

[0280] Update rules The expression is as follows:

[0281]

[0282] This update rule is used to calculate the new Backward variable value of the sequence S at time step t. It means that to calculate the new It is necessary to change the Backward variable β of the next time step t+1 t+1 (s) and β t+1 (s+1) is added. This update rule is used to recursively calculate β in the Backward process of the CTC algorithm. t (s), thereby estimating the probability of the output label sequence y given the input feature sequence x.

[0283] Among them, β t(s) is the backward variable, which indicates the probability of generating sequence S from time step t to the last time step T; t is the current time step, and its value range is [1, T], where T is the length of the input feature sequence x; s is the position of the current sequence S, and its value range is [1, |S|], where |S| indicates the length of the sequence S; π is the alignment path, which indicates the correspondence between each time step of the input feature sequence x and the output sequence S, and π is a sequence of length T; N T (s:|l|) is the set of all possible alignment paths, where each path π satisfies (π t:T )=l s:|l| ; B removes repeated labels and blank labels and converts the path π into a label sequence. For example, if the path π is "aa--b-bc", then B(π) = "abc"; l s:|l| is the subsequence from s to the last character of the target output sequence; t′ is the variable of time step, which is used to calculate the product of the observation probability from time step t to time step T, and the value range of t′ is [t,T]; is the probability that the input feature sequence x generates the label π(t') at time step t'. For example, if the probability that the input feature sequence x generates the character "a" at a certain time step is 0.8, then y("a") = 0.8.

[0284] Through these two formulas, the Backward variable β can be calculated in the Backward process of the CTC algorithm. t (s) to estimate the probability of outputting label sequence y given input feature sequence x. Combined with the forward variable α t (s) and Backward variable β t (s), the CTC loss function can be calculated and the parameters of the model can be optimized.

[0285] Step S55: Calculate the loss function. Based on the calculation results of the forward probability and the reverse probability, calculate the CTC loss. The CTC loss represents the alignment error between the given input sequence and the target sequence. By minimizing this loss, the model will learn to better align the input sequence, thereby improving the recognition effect. In this step, the forward probability α is used t (s) and the reverse probability β t (s), calculate the gradient of the CTC loss function. These gradient values ​​are used to optimize the parameters of the model, thereby improving the accuracy of the model in predicting the output label sequence y given the input feature sequence x. The expression of CTC loss is as follows:

[0286]

[0287] is the CTC loss function with respect to the startup value The gradient of , which needs to be calculated to optimize the model parameters; is the starting value of the model output (such as the output of a neural network) at time step t, corresponding to label t; is the probability of generating label k for input feature sequence x at time step t; p(l|x) is the probability of outputting label sequence y given input feature sequence x, which can be calculated by CTC Forward variable α t (s) and Backward variable β t (s) is obtained; l is the target output sequence; x is the input feature sequence; α t (s) is the CTC Forward variable, which is the probability of generating sequence s at time step t given the input feature sequence x; β t (s) is the CTC Backward variable, which is the probability of generating sequence s from time step t to the last time step T given the input feature sequence x; α t (s) and β t (s) is the normalized CTC Forward variable and Backward variable; lab(l,k) is the set of all sequences s containing label k; |l′| represents the length of the expanded output sequence; At time step t, the input feature sequence x generates the label l' in the expanded output sequence l' s probability.

[0288] This formula defines the calculation of the CTC loss function with respect to the startup value The gradient is used to optimize the model parameters to improve the accuracy of the model in predicting the output label sequence y given the input feature sequence x. In this formula, the gradient Used to update the parameters of the speech recognition module M33. Through the above steps, the CTC training unit M34 uses the connection time series classification loss function (CTC loss) to calculate the distance between the output sequence and the label sequence and updates the model parameters of the speech recognition module M33 through forward probability calculation and reverse probability calculation loss functions.

[0289] Through the architecture of parallel waveform neural network M31, MFCC unit M32, speech recognition module M33, and CTC trainer M34, the present invention has the effects of high efficiency (by using MFCC features and WAVENET model, the system can realize efficient speech recognition function), high accuracy (by using CTC LOSS, the system can achieve higher recognition accuracy when processing speech information with noise) and high real-time performance (the system can process speech information in real time and realize real-time speech recognition).

[0290] The position detection module M4 analyzes the user's position information based on the ambient sound to confirm whether the user is located in the immersive area or the non-immersive area.

[0291] For the functions and architecture of the location detection module M4, please refer to Fig.15 and Fig.16 , which are the architecture diagram (1) and architecture diagram (2) of the position detection module in the present invention, as shown in the figure: In one embodiment, the position detection module M4 includes a convolutional neural network classifier (CNN classification) M41, and the convolutional neural network classifier M41 is trained to map the input environmental sound into a voice command, and the source of the voice command comes from one of the two results of the immersive area or the non-immersive area.

[0292] The convolutional neural network classifier M41 classifies information based on the source of the generated audio information of the received ambient sound. Specifically, the convolutional neural network classifier M41 refers to a sound classifier that uses convolutional neural networks (CNNs) as the main algorithm. In the CNN classification task, the network is trained with labeled sounds to classify the received ambient sounds according to the characteristics of the ambient sounds.

[0293] Before the ambient audio is input to the convolutional neural network classifier M41, the processor first converts the ambient audio AW into a normalized time-frequency spectrum SP (spectrogram) through spectral analysis so that the convolutional neural network classifier M41 can identify the ambient audio AW.

[0294] Specifically, the sound signal can be converted into a time-frequency spectrum SP in the following manner: first, resample and convert the sound signal into stereo. Then, segment the sound signal, usually using a window function (such as a Hamming window) and an overlapping technique (such as 50% overlap) to divide it into multiple short time periods, and translate the audio signal through the audio augmentation time axis (Audio Augmentation TimeShift) to increase the generalization ability of the model, thereby completing the normalization. Then, for the sound signal in each short time period, use Fourier transform (FFT) to convert it into a spectrum signal. Since the spectrum signal obtained by Fourier transform is a continuous frequency signal from 0Hz to the sampling frequency, it needs to be divided into multiple discrete frequency bands for subsequent processing. This is usually implemented using short-time Fourier transform (STFT), which divides the spectrum signal of each short time period into a series of discrete frequency bands to form a two-dimensional matrix, where the horizontal axis is time and the vertical axis is frequency. Finally, for each frequency band in a short time period, its energy or intensity can be calculated and drawn into a color image or a grayscale image to represent the sound intensity distribution at different frequencies and times, that is, to obtain the time-frequency spectrum SP. Finally, Spectrogram Augmentation (SpecAugment) is performed to enhance the audio data through time masking, frequency masking, time shifting and frequency shifting to improve the generalization ability and robustness of the model.

[0295] After the ambient audio AW is converted into a time-frequency spectrum SP, it is input into a convolutional neural network CA. The convolutional neural network CA consists of a series of convolutional layers and pooling layers. The convolutional layer is used to detect features such as edges, shapes, and patterns. The pooling layer is used to reduce the dimension of the input data and finally establish the feature map (Features Maps) FM of the input time-frequency spectrum SP. The feature map FM contains the local information and global structure of the time-frequency spectrum SP and can be used for subsequent audio classification tasks.

[0296] After obtaining the feature map FM, the feature map FM is input into the fully connected layer FC. The fully connected layer FC is used for actual classification, mapping the extracted features to the corresponding category labels to obtain audio classification information. Specifically, the function of the fully connected layer FC is to flatten the extracted feature map FM, convert it into a one-dimensional vector, and then perform matrix multiplication with the weight matrix to obtain the final output result. Each neuron in the fully connected layer FC is connected to all neurons in the previous layer, and each connection has a weight. In this way, all features of the previous layer can be taken into account for more accurate classification or prediction.

[0297] The audio classification information may be one of the two results of being located in the immersive area or being located in the non-immersive area. In another embodiment, the audio classification information may be a weight value of the immersive area or the non-immersive area, for example, expressing the possibility or relative amount of the ambient sound coming from the immersive area or the non-immersive area in percentage or fraction, for example, the result of the audio classification information may be 60% for the immersive area, 40% for the non-immersive area, or 6 points for the immersive area, 4 points for the non-immersive area, etc., which is not limited in the present invention.

[0298] For the training method of the convolutional neural network classifier M41, please refer to Fig.17 , is a schematic diagram of the training process of the convolutional neural network classifier in the present invention, as shown in the figure: first, prepare multiple sets of original audio files for training (step S61). The original audio files can be, for example, but not limited to, MP3 audio files (MPEG-1 Audio Layer 3), WAV audio files (Waveform Audio File Format), AIFF audio files (Audio Interchange File Format), FLAC audio files (Free Lossless Audio Codec), AAC audio files (Advanced Audio Coding), OGG audio files (Ogg Vorbis), MIDI audio files (Musical Instrument Digital Interface), WMA audio files (Windows Media Audio) or other similar audio files, which are not limited in the present invention. In the step, a large amount of environmental audio can be recorded and divided into short time segments, each segment corresponding to a sample.

[0299] Next, use MFCC to extract time-frequency features for all samples (step S62); in this step, the following sub-steps are included: step SF1, perform short-time Fourier transform on all samples to obtain their spectral features; step SF2, based on the human ear's higher sensitivity to the low-frequency region, use the Mel scale to normalize the short-time Fourier transform spectrum to obtain the Mel cepstrum; step SF3, discretize the Mel cepstrum, and use the discretized Filter Bank to obtain MFCC features. These features are more in line with the nonlinear frequency perception of the human ear; step SF4, construct a two-dimensional time-frequency feature map for each sample based on the MFCC features; step SF5, construct an enhanced data set based on the MFCC feature time-frequency map to provide richer feature information for model training.

[0300] Next, define the convolutional neural network classifier M41 (step S63); in this embodiment, the CNN model includes 1. using 5 convolutional layers, the convolution kernel size decreases from 11×11 to 3×3, and the activation function is ReLU; 2. using 3 maximum pooling layers, the pooling kernel size is 3×3, and the step size is 2; 3. using 3 fully connected layers, the number of neurons in the 1st and 2nd layers is 1024, the number of neurons in the 3rd output layer is 2, representing the two categories of binary classification tasks, and adding a dropout regularization of 0.5; 4. selecting the ADAM optimizer, and the learning rate is set to 0.001; 5. ReLU is selected as the activation function of each layer of CNN, and the ADAM optimizer's β1=0.9, β2=0.999.

[0301] Next, the convolutional neural network classifier M41 is pre-trained (step S64); in this step, a large number of labeled samples are used to train the convolutional neural network classifier M41 so that it can learn the characteristic representation of the environmental sound; some samples are manually labeled as immersive areas or non-immersive areas to form a labeled sample set. It should be noted that since audio may produce different sound field information in different environments, in order to improve reliability, the trained environmental sound samples can be obtained on-site or in a general environment, which is not limited in the present invention.

[0302] Next, the convolutional neural network classifier M41 is trained and adjusted (step S65). In this step, the convolutional neural network classifier M41 is fine-tuned using a labeled sample set so that in addition to learning the audio features, it also learns the features for distinguishing between immersive areas and non-immersive areas.

[0303] The convolutional neural network classifier M41 is trained according to the following process: Step SG1, forward propagation: forward propagation is performed through the CNN model to map the input data to one of the classification results of the immersive area or the non-immersive area, and the difference between the predicted result and the true label is calculated; Step SG2, back propagation: through the back propagation algorithm, the model parameters of the convolutional neural network are adjusted according to the loss function to make the model's prediction result closer to the true label; Step SG3, parameter update: the model parameters of the convolutional neural network are updated according to the optimizer to minimize the loss function; and Step SG4, repeated training: repeated forward propagation, back propagation and parameter update operations until the model converges or reaches a predetermined number of training times.

[0304] Finally, the threshold logic is determined (step S66); in this step, based on a large number of manually labeled samples, the threshold or judgment logic for determining whether the new sample is an immersive area or a non-immersive area is determined. In principle, after the ambient sound enters the convolutional neural network classifier M41, the convolutional neural network classifier M41 will calculate the probability weight of an immersive area. When this probability weight is greater than a threshold, it is determined that the ambient sound is from the immersive area, otherwise it is from the non-immersive area.

[0305] The processing module M5 generates corresponding control instructions based on the user's voice instructions and location information, and switches the audio output mode of the audio processing module M2 or the reverse audio output mode of the sound elimination module M1 according to the control instructions. Specifically, the following implementation scenarios may be included:

[0306] Scenario 1: The processing module M5 receives a voice command from the immersive area. When receiving the voice command of "turn up the volume", the processing module M5 can respond by turning up the volume of the immersive area (i.e., increasing the gain of the audio output device 20) through the audio processing module M2;

[0307] Scenario 2: The processing module M5 receives a voice command from the immersive area. When receiving the voice command of "lower the volume", the processing module M5 can respond by lowering the volume of the immersive area (i.e., lowering the gain of the audio output device 20) through the audio processing module M2;

[0308] Scenario 3: When the processing module M5 receives a voice command from a non-immersive area and receives a voice command of "turn up the volume", the processing module M5 can respond by turning up the volume of the non-immersive area (i.e., reducing the gain of the reverse cancellation sound) through the sound cancellation module M1;

[0309] Scenario 4: When the processing module M5 receives a voice command from a non-immersive area and receives a voice command of "lower the volume", the processing module M5 can respond by lowering the volume of the non-immersive area through the sound cancellation module M1 (i.e., increasing the gain of the reverse cancellation sound);

[0310] Scenario 5: When the processing module M5 receives a voice command from the immersive area and receives a voice command of "improving the immersive effect", the processing module M5 can respond by eliminating the noise from the non-immersive area through the sound elimination module M1;

[0311] Scenario 6: When the processing module M5 receives a voice command from a non-immersive area and receives a voice command to "enhance the immersive effect", the processing module M5 can respond by increasing the gain of the reverse offset sound;

[0312] Scenario 7: When the processing module M5 receives a voice command from the immersive area and receives a voice command to "reduce immersion effect", the processing module M5 can respond by reducing the stereo output through the audio processing module M2.

[0313] It should be noted that the above scenario descriptions are merely examples of several embodiments of the present invention, and these embodiments are not intended to limit the scope of protection of the present invention.

[0314] In summary, please refer to Fig.18 , is a schematic diagram of the overall operation flow of the open space immersive system of the present invention, as shown in the figure: first, the user submits a voice message in the immersive space or the non-immersive space (step S71); the audio receiving device 10 receives the ambient sound including the voice command through the microphone 11 (step S72), and the received ambient sound is processed by the processor 41; after receiving the ambient sound, the processor 41 analyzes the voice information by running the intelligent voice recognition module M3 to obtain a voice command, and finds the text data corresponding to the voice command through the voice recognition module M33 (step S73); and step S At the same time as step 73 (it can also be before or after), the processor 41 analyzes the user's location information through the ambient sound by running the position detection module M5 to confirm that the user is in the immersive area or the non-immersive area (step S74); after confirming the text data of the voice command and the user's location information (in the immersive area or in the non-immersive area), the processor 41 generates corresponding control instructions based on the user's text data and location information by running the processing module M5, and switches the audio output mode of the audio processing module M2 or the reverse audio output mode of the sound elimination module M1 according to the control instruction (step S75). In one embodiment, before step S71, the processor's intelligent voice recognition module M3 can be triggered by specific designated voice information (such as short voice), and after being triggered, it switches from standby mode to working mode, thereby reducing the consumption caused by calculation in the normal state.

[0315] The following is a more detailed description of the embodiments of the specific configuration positions in the open space immersive audio system of the present invention. Please refer to Figures 19 to 22 , which are configuration schematic diagram (1), configuration schematic diagram (2), configuration schematic diagram (3) and configuration schematic diagram (4) of the open space immersive audio system of the present invention.

[0316] For the first configuration method, please refer to Fig.19In this embodiment, the reverse audio output device 30 can be configured at any position of the audio device AP to eliminate the audio played in the non-immersive space, where the immersive space is marked as A and the non-immersive space is marked as B. The position can be, for example, configured at the back side of the audio device AP (such as (a1)) to eliminate the audio played at the back of the audio device AP when the audio output device 20 plays the audio in the front direction; configured at the left side of the audio device AP (such as (a2)) to eliminate the audio played on the left side of the audio device AP; configured at the right side of the audio device AP (such as (a3)) to eliminate the audio played on the right side of the audio device AP; configured at the top side of the audio device AP (such as (a4)) to eliminate the audio played on the upper side of the audio device AP; configured at the bottom side of the audio device AP (such as (a5)) to eliminate the audio played on the bottom side of the audio device AP; or a combination of multiple and configured at any of the above positions, which is not limited in the present invention. In another embodiment, although not shown in the figure, the reverse audio output device 30 can also be set at the corner position or the corner position of the audio device AP, which is explained here first.

[0317] For the second configuration method, see Fig. 20 : On the audio device AP, the reverse audio output device 30 can be configured on both sides of the audio output device 20. By eliminating the playback audio on both sides of the audio output device 20, the directional effect of the audio output device 20 is increased, and the sound field disorder caused by the sound with too wide a propagation range is avoided. It can be seen from the accompanying drawings that the reverse audio output devices 30 on both sides will increase the range of the non-immersive space B1 on both sides, making the transmission range of the immersive space A1 narrower.

[0318] The third configuration mode is shown in the following figure. Fig.21 : In one embodiment, the angle θ1 between the reverse cancellation audio output direction L1 of the reverse audio output device 30 on the left and the virtual plane PH centered on the audio device AP and the immersive space A2 is 20 degrees to 45 degrees, and the angle θ2 between the reverse cancellation audio output direction L2 of the reverse audio output device 30 on the right and the virtual plane PH centered on the audio device AP and the immersive space A2 is 20 degrees to 45 degrees.

[0319] For the fourth configuration method, see Fig. 22: The reverse audio output device 30 can be distributed in an array on the front, back, left, right, top and bottom sides of the audio device AP. For example, in (b1), the reverse audio output device 30 is set on the surface positions of the front, back, left, right, top and bottom sides of the audio device AP; or, for example, in (b2), the reverse audio output device 30 is set on the edge position at each corner of the audio device AP; or, for example, in (b3), the reverse audio output device 30 is set on the vertex position at each corner of the audio device AP; or any combination of the above positions. It is not limited in the present invention to simultaneously set multiple reverse audio output devices 30 at the same surface position, edge position, or vertex position to form an array. In the described embodiment, the non-immersive space where the audio is to be eliminated can be selected according to the factory configuration or according to the setting, so as to configure the best sound field effect in accordance with the corresponding environment. Although the above is represented by dots in the drawings, the dots can be understood as the reverse audio output device 30 being a plurality of arrays distributed on the surfaces, edges, and corners, thereby more accurately adjusting the range of the non-immersive space or the direction of the sound output through the array.

[0320] In summary, the present invention provides a customized sound experience based on the user's area and voice information by determining whether the user is located in an immersive area or a non-immersive area, thereby better meeting the specific needs of individual users or adapting to different environments and situations.

[0321] The present invention has been described in detail above. The above is only one preferred embodiment of the present invention and should not be used to limit the scope of the present invention. That is, all equivalent changes and modifications made according to the scope of the patent application of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. An open space immersive system with high-speed dynamic intelligent voice, characterized in that: include: an audio receiving device; an audio output device; A reverse audio output device; as well as At least one processor executes a sound elimination module, an intelligent speech recognition module, an audio processing module and a processing module after loading at least one storage device, wherein: After receiving the error sound in the non-immersive area through the audio receiving device, the sound cancellation module generates a reverse audio signal based on the error sound and outputs it to the audio output device to generate a reverse cancellation sound; The audio processing module outputs a play audio to the at least one audio output device, so as to play the sound effect in the immersive area; After the audio receiving device receives the ambient sound, the intelligent voice recognition module obtains the user's voice information from the ambient sound, and obtains a voice command by analyzing the voice information. The intelligent voice recognition module includes a parallel waveform neural network that generates a real voice signal based on the ambient sound, an MFCC unit that extracts audio features from the real voice signal, and a voice recognition module that maps the audio features output by the MFCC unit to the voice command; The processing module generates a corresponding control instruction based on the voice instruction of the user, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

2. The open space immersive system with high-speed dynamic intelligent voice according to claim 1, characterized in that: The processor executes a position detection module after loading the at least one storage device. The position detection module analyzes the user's position information from the ambient sound to confirm whether the user is located in the immersive area or the non-immersive area. The processing module generates corresponding control instructions based on the user's voice command and the position information, and switches the audio output mode of the audio processing module or the reverse audio output mode of the sound elimination module according to the control instruction.

3. The open space immersive system with high-speed dynamic intelligent voice according to claim 2, characterized in that: The speech recognition module is a convolutional neural network model.

4. The open space immersive system with high-speed dynamic intelligent voice according to claim 3, characterized in that: The intelligent speech recognition module includes a CTC training unit, which is used to calculate the distance between the output sequence and the label sequence through a connection time series classification loss function, and to update the model parameters of the speech recognition module model through a back propagation algorithm.

5. The open space immersive system with high-speed dynamic intelligent voice according to claim 1, characterized in that: The position detection module includes a convolutional neural network, and the convolutional neural network is trained to map the input environmental sound into the voice command, and the source of the voice command comes from the immersive area or the non-immersive area.

6. The open space immersive system with high-speed dynamic intelligent voice according to claim 5, characterized in that: The convolutional neural network includes a plurality of convolutional layers for extracting the environmental sound features and a fully connected layer for mapping the plurality of environmental sound features to one of the classification results of the immersive area or the non-immersive area according to weights.

7. The open space immersive system with high-speed dynamic intelligent voice according to claim 6, characterized in that: The convolutional neural network is trained according to the following process: Forward propagation: forward propagation is performed through the CNN model to map the input data to one of the classification results of the immersive area or the non-immersive area, and the difference between the predicted result and the true label is calculated; Back propagation: Through the back propagation algorithm, the model parameters of the convolutional neural network are adjusted according to the loss function to make the model's prediction results closer to the true label; Parameter update: updating the model parameters of the convolutional neural network according to the optimizer to minimize the loss function; and Repeated training: Repeat forward propagation, backpropagation, and parameter update operations until the model converges or reaches a predetermined number of training times.

8. The open space immersive system with high-speed dynamic intelligent voice according to claim 1, characterized in that: The intelligent voice recognition module triggers a program for analyzing the voice information through designated voice information or human-machine interface instructions when in standby mode.

Images