Speech recognition, network model training method, device, equipment and storage medium

Abstract

The present application provides a speech recognition and network model training method, device, equipment and storage medium, which solves the problem of poor recognition effect of speech recognition system in the prior art. Based on the initial speaker beam and multi-channel speech signals, the first target speaker beam is determined, the initial speaker beam is obtained based on the beamforming process of the multi-channel speech signals, and the first target speaker beam indicates the sound information of each multi-channel speech signal in the direction of the initial speaker beam; based on the first target speaker beam and the multi-channel speech signals, the noise beam is determined, and the noise beam indicates the sound information in the multi-channel speech signals except the first target speaker beam; based on the noise beam and the wake-up beam, the first target speaker beam is filtered to obtain the second target speaker beam, and the wake-up beam indicates the beam containing the wake-up word; speech recognition is performed on the second target speaker beam to obtain the recognized text.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and specifically to a speech recognition method and device, a training method and device for a speech recognition network model, a computer device, and a storage medium. Background Art

[0002] Speech recognition is an important part of artificial intelligence. It uses machine learning methods to enable machines to automatically convert speech into corresponding text, thus giving machines a function similar to human hearing. As the use scenarios of speech input become increasingly complex, such as high-noise scenarios such as shopping malls, the requirements for existing speech recognition systems are getting higher and higher. Existing speech recognition systems have poor recognition effects in far-field multi-point noise scenarios. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a speech recognition method and device, a training method and device for a speech recognition network model, a computer device, and a storage medium to solve the problem of poor recognition effect of the speech recognition system in the prior art.

[0004] The first aspect of the present application provides a speech recognition method, including: determining a first target speaker beam based on an initial speaker beam and multi-channel speech signals, the initial speaker beam is obtained based on a beamforming process of the multi-channel speech signals, and the first target speaker beam indicates the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam; determining a noise beam based on the first target speaker beam and the multi-channel speech signals, the noise beam indicates the sound information in the multi-channel speech signals other than the first target speaker beam; filtering the first target speaker beam based on the noise beam and the wake-up beam to obtain a second target speaker beam, the wake-up beam indicating a beam containing a wake-up word; performing speech recognition on the second target speaker beam to obtain a recognized text.

[0005] In one embodiment, determining the first target speaker beam based on the initial speaker beam and the multi-channel speech signal includes: determining multiple fixed beams based on the multi-channel speech signal, the multiple fixed beams correspond to multiple preset directions one by one; determining multiple adaptive beams based on the multiple fixed beams, the multiple adaptive beams correspond to the multiple fixed beams one by one, and the adaptive beams include the speech information of the multiple fixed beams in the preset directions where the adaptive beams are located; filtering out the speech information outside the direction where the initial speaker beam is located in the multiple fixed beams and the multiple adaptive beams to obtain the first target speaker beam. In one embodiment, determining multiple adaptive beams based on multiple fixed beams includes: for each fixed beam, determining the first directional correlation coefficient vector of the fixed beam and the remaining fixed beams; determining the product of the first directional correlation coefficient vector and the fixed beam as the adaptive beam.

[0006] In one embodiment, filtering out speech information outside the direction of the initial speaker beam in the multiple fixed beams and the multiple adaptive beams to obtain the first target speaker beam includes: respectively determining the second directional correlation coefficient vectors of the multiple fixed beams and the multiple adaptive beams with the initial speaker beam; and determining the product of the second directional correlation coefficient vector and the initial speaker beam as the first target speaker beam.

[0007] In one embodiment, filtering multiple speech signals based on a first target speaker beam to obtain a noise beam includes: extracting acoustic features from the first target speaker beam and the multiple speech signals to obtain first feature information and second feature information, respectively; determining third feature information in the second feature information that does not match the first feature information; and determining that the sound signal corresponding to the third feature information is a noise beam.

[0008] In one embodiment, filtering the first target speaker beam based on the noise beam and the wake-up beam to obtain the second target speaker beam includes: performing acoustic feature extraction on the first target speaker beam, the noise beam, and the wake-up beam to obtain fourth feature information, fifth feature information, and sixth feature information, respectively; determining seventh feature information in the fourth feature information that matches the fifth feature information; determining eighth feature information in the feature information other than the seventh feature information in the fourth feature information that matches the sixth feature information; and determining that the sound signal corresponding to the eighth feature information is the second target speaker beam.

[0009] In one embodiment, extracting acoustic features from the first target speaker beam, the noise beam, and the wake-up beam includes: extracting energy spectrum acoustic features from the first target speaker beam, the noise beam, and the wake-up beam.

[0010] In one embodiment, after performing acoustic feature extraction on the first target speaker beam, the noise beam and the wake-up beam to obtain the fourth feature information, the fifth feature information and the sixth feature information respectively, it also includes: identifying the valid audio information in the fourth feature information, the fifth feature information and the sixth feature information respectively; determining the seventh feature information in the fourth feature information that matches the fifth feature information includes: determining the seventh feature information in the valid audio information of the fourth feature information that matches the valid audio information of the fifth feature information; determining the eighth feature information in the feature information other than the seventh feature information in the fourth feature information that matches the sixth feature information includes: determining the eighth feature information in the feature information other than the seventh feature information in the valid audio information of the fourth feature information that matches the valid audio information of the sixth feature information.

[0011] The second aspect of the present application provides a training method for a speech recognition network model, which includes a secondary beamforming module and a post-filtering module; the training method includes: inputting initial speaker beam samples and multi-channel speech signal samples into the secondary beamforming module to obtain first target speaker beam samples and noise beam samples, wherein the initial speaker beam samples are obtained after the multi-channel speech signal samples are processed by the front-end beamforming module, and the multi-channel speech signal samples include speaker speech samples; inputting first target speaker beam samples, noise beam samples and wake-up beam samples into the post-filtering module to obtain second target speaker beam samples, wherein the wake-up beam is obtained after the multi-channel speech signals are processed by the wake-up module; inputting speaker speech samples, noise beam samples and wake-up beam samples into the post-filtering module to obtain second target speaker beam parameters; and training the secondary beamforming module and the post-filtering module based on the second target speaker beam samples and the second target speaker beam parameters.

[0012] In one embodiment, the speaker speech sample is annotated with text data; the speech recognition network model also includes a speech recognition module; the training method also includes: inputting the second target speaker beam sample and the second target speaker beam parameter into the speech recognition module to obtain the recognition text and the recognition text parameter respectively; and performing back-propagation training on the speech recognition module, the post-filtering module and the secondary beamforming module based on the recognition text, the recognition text parameter and the text data.

[0013] The third aspect of the present application provides a speech recognition device, including: a first determination module, used to determine a first target speaker beam based on an initial speaker beam and a multi-channel speech signal, the initial speaker beam is obtained based on a beamforming process of the multi-channel speech signal, and the first target speaker beam indicates the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam; a second determination module, used to determine a noise beam based on the first target speaker beam and the multi-channel speech signal, the noise beam indicates the sound information in the multi-channel speech signal other than the first target speaker beam; a post-filtering module, used to filter the first target speaker beam based on the noise beam and the wake-up beam to obtain a second target speaker beam, the wake-up beam indicates a beam containing a wake-up word; a speech recognition module, used to perform speech recognition on the second target speaker beam to obtain a recognized text.

[0014] In a fourth aspect, the present application provides a training device for a speech recognition network model, comprising: a secondary beamforming module, which outputs a first target speaker beam sample and a noise beam sample based on an initial speaker beam sample and a multi-channel speech signal sample, wherein the initial speaker beam is obtained based on a beamforming process of a multi-channel speech signal, and the multi-channel speech signal sample includes a speaker speech sample; a post-filtering module, which outputs a second target speaker beam sample based on the first target speaker beam sample, the noise beam sample and the wake-up beam sample, wherein the wake-up beam is obtained by passing the multi-channel speech signal through the wake-up module; a second target speaker beam parameter is output based on the speaker speech sample, the noise beam sample and the wake-up beam sample through the post-filtering module; and a training module, which trains the secondary beamforming module and the post-filtering module based on the second target speaker beam sample and the second target speaker beam parameter.

[0015] The fifth aspect of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executed by the processor. When the processor executes the computer program, the steps of the speech recognition method and / or the steps of the training method of the speech recognition network model provided in any of the above embodiments are implemented.

[0016] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the speech recognition method and / or the steps of the speech recognition network model training method provided in any of the above embodiments are implemented.

[0017] According to the speech recognition method and apparatus, speech recognition network training method and apparatus, computer equipment, and storage medium provided in the present application, the initial speaker beam is reshaped based on the multi-channel speech signals collected by the microphone array to obtain the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam, that is, the first target speaker beam. The first target speaker beam and the multi-channel speech signals cover the same space and are comparable. Then, the first target speaker beam is filtered out from the multi-channel speech signals to obtain a noise beam. The first target speaker beam is post-filtered using the noise beam and the wake-up beam to obtain a second target speaker beam as the input of the back-end speech recognition module, thereby enhancing the speech information of the target speaker in the second target speaker beam, thereby achieving the purpose of accurately identifying and providing speaker identity information in high-noise scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The network architecture of a speech recognition system provided in one embodiment of the present application.

[0019] Figure 2 It is a flowchart of a speech recognition method provided in one embodiment of the present application.

[0020] Figure 3 A flowchart of a speech recognition method provided in the second embodiment of the present application.

[0021] Figure 4 A flowchart of a speech recognition method provided in the third embodiment of the present application.

[0022] Figure 5 A flowchart of a method for training a speech recognition network model provided in one embodiment of the present application.

[0023] Figure 6 A flowchart of a method for training a speech recognition network model provided in another embodiment of the present application.

[0024] Figure 7 This is a structural block diagram of a speech recognition device provided in one embodiment of the present application.

[0025] Figure 8 A structural block diagram of a training device for a speech recognition network model provided in one embodiment of the present application.

[0026] Fig. 9 It is a structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] Application Overview

[0029] The existing mainstream far-field speech recognition system includes a speech wake-up module, a front-end beamforming module, and a back-end speech recognition module. The speech wake-up module is used to recognize the preset wake-up word to wake up the terminal device. The front-end beamforming module retains the speech signal in the desired direction and suppresses the speech signal in the undesired direction based on the multi-channel speech signals collected by the entire microphone array, so as to suppress the interference source and increase the sound gain in the target direction. The back-end speech recognition module extracts the Mel-frequency cepstral coefficient features of the formed beam and sends it to the acoustic model for speech recognition to obtain the recognition result, which includes the text content corresponding to the speech signal of the target speaker.

[0030] In the above speech recognition process, the speaker beam output by the front-end beamforming module is inevitably mixed with the speech information of other speakers or other interference information. The reasons include: on the one hand, the voice wake-up module, the front-end beamforming module and the back-end speech recognition module are in a cascade relationship. When it is necessary to optimize the recognition effect, each module is independently optimized based on different optimization goals, resulting in the speech recognition system being unable to achieve optimal performance as a whole. On the other hand, the front-end beamforming algorithm is not strong enough in noise processing, which makes it difficult for its beamforming to effectively transplant interference sources, and a large amount of noise appears in the back-end recognition results.

[0031] In view of this, the present application provides a speech recognition method and device, a training method and device for a speech recognition network, a computer device, and a storage medium, and adds a secondary beamforming module and a post-filtering module between the front-end beamforming module and the back-end speech recognition module. Among them, the secondary beamforming module performs secondary shaping on the initial speaker beam output by the front-end beamforming module based on the multi-channel speech signals collected by the microphone array, and obtains the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam, that is, the first target speaker beam. The first target speaker beam and the multi-channel speech signal cover the same space and are comparable. Then, the first target speaker beam is filtered out from the multi-channel speech signal to obtain a noise beam. The first target speaker beam, the noise beam, and the wake-up beam output by the speech wake-up module are input into the post-filtering module, and the first target speaker beam is post-filtered using the noise beam and the wake-up beam, and the second target speaker beam is output as the input of the back-end speech recognition module, thereby obtaining the text content corresponding to the speech signal of the target speaker. Through the post-filtering process, the noise in the first target speaker beam and the interference of non-wake-up person voice fragments are deeply eliminated in a larger field of view at the back end, thereby improving the accuracy of personalized speech recognition in complex scenarios.

[0032] Figure 1 The network architecture of the speech recognition system provided in one embodiment of the present application. Figure 1 As shown, the speech recognition system 100 includes a speech wake-up module 110 , a front-end beamforming module (not shown in the figure), a secondary beamforming module 120 , a post-filtering module 130 and a back-end speech recognition module 140 .

[0033] Among them, the voice wake-up module 110 and the front-end beamforming module are conventional modules and are implemented by conventional algorithms. The voice wake-up module 110 outputs a wake-up beam based on the multi-channel voice signals collected by the microphone array. The wake-up beam refers to a beam containing a preset wake-up word. The wake-up word exists in the voice signal of the target speaker, so the wake-up beam contains the voiceprint feature information of the target speaker. The front-end beamforming module outputs the initial speaker beam by filtering and / or beamforming the multi-channel voice signals collected by the microphone array. The secondary beamforming module 120 generates a first target speaker beam and a noise beam based on the initial speaker beam and the multi-channel voice signals. The post-filtering module 130 generates a second target speaker beam based on the first target speaker beam, the noise beam and the wake-up beam. The back-end speech recognition module 140 performs speech recognition on the second target speaker beam to obtain a recognition result. The back-end speech recognition module 140 is a conventional module and is implemented by a conventional algorithm.

[0034] The following is a Figure 1 The specific execution process of the speech recognition system shown is described.

[0035] Figure 2 is a flow chart of a speech recognition method provided by an embodiment of the present application. The speech recognition method has Figure 1 The network architecture shown. The speech recognition method can be implemented as a program stored in a storage medium and called and executed by a processor. Figure 1 and Figure 2 As shown, the speech recognition method 200 includes the following steps:

[0036] Step S210, determining a first target speaker beam based on the initial speaker beam and the multi-channel speech signals. The initial speaker beam is obtained based on the beamforming process of the multi-channel speech signals, and the first target speaker beam indicates the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam.

[0037] like Figure 1As shown, the initial speaker beam is the output of the front-end beamforming module. The initial speaker beam contains the speech signals of the multi-channel speech signals in the preset main direction, and the preset main direction is the direction of the pre-specified main microphone. In other words, the spatial coverage of the initial speaker beam is a part of the coverage of the multi-channel speech signals. After step S210, the initial speaker beam is transformed into the first target speaker beam, and the first target speaker beam indicates the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam. In other words, the coverage of the first target speaker beam is consistent with the coverage of the multi-channel speech signals, so that the multi-channel speech signals and the speech signals in the preset main direction are comparable. It can be seen that step S210 is equivalent to the process of secondary shaping of the initial speaker beam, the purpose of which is to make the speech signals in the preset main direction and the multi-channel speech signals in the same space.

[0038] Step S220: determining a noise beam based on the first target speaker beam and the multi-channel speech signals, wherein the noise beam indicates sound information in the multi-channel speech signals other than the first target speaker beam.

[0039] like Figure 2 As shown, step S220 can be implemented by a neural sub-network of a Transformer structure. The neural sub-network of the Transformer structure is a standard network structure stacked with multiple convolutional layers and multi-head attention sub-modules, which is used to remove the first target speaker beam from the multi-channel speech signal to obtain the remaining noise beam.

[0040] In one embodiment, the processing process of the neural subnetwork of the Transformer structure includes: extracting acoustic features of the first target speaker beam and the multi-channel speech signal to obtain first feature information and second feature information respectively. Determining third feature information in the second feature information that does not match the first feature information. Determining that the sound signal corresponding to the third feature information is a noise beam.

[0041] Step S230 , filtering the first target speaker beam based on the noise beam and the wake-up beam to obtain a second target speaker beam, where the wake-up beam indicates a beam containing the wake-up word.

[0042] The wake-up beam is derived from the output of the wake-up module 110. The wake-up beam is used to filter the first target speaker beam, thereby realizing accurate recognition of the target speaker's voice while rejecting the voice of non-target speakers.

[0043] Due to the filtering process, e.g. Figure 1The filtering process of the neural subnetwork of the Transformer structure in is not a process that can be completely visualized as AB=C. Therefore, there is overlapping sound information between the first target speaker beam and the noise beam. In step S230, the first target speaker is filtered based on the noise beam and the wake-up beam, which can further enhance the sound signal of the target speaker in the preset main direction.

[0044] In one embodiment, Figure 1 As shown, step S230 is implemented by a hybridCNN neural sub-network. Since the neural sub-network of the Transformer structure is shallow and has a small field of view, that is, the features extracted by it are relatively short in the time domain. The hybridCNN neural sub-network is deeper and has a larger field of view, that is, it can extract more speech information in the time domain, so that the first target speaker beam can be further filtered.

[0045] Step S240: Perform speech recognition on the second target speaker beam to obtain a recognized text.

[0046] This step can be implemented using a conventional algorithm and will not be described in detail here.

[0047] According to the speech recognition method provided in this embodiment, the initial speaker beam output by the front-end beamforming module is secondary-formed based on the multi-channel speech signals collected by the microphone array to obtain the sound information of each of the multi-channel speech signals in the direction of the initial speaker beam, that is, the first target speaker beam. The first target speaker beam and the multi-channel speech signals cover the same space and are comparable. Then, the first target speaker beam is filtered out from the multi-channel speech signals to obtain the noise beam. The first target speaker beam is post-filtered using the noise beam and the wake-up beam output by the speech wake-up module to obtain the second target speaker beam as the input of the back-end speech recognition module, thereby enhancing the speech information of the target speaker in the second target speaker beam, thereby achieving the purpose of accurately identifying and providing speaker identity information in high-noise scenarios.

[0048] Figure 3 A flow chart of a speech recognition method provided in the second embodiment of the present application. Figure 1 and Figure 3 As shown, in this embodiment, step S210 is specifically performed as follows:

[0049] Step S310: determining a plurality of fixed beams based on the multi-channel voice signals, wherein the plurality of fixed beams correspond to a plurality of preset directions one by one.

[0050] The microphone array is composed of n microphones, where n is any positive integer greater than or equal to 2. The microphone array can be an array of any topological structure, such as a linear array, a ring array, a plane array, a spherical array, etc. Taking a plane array as an example, the number of microphones in the plane array is 8, which is composed of a ring composed of 7 microphones plus a central microphone. In this case, the 360° space can be divided into p sector areas, and the center line of each sector area corresponds to a preset direction.

[0051] In one embodiment, for each preset direction, a beamforming algorithm is used to obtain a fixed beam in the preset direction. The beamforming algorithm mentioned here may be a fixed beamforming algorithm.

[0052] In this embodiment, a complex convolution operation is performed on the frequency spectra in multiple preset directions to obtain multiple fixed beams.

[0053] Step S320: determining a plurality of adaptive beams based on the plurality of fixed beams, wherein the plurality of adaptive beams correspond to the plurality of fixed beams one-to-one, and the adaptive beams include voice information of the plurality of fixed beams in preset directions where the adaptive beams are located.

[0054] See also Figure 1 ,Firstly, for each fixed beam, a first directional correlation coefficient vector of the fixed beam and the remaining fixed beams is determined.

[0055] Specifically, taking each fixed beam as a reference, the inner product of the remaining fixed beams and the reference fixed beam is calculated to obtain the directional correlation coefficient vector of the reference vector. The result of the inner product of a fixed beam and the reference fixed beam is a value, which represents the projection of the fixed beam on the reference fixed beam, and also represents how much sound information on the reference fixed beam is contained in the fixed beam. The first directional correlation coefficient vector corresponding to fixed beam 1 can be expressed as [a 11 a 12 .....a 1p ], then multiple fixed beams correspond to a first-direction correlation coefficient matrix with p rows and p columns.

[0056] Secondly, the product of the first directional correlation coefficient vector and the fixed beam is determined as the adaptive beam.

[0057] The relationship between the adaptive beam and the fixed beam can be expressed as: adaptive beam AD1 = a 11 *FD1+a 12 *FD2+……+a 1p *FDp, where FDi, i = 1, 2, ....., p represents multiple fixed beams. It can be seen that the adaptive beam is the weighted sum of the energy information of all fixed beams in a certain direction.

[0058] Step S330 , filtering out speech information outside the direction of the initial speaker beam in the multiple fixed beams and the multiple adaptive beams to obtain a first target speaker beam.

[0059] First, second directional correlation coefficient vectors of each of a plurality of fixed beams and a plurality of adaptive beams and an initial speaker beam are determined respectively.

[0060] Specifically, the initial speaker beam is used as a reference beam, and the dot product between each fixed beam and the initial speaker beam and the dot product between each adaptive beam and the initial speaker beam are calculated to obtain a second directional correlation coefficient vector.

[0061] Secondly, the product of the second directional correlation coefficient and the initial speaker beam is determined as the first target speaker beam.

[0062] The first target speaker beam represents 2P beams, including p fixed beams and p adaptive beams, which are the weighted sum of the projections in the preset main direction. It is a filtering process for the speech information in the 2P beams that is not in the preset main direction. Finally, the first target speaker beam belonging to the same space as the multi-channel speech signal is obtained, making the two comparable.

[0063] Figure 4 This is a flow chart of the speech recognition method provided in the third embodiment of the present application. Figure 4 As shown, the speech recognition method 400 and Figure 2 The difference of the speech recognition method 200 is that, in this embodiment, the Figure 1 As shown, step S230 is specifically performed as follows:

[0064] Step S410, performing acoustic feature extraction on the first target speaker beam, the noise beam, and the wake-up beam to obtain fourth feature information, fifth feature information, and sixth feature information, respectively.

[0065] The extracted acoustic features may be energy spectrum features. The process of extracting energy spectrum features specifically includes: taking the first target speaker beam as an example. First, mirror-fill frames before and after the first target speaker beam. In one example, 256 sampling points are added before and after the first target speaker beam. Secondly, frame division is performed using a sliding window method. In one example, a Hamming window is used for sliding window operation. In one example, the window length is 512 sampling points and the step length is 160 sampling points. Thirdly, energy spectrum features are extracted for each speech frame. For example, Fourier transform is performed on the speech frames in turn, the sum of the squares of the real part and the imaginary part is calculated, the samples are downsampled to 128 sampling points, the logarithm is taken, and the energy spectrum features are obtained.

[0066] Step S420: Determine seventh feature information in the fourth feature information that matches the fifth feature information.

[0067] The seventh feature information in the fourth feature information that matches the fifth feature information represents the noise information in the first target speaker beam, which is equivalent to a filtering process of the first target speaker beam using the noise beam.

[0068] Step S430: determining eighth feature information in the fourth feature information other than the seventh feature information that matches the sixth feature information.

[0069] After filtering the first target speaker beam using the noise beam, the wake-up beam is used for secondary filtering to obtain the eighth feature information.

[0070] Step S440: determine that the sound signal corresponding to the eighth characteristic information is the second target speaker beam.

[0071] In one embodiment, before step S420, the process further includes: detecting a valid sound segment. Figure 1 After obtaining the fourth characteristic information, the fifth characteristic information and the sixth characteristic information according to step S410, respectively identify the valid audio information in the fourth characteristic information, the fifth characteristic information and the sixth characteristic information. That is, the frequency bands without valid voice in the fourth characteristic information, the fifth characteristic information and the sixth characteristic information, such as the frequency bands corresponding to the speaking intervals, are cut off. In this case, step S420 is specifically performed as follows: determining the seventh characteristic information in the valid audio information of the fourth characteristic information that matches the valid audio information of the fifth characteristic information. Step S430 is specifically performed as follows: determining the eighth characteristic information in the feature information other than the seventh characteristic information in the valid audio information of the fourth characteristic information that matches the valid audio information of the sixth characteristic information. Subsequently, the sound signal corresponding to the eighth characteristic information is determined as the second target speaker beam. The valid segment detection process can filter out the invalid voice information in the fourth characteristic information, the fifth characteristic information and the sixth characteristic information, and simplify the amount of data for subsequent operations.

[0072] Figure 5 A flow chart of a method for training a speech recognition network model provided in an embodiment of the present application. The speech recognition network model has Figure 1The network architecture shown includes a secondary beamforming module 120 and a post-filtering module 130. The training samples used in the training process include: multi-channel speech signal samples collected by the microphone array in real scenes and speaker speech samples annotated with text data. The speaker speech sample is clean data, which can be obtained by collecting data in a quiet scene or directly through text synthesis. The multi-channel speech signal samples include speaker speech samples. That is, the target speaker records a speech based on a preset text in a quiet scene as a speaker speech sample. The same target speaker records a speech based on the same preset text in a real scene as a multi-channel speech signal sample.

[0073] Combination Figure 1 and Figure 5 As shown, the training method 500 includes:

[0074] Step S510: the initial speaker beam samples and the multi-channel speech signal samples are input into the secondary beamforming module 120 to obtain the first target speaker beam samples and the noise beam samples. The initial speaker beam samples are obtained after the multi-channel speech signal samples are processed by the front-end beamforming module, and the multi-channel speech signal samples include the speaker speech samples.

[0075] For details on the operation of the secondary beamforming module 120 on the initial speaker beam samples and the multi-path speech signal samples, see Figure 2 and Figure 3 The execution process of step S210 and step S220 in the voice recognition method is not described here.

[0076] Step S520, the first target speaker beam sample, the noise beam sample and the wake-up beam sample are input into the post-filter module 130 to obtain the second target speaker beam sample. The wake-up beam is obtained after the multi-channel speech signal is processed by the wake-up module.

[0077] For details on the operations of the post-filter module 130 on the first target speaker beam sample, the noise beam sample and the wake-up beam sample, see Figure 2 and Figure 4 The execution process of step S230 in the voice recognition method is shown in FIG.

[0078] Step S530: The speaker speech sample, the noise beam sample and the wake-up beam sample are input into the post-filter module 130 to obtain the second target speaker beam parameter.

[0079] For details on the operations of the post-filter module 130 on the speaker speech sample, the noise beam sample and the wake-up beam sample, see Figure 2 and Figure 4 The execution process of step S230 in the voice recognition method is shown in FIG.

[0080] Step S540: training the secondary beamforming module and the post-filtering module based on the second target speaker beam samples and the second target speaker beam parameters.

[0081] Specifically, a loss function of the mean square error between the second target speaker beam samples and the second target speaker beam parameters is calculated so that the calculated second target speaker beam samples are as close as possible to the second target speaker beam parameters.

[0082] Figure 6 A flow chart of a method for training a speech recognition network model provided in another embodiment of the present application. Figure 6 As shown, the training method 600 is Figure 5 Based on the training method 500 shown, it further includes:

[0083] Step S610, see Figure 1 The second target speaker beam sample and the second target speaker beam parameter are input into the speech recognition module 140 to obtain the recognized text and the recognized text parameter respectively.

[0084] The specific operations of the speech recognition module 140 on the second target speaker beam samples and the second target speaker beam parameters may be conventional operations, which will not be described in detail here.

[0085] Step S620 , performing back propagation training on the speech recognition module 140 , the post-filtering module 130 and the secondary beamforming module 120 based on the recognition text, the recognition text parameters and the text data.

[0086] Specifically, for the convenience of description, the first target speaker beam sample, noise beam sample and wake-up beam sample are respectively denoted as beam1, beam2 and beam3. The speaker speech sample is denoted as clean, and its annotated text data is denoted as label. Figure 1 , the training process includes:

[0087] Step 1: feature extraction is performed on beam1, beam2, and beam3 respectively, and then the extracted features beam1, beam2, and beam3 are formed into a graph and sent to a sub-network consisting of multiple convolutional layers for effective audio detection and filtering to obtain the output a1 of the post-filter module 130.

[0088] Step 2: extract features of clean, beam2, and beam3 respectively, and then combine the extracted features of clean, beam2, and beam3 into a graph, and send it to a sub-network composed of multiple convolutional layers for effective audio detection and filtering to obtain the output b1 of the post-filter module 130.

[0089] Step 3: Calculate the first loss of the mean square error between a1 and b1.

[0090] Step 4: Data a1 passes through the backend speech recognition module 140, which is a submodule of a standard transformer structure, to obtain output a2.

[0091] Step 5: Data b1 passes through the backend speech recognition module 140, i.e., a standard transformer structure, to obtain output b2.

[0092] Step 6: Calculate the KLD error second loss of a2 and b2.

[0093] Step 7: Calculate the third loss and fourth loss of the error between a2 and b2 and label respectively.

[0094] Step 8: Calculate the gradient through the first loss, second loss, third loss and fourth loss, and use back propagation to guide the training so that beam2, beam3 and beam1 can be as close to clean as possible after interaction.

[0095] According to the training method of the speech recognition network model provided in this embodiment, by jointly training the speech recognition module 140, the post-filtering module 130 and the secondary beamforming module 120, and taking the recognition rate as the goal, the overall recognition effect of the speech recognition network model is improved.

[0096] Exemplary Devices

[0097] Figure 7 This is a structural block diagram of a speech recognition device provided in one embodiment of the present application. Figure 7 As shown, the speech recognition device 700 includes: a first determination module 710, a second determination module 720, a post-filtering module 730 and a speech recognition module 740. Figure 1 , the first determination module 710 and the second determination module 720 constitute the secondary beamforming module 120. The first determination module 710 is used to determine the first target speaker beam based on the initial speaker beam and the multi-channel voice signal. The initial speaker beam is obtained based on the beamforming process of the multi-channel voice signal. The first target speaker beam indicates the sound information of each of the multi-channel voice signals in the direction of the initial speaker beam. The second determination module 720 is used to determine the noise beam based on the first target speaker beam and the multi-channel voice signal. The noise beam indicates the sound information in the multi-channel voice signal other than the first target speaker beam. The post-filtering module 730 is used to filter the first target speaker beam based on the noise beam and the wake-up beam to obtain the second target speaker beam. The wake-up beam indicates the beam containing the wake-up word. The speech recognition module 740 is used to perform speech recognition on the second target speaker beam to obtain the recognized text.

[0098] In one embodiment, the first determination module 710 is specifically used to determine multiple fixed beams based on multiple voice signals, and the multiple fixed beams correspond one-to-one to multiple preset directions; determine multiple adaptive beams based on the multiple fixed beams, and the multiple adaptive beams correspond one-to-one to the multiple fixed beams, and the adaptive beams include the voice information of the multiple fixed beams in the preset directions of the adaptive beams; filter out the voice information outside the direction of the initial speaker beam in the multiple fixed beams and the multiple adaptive beams to obtain the first target speaker beam.

[0099] In one embodiment, determining multiple adaptive beams based on multiple fixed beams includes: determining, for each fixed beam, a first directional correlation coefficient vector of the fixed beam and other fixed beams; and determining the product of the first directional correlation coefficient vector and the fixed beam as an adaptive beam.

[0100] In one embodiment, filtering out speech information outside the direction of the initial speaker beam in the multiple fixed beams and the multiple adaptive beams to obtain the first target speaker beam includes: respectively determining the second directional correlation coefficient vectors of the multiple fixed beams and the multiple adaptive beams with the initial speaker beam; and determining the product of the second directional correlation coefficient vector and the initial speaker beam as the first target speaker beam.

[0101] In one embodiment, the second determination module 720 is specifically used to extract acoustic features from the first target speaker beam and the multi-channel speech signal to obtain first feature information and second feature information respectively; determine third feature information in the second feature information that does not match the first feature information; and determine that the sound signal corresponding to the third feature information is a noise beam.

[0102] In one embodiment, the post-filtering module 730 is specifically used to: perform acoustic feature extraction on the first target speaker beam, the noise beam and the wake-up beam to obtain fourth feature information, fifth feature information and sixth feature information respectively; determine seventh feature information in the fourth feature information that matches the fifth feature information; determine eighth feature information in the feature information other than the seventh feature information in the fourth feature information that matches the sixth feature information; and determine that the sound signal corresponding to the eighth feature information is the second target speaker beam.

[0103] In one embodiment, extracting acoustic features from the first target speaker beam, the noise beam, and the wake-up beam includes: extracting energy spectrum acoustic features from the first target speaker beam, the noise beam, and the wake-up beam.

[0104] In one embodiment, after performing acoustic feature extraction on the first target speaker beam, the noise beam and the wake-up beam to obtain the fourth feature information, the fifth feature information and the sixth feature information respectively, it also includes: identifying the valid audio information in the fourth feature information, the fifth feature information and the sixth feature information respectively; determining the seventh feature information in the fourth feature information that matches the fifth feature information includes: determining the seventh feature information in the valid audio information of the fourth feature information that matches the valid audio information of the fifth feature information; determining the eighth feature information in the feature information other than the seventh feature information in the fourth feature information that matches the sixth feature information includes: determining the eighth feature information in the feature information other than the seventh feature information in the valid audio information of the fourth feature information that matches the valid audio information of the sixth feature information.

[0105] The speech recognition device provided in this embodiment belongs to the same application concept as the speech recognition method provided in the embodiment of this application, can execute the speech recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the speech recognition method. For technical details not fully described in this embodiment, please refer to the speech recognition method provided in the embodiment of this application, and will not be repeated here.

[0106] Figure 8 A structural block diagram of a training device for a speech recognition network model provided in an embodiment of the present application. Figure 1 and Figure 8 As shown, the training device 800 includes: a secondary beamforming module 810, a post-filtering module 820 and a training module 830. Among them, the secondary beamforming module 810 is used to output a first target speaker beam sample and a noise beam sample based on the initial speaker beam sample and the multi-channel speech signal sample. The initial speaker beam is obtained based on the beamforming process of the multi-channel speech signal, and the multi-channel speech signal sample includes the speaker speech sample. The post-filtering module 820 is used to output a second target speaker beam sample based on the first target speaker beam sample, the noise beam sample and the wake-up beam sample. The wake-up beam is obtained by the multi-channel speech signal through the wake-up module; based on the speaker speech sample, the noise beam sample and the wake-up beam sample, the second target speaker beam parameter is output. The training module 830 is used to train the secondary beamforming module and the post-filtering module based on the second target speaker beam sample and the second target speaker beam parameter.

[0107] In one embodiment, the speech recognition network model further includes a speech recognition module 840, which is used to output recognition text and recognition text parameters based on the second target speaker beam sample and the second target speaker beam parameter. In this case, the training module 830 is specifically used to perform back propagation training on the speech recognition module, the post-filtering module and the secondary beamforming module based on the recognition text, the recognition text parameters and the text data.

[0108] The training device of the speech recognition network model provided in this embodiment belongs to the same application concept as the training method of the speech recognition network model provided in the embodiment of this application, and can execute the training method of the speech recognition network model provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the training method of the speech recognition network model. For the technical details not described in detail in this embodiment, please refer to the training method of the speech recognition network model provided in the embodiment of this application, and will not be repeated here.

[0109] Exemplary Electronic Devices

[0110] Fig. 9 is a structural block diagram of an electronic device provided by an embodiment of the present application. Fig. 9 As shown, electronic device 900 includes one or more processors 910 and memory 920 .

[0111] The processor 910 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 900 to perform desired functions.

[0112] The memory 920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 910 may run the program instructions to implement the speech recognition method of each embodiment of the present application described above and the training method of the speech recognition network model and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0113] In one example, the electronic device 900 may further include: an input device 930 and an output device 940 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0114] The output device 940 can output various information to the outside, including the determined distance information, direction information, etc. The output device 940 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0115] Of course, to simplify, Fig. 9Only some of the components related to the present application in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.

[0116] Exemplary computer program products and computer-readable storage media

[0117] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech recognition method and the training method of the speech recognition network model according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0118] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0119] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by the processor, the processor 11 executes the steps of the speech recognition method and the training method of the speech recognition network model according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0120] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0121] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0122] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0123] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0124] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0125] It should be understood that the qualifiers "first", "second", "third", "fourth", "fifth" and "sixth" used in the description of the embodiments of the present application are only used to more clearly explain the technical solutions and cannot be used to limit the scope of protection of the present application.

[0126] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A speech recognition method, characterized in that: include: Determine a first target speaker beam based on an initial speaker beam and multi-channel speech signals, wherein the initial speaker beam is obtained based on a beamforming process of the multi-channel speech signals, and the first target speaker beam indicates sound information of each of the multi-channel speech signals in a direction where the initial speaker beam is located; Determine a noise beam based on the first target speaker beam and the multi-channel speech signal, wherein the noise beam indicates sound information in the multi-channel speech signal other than the first target speaker beam; filtering the first target speaker beam based on the noise beam and the wake-up beam to obtain a second target speaker beam, wherein the wake-up beam indicates a beam containing a wake-up word; performing speech recognition on the second target speaker beam to obtain a recognized text; The determining of the noise beam based on the first target speaker beam and the multi-path speech signal comprises: Performing acoustic feature extraction on the first target speaker beam and the multi-channel speech signal to obtain first feature information and second feature information respectively; determining third feature information in the second feature information that does not match the first feature information; It is determined that the sound signal corresponding to the third characteristic information is the noise beam.

2. The speech recognition method according to claim 1, characterized in that: The determining of the first target speaker beam based on the initial speaker beam and the multi-path speech signal comprises: Determine a plurality of fixed beams based on the multi-path voice signals, wherein the plurality of fixed beams correspond to a plurality of preset directions one by one; Determine a plurality of adaptive beams based on the plurality of fixed beams, the plurality of adaptive beams corresponding to the plurality of fixed beams one-to-one, the adaptive beams including voice information of the plurality of fixed beams in preset directions where the adaptive beams are located; Speech information outside the direction of the initial speaker beam in the multiple fixed beams and the multiple adaptive beams is filtered out to obtain the first target speaker beam.

3. The speech recognition method according to claim 2, characterized in that: The determining a plurality of adaptive beams based on the plurality of fixed beams comprises: For each of the fixed beams, determining a first directional correlation coefficient vector between the fixed beam and the remaining fixed beams; The product of the first directional correlation coefficient vector and the fixed beam is determined as the adaptive beam.

4. The speech recognition method according to claim 2, characterized in that: The filtering out speech information outside the direction of the initial speaker beam in the multiple fixed beams and the multiple adaptive beams to obtain the first target speaker beam includes: respectively determining second directional correlation coefficient vectors of the multiple fixed beams and the multiple adaptive beams with the initial speaker beam; The product of the second directional correlation coefficient vector and the initial speaker beam is determined as the first target speaker beam.

5. The speech recognition method according to claim 1, characterized in that: The filtering the first target speaker beam based on the noise beam and the wake-up beam to obtain the second target speaker beam includes: Performing acoustic feature extraction on the first target speaker beam, the noise beam, and the wake-up beam to obtain fourth feature information, fifth feature information, and sixth feature information, respectively; Determining seventh feature information in the fourth feature information that matches the fifth feature information; determining eighth feature information matching the sixth feature information among the feature information other than the seventh feature information in the fourth feature information; Determine that the sound signal corresponding to the eighth characteristic information is the second target speaker beam.

6. The speech recognition method according to claim 5, characterized in that: The extracting acoustic features of the first target speaker beam, the noise beam and the wake-up beam includes: Energy spectrum acoustic feature extraction is performed on the first target speaker beam, the noise beam, and the wake-up beam.

7. The speech recognition method according to claim 5, characterized in that: After extracting acoustic features from the first target speaker beam, the noise beam, and the wake-up beam to obtain fourth feature information, fifth feature information, and sixth feature information, respectively, the method further includes: Respectively identifying valid audio information in the fourth feature information, the fifth feature information, and the sixth feature information; The determining the seventh characteristic information in the fourth characteristic information that matches the fifth characteristic information includes: Determine seventh feature information in the valid audio information of the fourth feature information that matches the valid audio information of the fifth feature information; The determining the eighth feature information matching the sixth feature information in the feature information other than the seventh feature information in the fourth feature information includes: Determine eighth feature information that matches the valid audio information of the sixth feature information among the feature information other than the seventh feature information in the valid audio information of the fourth feature information.

8. A method for training a speech recognition network model, characterized in that: The speech recognition network model includes a secondary beamforming module and a post-filtering module; the training method includes: The initial speaker beam sample and the multi-channel speech signal samples are input into the secondary beamforming module to obtain the first target speaker beam sample and the noise beam sample, wherein the initial speaker beam sample is obtained after the multi-channel speech signal samples are processed by the front-end beamforming module, and the multi-channel speech signal samples include speaker speech samples; The first target speaker beam sample, the noise beam sample and the wake-up beam sample are input into the post-filtering module to obtain a second target speaker beam sample, wherein the wake-up beam is obtained after the multi-channel speech signal is processed by the wake-up module; The speaker speech sample, the noise beam sample and the wake-up beam sample are input into the post-filter module to obtain the second target speaker beam parameter; Training the secondary beamforming module and the post-filtering module based on the second target speaker beam sample and the second target speaker beam parameter; The initial speaker beam samples and the multi-path speech signal samples are input into the secondary beamforming module to obtain the first target speaker beam samples and the noise beam samples, including: The initial speaker beam samples and multi-channel speech signal samples are input into the secondary beamforming module to obtain a first target speaker beam; Acoustic feature extraction is performed on the first target speaker beam and the multi-channel speech signal to obtain first feature information and second feature information respectively; third feature information in the second feature information that does not match the first feature information is determined; and the sound signal corresponding to the third feature information is obtained as the noise beam.

9. The training method according to claim 8, characterized in that: The speaker's speech sample is annotated with text data; the speech recognition network model also includes a speech recognition module; the training method also includes: The second target speaker beam sample and the second target speaker beam parameter are input into the speech recognition module to obtain a recognized text and a recognized text parameter respectively; The speech recognition module, the post-filtering module and the secondary beamforming module are back-propagation trained based on the recognition text, the recognition text parameters and the text data.

10. A speech recognition device, characterized in that: include: A first determination module is used to determine a first target speaker beam based on an initial speaker beam and multi-channel speech signals, wherein the initial speaker beam is obtained based on a beamforming process of the multi-channel speech signals, and the first target speaker beam indicates sound information of each of the multi-channel speech signals in a direction where the initial speaker beam is located; A second determination module, configured to determine a noise beam based on the first target speaker beam and the multi-channel speech signal, wherein the noise beam indicates sound information in the multi-channel speech signal other than the first target speaker beam; a post-filtering module, configured to filter the first target speaker beam based on the noise beam and the wake-up beam to obtain a second target speaker beam, wherein the wake-up beam indicates a beam containing a wake-up word; A speech recognition module, used for performing speech recognition on the second target speaker beam to obtain a recognized text; The second determination module is also used to perform acoustic feature extraction on the first target speaker beam and the multi-channel speech signal to obtain first feature information and second feature information respectively; determine third feature information in the second feature information that does not match the first feature information; and determine that the sound signal corresponding to the third feature information is the noise beam.

11. A training device for a speech recognition network model, characterized in that: include: A secondary beamforming module outputs a first target speaker beam sample and a noise beam sample based on an initial speaker beam sample and multi-channel speech signal samples, wherein the initial speaker beam is obtained based on a beamforming process of the multi-channel speech signal, and the multi-channel speech signal samples include speaker speech samples; A post-filtering module, which outputs a second target speaker beam sample based on the first target speaker beam sample, the noise beam sample and the wake-up beam sample, wherein the wake-up beam is obtained by the multi-channel speech signal through the wake-up module; Outputting a second target speaker beam parameter based on the speaker speech sample, the noise beam sample and the wake-up beam sample; A training module, which trains the secondary beamforming module and the post-filtering module based on the second target speaker beam sample and the second target speaker beam parameter; The secondary beamforming module is further used to output the first target speaker beam sample based on the initial speaker beam sample and the multi-channel speech signal sample; perform acoustic feature extraction on the first target speaker beam and the multi-channel speech signal to obtain first feature information and second feature information respectively; Determine third feature information in the second feature information that does not match the first feature information; and output the sound signal corresponding to the third feature information as the noise beam.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executed by the processor, characterized in that: When the processor executes the computer program, the steps of the speech recognition method according to any one of claims 1 to 7 and / or the steps of the training method of the speech recognition network model according to one of claims 8 and 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech recognition method according to any one of claims 1 to 7 and / or the steps of the training method of the speech recognition network model according to one of claims 8 and 9 are implemented.

Images