Speech recognition method and system, electronic equipment and storage medium

Abstract

The invention provides a speech recognition method and system, electronic equipment and a storage medium. The voice recognition method comprises the following steps: acquiring voice information; performing multiple times of circular processing on the voice information according to the dual-path recurrent neural network to obtain a plurality of first slice vectors; the multiple first slice vectors are connected in series to determine an embedded vector, and the embedded vector corresponds to the target user; acquiring acoustic characteristics of the reference voice; and inputting the acoustic features and the embedded vector into a personal voice activity detection model, and determining an identification result. According to the method, after the voice information is received, the voice information is processed by adopting the double-path recurrent neural network, and the feature information of the target user is extracted from the continuously updated recurrent neural network state, so that good performance can be obtained even if the voice information is very short; and the problem that the enthusiasm of the user is reduced due to overlong registration time is avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a speech recognition method and system, an electronic device and a storage medium. Background Art

[0002] In the related art, when selecting reference speech, a longer duration is often preferred. Although longer reference speech can obtain better speaker embedding (embedding vector) through pre-training SV (Speaker Verification) model, thereby accurately reflecting the attributes of the target speaker, in actual applications, too long registration time will reduce user enthusiasm and lead to poor user experience. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0004] To this end, the first aspect of the present invention proposes a speech recognition method.

[0005] A second aspect of the present invention provides a speech recognition system.

[0006] A third aspect of the present invention provides an electronic device.

[0007] A fourth aspect of the present invention provides a storage medium.

[0008] A fifth aspect of the present invention provides a computer program product.

[0009] In view of this, according to the first aspect of the present invention, a speech recognition method is proposed, including: obtaining speech information, wherein the speech information corresponds to a target user; performing multiple loop processing on the speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors; concatenating the multiple first slice vectors to determine an embedding vector, wherein the embedding vector corresponds to the target user; obtaining acoustic features of a reference speech; and inputting the acoustic features and the embedding vector into a personal voice activity detection model to determine a recognition result.

[0010] The speech recognition method provided by the present invention mainly includes: first obtaining speech information, wherein the speech information is the speech information input by the target user, then inputting the speech information into a dual-path recurrent neural network (DPRNN), and performing multiple loop processing to obtain multiple first slice vectors, that is, the speech information is iteratively updated multiple times by the dual-path recurrent neural network, and then slicing the vector output by the dual-path recurrent neural network to obtain the first slice vector. After obtaining multiple first slice vectors, the multiple first slice vectors are connected in series to form the embedding (embedding vector) of the final target user, that is, the embedding vector. Then, the acoustic features of the reference speech are obtained, wherein the reference speech can be set in advance by the target speaker, and the acoustic features of the reference speech are determined by processing the reference speech. Then, the acoustic features and the embedding vector are input into a personal voice activity detection (PVAD) model to obtain a matching result, that is, to determine whether the currently input speech information is the speech information input by the target speaker. After receiving voice information, the present invention uses a network based on DPRNN (Dual-Path Recurrent Neural Network) to process the voice information, and extracts the characteristic information of the target speaker, i.e., the embedding vector, from the continuously updated RNN (Recurrent Neural Network) state, i.e., the first slice vector, thereby ensuring that good performance can be obtained even if the voice information is very short, and avoiding the problem of reducing user enthusiasm due to excessively long registration time.

[0011] The above-mentioned speech recognition method according to the present invention may also have the following technical features:

[0012] In some technical schemes, optionally, the step of performing multiple loop processing on speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors includes: inputting the speech information into the dual-path recurrent neural network to obtain a first feature vector; determining a first slice vector in the first feature vector; re-inputting the first feature vector into the dual-path recurrent neural network to obtain the first feature vector again, and determining the first slice vector, until the number of loops reaches a preset number and stops.

[0013] In the technical scheme, the steps of performing multiple cyclic processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors include: first, inputting the speech information into the dual-path recurrent neural network to obtain the first feature vector, wherein the first feature vector refers to the feature information obtained by inputting the original speech information into the dual-path recurrent neural network, and then determining the first slice vector in the first feature vector, that is, obtaining the layer output slice of the first feature vector in the first feature vector. Then the first feature vector is input into the dual-path recurrent neural network again to obtain a new first feature vector, and a new first slice vector is obtained from the first feature vector obtained again, and this is repeated until the number of cycles reaches a preset number of times and stops. That is, the speech information is input into the dual-path recurrent neural network to obtain an output, and then the output is re-input into the dual-path recurrent neural network again to obtain another output, and after repeating this for a preset number of times, the layer output slice of each output, that is, the first slice vector, is obtained. The input is iteratively updated multiple times by PRNN, and the layer output slice obtained by each new iteration is extracted. Finally, the embedding vector, that is, the embedding (embedding vector) of the target speaker is obtained by connecting multiple layer output slices in series, thereby ensuring the accuracy of the feature information of the target speaker obtained in the end.

[0014] In some technical schemes, optionally, the dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the step of inputting speech information into the dual-path recurrent neural network to obtain a first feature vector includes: segmenting the speech information to obtain multiple sub-speech information; processing the multiple sub-speech information according to the first gated recurrent unit to obtain a second feature vector in each sub-speech information; determining a third feature vector according to the multiple second feature vectors and the speech information; segmenting the third feature vector to obtain multiple fourth feature vectors; processing the multiple fourth feature vectors according to the second gated recurrent unit to obtain multiple fifth feature vectors; and obtaining the first feature vector based on the multiple fifth feature vectors and the third feature vector.

[0015] In the technical scheme, the dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the step of inputting the speech information into the dual-path recurrent neural network to obtain the first feature vector includes: firstly, the speech information is cut into a plurality of sub-speech information. Then, the plurality of sub-speech information is processed by the first gated recurrent unit to obtain the second feature vector in each sub-speech information. Then, the plurality of second feature vectors and the speech information are superimposed to obtain the third feature vector. Then, the third feature vector is segmented to obtain a plurality of fourth feature vectors, and then, the plurality of fourth feature vectors are processed by the second gated recurrent unit to obtain feature information between adjacent fourth feature vectors, thereby obtaining a plurality of fifth feature vectors. Finally, the plurality of fifth feature vectors are superimposed with the third feature vector to obtain the first feature vector, and then the plurality of processed fifth feature vectors are superimposed with the third feature vector to obtain the first feature vector. In the present invention, the speech information is processed in sequence by two gated recurrent units to obtain the first feature vector, thereby ensuring the accuracy of the feature information of the target speaker obtained in the end.

[0016] In some technical solutions, optionally, the step of determining a first slice vector in a first feature vector includes: cutting the first feature vector by frame to obtain multiple second slice vectors; and using the second slice vector containing the recurrent neural network state as the first slice vector.

[0017] In the technical solution, the step of determining the first slice vector in the first feature vector includes: firstly, cutting the first feature vector to obtain multiple second slice vectors, wherein the cutting method is cutting by frame. Then, the second slice vector containing the recurrent neural network state is used as the first slice vector, wherein the second slice vector of the last frame may be used as the first slice vector. By cutting the first feature vector by frame, and then using the second slice vector containing the recurrent neural network state as the first slice vector, it is ensured that the first slice vector can contain the RNN state.

[0018] In some technical schemes, optionally, the personal voice activity detection model includes: a deformer block module, a cross-attention module, a feature linear modulation module and a fully connected layer; the acoustic features and the embedding vectors are input into the personal voice activity detection model, and the step of determining the recognition result includes: inputting the acoustic features into the deformer block module to obtain the acoustic feature vector; inputting the acoustic feature vector and the embedding vector into the cross-attention module to obtain the first user feature vector; inputting the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain the second user feature vector; inputting the second user feature vector into the fully connected layer to obtain the recognition result.

[0019] In the technical solution, the personal voice activity detection model includes: a deformer block module, a cross attention module, a feature linear modulation module and a fully connected layer; according to the acoustic features, the embedding vector and the personal voice activity detection model, the step of determining the recognition result includes: firstly inputting the acoustic features into the deformer block module, processing the acoustic features through the deformer block module, thereby obtaining an acoustic feature vector. After obtaining the acoustic feature vector, the acoustic feature vector and the embedding vector are input into the cross attention (CA) module to obtain the first user feature vector, and the acoustic feature vector and the embedding vector are associated through the cross attention module to determine the interaction relationship between the acoustic feature vector and the embedding vector. After obtaining the first user feature vector, the first user feature vector and the acoustic feature vector are input into the feature linear modulation (FiLM) module to obtain the second user feature vector, that is, the first user feature vector and the acoustic feature vector are integrated through the feature linear modulation module to obtain the second user feature vector, and after obtaining the second user feature vector, the second user feature vector is input into the fully connected layer, thereby obtaining the final matching result. The present invention processes acoustic features and embedded vectors using a PVAD model and obtains a final recognition result, thereby realizing recognition of speech information.

[0020] In some technical schemes, optionally, the step of inputting the acoustic feature vector and the embedding vector into the cross-attention module to obtain the first user feature vector includes: inputting the acoustic feature vector as query information into the cross-attention module; inputting the embedding vector as key value information into the cross-attention module; and obtaining the first user feature vector based on the query information and the key value information.

[0021] In the technical solution, the step of inputting the acoustic feature vector and the embedding vector into the cross-attention module to obtain the first user feature vector includes: firstly, inputting the acoustic feature vector as query information into the cross-attention module, and inputting the embedding vector into the cross-attention module as the key and value in the cross-attention module. Then, the cross-attention module determines the weighted sum of the value, i.e., the embedding vector, i.e., the first user feature vector, by the similarity score between the query information, i.e., the acoustic feature vector, and the key, i.e., the embedding vector. The present invention realizes the fusion between multimodal information, i.e., speech information, and acoustic features by using the cross-attention module.

[0022] In some technical schemes, optionally, the step of inputting the first user feature vector and the acoustic feature vector into a feature linear modulation module to obtain a second user feature vector includes: inputting the first user feature vector into the feature linear modulation module to obtain a scaling vector and a movement vector of the first user feature vector; multiplying the scaling vector by the acoustic feature vector and then superimposing the resultant with the movement vector to obtain a second user feature vector.

[0023] In the technical solution, the step of inputting the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain the second user feature vector includes: the feature linear modulation module can fuse the first user feature vector and the acoustic feature vector, and the feature linear modulation module includes scaling and moving operations, therefore, it is necessary to first input the first user feature vector into the feature linear modulation module to obtain the scaling vector and the moving vector. Then, the second user feature vector is obtained according to the scaling vector, the moving vector and the acoustic feature vector. The specific process can be that the scaling vector is first multiplied by the acoustic feature vector and then superimposed with the moving vector to obtain the second user feature vector. The fusion between speech information and acoustic features is achieved by utilizing the feature linear modulation module.

[0024] In some technical schemes, optionally, before the step of performing multiple loop processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors, it includes: processing the speech information according to a convolutional layer and a linear rectification function; and performing layer normalization and batch normalization on the processed speech information.

[0025] In this technical solution, before the step of performing multiple cyclic processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors, it includes: after obtaining the speech information, firstly using the convolution layer and the linear rectification function to process the speech information, and then sequentially performing layer normalization processing and batch normalization processing on the processed speech information. By performing the above processing on the speech information before inputting it into the dual-path recurrent neural network, the stability and expression ability of the dual-path recurrent neural network are increased.

[0026] According to a second aspect of the present invention, a speech recognition system is proposed, wherein the speech recognition system includes: a first acquisition module, the first acquisition module is used to acquire speech information, wherein the speech information corresponds to a target user; a first processing module, the first processing module is used to perform multiple loop processing on the speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors; a first determination module, the first determination module is used to determine an embedding vector by serially connecting multiple first slice vectors, wherein the embedding vector corresponds to the target user; a second acquisition module, the second acquisition module is used to acquire acoustic features of a reference speech; and a second determination module, the second determination module is used to input the acoustic features and the embedding vector into a personal voice activity detection model to determine a recognition result.

[0027] The speech recognition system provided by the present invention mainly includes: a first acquisition module, a first processing module, a first determination module, a second acquisition module and a second determination module. Among them, the first acquisition module can acquire speech information, wherein the speech information is the speech information input by the target user. Then the first processing module inputs the speech information into a dual-path recurrent neural network (DPRNN) and performs multiple loop processing to obtain multiple first slice vectors, that is, the speech information is iteratively updated multiple times by the dual-path recurrent neural network. After obtaining multiple first slice vectors, the first determination module connects the multiple first slice vectors in series to form the final embedding (embedding vector) of the target speaker, that is, the embedding vector. Then the second acquisition module acquires the acoustic features of the reference speech, wherein the reference speech can be set in advance by the target speaker, and the acoustic features of the reference speech are determined by processing the reference speech. Then the second determination module inputs the acoustic features and the embedding vector into a personal voice activity detection (PVAD) model to obtain a matching result, that is, to determine whether the currently input speech information is the speech information input by the target speaker. After receiving voice information, the present invention uses a network based on DPRNN (Dual-Path Recurrent Neural Network) to process the voice information, and extracts the characteristic information of the target speaker, i.e., the embedding vector, from the continuously updated RNN (Recurrent Neural Network) state, i.e., the first slice vector, thereby ensuring that good performance can be obtained even if the voice information is very short, and avoiding the problem of reducing user enthusiasm due to excessively long registration time.

[0028] According to a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned speech recognition methods when executing the computer program.

[0029] The electronic device provided by the present invention implements the steps of the above-mentioned speech recognition method when the processor executes the computer program, which can achieve the technical effects of any of the above-mentioned technical solutions and will not be repeated here.

[0030] According to a fourth aspect of the present invention, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned speech recognition methods are implemented.

[0031] The storage medium provided by the present invention implements the steps of the above-mentioned speech recognition method when the computer program is executed by the processor, and can achieve the technical effects of any of the above-mentioned technical solutions, which will not be repeated here.

[0032] According to a fifth aspect of the present invention, a computer program product is proposed, comprising a computer program, which, when executed by a processor, implements the steps of the speech recognition method in any of the above technical solutions.

[0033] The computer program product provided by this technical solution implements the steps of the speech recognition method of any technical solution of the present invention, and thus has all the beneficial effects of the speech recognition method of any technical solution of the present invention, which will not be repeated here.

[0034] Additional aspects and advantages of the present invention will become apparent in the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0036] Figure 1 A schematic diagram showing a flow chart of a speech recognition method according to an embodiment of the present invention;

[0037] Figure 2 A schematic flow chart showing the steps of performing multiple loop processing on speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors in a speech recognition method according to an embodiment of the present invention;

[0038] Figure 3 A schematic flow chart showing the step of inputting speech information into a dual-path recurrent neural network to obtain a first feature vector in a speech recognition method according to an embodiment of the present invention;

[0039] Figure 4 A schematic flow chart showing a step of determining a first slice vector in a first feature vector in a speech recognition method according to an embodiment of the present invention;

[0040] Figure 5 A flow chart showing the steps of inputting acoustic features and embedding vectors into a personal voice activity detection model and determining a recognition result in a voice recognition method according to an embodiment of the present invention;

[0041] Figure 6 A schematic flow chart showing a step of inputting an acoustic feature vector and an embedding vector into a cross attention module to obtain a first user feature vector in a speech recognition method according to an embodiment of the present invention;

[0042] Figure 7A schematic flow chart showing a step of inputting a first user feature vector and an acoustic feature vector into a feature linear modulation module to obtain a second user feature vector in a speech recognition method according to an embodiment of the present invention;

[0043] Figure 8 A flow chart showing a step before a step of performing multiple loop processing on speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors in a speech recognition method according to an embodiment of the present invention;

[0044] Fig. 9 One of the principle schematic diagrams of a speech recognition method according to an embodiment of the present invention is shown;

[0045] Fig.10 A second schematic diagram showing the principle of a speech recognition method according to an embodiment of the present invention;

[0046] FIG. 11-a is a schematic diagram showing the result of a speech recognition method according to an embodiment of the present invention;

[0047] FIG. 11-b is a schematic diagram showing the result of a speech recognition method in the related art;

[0048] Fig.12 A schematic block diagram of a speech recognition system according to an embodiment of the present invention is shown;

[0049] Fig.13 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0052] Figure 1 A flow chart of a speech recognition method according to an embodiment of the present invention is shown. The method comprises:

[0053] Step 102: Acquire voice information, wherein the voice information corresponds to a target user;

[0054] Step 104: performing multiple cycles of processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors;

[0055] Step 106: Concatenate multiple first slice vectors to determine an embedding vector, wherein the embedding vector corresponds to the target user;

[0056] Step 108: Acquire acoustic features of reference speech;

[0057] Step 110: Input the acoustic features and the embedding vector into a personal voice activity detection model to determine a recognition result.

[0058] The speech recognition method provided by the present invention mainly includes: first obtaining speech information, wherein the speech information is the speech information input by the target user, and the time length of the speech information can be 2 seconds. Then the speech information is input into a dual-path recurrent neural network (DPRNN), and multiple cycles are processed to obtain multiple first slice vectors, that is, the speech information is iteratively updated multiple times by the dual-path recurrent neural network, and then the vector output by the dual-path recurrent neural network is sliced ​​to obtain the first slice vector. Among them, the number of iterative updates can be 6 times. Among them, DPRNN is a well-known speech separation network and target speaker extraction network, which can divide the input speech information into small pieces for long sequence modeling. At the same time, the network output of DPRNN, that is, the first slice vector can be an RNN (Recurrent Neural Network, recurrent neural network) state. Since the RNN state can memorize historical information and be used to guide the operation of the next time step, it can also be continuously updated when the network is running. Therefore, using the RNN state as a speaker feature means that the feature information of the target speaker can be continuously updated in the VAD (Voice Activity Detection) step after registration. Therefore, in the registration step, only a reference speech of shorter duration is needed as the starting condition, and the final RNN state can retain its characteristic information and guide the network to detect the speech activity of the target speaker. After obtaining multiple first slice vectors, multiple first slice vectors are connected in series to form the final embedding of the target speaker, that is, the embedding vector. Then, the acoustic features of the reference speech are obtained, wherein the reference speech can be set in advance by the target speaker, and the acoustic features of the reference speech are determined by processing the reference speech. Then, the acoustic features and the embedding vector are input into the Personalized VoiceActivity Detection (PVAD) model to obtain the matching result, that is, to determine whether the currently input voice information is the voice information input by the target speaker. After receiving voice information, the present invention uses a network based on DPRNN (Dual-Path Recurrent Neural Network) to process the voice information, and extracts the characteristic information of the target speaker, i.e., the embedding vector, from the continuously updated RNN (Recurrent Neural Network) state, i.e., the first slice vector, thereby ensuring that good performance can be obtained even if the voice information is very short, and avoiding the problem of reducing user enthusiasm due to excessively long registration time.

[0059] Figure 2 A flow chart showing a step of performing multiple loop processing on speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors in a speech recognition method according to an embodiment of the present invention is shown; wherein the step of performing multiple loop processing on speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors includes:

[0060] Step 202: Input the speech information into the dual-path recurrent neural network to obtain a first feature vector;

[0061] Step 204: determining a first slice vector in the first feature vector;

[0062] Step 206: re-input the first feature vector into the dual-path recurrent neural network, re-obtain the first feature vector, and determine the first slice vector, until the number of cycles reaches a preset number and stops.

[0063] In this embodiment, the step of performing multiple cyclic processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors includes: first, inputting the speech information into the dual-path recurrent neural network to obtain a first feature vector, wherein the first feature vector refers to the feature information obtained by inputting the original speech information into the dual-path recurrent neural network, and then determining the first slice vector in the first feature vector, that is, obtaining the layer output slice of the first feature vector in the first feature vector. Then the first feature vector is input into the dual-path recurrent neural network again to obtain a new first feature vector, and a new first slice vector is obtained from the first feature vector obtained again, and this is repeated until the number of cycles reaches a preset number of times and stops. That is, the speech information is input into the dual-path recurrent neural network to obtain an output, and then the output is re-input into the dual-path recurrent neural network again to obtain another output, and after repeating this for a preset number of times, the layer output slice of each output, that is, the first slice vector, is obtained. The input is iteratively updated multiple times by PRNN, and the layer output slice obtained in each new iteration is extracted. Finally, the embedding vector, that is, the embedding (embedding vector) of the target speaker is obtained by connecting multiple layer output slices in series, thereby ensuring the accuracy of the feature information of the target speaker obtained in the end.

[0064] Figure 3 A flow chart showing a step of inputting speech information into a dual-path recurrent neural network to obtain a first feature vector in a speech recognition method according to an embodiment of the present invention; wherein the dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the step of inputting speech information into the dual-path recurrent neural network to obtain a first feature vector includes:

[0065] Step 302: Segment the voice information to obtain a plurality of sub-voice information;

[0066] Step 304: Processing the plurality of sub-speech information according to the first gated recurrent unit to obtain a second feature vector in each sub-speech information;

[0067] Step 306: determining a third feature vector according to the plurality of second feature vectors and the speech information;

[0068] Step 308: Segment the third eigenvector to obtain a plurality of fourth eigenvectors;

[0069] Step 310: Processing the plurality of fourth eigenvectors according to the second gated recurrent unit to obtain a plurality of fifth eigenvectors;

[0070] Step 312: Obtain a first eigenvector according to the plurality of fifth eigenvectors and the third eigenvector.

[0071] In this embodiment, the dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the step of inputting the speech information into the dual-path recurrent neural network to obtain the first feature vector includes: firstly cutting the speech information into a plurality of sub-speech information. Then, the plurality of sub-speech information is processed by the first gated recurrent unit to obtain the second feature vector in each sub-speech information, wherein the first gated recurrent unit may be a BIGRU (Bidirectional Gated Recurrent Unit, bidirectional gated recurrent unit), and the first gated recurrent unit may extract the local feature information inside the sub-speech information. Then, the plurality of second feature vectors and the speech information are superimposed to obtain the third feature vector. Wherein, when superimposing, the plurality of second feature vectors may first be processed by a fully connected layer (Fully Connected layer, FC) and layer norm (layer normalization) in sequence, and then the processed plurality of second feature vectors and the speech information are superimposed to obtain the third feature vector. Then, the third feature vector is segmented to obtain a plurality of fourth feature vectors, and then, the plurality of fourth feature vectors are processed by the second gated recurrent unit to obtain, and the feature information between adjacent fourth feature vectors is extracted to obtain a plurality of fifth feature vectors. Among them, the second gated recurrent unit can be a BIGRU (Bidirectional Gated Recurrent Unit). Finally, the first feature vector is obtained by superimposing the plurality of fifth eigenvectors and the third eigenvector. When superimposing, the plurality of fifth eigenvectors can first be processed by a fully connected layer (FC) and layer norm in sequence, and then the processed plurality of fifth eigenvectors and the third eigenvector are superimposed to obtain the first feature vector. In the present invention, the speech information is processed in sequence by two gated recurrent units to obtain the first feature vector, thereby ensuring the accuracy of the feature information of the target speaker obtained in the end.

[0072] Figure 4 A schematic flow chart of a step of determining a first slice vector in a first feature vector in a speech recognition method according to an embodiment of the present invention is shown; wherein the step of determining a first slice vector in a first feature vector includes:

[0073] Step 402: Slice the first feature vector by frame to obtain a plurality of second slice vectors;

[0074] Step 404: Use the second slice vector containing the recurrent neural network state as the first slice vector.

[0075] In this embodiment, the step of determining the first slice vector in the first feature vector includes: firstly, cutting the first feature vector to obtain multiple second slice vectors, wherein the cutting method is cutting by frame. Then, the second slice vector containing the recurrent neural network state is used as the first slice vector, wherein the second slice vector of the last frame may be used as the first slice vector. By cutting the first feature vector by frame, and then using the second slice vector containing the recurrent neural network state as the first slice vector, it is ensured that the first slice vector can contain the RNN state.

[0076] Figure 5 A flow chart of the step of determining a recognition result according to acoustic features, an embedding vector, and a personal voice activity detection model in a speech recognition method according to an embodiment of the present invention is shown; wherein the step of inputting the acoustic features and the embedding vector into the personal voice activity detection model to determine the recognition result includes:

[0077] Step 502: input the acoustic feature into the deformer block module to obtain an acoustic feature vector;

[0078] Step 504: Input the acoustic feature vector and the embedding vector into a cross-attention module to obtain a first user feature vector;

[0079] Step 506: input the first user feature vector and the acoustic feature vector into a feature linear modulation module to obtain a second user feature vector;

[0080] Step 508: Input the second user feature vector to the fully connected layer to obtain a recognition result.

[0081] In this embodiment, the personal voice activity detection model includes: a deformer block module, a cross attention module, a feature linear modulation module and a fully connected layer; according to the acoustic features, the embedding vector and the personal voice activity detection model, the step of determining the recognition result includes: firstly inputting the acoustic features into the deformer block module, processing the acoustic features through the deformer block module, thereby obtaining an acoustic feature vector, wherein the deformer block module can be a four-layer stacked Conformer (convolution enhanced transformer speech recognition) module. After obtaining the acoustic feature vector, the acoustic feature vector and the embedding vector are input into the cross attention (CA) module to obtain the first user feature vector, and the acoustic feature vector and the embedding vector are associated through the cross attention module to determine the interactive relationship between the acoustic feature vector and the embedding vector. After obtaining the first user feature vector, the first user feature vector and the acoustic feature vector are input into the feature-wise linear modulation (FiLM) module to obtain the second user feature vector, that is, the first user feature vector and the acoustic feature vector are integrated through the feature linear modulation module to obtain the second user feature vector, and after obtaining the second user feature vector, the second user feature vector is input into the fully connected layer, thereby obtaining the final matching result. The present invention processes the acoustic features and the embedded vector using the PVAD model and obtains the final recognition result, thereby realizing the recognition of speech information.

[0082] Figure 6 A flow chart of a step of inputting an acoustic feature vector and an embedding vector into a cross-attention module to obtain a first user feature vector in a speech recognition method according to an embodiment of the present invention is shown; wherein the step of inputting the acoustic feature vector and the embedding vector into the cross-attention module to obtain the first user feature vector includes:

[0083] Step 602: Input the acoustic feature vector as query information into the cross attention module;

[0084] Step 604: Input the embedding vector as key information into the cross attention module;

[0085] Step 606: Obtain a first user feature vector according to the query information and the key value information.

[0086] In this embodiment, the step of inputting the acoustic feature vector and the embedding vector into the cross-attention module to obtain the first user feature vector includes: firstly, inputting the acoustic feature vector as query information into the cross-attention module, and inputting the embedding vector into the cross-attention module as the key and value in the cross-attention module. Then the cross-attention module determines the weighted sum of the value, i.e., the embedding vector, i.e., the first user feature vector, by the similarity score between the query information, i.e., the acoustic feature vector, and the key, i.e., the embedding vector. The present invention realizes the fusion between multimodal information, i.e., speech information and acoustic features by using the cross-attention module.

[0087] Figure 7 A flow chart showing a step of inputting a first user feature vector and an acoustic feature vector into a feature linear modulation module to obtain a second user feature vector in a speech recognition method according to an embodiment of the present invention is shown; wherein the step of inputting the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain the second user feature vector comprises:

[0088] Step 702: input the first user feature vector into a feature linear modulation module to obtain a scaling vector and a movement vector of the first user feature vector;

[0089] Step 704: multiply the scaling vector by the acoustic feature vector and then superimpose the resultant vector with the moving vector to obtain a second user feature vector.

[0090] In this embodiment, the step of inputting the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain the second user feature vector includes: the feature linear modulation module can fuse the first user feature vector and the acoustic feature vector, and the feature linear modulation module includes scaling and moving operations, therefore, it is necessary to first input the first user feature vector into the feature linear modulation module to obtain the scaling vector and the moving vector. Then, the second user feature vector is obtained according to the scaling vector, the moving vector and the acoustic feature vector. The specific process can be firstly to multiply the scaling vector by the acoustic feature vector and then to superimpose the scaling vector with the moving vector to obtain the second user feature vector. The fusion between speech information and acoustic features is achieved by utilizing the feature linear modulation module.

[0091] Figure 8 A flow chart showing a process before a step of performing multiple loop processing on speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors in a speech recognition method according to an embodiment of the present invention is shown; wherein, before the step of performing multiple loop processing on speech information according to a dual-path recurrent neural network to obtain multiple first slice vectors, the process includes:

[0092] Step 802: Processing the speech information according to the convolution layer and the linear rectification function;

[0093] Step 804: performing layer normalization and batch normalization on the processed speech information.

[0094] In this embodiment, before the step of performing multiple cyclic processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors, it includes: after obtaining the speech information, firstly using a convolution layer and a linear rectification function to process the speech information, and then sequentially performing layer normalization processing and batch normalization processing on the processed speech information. By performing the above processing on the speech information before inputting it into the dual-path recurrent neural network, the stability and expression ability of the dual-path recurrent neural network are increased.

[0095] Fig. 9 FIG. 1 shows one of the principle schematic diagrams of a speech recognition method according to an embodiment of the present invention; Fig. 9 As shown, the present invention mainly includes two parts: an ultra-short speech extractor and a PVAD (Personalized Voice Activity Detection) backbone network. Among them, the voice information first obtained in the ultra-short speech extractor can be an ultra-short speech, and then the ultra-short speech is input into the convolution layer and the linear rectification function for preprocessing, and then the preprocessed speech information is subjected to layer normalization processing and batch normalization processing, and finally the processed speech information is obtained. Then the processed speech information is input into the dual-path recurrent neural network (Dual-Path Recurrent Neural Network, DPRNN), and the input is iteratively updated 6 times by DPRNN, and the layer output slices obtained in each new iteration are extracted, and they are connected in series to form the final embedding of the target speaker. Among them, as Fig.10As shown in the figure, the process of DPRNN processing the input speech information is to first divide the speech information into small blocks, then use a bidirectional gated recurrent unit (BIGRU) to extract intra-block features, then use another bidirectional gated recurrent unit to extract inter-block features, and finally superimpose to obtain the final feature information. Among them, since the RNN state can remember historical information and be used to guide the operation of the next time step, it can also be continuously updated when the network is running. Using the RNN state as the speaker feature means that the feature information of the target speaker can be continuously updated in the VAD step after registration. Therefore, in the registration step, only a short-duration reference speech is needed as the starting condition, and the final RNN state can retain its feature information and guide the network to detect the speech activity of the target speaker. The initial states of both local and global BiGRU are set to zero. Since the outputs of the FC layer and the layer normalization are more robust and also contain RNN states, we only use the slices containing the RNN state in the layer output, and the global BiGRU and local BiGRU states are discarded. Among them, the formula of DPRNN can be:

[0096]

[0097] Among them, Seq in and Seq out are the input and output of the i-th (i=0...5) DPRNN iteration. The input reference speech is divided into R blocks, each of which has a length of P. Since this is a BiGRU layer with two directions, the output in each direction is K-dimensional. Each iteration extracts a segment from the second-dimensional layer output and obtains Finally, the six fragments are spliced ​​together to obtain As speaker embedding for ultra-short reference speech.

[0098] Then it enters the PVAD backbone network, where it first processes the acoustic features of the concatenated speech through a four-layer stacked Conformer module to generate E concat Next, we will obtain the speaker feature E from the ultra-short speech extractor us As key and value, E concat The query is fed into the Cross Attention (CA) module. Then, the output of the CA module is used to concat Feature Linear Modulation (FiLM) is performed to fuse these two sets of features. FiLM includes scaling and shifting operations, where E CA is the output of CA, γ() and β() are the scaling and translation vectors of FiLM respectively. The FiLM process can be expressed as:

[0099] FiLM(E concat ,E CA )=γ(E CA )×E concat +β(E CA );

[0100] Finally, the fused features are input into the fully connected layer to generate the decision p of the PVAD model. The whole process can be described as follows:

[0101] p=FC(FiLM(Con(F concat ),CA(Con(F concat ),E us )));

[0102] Among them, Con represents the Conformer module, F concat It is the acoustic feature of the concatenated speech, and FC is the fully connected layer.

[0103] In order to illustrate that the speech recognition method provided by the present invention can better process ultra-short reference speech, the present invention compares the US-PVAD (Ultra Short Personalized Voice Activity Detection) model proposed by the present invention and the use of several different length reference speech in Table 1.

[0104] Benchmark models for speech, such as LSTM+Concat (Long Short Term Memory Concatenate), Conformer+Concat (Convolutional Hybrid Network Connection), and Conformer+FiLM (Convolutional Hybrid Network Feature Linear Modulation).

[0105] Table 1

[0106]

[0107]

[0108] From the results in Table 1, we can see that within the reference speech duration of 0.2 seconds, US-PVAD outperforms the baseline system in all indicators. Especially in the extreme case with additional noise, the REC (Record) of the baseline system is around 50% (E13-E15), while US-PVAD still reaches 60% (E16: 60.91%), which is even higher than the baseline system with a reference duration of 2 seconds (E1: 57.01%). In terms of overall performance (F1 and AP), US-PVAD not only outperforms the baseline system, but also maintains a high non-speech (NS) detection efficiency after adding noise (E16: 96.29%). In addition, US-PVAD is also highly efficient in detecting target speaker speech (TSS), with 0.2-second reference speech (E16: 77.65%) outperforming the baseline systems using more than 1-second reference speech (E1: 69.75%, E5: 64.43%), and being comparable to the other two baseline systems using 1-second reference speech (E6: 79.21%, E3: 82.49%). In addition, the present invention also evaluates the performance of US-PVAD under 1-second and 0.5-second reference speech. In particular, for 0.5 seconds (which is a very short duration and can be used as a reference speech in practical applications), the performance of US-PVAD is significantly better than that of the baseline systems. The results show that the baseline systems perform worse than US-PVAD in all indicators, especially in the AP indicator of target speaker speech (TSS), where US-PVAD (E12) using 0.5-second reference speech outperforms all baseline systems (E5, E7) using 1-second reference speech. This shows that US-PVAD can well support ultra-short reference speech due to the continuous updating of the target speaker embedding in the ultra-short speech extraction step.

[0109] In addition, changing the registration duration from a longer speech of 2 seconds to an ultra-short speech of 0.2 seconds has different effects on different models. Compared with the baseline model, the performance indicators of US-PVAD change less, especially under noise-free conditions. Specifically, the PRE of US-PVAD only dropped by 5.77% (E4: 83.19% to E16: 77.42%), while the PRE of the baseline model dropped by a maximum of 18.70% (E2: 78.65% to E14: 59.95%) and a minimum of 11.16% (E3: 81.99% to E15: 70.83%). This shows that compared with the baseline model, US-PVAD can maintain more stable performance when facing shorter registration times.

[0110] It is worth noting that when using 2-second reference speech, US-PVAD (E4) outperforms the baseline model in PRE and overall performance, but is less effective than the two baselines (E2, E3) in REC. This may be because the baseline system relies on an external SV model to extract speaker embeddings, while the embeddings extracted from the 2-second reference speech contain more speaker information, so REC is slightly higher than US-PVAD. Since US-PVAD does not rely on the SV model, and the ultra-short speech extractor and PVAD backbone network are trained together, the observed performance difference is in line with the research expectations of the present invention. In addition, compared with the baseline model that relies on the SV model, US-PVAD has significantly fewer total parameters (1.93 / 0.00), making it more suitable for lightweight deployment. Overall, US-PVAD can better handle ultra-short speech while maintaining good performance for reference speech of normal length.

[0111] Comparison of Bt-SNE. The quality of the target speaker embedding extracted from ultra-short speech is crucial to the performance of the PVAD backbone network. The present invention compares the t-SNE visualization effects of the embedding vectors extracted by the ultra-short speech extractor of US-PVAD and ResNet34. As shown in Figures 11-a and 11-b, Figure 11-a is a rendering of the personal ultra-short speech activity detection technology proposed by the present invention, and Figure 11-b is a rendering of the related technology. It can be seen from Figures 11-a and 11-b that for 0.2 seconds of speech, the speaker features generated by US-PVAD show obvious clustering and distribution according to different speakers, and compared with ResNet34, it can better separate the embedding vectors from different speakers. This is the key factor for US-PVAD to perform well in processing ultra-short speech.

[0112] Fig.12 A schematic block diagram of a speech recognition system according to an embodiment of the present invention is shown; wherein the speech recognition system 120 comprises:

[0113] A first acquisition module 1202, configured to acquire voice information, wherein the voice information corresponds to a target user;

[0114] A first processing module 1204 is used to perform multiple cycles of processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors;

[0115] A first determination module 1206, configured to concatenate multiple first slice vectors to determine an embedding vector, wherein the embedding vector corresponds to a target user;

[0116] A second acquisition module 1208, configured to acquire acoustic features of a reference speech;

[0117] The second determination module 1210 is used to input the acoustic features and the embedding vector into the personal voice activity detection model to determine the recognition result.

[0118] The speech recognition system 120 provided by the present invention mainly includes: a first acquisition module 1202, a first processing module 1204, a first determination module 1206, a second acquisition module 1208 and a second determination module 1210. Among them, the first acquisition module 1202 can obtain voice information, wherein, the voice information is the voice information input by the target user, and the time length of the voice information can be 2 seconds. Then the first processing module 1204 inputs the voice information into a dual-path recurrent neural network (DPRNN) and performs multiple loop processing to obtain multiple first slice vectors, that is, the voice information is iteratively updated multiple times by the dual-path recurrent neural network. Among them, the number of iterative updates can be 6 times. Among them, DPRNN is a well-known speech separation network and target speaker extraction network, which can divide the input voice information into small pieces for long sequence modeling. At the same time, the network output of DPRNN, that is, the first slice vector can be an RNN (Recurrent Neural Network) state. Since the RNN state can memorize historical information and be used to guide the operation of the next time step, it can also be continuously updated when the network is running. Therefore, using the RNN state as the speaker feature means that the feature information of the target speaker can be continuously updated in the VAD step after registration. Therefore, in the registration step, only a reference speech of a shorter duration is needed as the starting condition, and the final RNN state can retain its feature information and guide the network to detect the voice activity of the target speaker. After obtaining multiple first slice vectors, the first determination module 1206 connects the multiple first slice vectors in series to form the final embedding of the target speaker. Then the second acquisition module 1208 obtains the acoustic features of the reference speech, wherein the reference speech can be set in advance by the target speaker, and the acoustic features of the reference speech are determined by processing the reference speech. Then the second determination module 1210 inputs the acoustic features and the embedding vector into the personal voice activity detection model to obtain a matching result, that is, to determine whether the currently input voice information is the voice information input by the target speaker. After receiving the voice information, the present invention uses a DPRNN-based network to process the voice information, and extracts the characteristic information of the target speaker, i.e., the embedding vector, from the continuously updated RNN state, i.e., the first slice vector, thereby ensuring good performance even if the voice information is very short, and avoiding the problem of reducing user enthusiasm due to excessively long registration time.

[0119] In some embodiments, optionally, the first processing module 1204 is used to input speech information into a dual-path recurrent neural network to obtain a first feature vector; determine a first slice vector in the first feature vector; re-input the first feature vector into the dual-path recurrent neural network, re-obtain the first feature vector, and determine the first slice vector, until the number of cycles reaches a preset number and stops.

[0120] In this embodiment, the first processing module 1204 is specifically used to input the voice information into the dual-path recurrent neural network to obtain the first feature vector, wherein the first feature vector refers to the feature information obtained by inputting the original voice information into the dual-path recurrent neural network, and then determining the first slice vector in the first feature vector, that is, obtaining the layer output slice of the first feature vector in the first feature vector. Then the first feature vector is input into the dual-path recurrent neural network again to obtain a new first feature vector, and a new first slice vector is obtained from the first feature vector obtained again, and this is repeated until the number of cycles reaches a preset number of times and stops. That is, the voice information is input into the dual-path recurrent neural network to obtain an output, and then the output is re-input into the dual-path recurrent neural network again to obtain another output, and after repeating this for a preset number of times, the layer output slice of each output, that is, the first slice vector, is obtained. The input is iteratively updated multiple times by PRNN, and the layer output slice obtained in each new iteration is extracted. Finally, the embedding vector, that is, the embedding (embedding vector) of the target speaker is obtained by connecting multiple layer output slices in series, thereby ensuring the accuracy of the feature information of the target speaker obtained in the end.

[0121] In some embodiments, optionally, the dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the first processing module 1204 is specifically used to segment the speech information to obtain multiple sub-speech information; process the multiple sub-speech information according to the first gated recurrent unit to obtain the second eigenvector in each sub-speech information; determine the third eigenvector according to the multiple second eigenvectors and the speech information; segment the third eigenvector to obtain multiple fourth eigenvectors; process the multiple fourth eigenvectors according to the second gated recurrent unit to obtain multiple fifth eigenvectors; and obtain the first eigenvector based on the multiple fifth eigenvectors and the third eigenvector.

[0122] In this embodiment, the dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the first processing module 1204 is specifically used to cut the voice information into multiple sub-voice information. Then, the first gated recurrent unit is used to process the multiple sub-voice information to obtain the second feature vector in each sub-voice information, wherein the first gated recurrent unit can be a BIGRU, and the first gated recurrent unit can extract local feature information inside the sub-voice information. Then, the multiple second feature vectors and the voice information are superimposed to obtain the third feature vector. Wherein, when superimposing, the multiple second feature vectors can first be processed by the full connection layer (and layer norm in sequence, and then the processed multiple second feature vectors and the speech information are superimposed to obtain the third feature vector. Then the third feature vector is segmented to obtain multiple fourth feature vectors, and then the multiple fourth feature vectors are processed according to the second gated recurrent unit to obtain, and the feature information between adjacent fourth feature vectors is extracted to obtain multiple fifth feature vectors. Wherein, the second gated recurrent unit can be BIGRU. Finally, the multiple fifth feature vectors and the third feature vector are superimposed to obtain the first feature vector, wherein, when superimposing, the multiple fifth feature vectors can first be processed by the full connection layer and layer norm in sequence, and then the processed multiple fifth feature vectors and the third feature vector are superimposed to obtain the first feature vector. In the present invention, the speech information is processed in sequence by two gated recurrent units to obtain the first feature vector, thereby ensuring the accuracy of the feature information of the target speaker obtained in the end.

[0123] In some embodiments, optionally, the first processing module 1204 is further specifically used to cut the first feature vector by frame to obtain multiple second slice vectors; and use the second slice vector containing the recurrent neural network state as the first slice vector.

[0124] In this embodiment, the first processing module 1204 is also specifically used to cut the first feature vector to obtain multiple second slice vectors, wherein the cutting method is cutting according to the frame. Then the second slice vector containing the recurrent neural network state is used as the first slice vector, wherein the second slice vector of the last frame can be used as the first slice vector. By cutting the first feature vector according to the frame, and then using the second slice vector containing the recurrent neural network state as the first slice vector, it is ensured that the first slice vector can contain the RNN state.

[0125] In some embodiments, optionally, the personal voice activity detection model includes: a deformer block module, a cross-attention module, a feature linear modulation module and a fully connected layer; the second determination module 1210 is used to input the acoustic feature into the deformer block module to obtain an acoustic feature vector; input the acoustic feature vector and the embedding vector into the cross-attention module to obtain a first user feature vector; input the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain a second user feature vector; input the second user feature vector into the fully connected layer to obtain a recognition result.

[0126] In this embodiment, the personal voice activity detection model includes: a deformer block module, a cross-attention module, a feature linear modulation module and a fully connected layer; the second determination module 1210 is used to input the acoustic feature into the deformer block module, and process the acoustic feature through the deformer block module to obtain an acoustic feature vector, wherein the deformer block module can be a four-layer stacked Conformer module. After obtaining the acoustic feature vector, the acoustic feature vector and the embedding vector are input into the cross-attention module to obtain the first user feature vector, and the acoustic feature vector and the embedding vector are associated through the cross-attention module to determine the interaction relationship between the acoustic feature vector and the embedding vector. After obtaining the first user feature vector, the first user feature vector and the acoustic feature vector are input into the feature linear modulation module to obtain the second user feature vector, that is, the first user feature vector and the acoustic feature vector are integrated through the feature linear modulation module to obtain the second user feature vector, and after obtaining the second user feature vector, the second user feature vector is input into the fully connected layer, thereby obtaining the final matching result. The present invention processes acoustic features and embedded vectors using a PVAD model and obtains a final recognition result, thereby realizing recognition of speech information.

[0127] In some embodiments, optionally, the second determination module 1210 is specifically used to input the acoustic feature vector as query information into the cross-attention module; input the embedded vector as key value information into the cross-attention module; and obtain the first user feature vector based on the query information and the key value information.

[0128] In this embodiment, the second determination module 1210 is specifically used to input the acoustic feature vector as query information into the cross attention module, and input the embedding vector as the key and value in the cross attention module into the cross attention module. Then the cross attention module determines the weighted sum of the value, i.e., the embedding vector, i.e., the first user feature vector, by the similarity score between the query information, i.e., the acoustic feature vector, and the key, i.e., the embedding vector. The present invention realizes the fusion between multimodal information, i.e., speech information, and acoustic features by using the cross attention module.

[0129] In some embodiments, optionally, the second determination module 1210 is further specifically used to input the first user feature vector into the feature linear modulation module to obtain a scaling vector and a movement vector of the first user feature vector; multiply the scaling vector by the acoustic feature vector and then superimpose the resultant with the movement vector to obtain a second user feature vector.

[0130] In this embodiment, the feature linear modulation module can fuse the first user feature vector and the acoustic feature vector, and the feature linear modulation module includes scaling and moving operations. Therefore, the second determination module 1210 needs to first input the first user feature vector into the feature linear modulation module to obtain a scaling vector and a moving vector. Then, the second user feature vector is obtained according to the scaling vector, the moving vector and the acoustic feature vector. The specific process can be to first multiply the scaling vector by the acoustic feature vector and then superimpose it with the moving vector to obtain the second user feature vector. The fusion between speech information and acoustic features is achieved by using the feature linear modulation module.

[0131] In some embodiments, optionally, the speech recognition system 120 further includes: a second processing module for processing speech information according to a convolutional layer and a linear rectification function; and a third processing module for performing layer normalization processing and batch normalization processing on the processed speech information.

[0132] In this embodiment, the speech recognition system 120 further includes: a second processing module and a third processing module. After obtaining the speech information, the second processing module processes the speech information using a convolutional layer and a linear rectification function, and then the third processing module sequentially performs layer normalization processing and batch normalization processing on the processed speech information. By performing the above processing on the speech information before inputting it into the dual-path recurrent neural network, the stability and expression ability of the dual-path recurrent neural network are increased.

[0133] Fig.13 A schematic block diagram of an electronic device of an embodiment of the present invention is shown; wherein the electronic device 130 includes a memory 1302, a processor 1304, and a computer program stored in the memory 1302 and executable on the processor 1304, and when the processor 1304 executes the computer program, the steps of the speech recognition method as described above are implemented.

[0134] The electronic device 130 provided by the present invention, in which the processor 1304 implements the steps of the above-mentioned speech recognition method when executing the computer program, can achieve the technical effects of any of the above-mentioned embodiments, and will not be described in detail.

[0135] An embodiment of the present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned speech recognition methods are implemented.

[0136] The storage medium provided by the present invention implements the steps of the above-mentioned speech recognition method when the computer program is executed by the processor, and can achieve the technical effects of any of the above-mentioned embodiments, which will not be repeated here.

[0137] An embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the speech recognition method in any of the above embodiments when executed by a processor.

[0138] The computer program product provided in this embodiment implements the steps of the speech recognition method of any embodiment of the present invention, and thus has all the beneficial effects of the speech recognition method of any embodiment of the present invention, which will not be described in detail here.

[0139] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance unless otherwise clearly specified and limited; the terms "connection", "installation", "fixation" and the like should be understood in a broad sense, for example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0140] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A speech recognition method, characterized in that: include: Acquiring voice information, wherein the voice information corresponds to a target user; Performing multiple cycles of processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors; Concatenate a plurality of the first slice vectors to determine an embedding vector, wherein the embedding vector corresponds to the target user; Acquiring acoustic features of reference speech; The acoustic features and the embedding vector are input into a personal voice activity detection model to determine a recognition result.

2. The speech recognition method according to claim 1, characterized in that: The step of performing multiple cycles of processing on the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors includes: Inputting the speech information into the dual-path recurrent neural network to obtain a first feature vector; determining the first slice vector in the first feature vector; The first feature vector is re-input into the dual-path recurrent neural network to obtain the first feature vector again, and the first slice vector is determined until the number of cycles reaches a preset number and stops.

3. The speech recognition method according to claim 2, characterized in that: The dual-path recurrent neural network includes a first gated recurrent unit and a second gated recurrent unit, and the step of inputting the speech information into the dual-path recurrent neural network to obtain a first feature vector includes: Segmenting the voice information to obtain a plurality of sub-voice information; Processing the plurality of sub-speech information according to the first gated recurrent unit to obtain a second feature vector in each of the sub-speech information; Determine a third feature vector according to the plurality of second feature vectors and the speech information; Segmenting the third eigenvector to obtain a plurality of fourth eigenvectors; Processing the plurality of fourth eigenvectors according to the second gated recurrent unit to obtain a plurality of fifth eigenvectors; The first eigenvector is obtained according to a plurality of the fifth eigenvectors and the third eigenvector.

4. The speech recognition method according to claim 2, characterized in that: The step of determining the first slice vector in the first feature vector comprises: Cutting the first feature vector by frame to obtain a plurality of second slice vectors; The second slice vector containing the recurrent neural network state is used as the first slice vector.

5. The speech recognition method according to claim 1, characterized in that: The personal voice activity detection model comprises: a deformer block module, a cross attention module, a feature linear modulation module and a fully connected layer; the acoustic features and the embedding vector are input into the personal voice activity detection model, and the step of determining the recognition result comprises: Inputting the acoustic feature into the deformer block module to obtain an acoustic feature vector; Inputting the acoustic feature vector and the embedding vector into a cross attention module to obtain a first user feature vector; Inputting the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain a second user feature vector; The second user feature vector is input into the fully connected layer to obtain the recognition result.

6. The speech recognition method according to claim 5, characterized in that: The step of inputting the acoustic feature vector and the embedding vector into a cross attention module to obtain a first user feature vector comprises: inputting the acoustic feature vector as query information into the cross-attention module; Inputting the embedding vector as key information into the cross attention module; The first user feature vector is obtained according to the query information and the key value information.

7. The speech recognition method according to claim 5, characterized in that: The step of inputting the first user feature vector and the acoustic feature vector into the feature linear modulation module to obtain a second user feature vector comprises: Inputting the first user feature vector into the feature linear modulation module to obtain a scaling vector and a movement vector of the first user feature vector; The scaling vector is multiplied by the acoustic feature vector and then superimposed with the movement vector to obtain the second user feature vector.

8. The speech recognition method according to any one of claims 1 to 7, characterized in that: Before the step of performing multiple cycles of processing the speech information according to the dual-path recurrent neural network to obtain multiple first slice vectors, the speech recognition method includes: Processing the speech information according to a convolutional layer and a linear rectification function; The processed speech information is subjected to layer normalization processing and batch normalization processing.

9. A speech recognition system, characterized in that: include: A first acquisition module, the first acquisition module is used to acquire voice information, wherein the voice information corresponds to a target user; A first processing module, the first processing module is used to perform multiple loop processing on the voice information according to a dual-path recurrent neural network to obtain multiple first slice vectors; A first determination module, the first determination module is used to concatenate multiple first slice vectors to determine an embedding vector, wherein the embedding vector corresponds to the target user; A second acquisition module, the second acquisition module is used to acquire acoustic features of the reference speech; The second determination module is used to input the acoustic features and the embedding vector into a personal voice activity detection model to determine a recognition result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the speech recognition method according to any one of claims 1 to 8 are implemented.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech recognition method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the speech recognition method according to any one of claims 1 to 8 are implemented.

Images