Speech recognition method and device

Abstract

The present application discloses a speech recognition method and apparatus. The method comprises: obtaining multiple speech data, wherein the multiple speech data have the same semantics but different acoustic features; extracting speech spectrum features of each speech data; performing shared encoding on the speech spectrum features of each speech data to obtain shared speech spectrum features after encoding; constructing a comprehensive search graph based on the shared speech spectrum features; and using the comprehensive search graph to perform speech decoding on the speech data to be recognized to obtain a text sequence corresponding to the speech data to be recognized. The present application solves the technical problem of difficulty in modeling speech recognition models for dialects due to the complexity, diversity, and multiplicity of different dialects.

Description

Technical Field

[0001] The present application relates to the field of speech recognition, and more specifically, to a speech recognition method and apparatus. Background Art

[0002] With the development of artificial intelligence technology, speech recognition has been applied in many fields of industry, enabling the development of various industries. Speech recognition technology is a key technology for human-computer interaction in information technology and is currently being applied in various fields of industry.

[0003] Since different dialects contain multiple levels of complexity, differences, and diversity in acoustics, lexical structure, language, and other aspects, this brings additional difficulties to the modeling of speech recognition models. Achieving good speech recognition performance in complex multi-dialect scenarios has become an issue that needs to be explored jointly by the research community and the industry.

[0004] Currently, no effective solution has been proposed to the above-mentioned problem of difficulty in modeling speech recognition models for dialects due to the complexity, diversity and diversity of different dialects. Summary of the Invention

[0005] The embodiments of the present application provide a speech recognition method and apparatus to at least solve the technical problem of difficulty in modeling speech recognition models for dialects due to the complexity, diversity, and multiplicity of different dialects.

[0006] According to one aspect of an embodiment of the present application, a speech recognition method is provided, comprising: obtaining a plurality of speech data, wherein the plurality of speech data have the same semantics but different acoustic features; extracting speech spectrum features of each speech data respectively; performing shared encoding on the speech spectrum features of each speech data to obtain shared speech spectrum features after encoding; constructing a comprehensive search graph based on the shared speech spectrum features; and performing speech decoding on the speech data to be recognized using the comprehensive search graph to obtain a text sequence corresponding to the speech data to be recognized.

[0007] Optionally, the speech spectrum features of each speech data are shared and encoded, including: inputting the speech spectrum features of each speech data into the self-attention mechanism layer to obtain a score of the strength of the correlation between the speech spectrum features of different speech data; and performing weighted summation on the scores to obtain a feature vector including context information.

[0008] Optionally, after obtaining the feature vector including context information, the method further includes: inputting the feature vector including context information into a bidirectional recurrent gate unit, encoding the feature vector including context information into a compressed vector; calculating the CTC loss function of the compressed vector to obtain a CTC label with a sequence prediction probability.

[0009] Optionally, a comprehensive search graph is constructed based on shared speech spectrum features, including: obtaining a speech dictionary of multiple speech data and a language model trained using each speech data as a data set, wherein the language model is used to identify the corresponding speech data; representing the speech dictionary, language model and CTC label with weighted finite state machines respectively to obtain their respective corresponding networks; and fusing the networks corresponding to the speech dictionary, language model and CTC label to obtain a comprehensive search graph.

[0010] Optionally, the speech dictionary, language model, and networks corresponding to the CTC label are fused to obtain a comprehensive search graph, including: merging the network corresponding to the speech dictionary and the network corresponding to the language model through a combination operation to obtain a merged network; performing deterministic operations and minimization operations on the merged network to obtain a processed network; merging the processed network with the network corresponding to the CTC label through a combination operation to obtain a comprehensive search graph.

[0011] Optionally, the speech data to be recognized is speech-decoded using a comprehensive search graph to obtain a text sequence corresponding to the speech data to be recognized, including: inputting the speech data to be recognized into the comprehensive search graph to obtain multiple text sequences; scoring the multiple text sequences respectively, and taking the text sequence with the highest score among the multiple text sequences as the text sequence corresponding to the speech data to be recognized.

[0012] Optionally, the speech spectrum features of each speech data are extracted respectively, including performing the following processing on each speech data: pre-emphasis processing, framing processing, windowing processing, fast Fourier transform processing, filtering processing, logarithmic operation processing and discrete Fourier transform processing.

[0013] According to another aspect of an embodiment of the present application, a speech recognition device is also provided, including: an acquisition module for acquiring multiple speech data, wherein the multiple speech data have the same semantics but different acoustic features; an extraction module for respectively extracting the speech spectrum features of each speech data; an encoding module for shared encoding of the speech spectrum features of each speech data to obtain shared speech spectrum features after encoding; a construction module for constructing a comprehensive search graph based on the shared speech spectrum features; and a decoding module for using the comprehensive search graph to perform speech decoding on the speech data to be recognized to obtain a text sequence corresponding to the speech data to be recognized.

[0014] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above speech recognition method.

[0015] According to another aspect of the embodiments of the present application, a processor is further provided, which is used to run a program stored in a memory, wherein the above speech recognition method is executed when the program is run.

[0016] In an embodiment of the present application, a plurality of speech data are obtained, wherein the plurality of speech data have the same semantics but different acoustic features; the speech spectrum features of each speech data are extracted respectively; the speech spectrum features of each speech data are shared and encoded to obtain the shared speech spectrum features after encoding; a comprehensive search graph is constructed based on the shared speech spectrum features; the speech data to be recognized is speech-decoded using the comprehensive search graph to obtain a text sequence corresponding to the speech data to be recognized. By shared encoding and decoding of the speech data, the purpose of recognizing different dialects is achieved, thereby achieving the technical effects of accurate speech recognition in multi-dialect scenarios, reducing the computational complexity of the speech recognition algorithm, and improving the efficiency of speech recognition, thereby solving the technical problem of difficulty in modeling speech recognition models for dialects due to the complexity, diversity and diversity of different dialects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a speech recognition method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a speech recognition method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an optional speech recognition method according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of an optional network represented by a weighted finite state machine according to an embodiment of the present application;

[0022] Figure 5 This is a structural block diagram of a language recognition device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Currently, the industry's main solutions to the difficulty in building speech recognition models for different dialects due to their complexity, diversity, and diversity can be divided into the following two categories:

[0026] One approach is to directly mix data from multiple dialects and input it into the model for training. However, this approach cannot guarantee the accuracy of the model on data from different dialects. However, this application introduces a multi-task learning method through a multi-dialect decoding module to ensure model accuracy.

[0027] The second approach is to build corresponding dialect speech recognition models for dialects in different regions. However, this approach not only leads to problems such as waste of training resources and excessive load on the engine deployment end, but also the scarcity of training data for some ethnic minorities. However, the shared encoding module of this application introduces an attention mechanism and fully utilizes the correlation between different dialects. This not only improves model accuracy and facilitates speech recognition for dialects that lack training data, but also reduces the computational workload of the encoding module and reduces the engine burden.

[0028] The comparison between this application and the existing patent 1 is as follows:

[0029] (1) Similarities: The existing patent first extracts multiple speech features from various dialect audios, then uses the extracted feature information to input into a neural network model for training to obtain neural network models for various dialects. In essence, the present application also first extracts features from speech signals of different dialects, then inputs the extracted features into a constructed shared encoder and multiple decoders for training to obtain network structures for different dialects.

[0030] (2) Distinctions:

[0031] 1) In terms of architecture, the method in the existing patent 1 trains 18 combined models for each dialect separately to select the optimal single model. Since 18 models need to be screened for each dialect, the training scale is very large.

[0032] 2) Regarding multi-task performance, the existing method in Patent 1 only combines the LOSS function at the end and simply performs averaging, which does not reflect the strength of the correlation between different dialects and does not fully utilize the hidden correlation information between different dialects;

[0033] 3) This application introduces the idea of attention mechanism in the shared encoding module, fully explores the hidden correlation information between different dialects, and obtains common speech features; introduces the idea of weighted finite state machine in the multi-dialect decoding module, fully integrates the unique features and shared features of different dialects, reduces the amount of calculation while ensuring the recognition results, and improves recognition efficiency.

[0034] The comparison between this application and the existing patent 2 is as follows:

[0035] (1) Similarities: The existing patent 2 uses a CNN simulated auditory model to extract features from the time-domain sound waveform of the dialect, then inputs the extracted fixed-length feature sequence into the RNN, and then connects it to the fully connected layer and performs softmax to predict the dialect category. This application also essentially extracts features from speech signals of different dialects, solving the speech recognition problem of multiple dialects.

[0036] (2) Distinctions:

[0037] 1) Existing Patent 2 primarily classifies different dialects to improve noise immunity during dialect recognition. However, it also fails to reflect the strength of the correlation between different dialects and does not fully utilize the hidden correlation information between different dialects.

[0038] 2) This application introduces the idea of attention mechanism in the shared encoding module, fully explores the hidden correlation information between different dialects, and obtains common speech features; introduces the idea of weighted finite state machine in the multi-dialect decoding module, fully integrates the unique features and shared features of different dialects, reduces the amount of calculation while ensuring the recognition results, and improves recognition efficiency.

[0039] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a speech recognition method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the speech recognition method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] According to an embodiment of the present application, an embodiment of a speech recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Figure 2 is a flow chart of a speech recognition method according to an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0046] Step S202: Acquire multiple speech data, wherein the multiple speech data have the same semantics but different acoustic features;

[0047] Step S204, extracting speech spectrum features of each speech data;

[0048] Step S206, performing shared encoding on the speech spectrum features of each speech data to obtain encoded shared speech spectrum features;

[0049] In this step, a shared encoder is constructed to perform shared encoding on the acquired multi-dialect speech features.

[0050] Step S208, constructing a comprehensive search graph based on the shared speech spectrum features;

[0051] Step S210 , performing speech decoding on the speech data to be recognized using the comprehensive search graph to obtain a text sequence corresponding to the speech data to be recognized.

[0052] Through the above steps, by sharing the encoding and decoding of voice data, the purpose of identifying different dialects is achieved, thereby achieving the technical effects of accurately recognizing voice in multi-dialect scenarios, reducing the computational complexity of the voice recognition algorithm, and improving the efficiency of voice recognition.

[0053] According to an optional embodiment of the present application, step S206 is executed to perform shared encoding on the speech spectrum features of each speech data, including: inputting the speech spectrum features of each speech data into the self-attention mechanism layer to obtain a score of the strength of the correlation between the speech spectrum features of different speech data; and performing weighted summation on the scores to obtain a feature vector including context information.

[0054] Optionally, the self-attention mechanism layer is a self-attantion layer. In this step, a self-attantion layer is introduced into the shared speech coding module. The self-attantion layer uses the attention mechanism to pay more attention to the relevant parts of the input features and less attention to the irrelevant parts, dynamically generate weights for different connections, and selectively use relevant information. The speech data features are input into the shared speech coding module, and the speech features x1, x2...xn are input into the self-attention layer. The scores of the correlation strength between different speech features are dynamically calculated through the attention mechanism, and the scores are weighted and summed to obtain the feature vectors z1, z2...zn containing context information, such as Figure 3 shown.

[0055] Since dialects are richly correlated, joint training of speech features of different dialects can fully obtain the hidden correlation information between different dialects, which can improve the accuracy of the model in different dialects, reduce the risk of overfitting, and reduce model complexity.

[0056] According to another optional embodiment of the present application, after obtaining the feature vector including context information, the method also includes: inputting the feature vector including context information into a bidirectional recurrent gate unit, encoding the feature vector including context information into a compressed vector; calculating the CTC loss function of the compressed vector to obtain a CTC label with a sequence prediction probability.

[0057] Optionally, the bidirectional recurrent gate unit is a bidirectional GRU unit. In this step, a bidirectional GRU layer is also introduced in the shared speech coding module. Bidirectional GRU is easier to calculate than LSTM, and can greatly improve training efficiency. It can not only obtain information from previous moments, but also capture information from future moments, thereby making full use of the contextual information of the speech. GRU is a deformation of RNN and a simplification of LSTM. It consists of a reset gate and an update gate. The reset gate determines how to combine new input information with previous memory, and the update gate defines the amount of previous memory saved to the current time step. The speech features z1, z2...zn containing contextual information are input into the bidirectional GRU layer. The two-layer GRU can fully capture the information from previous and future moments, and encode the speech features z1, z2...zn into compressed vectors h1, h2...hn, such as Figure 3 shown.

[0058] The CTC loss function is calculated for the compressed vectors h1,h2...hn, automatically aligning the input sequence with the output label. The network output is converted into a conditional probability, and the CTC label with the sequence prediction probability is output. The objective function of CTC is to adjust the internal parameters of the shared encoder to maximize the logarithmic probability of the output label sequence given the input speech features x1,x2...xn and the corresponding output label sequence y1,y2...yn, that is:

[0059]

[0060] The validation set LOSS results after each round of training for different dialect tasks are fed back to the shared speech encoding module to train the parameters of the shared speech encoding module. CTCLOSS is used to calculate the LOSS for each dialect task, and the loss functions obtained for different dialect tasks are dynamically weighted averaged. The loss function formula is as follows:

[0061] The loss function for a single dialect decoder is:

[0062]

[0063] For multi-dialect decoders there are:

[0064]

[0065] in,

[0066]

[0067] Afterwards, the multi-task learning training method is used to train the model with the goal of minimizing the above loss function.

[0068] According to another optional embodiment of the present application, step S208 is executed to construct a comprehensive search graph based on the shared speech spectrum features, including: obtaining a speech dictionary of multiple speech data and a language model trained by each speech data as a data set, wherein the language model is used to identify the corresponding speech data; representing the speech dictionary, language model and CTC label with weighted finite state machines respectively to obtain their corresponding networks; and fusing the networks corresponding to the speech dictionary, language model and CTC label to obtain a comprehensive search graph.

[0069] In this step, multiple decoders for specific dialects are constructed to decode different dialects separately, where the size of N is equal to the number of dialect categories, such as Figure 3 shown.

[0070] Optionally, a weighted finite state machine (WFST) is used to express the dialect dictionaries of different dialects, the language models trained by different dialects, and the CTC labels output by the shared encoder in the form of WFST. The network formed by the expression is as follows: Figure 4 As shown, ci represents character i, wi represents weight i, <blank>is the empty label of CTC, <eps>Represents an empty input or output.

[0071] Dialect dictionaries and language models of different dialects are obtained from the database. By using a weighted finite state machine (WFST), the unique features of different dialects and the shared features obtained from the shared coding layer are expressed and integrated in a unified manner, and a comprehensive search graph is constructed for joint speech decoding. In addition, the use of multi-task learning and training methods can effectively ensure information sharing between different dialects and the integrity of unique features, while greatly improving decoding efficiency.

[0072] In some optional embodiments of the present application, the speech dictionary, language model, and networks corresponding to the CTC label are fused to obtain a comprehensive search graph, including: merging the network corresponding to the speech dictionary and the network corresponding to the language model through a combination operation to obtain a merged network; performing deterministic operations and minimization operations on the merged network to obtain a processed network; merging the processed network with the network corresponding to the CTC label through a combination operation to obtain a comprehensive search graph.

[0073] During the decoding process, the results of the three networks are fused. First, the dialect dictionary network L and the speech model network G are combined through a combination operation to form the LG network. Next, the LG network is subjected to determinization and minimization operations. Finally, it is combined with the CTC label network C to construct a comprehensive search graph S for speech decoding. Where min, det, and . represent minimization, determinization, and combination operations, respectively. The system ultimately selects the text sequence with the highest overall score S in the comprehensive search graph as the recognition result. The specific calculation formula is as follows:

[0074]

[0075] It should be noted that it is necessary to build a shared speech encoding network and a multi-dialect decoding network and perform model training according to the above training process based on AI frameworks such as pytorch and tensorflow.

[0076] In other optional embodiments of the present application, step S210 is executed to use the comprehensive search graph to perform voice decoding on the voice data to be recognized to obtain a text sequence corresponding to the voice data to be recognized, including: inputting the voice data to be recognized into the comprehensive search graph to obtain multiple text sequences; scoring the multiple text sequences respectively, and taking the text sequence with the highest score among the multiple text sequences as the text sequence corresponding to the voice data to be recognized.

[0077] Speech recognition decoding is performed according to the above decoding process to obtain multiple recognized texts. Multiple text sequences are scored separately, and the system ultimately selects the text sequence with the highest overall score in the comprehensive search graph as the recognition result.

[0078] According to an optional embodiment of the present application, step S204 is executed to extract the speech spectrum features of each speech data respectively, including performing the following processing on each speech data: pre-emphasis processing, framing processing, windowing processing, fast Fourier transform processing, filtering processing, logarithmic operation processing and discrete Fourier transform processing.

[0079] The speech data sample sets of various dialects are subjected to feature extraction respectively. For example, the speech data is subjected to pre-emphasis, framing, windowing, FFT, MEL filter bank, logarithmic operation, and DFT to generate 39-dimensional MFCC features x1, x2...xn, such as Figure 3 As shown in the figure, the voice data features can be extracted using MFCC features, FBANK features, PLP features, etc. After the voice spectrum features are extracted from the voice data, they are output to the shared coding module for further data characterization and extraction.

[0080] It's important to note that data preprocessing is required before feature extraction. The data preprocessing module preprocesses the input speech data and outputs speech data that meets the input requirements of the feature extraction module. This data is then sent to the feature extraction module for feature extraction. Data preprocessing can be performed using tools such as sox and ffmpeg.

[0081] This application provides a multi-dialect speech recognition training method based on multi-task learning, aiming to address the multi-dialect problem. The system primarily comprises a data preprocessing module, a feature extraction module, a shared speech encoding module, and a multi-dialect decoding module. The shared speech encoding module and multi-dialect decoding module provided in this application can be used for speech recognition in various dialect scenarios, reducing computational effort while ensuring accurate recognition results and improving recognition efficiency.

[0082] Figure 5 is a structural block diagram of a speech recognition device according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0083] An acquisition module 50 is configured to acquire a plurality of speech data, wherein the plurality of speech data have the same semantics but different acoustic features;

[0084] Extraction module 52, used to extract speech spectrum features of each speech data;

[0085] The encoding module 54 is used to perform shared encoding on the speech spectrum features of each speech data to obtain the encoded shared speech spectrum features;

[0086] A construction module 56 is configured to construct a comprehensive search graph based on the shared speech spectrum features;

[0087] The decoding module 58 is used to perform speech decoding on the speech data to be recognized using the comprehensive search graph to obtain a text sequence corresponding to the speech data to be recognized.

[0088] It should be noted that Figure 5 The preferred implementation of the embodiment shown can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0089] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above speech recognition method.

[0090] The above-mentioned non-volatile storage medium is used to store a program that performs the following functions: obtaining multiple voice data, wherein the multiple voice data have the same semantics but different acoustic features; extracting the voice spectrum features of each voice data respectively; performing shared encoding on the voice spectrum features of each voice data to obtain shared voice spectrum features after encoding; constructing a comprehensive search graph based on the shared voice spectrum features; using the comprehensive search graph to perform voice decoding on the voice data to be recognized to obtain a text sequence corresponding to the voice data to be recognized.

[0091] An embodiment of the present application further provides a processor, which is used to run a program stored in a memory, wherein the above speech recognition method is executed when the program is run.

[0092] The above-mentioned processor is used to run a program that performs the following functions: obtaining multiple voice data, wherein the multiple voice data have the same semantics but different acoustic features; extracting the voice spectrum features of each voice data respectively; performing shared encoding on the voice spectrum features of each voice data to obtain the shared voice spectrum features after encoding; constructing a comprehensive search graph based on the shared voice spectrum features; using the comprehensive search graph to perform voice decoding on the voice data to be recognized to obtain a text sequence corresponding to the voice data to be recognized.

[0093] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0094] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0096] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0097] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0099] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.< / eps> < / blank>

Claims

1. A speech recognition method, characterized in that: include: Acquiring a plurality of speech data, wherein the plurality of speech data have the same semantics but different acoustic features; Extracting speech spectrum features of each speech data respectively; The speech spectrum features of each speech data are shared and encoded to obtain shared speech spectrum features after encoding, wherein the shared encoding of the speech spectrum features of each speech data includes: inputting the speech spectrum features of each speech data into a self-attention mechanism layer to obtain scores of correlation strength between speech spectrum features of different speech data; performing weighted summation on the scores to obtain a feature vector including context information; inputting the feature vector including context information into a bidirectional recurrent gate unit to encode the feature vector including context information into a compressed vector; and calculating a CTC loss function of the compressed vector to obtain a CTC label with a sequence prediction probability. constructing a comprehensive search graph based on the shared speech spectrum features; The integrated search graph is used to perform speech decoding on the speech data to be recognized to obtain a text sequence corresponding to the speech data to be recognized.

2. The method according to claim 1, characterized in that Constructing a comprehensive search graph based on the shared speech spectrum features, including: Obtaining a speech dictionary of the plurality of speech data and a language model trained using each of the speech data as a data set, wherein the language model is used to recognize the corresponding speech data; The speech dictionary, the language model, and the CTC label are respectively represented by weighted finite state machines to obtain corresponding networks; The speech dictionary, the language model, and the network corresponding to the CTC label are fused to obtain the comprehensive search graph.

3. The method according to claim 2, characterized in that The speech dictionary, the language model, and the network corresponding to the CTC label are integrated to obtain the comprehensive search graph, including: Merging the network corresponding to the speech dictionary and the network corresponding to the language model through a combination operation to obtain a merged network; performing a deterministic operation and a minimization operation on the merged network to obtain a processed network; The processed network is merged with the network corresponding to the CTC label through the combination operation to obtain the comprehensive search graph.

4. The method according to claim 1, wherein Using the comprehensive search graph to perform speech decoding on the speech data to be recognized to obtain a text sequence corresponding to the speech data to be recognized, including: Inputting the speech data to be recognized into the comprehensive search graph to obtain multiple text sequences; The multiple text sequences are scored respectively, and the text sequence with the highest score among the multiple text sequences is used as the text sequence corresponding to the speech data to be recognized.

5. The method according to claim 1, wherein The speech spectrum features of each speech data are extracted respectively, including performing the following processing on each speech data: pre-emphasis processing, framing processing, windowing processing, fast Fourier transform processing, filtering processing, logarithmic operation processing and discrete Fourier transform processing.

6. A speech recognition device, characterized in that: include: An acquisition module, configured to acquire a plurality of speech data, wherein the plurality of speech data have the same semantics but different acoustic features; An extraction module, used to extract the speech spectrum features of each speech data; The encoding module is configured to perform shared encoding on the speech spectrum features of each speech data to obtain shared speech spectrum features after encoding, wherein the shared encoding of the speech spectrum features of each speech data comprises: inputting the speech spectrum features of each speech data into a self-attention mechanism layer to obtain scores indicating the strength of correlation between the speech spectrum features of different speech data; performing weighted summation on the scores to obtain a feature vector including context information; inputting the feature vector including context information into a bidirectional recurrent gate unit to encode the feature vector including context information into a compressed vector; and calculating a CTC loss function of the compressed vector to obtain a CTC label with a sequence prediction probability. A construction module, configured to construct a comprehensive search graph based on the shared speech spectrum features; The decoding module is used to perform speech decoding on the speech data to be recognized using the comprehensive search graph to obtain a text sequence corresponding to the speech data to be recognized.

7. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the speech recognition method according to any one of claims 1 to 5.

8. A processor, characterized in that: The processor is configured to run a program stored in a memory, wherein the program executes the speech recognition method according to any one of claims 1 to 5 when running.

Images