Speech recognition method and apparatus, computer device, and computer program

Abstract

A speech recognition method, apparatus, computer device, and storage medium include: a step (21) of receiving streaming speech data; a step (22) of processing the streaming speech data with a speech recognition model to obtain a speech-recognized text, the speech recognition model being obtained by performing a neural network structure search on an initial network, the initial network including a plurality of feature aggregation nodes connected by a first type of operation element, the operation space corresponding to the first type of operation element being a first operation space, and specific operations dependent on context information in the first operation space being designed to be independent of future data; and a step (23) of outputting the speech-recognized text. The above technical measures can ensure the accuracy of speech recognition, reduce recognition delay in streaming speech recognition scenarios, and improve the effectiveness of streaming speech recognition.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to a Chinese patent application filed on January 12, 2021, bearing application number "202110036471.8" and titled "Speech recognition method and apparatus, as well as computer device and storage medium", the entire contents of which are incorporated by reference into the embodiments of this application.

[0002] The present application relates to the technical field of voice recognition, and in particular to a voice recognition method, apparatus, computer device and storage medium. [Background technology]

[0003] Speech recognition is a technology that recognizes speech as text, and it is widely used in various artificial intelligence (AI) scenes.

[0004] In the related technology, in order to ensure the accuracy of voice recognition, the voice recognition model needs to refer to the context information of the voice in the process of recognizing the input voice, that is, when recognizing voice data, it needs to simultaneously combine and recognize the history and future information of the voice data.

[0005] In the above technical solutions, the speech recognition model introduces future information into the speech recognition process, which results in a certain delay, thereby limiting the application of the speech recognition model to stream speech recognition. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a speech recognition method, apparatus, computer device and storage medium that can reduce the recognition delay in streaming speech recognition scenes and improve the effect of streaming speech recognition, and the technical means are as follows: [Means for solving the problem]

[0007] In one aspect, embodiments of the present application provide a speech recognition method implemented by a computing device, the method comprising: receiving streaming audio data; processing the stream audio data with a speech recognition model to obtain a speech-recognized text corresponding to the stream audio data, the speech recognition model being obtained by performing a neural network structure search on an initial network, the initial network including a plurality of feature aggregation nodes connected by a first type of operation element (also simply referred to as "operation"), an operation space (also referred to as "operation space") corresponding to the first type of operation element is a first operation space, and a specific operation dependent on context information in the first operation space is designed to be independent of future data; and outputting the speech-recognized text.

[0008] In another aspect, embodiments of the present application provide a speech recognition method implemented by a computing device, the method comprising: obtaining speech training samples including speech samples and speech recognition tags corresponding to the speech samples; performing neural network structure exploration on an initial network based on the speech training samples to obtain a network exploration model, the initial network including a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation dependent on context information in the first operation space is designed to be independent of future data; A step of constructing a speech recognition model based on the network search model, wherein the speech recognition model processes the input stream audio data to obtain a speech recognition text corresponding to the stream audio data.

[0009] In another aspect, embodiments of the present application provide a speech recognition device, the device comprising: an audio data receiving module for receiving streaming audio data; a speech data processing module for processing the stream speech data with a speech recognition model to obtain a speech recognition text corresponding to the stream speech data, the speech recognition model being obtained by performing a neural network structure search on an initial network, the initial network including a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation depending on context information in the first operation space is designed to be independent of future data; and a text output module for outputting the speech-recognized text.

[0010] In another aspect, embodiments of the present application provide a speech recognition device, the device comprising: a sample acquisition module for acquiring speech training samples including speech samples and speech recognition tags corresponding to the speech samples; A network exploration module performs neural network structure exploration on an initial network based on the speech training sample to obtain a network exploration model, the initial network includes a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation depending on context information in the first operation space is designed to be independent of future data; and a model construction module for constructing a speech recognition model based on the network search model, the speech recognition model processing input stream audio data to obtain speech recognition text corresponding to the stream audio data.

[0011] In another aspect, an embodiment of the present application provides a computing device including a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to realize the above-mentioned speech recognition method.

[0012] In another aspect, an embodiment of the present application provides a computer-readable storage medium having at least one computer instruction stored therein, the at least one computer instruction being loaded and executed by a processor to realize the speech recognition method described above.

[0013] In another aspect, embodiments of the present application provide a computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, the processor executing the computer instructions to cause the computing device to perform the speech recognition method. Effect of the Invention

[0014] A specific operation that needs to depend on context information in the operation space corresponding to the first type operation element in the initial network is set to be independent of future data, and then a neural network structure search is performed on the initial network to construct a speech recognition model. Since a specific operation that does not depend on future data is introduced into the model and a highly accurate model structure can be searched for by the neural network structure search, the above technical means can ensure the accuracy of speech recognition, reduce the recognition delay in the stream speech recognition scene, and improve the effect of stream speech recognition. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a block diagram of model search and speech recognition according to an exemplary embodiment. [Diagram 2] 1 is a flowchart of a speech recognition method according to an exemplary embodiment. [Diagram 3] 1 is a flowchart of a speech recognition method according to an exemplary embodiment. [Figure 4] 1 is a flowchart of a speech recognition method according to an exemplary embodiment. [Diagram 5] FIG. 5 is a schematic diagram of a network structure according to the embodiment shown in FIG. 4. [Figure 6] FIG. 5 is a schematic diagram of a folding operation according to the embodiment shown in FIG. 4. [Figure 7] FIG. 5 is a schematic diagram of another convolution operation according to the embodiment shown in FIG. 4. [Figure 8] FIG. 5 is a schematic diagram of causal folding according to the embodiment shown in FIG. 4. [Figure 9] FIG. 5 is a schematic diagram of another causal convolution according to the embodiment shown in FIG. [Figure 10] FIG. 1 is a schematic diagram of a model building and speech recognition framework according to an exemplary embodiment. [Figure 11] 1 is a block diagram illustrating a configuration of a voice recognition device according to an exemplary embodiment. [Figure 12]1 is a block diagram illustrating a configuration of a voice recognition device according to an exemplary embodiment. [Figure 13] FIG. 1 is a schematic diagram of a computing device according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Before describing each embodiment shown in this application, some concepts related to this application will first be explained.

[0017] 1)Artificial Intelligence (AI) Artificial intelligence is the theory, method, technology, and application system that uses digital computers or devices controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that aims to understand the nature of intelligence and create new intelligent devices that can react in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent devices to enable them to have the functions of sensing, reasoning, and decision-making.

[0018] Artificial intelligence technology is a comprehensive discipline, and the related fields are broad, including not only hardware technology but also software technology. The basic technologies of artificial intelligence generally include sensors, specialized artificial intelligence chips, cloud computing, distributed memory, big data processing technology, operating / interaction systems, mechatronics and other technologies. The software technology of artificial intelligence mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning.

[0019] 2) Neural Architecture Search (NAS) Neural network structure search is a policy for designing a neural network using an algorithm, i.e., when the length and structure of the network are not determined, a certain search space is artificially set, and a network structure that has the optimal expression in the validation set is searched for from the search space according to the designed search policy.

[0020] Neural network structure search technology includes three parts in terms of structure: search space, search policy, and evaluation estimation, and in terms of implementation, it is further divided into reinforcement learning-based NAS, genetic algorithm-based NAS (also called evolution-based NAS), and differentiable NAS (also called gradient-based NAS).

[0021] The NAS based on reinforcement learning uses a recurrent neural network as a controller to generate a sub-network, then trains and evaluates the sub-network to obtain its network performance (e.g., accuracy rate), and finally updates the parameters of the controller. However, the performance of the sub-network is not differentiable, and the controller cannot be directly optimized, and the controller parameters can only be updated based on the policy gradient method in the reinforcement learning manner. However, due to the nature of discrete optimization, such a method consumes a large amount of computational resources, because in such a NAS algorithm, in order to fully explore the "potential" of each sub-network, the controller initializes the network weights every time it samples a sub-network, trains it from scratch, and then verifies its performance. In comparison, the differentiable NAS based on gradient optimization shows a very high efficiency advantage. Differentiable NAS based on gradient optimization does not sample one subnetwork alone, train it from scratch, and then verify its performance. Instead, it constructs the entire search space into one super-network, and then models the training and search process into a bi-level optimization problem. In addition, since the super-network itself is composed of a set of subnetworks, the performance of the subnetwork with the maximum current probability is approximated by the accuracy rate of the current super-network, which has extremely high search efficiency and performance, and is gradually becoming the mainstream neural network structure search method.

[0022] 3) Super-network A super-network is a set that contains all possible sub-networks in a differentiable NAS. Developers can design one large search space, which constitutes one super-network, which contains multiple sub-networks. After training each sub-network, its performance index can be evaluated, and the neural network structure search can find the sub-network with the highest performance index from these sub-networks.

[0023] 4) Speech Technology (ST) The key technologies of voice technology include Automatic Speech Recognition (ASR), Text To Speech (TTS) and voiceprint recognition. Enabling computers to hear, see, speak and sense is the future development direction of human-computer interaction, among which voice will be one of the most promising human-computer interaction methods in the future.

[0024] The technical means according to the embodiment of the present application includes a model search stage and a voice recognition stage. Figure 1 is a block diagram of model search and voice recognition according to an exemplary embodiment. As shown in Figure 1, in the model search stage, the model training device 110 performs neural network structure search on a pre-defined initial network by a pre-defined voice training sample, and builds a highly accurate voice recognition model according to the search result; in the voice recognition stage, the voice recognition device 120 recognizes the voice recognition text in the stream voice data according to the built voice recognition model and the input stream voice data.

[0025] The initial network may be a search space or a super-network in a neural network structure search, and the searched speech recognition model may be a sub-network in the super-network.

[0026] The model training device 110 and the voice recognition device 120 may be a computing device having machine learning capabilities, for example, the computing device may be a desktop computing device such as a personal computer, a server, or the computing device may be a portable computing device such as a tablet computer, an e-book reader, etc.

[0027] Preferably, the model training device 110 and the voice recognition device 120 may be the same device, or the model training device 110 and the voice recognition device 120 may be different devices. In addition, when the model training device 110 and the voice recognition device 120 are different devices, the model training device 110 and the voice recognition device 120 may be the same type of device, for example, the model training device 110 and the voice recognition device 120 may be both personal computers, or the model training device 110 and the voice recognition device 120 may be different types of devices. For example, the model training device 110 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, and may further be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The voice recognition device 120 may be, but is not limited to, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal and the server may be directly or indirectly connected by wired or wireless communication, which is not limited in this application.

[0028] In the technical means described in each embodiment of the present application, the model training device performs a neural network structure search on the initial network, and constructs and obtains a voice recognition model based on the search result, and the application scenarios thereof include, but are not limited to, the following application scenarios:

[0029] 1. Network conference scene In international network conferences, the general application of speech recognition is, for example, for stream conference voice, a speech recognition model is used to recognize speech recognition text, and the speech recognition text is displayed on the display screen of the network conference, and if necessary, the recognized speech recognition text can be translated and then displayed (for example, displayed by text or voice).The speech recognition model of the present application can perform low-latency speech recognition, thereby meeting the real-time speech recognition in network conference scenes.

[0030] 2. Live video / audio broadcasting scenes Regarding the application of voice recognition in network live broadcasting, for example, live broadcast scenes generally need to add subtitles to live broadcast screens. In general, the voice recognition model of the present application realizes low-latency recognition of voice in live broadcast streams, so that subtitles can be generated and added to the live broadcast data stream as quickly as possible, which is of great significance to reducing the delay of live broadcasting.

[0031] 3. Real-time translation scene In many meetings, when two or more of the participants speak different languages, it is generally necessary to have a professional translator to interpret. The speech recognition model of the present application can perform low-latency recognition of the speech spoken by the participants, and quickly display the recognized text and display it on a display screen or translated speech, thereby realizing automated real-time translation.

[0032] 2 is a flowchart of a voice recognition method according to an exemplary embodiment. This method may be performed by the voice recognition device in the embodiment shown in FIG 1. As shown in FIG 2, the voice recognition method may include the following steps 21 to 23.

[0033] In step 21, streaming audio data is received.

[0034] Preferably, the streaming audio data is audio stream data generated by encoding real-time audio, and there is a high latency requirement for voice recognition of the streaming audio data, i.e., it is necessary to ensure that there is a short delay between inputting the streaming audio data and outputting the voice recognition result.

[0035] In step 22, the streaming audio data is processed by a speech recognition model to obtain a speech recognition text corresponding to the streaming audio data, and the speech recognition model is obtained by performing a neural network structure search on an initial network, the initial network includes a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation dependent on context information in the first operation space is designed to be independent of future data.

[0036] The speech recognition model is a streaming ASR model, which supports real-time feedback of speech recognition results when processing streaming speech data, unlike the non-streaming speech recognition model, which must feed back speech recognition results after processing the audio of a complete sentence when processing non-streaming speech data.

[0037] The future data refers to other audio data located after the currently recognized audio data in the time domain. For a certain operation that depends on the future data, when the certain operation recognizes the current audio data, it needs to wait for the arrival of the future data so as to complete the recognition of the current audio data, thus resulting in a certain delay, and with the increase of such operations, the delay of completing the recognition of the current audio data also increases accordingly.

[0038] For a specific operation that does not depend on future data, when the specific operation recognizes current voice data, recognition of the current voice data can be completed without waiting for the arrival of future data, and no delay is introduced in the process due to waiting for future data.

[0039] In one possible embodiment, the specific operation that does not depend on future data refers to an operation that, during the process of performing feature processing on audio data, can complete the processing process based on current audio data and data from the history prior to the current audio data.

[0040] In step 23, the speech recognition text is output.

[0041] As described above, the technical means according to the embodiment of the present application sets a specific operation that needs to depend on context information in the operation space corresponding to the first type operation element in the initial network to be independent of future data, and then performs neural network structure search on the initial network to build a speech recognition model. Since a specific operation that does not depend on future data is introduced into the model and a highly accurate model structure can be searched for by neural network structure search, the above technical means can ensure the accuracy of speech recognition, reduce the recognition delay in the streaming speech recognition scene, and improve the effect of streaming speech recognition.

[0042] 3 is a flowchart of a speech recognition method according to an exemplary embodiment. The method may be performed by the model training device in the embodiment shown in FIG. 1, and the speech recognition method may be a method based on neural network structure search. As shown in FIG. 3, the speech recognition method may include the following steps 31 to 33.

[0043] In step 31, speech training samples are obtained that include speech samples and speech recognition tags that correspond to the speech samples.

[0044] In step 32, a neural network structure search is performed on the initial network based on the voice training sample to obtain a network search model, the initial network includes a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation dependent on context information in the first operation space is designed to be independent of future data.

[0045] In order to reduce the voice recognition delay, the embodiment of the present application improves the conventional NAS technical means to design the original specific operation (neural network operation) that depends on historical data and future data in the operation space to depend only on historical data, that is, to design the specific operation in a delay-free manner, thereby searching for a low-latency neural network structure in the subsequent neural network structure search process.

[0046] Preferably, the first type of operation element is obtained by combining at least one operation in a first operation space.

[0047] In step 33, a speech recognition model is constructed based on the network search model, and the speech recognition model processes the input stream speech data to obtain a speech recognition text corresponding to the stream speech data.

[0048] As described above, the technical means according to the embodiment of the present application sets a specific operation that needs to depend on context information in the operation space corresponding to the first type operation element in the initial network to be independent of future data, and then performs neural network structure search on the initial network to build a speech recognition model. Since a specific operation that does not depend on future data is introduced into the model and a highly accurate model structure can be searched for by neural network structure search, the above technical means can ensure the accuracy of speech recognition, reduce the recognition delay in the streaming speech recognition scene, and improve the effect of streaming speech recognition.

[0049] 4 is a flowchart of a speech recognition method according to an exemplary embodiment. The method may be performed by a model training device and a speech recognition device, and the model training device and the speech recognition device may be implemented as one computer device or belong to different computer devices. The method may include the following steps 401 to 406.

[0050] In step 401, a model training device obtains speech training samples including speech samples and speech recognition tags corresponding to the speech samples.

[0051] The speech training samples are a set of samples collected in advance by a developer, and the speech training samples include each speech sample and a speech recognition tag corresponding to the speech sample, which is used to train and evaluate the model in the subsequent network structure exploration process.

[0052] In one possible embodiment, the speech recognition tag includes acoustic recognition information of the speech sample, the acoustic recognition information including phonemes, syllables, or demi-syllables.

[0053] In the technical means disclosed in the present application, when the purpose of performing model search on the initial network is to construct a highly accurate acoustic model, the speech recognition tag may be information corresponding to the output result of the acoustic model, such as a phoneme, a syllable, or a semisyllable.

[0054] In one possible embodiment, the voice sample may be pre-divided into a number of short-time speech segments (also called speech frames) with overlapping parts, and each speech frame corresponds to a respective phoneme, syllable or semi-syllable. For example, in general, for a voice with a sampling rate of 16K, the length of one frame of voice after division is 25 ms, and the inter-frame overlap is 15 ms, and this process is also called "frame division".

[0055] In step 402, the model training device performs a neural network structure search on the initial network based on the voice training sample to obtain a network search model.

[0056] The initial network includes a plurality of feature aggregation nodes connected by operation elements, the operation elements between the plurality of feature aggregation nodes include a first type of operation element, a specific operation dependent on context information included in a first operation space corresponding to the first type of operation element is designed to be independent of future data, a combination of one or more operations in the first operation space is used to realize the first type of operation element, and the specific operation is a neural network operation dependent on context information.

[0057] In the embodiments of the present application, the first operational space may include, in addition to including certain operations that depend on context information, operations that do not depend on context information, such as residual connection operations, and the embodiments of the present application do not limit the operation types included in the first operational space.

[0058] In one possible embodiment, the initial network includes n unit networks, the n unit networks including at least one first unit network, the first unit network including an input node, an output node, and at least one said feature aggregation node connected by said first type of operation elements.

[0059] In one exemplary technical means, the above initial network may be divided into unit networks, and each unit network includes an input node, an output node, and one or more feature summarization nodes between the input node and the output node.

[0060] The search space of each unit network in the initial network may be the same or different.

[0061] In one possible embodiment, between the n unit networks: The connections are made in at least one of the following connection styles: bi-chain-styled, single-link-style, and densely-connected.

[0062] In one exemplary technical means, the unit networks in the above initial network are connected by a pre-set linking method, and the linking methods between different unit networks may be the same or different.

[0063] In the technical solutions disclosed in the embodiments of the present application, the connection manner between each unit network in the initial network is not limited.

[0064] In one possible embodiment, the n unit networks include at least one second unit network, which includes an input node, an output node, and at least one feature aggregation node connected by a second type of operation element, a second operation space corresponding to the second type of operation element includes the specific operation depending on future data, and a combination of one or more operations in the second operation space realizes the second type of operation element.

[0065] Preferably, in addition to the above-mentioned specific operations that do not depend on future information (low latency / latency controllable), the search space of the initial network may further include some specific operations that need to depend on future information (high latency / latency uncontrollable), i.e., the above-mentioned specific operations that depend on future data, thereby reducing the speech recognition delay and ensuring the utilization of future information of current speech data, thereby ensuring the accuracy of speech recognition.

[0066] In one possible embodiment, a topology structure is shared between at least one of the first unit networks, or a topology structure and network parameters are shared between at least one of the first unit networks, and a topology structure is shared between at least one of the second unit networks, or a topology structure and network parameters are shared between at least one of the second unit networks.

[0067] In one exemplary technical means, when the initial network is divided into unit networks and divided into two or more different types of unit networks, in order to reduce the complexity of network search, the topology structure and network parameters can be shared in the same type of unit networks during the search process.

[0068] In another possible embodiment, the search process may involve sharing topology structures or network parameters among unit networks of the same type.

[0069] In another possible embodiment, the topology structure and network parameters may be shared between some unit networks in the same type of unit network, for example, assuming that an initial network includes four first unit networks, one set of topology structure and network parameters is shared between two first unit networks, and another set of topology structure and network parameters is shared between the other two first unit networks.

[0070] In another possible embodiment, each unit network in the initial network may not share network parameters.

[0071] In one possible embodiment, the particular operation designed to be independent of future data is a causal particular operation; Or, Certain operations that are designed to be independent of future data are mask-based certain operations.

[0072] The independence of a particular operation from future data may be achieved by a causal approach or a mask-based approach. Of course, other possible approaches can be used besides using the causal and mask approach to ensure that a particular operation is independent of future data, and the embodiments of the present application are not limited thereto.

[0073] In one possible embodiment, the feature aggregation node performs at least one of an addition operation, a stitching operation, and a multiplication operation on the input data.

[0074] In one exemplary technical means, the operation corresponding to each feature summarization node in the initial network may be fixedly set to one operation, for example, fixedly set to an addition operation.

[0075] Alternatively, in another possible embodiment, the operations corresponding to the feature summarization nodes may be set to different operations, for example, some feature summarization nodes are set to an addition operation and some feature summarization nodes are set to a stitching operation.

[0076] Alternatively, in another possible embodiment, the operations corresponding to the feature summarizing nodes may not be fixed to a specific operation, and the operations corresponding to each feature summarizing node may be determined during the neural network structure search process.

[0077] In one possible embodiment, the specific operation includes at least one of a convolution operation, a pooling operation, an operation based on a Long Short-Term Memory (LSTM) artificial neural network, and an operation based on a Gated Recurrent Unit (GRU). Alternatively, the specific operation may include a convolutional neural network operation that depends on other context information, and the embodiment of the present application does not limit the operation type of the specific operation.

[0078] In an embodiment of the present application, the model training device performs neural network structure exploration based on the initial network to determine a network exploration model with high accuracy. In the above exploration process, the model training device performs machine learning training and evaluation on each sub-network in the initial network using voice training samples to determine information such as whether a feature aggregation node in the initial network is reserved, whether each operation element between the reserved feature aggregation nodes is reserved, the operation type corresponding to the reserved operation element, and the parameters of each operation element and feature aggregation node. Further, a sub-network with an appropriate topology structure and an accuracy that meets the requirements is determined from the initial network, and the network exploration model obtained by the exploration is obtained.

[0079] Referring to Fig. 5, a schematic diagram of a network structure according to an embodiment of the present application is shown. As shown in Fig. 5, taking a conventional neural architecture search (NAS) method based on a cell structure as an example, Fig. 5 shows a schematic diagram of a NasNet-based search space, in which the connection between cells (unit networks) in the macro part 51 is a bi-chain-styled method, and the node structure of the micro part 52 is op_type (operation type) + connection (connection point).

[0080] The technical means shown in the embodiments of the present application are based on the topology structure shown in Figure 5, and the following description of the search space will be described using such a topology structure as an example. As shown in Figure 5, the construction of the search space is generally divided into two steps: macro architecture and micro architecture.

[0081] The linking method of the macro structure part is bi-chain-styled, and the input of each cell is the output of the previous two cells. The linking method is a fixed artificially designed topology and is not involved in the search. The number of layers of cells is variable and may be different in the search phase and the evaluation phase (based on the searched structure). The number of layers of cells may also be different when targeting different tasks.

[0082] It should be noted that in some NAS algorithms, the linking scheme of the macro structure may also participate in the search, i.e., a non-fixed bi-chain-styled linking scheme, and the embodiments of the present application are not limited thereto.

[0083] The micro structure is a topological structure within a cell, and can be considered as a directed acyclic graph, as shown in Figure 5. Nodes IN(1) and IN(2) are input nodes of the cell, and node1, node2, node3, and node4 are intermediate nodes corresponding to the feature aggregation nodes (the number of which is variable), and the inputs of each node are the outputs of all previous nodes, i.e., the inputs of node1 are IN(1) and IN(2), the inputs of node2 are IN(1), IN(2), and node1, and so on, and node OUT is an output node, and its inputs are the outputs of all intermediate nodes.

[0084] The NAS algorithm searches for an optimal link relationship (i.e., a topological structure) based on the link relationship in the initial model. A fixed set of candidate operations (i.e., an operation space) is predefined between two nodes, such as 3×3 convolution, 3×3 average pooling, etc., which are used to process the input of the node, and the candidate operations predefine a set of summarization functions (i.e., various feature aggregation operations) after processing the input, such as sum (addition), concat (merging), product (multiplication), etc. When the NAS algorithm searches for a neural network structure based on a training sample, it reserves the optimal candidate operation / function based on all candidate operations / functions. Note that the application example of this technical means can select a summarization function=sum function fixedly, and only the topological structure and candidate operations in the cell are searched for, and the following explanation of the search algorithm will all be explained using such a search space as an example. Preferably, the summarization function may be fixedly set to another function, or the summarization function may not be fixedly set.

[0085] In the task of stream speech recognition, the traditional NAS method has difficulty in generating a low-latency stream speech recognition model network structure. Taking the DARTS-based search space as an example, the macro structure is: Two types of cell structures are designed: a normal cell, in which the time-frequency domain resolution of the input and output remains unchanged, and a reduction cell, in which the time-frequency domain resolution of the output is half that of the input.

[0086] The reduction cells are fixed to two layers, located at 1 / 3 and 2 / 3 of the entire network, respectively, and the remaining locations are normal cells. The application example shown in the embodiment of the present application is described as an example in which the macro structure and the DARTS method are the same, and the following description of the macro structure is omitted because it is the above-mentioned topology structure. Based on the above search space, the search algorithm generates a final micro structure, and the normal cells share the same topology structure and corresponding operations, and the reduction cells share the same topology structure and corresponding operations. In the DARTS-based search space, the convolution operation and the pooling operation both depend on future information (for the current time), so in the network structure generated by the NAS algorithm, the normal cells and the reduction cells each generate delays, and for different tasks, the number of layers of the normal cells changes, so the delay also changes accordingly, and based on the above principle, the generated network structure delay increases with the increase in the number of network layers. To explain the above concept of delay more clearly, take the delay of normal cell in the generated network structure as an example, and the delay of reduction cell is 4 frames, and the delay of reduction cell is 6 frames, then calculate the network delay of 5 layers of cells, the delay=4+6+2*(4+6+2*(4))=46 frames, where the number 2 is the multiplication coefficient added due to the reduction cell's halving of the resolution in the time-frequency domain, and further calculate the network delay of 8 layers of cells, the delay=(4+4)+6+2*((4+4)+6+2*(4+4))=74 frames, and so on. Obviously, when increasing the number of layers of cells, the delay of the entire network also increases rapidly.

[0087] In order to clearly understand the concept of speech delay in the NAS algorithm, the convolution operation in the convolutional neural network is taken as an example to describe the implementation process of a specific operation. In the application example of the embodiment of the present application, the search space is mainly the convolutional neural network, and the input speech feature is a feature map (which can be regarded as an image), that is, the speech feature is FBank second-order differential feature (40-dimensional log Mel-filterbank features with the first-order and the second-order derivatives), the first-order and second-order differential features respectively correspond to additional channels (channel concept in images), and the width of the feature map of the speech feature corresponds to the frequency domain resolution (40 dimensions), and the height corresponds to the length of the speech (number of frames).

[0088] When performing conventional candidate operation processing, the speech feature map generally depends on future information. Referring to FIG. 6, a schematic diagram of a convolution operation according to an embodiment of the present application is shown. As shown in FIG. 6, taking a 3*3 convolution operation as an example, the first row on the bottom is the input (each column is one frame), the middle is the hidden layer (each layer performs one 3*3 convolution operation), the top is the output, and the dots filled with the pattern on the left side are the padding frame. FIG. 6 shows a schematic diagram of applying a three-layer 3*3 convolution operation, where the unfilled dots in the Output layer are the output of the first frame, and the coverage of the solid arrow in the Input layer is all the dependent information, that is, it requires three future frames of input information. The logic of other candidate operations is similar, and the dependency on future information increases with the increase in the hidden layer.

[0089] For more intuition, refer to FIG. 7, which shows a schematic diagram of another convolution operation according to an embodiment of the present application. As shown in FIG. 7, the input speech data needs to pass through two hidden layers, the first hidden layer includes one 3*3 convolution operation, the second hidden layer includes one 5*5 convolution operation, the first 3*3 convolution operation needs to use one frame of historical information and one frame of future information to calculate the output of the current frame, and the second 5*5 convolution operation needs to use the input of the output of the first hidden layer, two frames of historical information and two frames of future information to calculate the output of the current frame.

[0090] Based on the above description, the conventional NAS method has difficulty in effectively controlling the delay of the network structure obtained by searching, especially in a large-scale speech recognition task, the number of cell layers in the network structure is larger, and the corresponding delay increases linearly. For the problems existing in the conventional NAS algorithm for stream speech recognition tasks, the embodiment of the present application provides a latency-controlled NAS algorithm. Different from the normal cell and reduction cell structure design in the conventional algorithm, the algorithm shown in the embodiment of the present application provides a latency-controlled cell structure instead of a normal cell, that is, the macro structure of the new algorithm is composed of both latency-free cells and reduction cells. Since the latency-free cell structure is a structure design without delay, no matter what topology structure and candidate operation the micro structure finally obtained by searching by the NAS algorithm is, the cell itself does not generate delay. The advantage of this structural design is that when the network structure obtained by the search is migrated to various tasks, the overall network delay does not change even if the number of latency-free cells is increased or decreased, and the delay is entirely determined by a fixed number of reduction cells, thereby reducing the delay and enabling delay control.

[0091] In the application of the embodiment of the present application, the means for realizing the latency-free cell structure design is to design the candidate operations (i.e., the operation space, for example, the convolution operation, the pooling operation, etc.) in the cell to be a latency-free operation method.

[0092] Taking the convolution operation as an example, the design means without delay may be to change the convolution operation from a conventional convolution operation to a causal convolution. For the conventional convolution operation, reference can be made to the above FIG. 6 and FIG. 7 and the corresponding description of future information. Referring to FIG. 8, a schematic diagram of the causal convolution according to the embodiment of the present application is shown. As shown in FIG. 8, the causal convolution differs from the general convolution method in that the output of the white filled dots in the Output layer corresponds to the coverage of the solid arrow in the Input layer, that is, the calculation of the current time depends only on the past information and does not depend on the future information. In addition to the convolution operation, other candidate operations that depend on future information (e.g., pooling operations) can all use the above similar causal processing method, that is, the calculation of the current time depends only on the past information. Further, for example, referring to FIG. 9, a schematic diagram of another causal convolution according to an embodiment of the present application is shown. As shown in FIG. 9, compared with the conventional operation, the input of the causal convolution passes through two hidden layers, the first hidden layer includes one 3*3 convolution operation, and the second hidden layer includes one 5*5 convolution operation, and the first 3*3 convolution operation needs to use the historical information of two frames to calculate the output of the current frame, and the second 5*5 convolution operation has the output of the first hidden layer as its input and needs to use the historical information of four frames to calculate the output of the current frame.

[0093] In the above latency-controlled NAS algorithm according to the embodiment of the present application, the macro structure is composed of latency-free cells and reduction cells, and the micro structure of the latency-free cells forms a search space with no-latency candidate operations. The neural network structure obtained by searching with the new algorithm has a model latency determined only by a fixed number of reduction cells, and can generate a low-latency stream speech recognition model network structure.

[0094] As described above, the application example in the embodiment of the present application is realized by a bi-chain-styled cell structure, and can be preferably expanded to more structures in the following manner.

[0095] 1) At the macro structure level, it is based on the design of the cell structure, and the linking method between cells may further include chain-styled, densely-connected, etc.

[0096] 2) At the macro structure level, the structural design is similar to the cell structure.

[0097] 3) At the micro-structure design level, the candidate operation design without delay is performed. The application example of the present embodiment is a causal method. Preferably, the candidate operation design without delay can also be realized by a mask-based method. For example, the above convolution operation can be realized as a convolution operation based on a Pixel Convolutional Neural Network (Pixel CNN).

[0098] In step 403, the model training device constructs a speech recognition model based on the network search model.

[0099] The speech recognition model processes the input stream audio data to obtain a speech recognition text corresponding to the stream audio data.

[0100] In the technical means disclosed in the present application, when the purpose of performing model search on an initial network is to build a highly accurate acoustic model, a model training device can build an acoustic model based on the network search model, and the acoustic model processes the streaming voice data to obtain acoustic recognition information of the streaming voice data, and then builds a speech recognition model based on the acoustic model and a decoding diagram.

[0101] A speech recognition model generally includes an acoustic model and a decoding diagram, in which the acoustic model recognizes acoustic recognition information such as phonemes, syllables, etc. from input speech data, and the decoding diagram obtains corresponding recognized text based on the acoustic recognition information recognized by the acoustic model.

[0102] The decoding diagram generally includes, but is not limited to, a phoneme / syllable dictionary and a language model, and the phoneme / syllable dictionary generally includes a mapping from a character or word to a phoneme / syllable sequence. For example, when a series of syllable sequences is input, the syllable dictionary can output the corresponding character or word. Generally, the phoneme / syllable dictionary is not related to the field of the text and is a general-purpose part in different recognition tasks. The language model is generally transformed by an n-gram (n-ary) language model, calculates the probability that a sentence occurs, and is trained with training data and statistical methods. Generally, for texts of different fields, such as news and spoken conversation texts, there is a large difference in the combination between common words and words, so when performing speech recognition of different fields, the language model can be changed to achieve matching.

[0103] In the latency-controlled NAS algorithm according to the embodiment of the present application, the delay of the neural network structure obtained by the search is determined only by a fixed number of reduction cells, and when the model structure is migrated to various application directions of speech recognition, the model delay after migration does not change with the change in the number of cell layers in the model structure, especially when it is directed to large-scale speech recognition tasks, the model structure after migration is very complex (the number of cell layers is large), and the conventional NAS algorithm has difficulty in effectively controlling the delay. The design of the new algorithm can ensure that the delay of the model structure after migration is fixed, and it can be adapted to various speech recognition tasks, including large-scale speech recognition tasks, and the application example of the present application can generate a low-latency stream recognition model network structure for large-scale speech recognition tasks.

[0104] In step 404, a voice recognition device receives the streaming voice data.

[0105] After the construction of the voice recognition model is completed, it can be deployed to a voice recognition device to perform the task of recognizing stream voice. In the stream voice recognition task, a voice collection device in the stream voice recognition scene can continuously collect stream voice and input it to the voice recognition device.

[0106] In step 405, the voice recognition device processes the streaming voice data with a voice recognition model to obtain a voice-recognized text corresponding to the streaming voice data.

[0107] In one possible embodiment, the speech recognition model includes an acoustic model and a decoding diagram, the acoustic model being constructed based on the network search model; The speech recognition device can process the streaming speech data with the acoustic model to obtain acoustic recognition information of the streaming speech data, where the acoustic recognition information includes phonemes, syllables or semi-syllables, and then process the acoustic recognition information of the streaming speech data with the decoding diagram to obtain the speech recognition text.

[0108] In an embodiment of the present application, if the acoustic model in the above speech recognition model is a model constructed by the neural network structure search in the above step, in the speech recognition process, the speech recognition device can process the stream speech data through the acoustic model in the speech recognition model to obtain acoustic recognition information such as corresponding syllables or phonemes, and then input the acoustic recognition information into a decoding diagram composed of a speech dictionary, a language model, etc. for decoding, and obtain a corresponding speech recognition text.

[0109] In step 406, the speech recognition device outputs the speech-recognized text.

[0110] In an embodiment of the present application, after the voice recognition device outputs the voice recognition text, the voice recognition text can be applied to subsequent processing, for example, displaying the voice recognition text or its translation text as subtitles, or converting the translation text of the voice recognition text into speech and then playing it.

[0111] As described above, the technical means according to the embodiment of the present application sets a specific operation that needs to depend on context information in the operation space of the first type operation element in the initial network to a specific operation that does not depend on future data, and then performs neural network structure search on the initial network to build a speech recognition model. Since the specific operation that does not depend on future data is introduced into the model and a highly accurate model structure can be searched for by neural network structure search, the above technical means can ensure the accuracy of speech recognition, reduce the recognition delay in stream speech recognition scenes, and improve the effect of stream speech recognition.

[0112] Taking the technical means illustrated in FIG. 4 above as an example for applying to stream speech recognition task, reference is made to FIG. 10, which is a schematic diagram of a model building and speech recognition framework according to an exemplary embodiment.

[0113] In the model training device, firstly, a preset operation space 1012 (a specific operation is designed to be independent of future data) is read from the operation space memory 1011, and a preset voice training sample (including voice sample and corresponding syllable information) is read from the sample set memory, and based on the preset voice training sample and the preset operation space 1012, a neural network structure search is performed on a preset initial network 1013 (for example, the network shown in FIG. 5 above) to obtain a network search model 1014.

[0114] Next, the model training device constructs an acoustic model 1015 based on the network search model 1014, the input of which may be the speech data and syllables corresponding to the historical recognition results of the speech data, and the output is the predicted syllables of the current speech data.

[0115] The model training device constructs a voice recognition model 1017 based on the acoustic model 1015 and a preset decoding diagram 1016, and deploys the voice recognition model 1017 to a voice recognition device.

[0116] In the voice recognition device, the voice recognition device acquires the streaming voice data 1018 collected by the voice collection device, divides the streaming voice data 1018, and then inputs each voice frame obtained by the division into a voice recognition model 1017, recognizes the voice using the voice recognition model 1017 to acquire a voice recognition text 1019, and outputs the voice recognition text 1019, thereby performing operations such as display / translation / natural language processing on the voice recognition text 1019.

[0117] 11 is a block diagram of a voice recognition device according to an exemplary embodiment. The voice recognition device can implement all or part of the steps in the method according to the embodiment shown in FIG. 2 or FIG. 4. The voice recognition device includes: an audio data receiving module 1101 for receiving stream audio data; a speech data processing module 1102 for processing the stream speech data with a speech recognition model to obtain a speech recognition text corresponding to the stream speech data, the speech recognition model being obtained by performing a neural network structure search on an initial network, the initial network including a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation dependent on context information in the first operation space is designed to be independent of future data; and a text output module 1103 for outputting the speech-recognized text.

[0118] In one possible embodiment, the initial network includes n unit networks, the n unit networks including at least one first unit network, the first unit network including an input node, an output node, and at least one of the feature aggregation nodes connected by the first type of operation elements.

[0119] In one possible embodiment, between the n unit networks: The connection is made by at least one of the following connection methods: double link method, single link method, and dense link method.

[0120] In one possible embodiment, the n unit networks include at least one second unit network, the second unit network includes an input node, an output node, and at least one of the feature aggregation nodes connected by a second type of operation element, a second operation space corresponding to the second type of operation element includes the specific operation depending on future data, and a combination of one or more operations in the second operation space realizes the second type of operation element.

[0121] In one possible embodiment, a topology structure and network parameters are shared between at least one of the first unit networks, and a topology structure and network parameters are shared between at least one of the second unit networks.

[0122] In one possible embodiment, the particular operation designed to be independent of future data is the particular operation based on causality, Or, Certain operations that are designed to be independent of future data are certain operations that are based on a mask.

[0123] In one possible embodiment, the feature aggregation node performs at least one of an addition operation, a stitching operation and a multiplication operation on the input data.

[0124] In one possible embodiment, the specific operations include at least one of a convolution operation, a pooling operation, an operation based on a long short-term memory artificial neural network LSTM, and an operation based on a gating cycle unit GRU.

[0125] In one possible embodiment, the speech recognition model includes an acoustic model and a decoding diagram, the acoustic model is constructed based on a network search model, and the network search model is obtained by performing a neural network structure search on the initial network using speech training samples; The voice data processing module 1102 includes: processing the streaming audio data with the acoustic model to obtain acoustic recognition information for the streaming audio data, the acoustic recognition information including phonemes, syllables, or semi-syllables; The acoustic recognition information of the stream voice data is processed according to the decoding diagram to obtain the voice recognition text.

[0126] As described above, the technical means according to the embodiment of the present application sets a specific operation that needs to depend on context information in the operation space corresponding to the first type operation element in the initial network to be independent of future data, and then performs neural network structure search on the initial network to build a speech recognition model. Since a specific operation that does not depend on future data is introduced into the model and a highly accurate model structure can be searched for by neural network structure search, the above technical means can ensure the accuracy of speech recognition, reduce the recognition delay in the streaming speech recognition scene, and improve the effect of streaming speech recognition.

[0127] 12 is a block diagram of a voice recognition device according to an exemplary embodiment. The voice recognition device can implement all or part of the steps in the method according to the embodiment shown in FIG. 3 or FIG. 4. The voice recognition device includes: a sample acquisition module 1201 for acquiring speech training samples including speech samples and speech recognition tags corresponding to the speech samples; A network exploration module 1202 for performing neural network structure exploration on an initial network based on the voice training sample to obtain a network exploration model, the initial network includes a plurality of feature aggregation nodes connected by a first type of operation element, an operation space corresponding to the first type of operation element is a first operation space, and a specific operation that depends on context information in the first operation space is designed to be independent of future data; and a model construction module 1203 for constructing a speech recognition model based on the network search model, the speech recognition model processing input stream audio data to obtain a speech recognition text corresponding to the stream audio data.

[0128] In one possible embodiment, the speech recognition tag comprises acoustic recognition information of the speech sample, the acoustic recognition information comprising phonemes, syllables or semi-syllables; The model construction module 1203 is Construct an acoustic model based on the network search model, and the acoustic model processes the streaming voice data to obtain acoustic recognition information of the streaming voice data; The speech recognition model is constructed based on the acoustic model and the decoding diagram.

[0129] As described above, the technical means according to the embodiment of the present application sets a specific operation that needs to depend on context information in the operation space corresponding to the first type operation element in the initial network to be independent of future data, and then performs neural network structure search on the initial network to build a speech recognition model. Since a specific operation that does not depend on future data is introduced into the model and a highly accurate model structure can be searched for by neural network structure search, the above technical means can ensure the accuracy of speech recognition, reduce the recognition delay in the streaming speech recognition scene, and improve the effect of streaming speech recognition.

[0130] 13 is a schematic diagram of a computer device according to an exemplary embodiment. The computer device may be implemented as a model training device and / or a voice recognition device in the embodiments of the above methods. The computer device 1300 includes a central processing unit 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 to the central processing unit 1301. The computer device 1300 further includes a basic input / output system 1306 that helps transfer information between elements in the computer, and a mass storage device 1307 for storing an operating system 1313, application programs 1314, and other program modules 1315.

[0131] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) that is connected to the system bus 1305. The mass storage device 1307 and its associated computer readable media provide non-volatile storage for the computing device 1300. That is, the mass storage device 1307 may include a computer readable medium (not shown), such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0132] Without loss of generality, the computer readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, flash memory or other solid state storage technology, CD-ROM or other optical storage, tape cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media is not limited to the above types. The system memory 1304 and mass storage device 1307 may be collectively referred to as memory.

[0133] The computing device 1300 can be connected to the Internet and other network devices via a network interface unit 1311 connected to the system bus 1305 .

[0134] The memory further includes at least one computer instruction, and the at least one computer instruction is stored in the memory, and the processor loads and executes the at least one computer instruction to realize all or some of the steps of the method shown in FIG. 2, FIG. 3, or FIG. 4.

[0135] In an exemplary embodiment, a non-transitory computer readable storage medium containing instructions is further provided, such as a memory containing computer programs (instructions), which, when executed by a processor of a computing device, can perform the methods illustrated in the embodiments of the present application. For example, the non-transitory computer readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0136] In an exemplary embodiment, a computer program product or computer program is further provided, the computer program product or computer program including computer instructions stored in a computer readable storage medium, a processor of a computing device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computing device to perform the methods illustrated in the above embodiments.

Claims

1. 1. A method for speech recognition implemented by a computing device, comprising: receiving streaming audio data; processing the stream audio data with a speech recognition model to obtain a speech recognition text corresponding to the stream audio data, the speech recognition model being constructed based on a network search model, the network search model being obtained by performing a neural network structure search on an initial network with a speech training sample, the initial network including a plurality of feature aggregation nodes connected by operation elements, the operation elements between the plurality of feature aggregation nodes including a first type of operation element, a first operation space corresponding to the first type of operation element includes a specific operation independent of future data, a combination of one or more operations in the first operation space is used to realize the first type of operation element, the specific operation is a neural network operation dependent on context information, the speech training sample includes a speech sample and a speech recognition tag corresponding to the speech sample; and outputting the speech-recognized text.

2. 2. The speech recognition method of claim 1, wherein the initial network includes n unit networks, the n unit networks including at least one first unit network, the first unit network including an input node, an output node, and at least one of the feature summarization nodes connected by the first type of operation elements.

3. Between the n unit networks, 3. The speech recognition method according to claim 2, wherein the connections are made by at least one of a double-link method, a single-link method, and a dense-link method.

4. 3. The speech recognition method of claim 2, wherein the n unit networks include at least one second unit network, the second unit network including an input node, an output node, and at least one of the feature aggregation nodes connected by a second type of operation element, a second operation space corresponding to the second type of operation element includes the specific operation depending on future data, and a combination of one or more types of operations in the second operation space is used to realize the second type of operation element.

5. A topology structure is shared between at least one of the first unit networks, or a topology structure and network parameters are shared between at least one of the first unit networks; The speech recognition method according to claim 4 , wherein a topology structure is shared between at least one of the second unit networks, or a topology structure and network parameters are shared between at least one of the second unit networks.

6. The particular operation that does not depend on future data is a causal particular operation; Or, The method of claim 1 , wherein the specific operation that does not depend on future data is a specific operation based on a mask.

7. The method of claim 1 , wherein the feature summarization node is used to perform at least one of an addition operation, a stitching operation, and a multiplication operation on input data.

8. The speech recognition method according to any one of claims 1 to 7, wherein the specific operations include at least one of a convolution operation, a pooling operation, an operation based on a long short-term memory artificial neural network (LSTM), and an operation based on a gating cycle unit (GRU).

9. The speech recognition model includes an acoustic model and a decoding diagram, and the acoustic model is constructed based on the network search model; The step of processing the stream of audio data with the speech recognition model to obtain a speech-recognized text corresponding to the stream of audio data includes: processing the streaming audio data with the acoustic model to obtain acoustic recognition information of the streaming audio data, the acoustic recognition information including phonemes, syllables, or semi-syllables; The speech recognition method according to any one of claims 1 to 7, further comprising the step of: processing acoustic recognition information of the stream speech data according to the decoding diagram to obtain the speech recognition text.

10. 1. A method for speech recognition implemented by a computing device, comprising: obtaining audio training samples, the audio training samples including audio samples and audio recognition tags corresponding to the audio samples; performing neural network structure exploration on an initial network based on the speech training samples to obtain a network exploration model, the initial network includes a plurality of feature summarization nodes connected by operation elements, the operation elements between the plurality of feature summarization nodes include a first type of operation element, a first operation space corresponding to the first type of operation element includes a specific operation that does not depend on future data, a combination of one or more operations in the first operation space is used to realize the first type of operation element, and the specific operation is a neural network operation that depends on context information; and constructing a speech recognition model based on the network search model, the speech recognition model being used to process input streaming audio data to obtain speech recognition text corresponding to the streaming audio data.

11. the speech recognition tag comprises acoustic recognition information for the speech sample, the acoustic recognition information comprising phonemes, syllables or semi-syllables; The step of constructing a speech recognition model based on the network search model includes: constructing an acoustic model based on the network search model, the acoustic model being used to process the streaming audio data to obtain acoustic recognition information of the streaming audio data; and constructing the speech recognition model based on the acoustic model and the decoding diagram.

12. A speech recognition device, comprising: an audio data receiving module for receiving streaming audio data; a speech data processing module for processing the stream speech data through a speech recognition model to obtain a speech recognition text corresponding to the stream speech data, the speech recognition model being constructed based on a network search model, the network search model being obtained by performing a neural network structure search on an initial network through a speech training sample, the initial network including a plurality of feature aggregation nodes connected by operation elements, the operation elements between the plurality of feature aggregation nodes including a first type of operation element, a first operation space corresponding to the first type of operation element includes a specific operation independent of future data, a combination of one or more operations in the first operation space is used to realize the first type of operation element, the specific operation is a neural network operation dependent on context information, and the speech training sample includes a speech sample and a speech recognition tag corresponding to the speech sample; a text output module for outputting the speech-recognized text.

13. A speech recognition device, comprising: a sample acquisition module for acquiring speech training samples including speech samples and speech recognition tags corresponding to the speech samples; a network exploration module for performing neural network structure exploration on an initial network based on the voice training sample to obtain a network exploration model, the initial network includes a plurality of feature summarization nodes connected by operation elements, the operation elements between the plurality of feature summarization nodes include a first type of operation element, a first operation space corresponding to the first type of operation element includes a specific operation that does not depend on future data, a combination of one or more types of operations in the first operation space is used to realize the first type of operation element, and the specific operation is a neural network operation that depends on context information; and a model construction module that constructs a speech recognition model based on the network search model, the speech recognition model being used to process input streaming audio data to obtain speech recognition text corresponding to the streaming audio data.

14. A computing device including a processor and a memory, A computer device, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to realize the speech recognition method according to any one of claims 1 to 11.

15. A program for causing a computer to execute the speech recognition method according to any one of claims 1 to 11.

Images