A model configuration method, device and storage medium of a speech recognition system

Abstract

The application provides a model configuration method, device and storage medium of a speech recognition system, and relates to the technical field of intelligent chips. The model configuration method of the speech recognition system is run on a neural network processing unit of a chip of a target device, and the speech recognition system comprises a speech activity detection model, an automatic speech recognition model and a punctuation recovery model. The model configuration method comprises the following steps: determining basic parameters of the speech activity detection model; performing model conversion on the speech activity detection model with the determined basic parameters; performing character conversion packaging on the speech activity detection model that has undergone the model conversion, so as to deploy the speech activity detection model on the neural network processing unit; obtaining output data of the packaged speech activity detection model, and performing character conversion based on the output data through the automatic speech recognition model. According to the embodiment of the application, the whole process from audio collection input to text output with added punctuation can be realized.

Description

Technical Field

[0001] This application relates to the field of intelligent chip technology, and more specifically, to a model configuration method, device, and storage medium for a speech recognition system. Background Technology

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, the form of terminal devices is becoming increasingly diversified. In addition to traditional smartphones, new human-computer interaction devices such as smartwatches, AR / VR glasses, in-vehicle terminals, and smart home control panels are constantly emerging. These non-smartphone terminals are generally limited by small screen sizes and input methods, relying more on voice commands or peripherals (such as gamepads and styluses) for interaction. In the intelligent era, voice recognition, as a core capability for achieving natural and efficient human-computer dialogue, has become an indispensable foundational technology for smart terminals, widely used in key scenarios such as real-time speech-to-text, voice search, voice assistants, and multi-turn dialogue systems.

[0003] Typical speech recognition systems usually include VAD (Voice Activity Detection) models, ASR (Automatic Speech Recognition) models, and PUNC (Punctuation Restoration) models. In practical applications, especially in environments with weak network signals, high latency, or complete offline operation, these tasks require low latency, high accuracy, and low power consumption to operate collaboratively. Particularly in weak network or offline environments, voice streaming over the network and cloud-based speech recognition cannot meet the requirements of many scenarios, and may even experience voice packet loss and inability to recognize speech. Therefore, performing model inference directly on the edge of the speech recognition system becomes essential.

[0004] However, deploying high-quality edge speech recognition systems on resource-constrained, multi-form terminals faces multiple technical challenges. First, terminal devices generally face performance bottlenecks, limited memory, and power sensitivity. Second, the system-on-chips (SoCs) used by different device manufacturers exhibit significant differences in instruction sets, memory layouts, operator support, and compilation toolchains, making it difficult to efficiently reuse models across platforms. Furthermore, ecosystem fragmentation further exacerbates the complexity of algorithm development, model compression, deployment optimization, and continuous maintenance. Summary of the Invention

[0005] This application provides a model configuration method, device, and storage medium for a speech recognition system, which can be adapted to heterogeneous hardware architectures, balance efficiency and accuracy, and support edge speech recognition with multi-task collaborative optimization.

[0006] According to one aspect of this application, a model configuration method for a speech recognition system is provided. The speech recognition system runs on the neural network processing unit of a chip in a target device, and the speech recognition system includes a speech activity detection model, an automatic speech recognition model, and a punctuation recovery model. The method includes: determining the basic parameters of the speech activity detection model; performing model conversion on the speech activity detection model with the determined basic parameters; encapsulating the converted speech activity detection model into text for deployment on the neural network processing unit; acquiring the output data of the encapsulated speech activity detection model; and performing text conversion based on the output data using the automatic speech recognition model.

[0007] According to some embodiments, the basic parameters of the speech activity detection model include the types of input and output data of the speech activity detection model, as well as the structure of the speech activity detection model.

[0008] According to some embodiments, a model conversion is performed on a speech activity detection model with determined basic parameters, including: decomposing a preset activation function in the speech activity detection model into multiple basic operators; performing graph optimization on the multiple basic operators; statically fixing the shape of the input data of the speech activity detection model; and after the graph optimization of the multiple basic operators and the static fixing of the input data shape have been performed, performing model conversion on the speech activity detection model using a preset conversion tool.

[0009] According to some embodiments, the shape of the input data of the speech activity detection model is statically fixed, including: obtaining the dynamic range of the input data of the speech activity detection model; obtaining the static value in the dynamic range according to a preset rule; and statically fixing the shape of the input data based on the static value.

[0010] According to some embodiments, a text conversion encapsulation is performed on a speech activity detection model that has undergone model conversion, including: extracting features from the input data to obtain feature vectors of the input data; aligning the feature vectors based on the statically fixed shape of the input data; converting the feature vectors that have undergone feature alignment into input tensors; and performing inference through the speech activity detection model based on the input tensors and preset text conversion inference rules to obtain output data, wherein the output data is the validity probability distribution data of audio segments.

[0011] According to some embodiments, feature extraction is performed on the input data to obtain a feature vector of the input data, including: normalizing the input data, where the input data is real-time audio frame data; obtaining preset acoustic features corresponding to the normalized input data using a preset feature extraction tool, and performing mean-variance normalization on the preset acoustic features; downsampling the input data to splice multiple consecutive audio frames in the input data to form an audio segment; and generating a feature vector based on the audio segment and the preset acoustic features that have undergone mean-variance normalization.

[0012] According to some embodiments, the output data of the encapsulated speech activity detection model is obtained, and text conversion is performed based on the output data using an automatic speech recognition model, including: determining the state of each audio frame in the input data based on the output data using preset parameters to determine the time boundary of the audio segment; pruning the audio segment according to the time boundary; and performing text conversion on the pruned audio segment using an automatic speech recognition model.

[0013] According to some embodiments, the punctuation recovery model is pruned based on the vocabulary of the automatic speech recognition model.

[0014] According to some embodiments, based on the vocabulary of the automatic speech recognition model, a vocabulary pruning process is performed on the punctuation recovery model, including: determining whether the original vocabulary of the punctuation recovery model can be pruned; if the original vocabulary can be pruned, constructing a new vocabulary of the punctuation recovery model and a mapping relationship between the original vocabulary and the new vocabulary; based on the mapping relationship, pruning the vocabulary of the embedding layer and the output layer of the punctuation recovery model; and validating the punctuation recovery model after vocabulary pruning.

[0015] According to some embodiments, determining whether the original vocabulary of the punctuation recovery model can be truncated includes: determining the relationship between the vocabulary of the automatic speech recognition model and the original vocabulary of the punctuation recovery model; determining that the original vocabulary of the punctuation recovery model can be truncated when the vocabulary of the automatic speech recognition model is a fully contained subset of the original vocabulary of the punctuation recovery model; obtaining words that are contained in the vocabulary of the automatic speech recognition model but not in the original vocabulary of the punctuation recovery model when the vocabulary of the automatic speech recognition model is a partially contained subset of the original vocabulary of the punctuation recovery model; and determining that the original vocabulary of the punctuation recovery model can be truncated when words that are contained in the vocabulary of the automatic speech recognition model but not in the original vocabulary of the punctuation recovery model are not valid.

[0016] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described above.

[0017] According to one aspect of this application, a computer-readable storage medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0018] According to embodiments of this application, a complete end-to-end process of audio acquisition, audio input, effective audio segment detection, noise filtering, feature extraction, speech-to-text conversion, punctuation addition, and text output can be achieved through pure edge-side inference. This avoids network transmission of audio packets, eliminates limitations of cloud computing resources, solves the problem of high latency in cloud-based speech recognition, and ensures user privacy and security.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.

[0021] Figure 1 A schematic diagram of a speech recognition system according to an example embodiment of this application is shown.

[0022] Figure 2 A flowchart illustrating a model configuration method for a speech recognition system according to an embodiment of this application is shown.

[0023] Figure 3 A flowchart illustrating a model configuration method for a speech recognition system according to another embodiment of this application is provided.

[0024] Figure 4 A schematic diagram of an electronic device according to an example embodiment of this application is shown. Detailed Implementation

[0025] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0027] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0030] This application provides a model configuration method, device, and storage medium for a speech recognition system. It can effectively reduce end-to-end recognition latency and reduce network transmission costs and cloud computing resource consumption caused by voice data uploading through the neural network processing unit (NPU) of the end-to-end joint chip, while strengthening user privacy and data security.

[0031] The following will describe in detail, with reference to the accompanying drawings, a model configuration method, device, and storage medium of a speech recognition system according to an embodiment of this application.

[0032] Figure 1 A schematic diagram of a speech recognition system according to an example embodiment of this application is shown.

[0033] like Figure 1As shown, the speech recognition system 100 includes a speech activity detection model 110, an automatic speech recognition model 120, and a punctuation recovery model 130.

[0034] The speech recognition system 100 runs on the neural network processing unit 200 of the chip 10 of the target device. The target device includes human-computer interaction terminal devices such as smartphones, smartwatches, and in-vehicle terminals.

[0035] The speech recognition system 100 acquires audio stream data, performs model inference in conjunction with the neural network processing unit 200, and outputs speech-converted text with appropriate punctuation added.

[0036] Figure 2 A flowchart illustrating a model configuration method for a speech recognition system according to an embodiment of this application is shown.

[0037] like Figure 2 As shown, in step S100, the basic parameters of the speech activity detection model are determined.

[0038] According to some embodiments, the basic parameters of the speech activity detection model include the types of input and output data of the speech activity detection model, as well as the structure of the speech activity detection model.

[0039] For example, a speech activity detection model can be based on one of the open-source PyTorch models. Its input data consists of real-time acquired audio frame data, and the audio sampling rate can be set to 16kHz. The output data of the speech activity detection model is the validity probability distribution data of an audio segment composed of consecutive audio frames.

[0040] For example, a speech activity detection model can be composed of four layers of FSMN (Feedforward Sequential Memory Neural Network) stacked together, with each layer of FSMN containing a memory block.

[0041] For example, the speech activity detection model also includes caching modules in_cache and out_cache to cache input and output data, and these modules are projections of the model's memory block. When inputting audio data into the speech activity detection model and performing inference, the memory block can be used to perform lookback and lookforward, thereby extracting valid audio segments.

[0042] In step S200, the speech activity detection model with the determined basic parameters is transformed.

[0043] The pre-defined activation function in the speech activity detection model is decomposed into multiple basic operators.

[0044] According to some embodiments, the preset activation function can be the GeLU activation function. The GeLU activation function is explicitly decomposed into basic operators such as Erf, Mul, and Add.

[0045] Graph optimization is performed on several basic operators to prepare for the transformation of the speech activity detection model.

[0046] According to some embodiments, graph optimization of multiple basic operators can be performed using preset graph optimization rules. For example, semantically identical operators can be standardized into a standard format, and some operators can be converted into equivalent operators supported by the model through an equivalent converter to achieve operator standardization. Then, constant folding is performed to pre-compute operations related to data shape by merging constant operations. Furthermore, the positions of some operators are adjusted to be closer to operators that support fusion, facilitating operator fusion.

[0047] Speech activity detection models typically support dynamic input to suit different application scenarios. However, if inference needs to be performed on the neural network processing unit of the target device's chip, the shape of the input data for the speech activity detection model must be fixed to a static form. According to embodiments of this application, statically fixing the shape of the input data for the speech activity detection model includes the following steps.

[0048] First, obtain the dynamic range of the input data for the speech activity detection model.

[0049] According to some embodiments, common input data cases of the voice activity detection model in different application scenarios can be obtained and inference verification can be performed to obtain the dynamic range of the input data and the size of each input data in the input data case.

[0050] Then, the static values ​​in the dynamic range of the input data are obtained according to preset rules.

[0051] According to some embodiments, a static value can be determined in the dynamic range of the input data based on the proportion of the size of each input data in the dynamic range of the input data.

[0052] Finally, based on the obtained static values, the shape of the input data is statically fixed.

[0053] According to some embodiments, the difference between the input data shape and the static value can be calculated first, and the input data shape can be aligned by supplementing invalid values ​​(usually 0). Alternatively, the input data shape can be truncated in units of the static value, and if the remaining data after truncating the input data shape is less than the static value, the input data shape can be padded.

[0054] For example, the input tensor of the statically fixed input data can be denoted as (1, 10, 400), where 1 represents that the batch of input data is 1 (i.e., the first bag), 10 represents that the feature length of the input data is 10, and 400 represents that the dimension of the single feature data of the input data is 400.

[0055] According to some embodiments, after the shape of the input data of the speech activity detection model is statically fixed, the speech activity detection model can be optimized by a preset model simplification method so as to facilitate direct deployment on the neural network processing unit of the target device's chip.

[0056] For example, multiple constant nodes in a speech activity detection model can be merged into one constant node, duplicate nodes can be merged into one node, and unnecessary nodes can be deleted to reduce the number of nodes and computational cost. Furthermore, duplicate layers in the speech activity detection model can be merged into one layer to reduce computational cost and memory usage.

[0057] After graph optimization of multiple basic operators and static fixation of the input data shape, the speech activity detection model is transformed using a pre-defined transformation tool. This transformation tool could be, for example, qnn-{onnx / pytorch}-converter.

[0058] In step S300, the speech activity detection model that has undergone model conversion is encapsulated into text to deploy the speech activity detection model on the neural network processing unit.

[0059] The input data of the speech activity detection model is normalized.

[0060] According to some embodiments, the input data can be truncated into multiple data blocks at preset intervals, and each data block can be normalized. The truncating interval for the data blocks can be set to truncate once every 1600 sampling points.

[0061] Using a preset feature extraction tool, the preset acoustic features corresponding to the normalized input data are obtained, and the mean and variance of the acoustic features are normalized.

[0062] According to some embodiments, the feature extraction tool can use a FeatureExtractor that can be shared by the speech activity detection model and the automatic speech recognition model to convert the normalized input data into FBANK (Filter Bank) acoustic features, and perform mean and variance normalization on the acoustic features using CMVN (Cepstral Mean and Variance Normalization).

[0063] At the same time, the normalized input data is downsampled to splice multiple consecutive audio frames in the input data into an audio segment.

[0064] According to some embodiments, normalized input data can be downsampled at a low frame rate.

[0065] For example, assuming a frame length of 400 and a frame shift of 160, the maximum number of valid audio frames that can be extracted from 1600 sampling points is 8. Each low frame rate output audio frame needs to be concatenated with 5 consecutive input audio frames. Since the first packet of input data lacks left context information, the first frame is copied to fill 2 frames as context information. The process stops when there are fewer than 5 remaining frames. For real-time streaming recognition, the last 4 frames are retained as context for the next packet, resulting in a final extracted feature length of 6. Using the remaining audio frames from the previous packet as context information, the feature length extracted from non-first packet data in the input data is 10.

[0066] Furthermore, feature vectors are generated based on audio segments from the input data and preset acoustic features that have undergone mean-variance normalization. The dimension of the feature vectors is fixed.

[0067] Based on the statically fixed shape of the input data, feature alignment is performed on the feature vectors.

[0068] For example, if the feature length of the first packet in the input data is only 6 (i.e., the input tensor corresponding to the first packet is (1, 6, 400)), while the input tensor of the speech activity detection model is fixed at (1, 10, 400), then the part of the feature vector with insufficient feature length will be padded with 0s. The output data corresponding to the padded positions does not contain valid information and can be directly pruned and removed after inference, retaining only the output data corresponding to the original valid input data.

[0069] The feature vectors that have been aligned with the features are converted into input tensors. Based on the input tensors and the preset text conversion inference rules, the speech activity detection model performs inference on the neural network processing unit and obtains the output data.

[0070] According to some embodiments, the preset text-to-text inference rules include the inference logic of the speech activity detection model. The input cache (in_cache) and output cache (out_cache) of the speech activity detection model are internally self-managed by the service runtime context module in the model. The speech activity detection model takes an input tensor as input data, performs inference on the neural network processing unit based on the text-to-text inference rules, and obtains the validity probability distribution data of audio segments in the input data, which is used as the model's output data.

[0071] In step S400, the output data of the encapsulated speech activity detection model is obtained, and text conversion is performed based on the output data using an automatic speech recognition model.

[0072] Based on the output data of the speech activity detection model, the state of each audio frame in the input data is determined by preset parameters in order to determine the time boundary of the audio segment.

[0073] According to some embodiments, preset parameters such as silence duration threshold and speech-to-noise ratio can be obtained, and combined with the original waveform of the audio in the input data, the state of each audio frame in the input data can be determined, and the start and end time boundaries of the corresponding audio segment can be determined. Based on the validity probability distribution data output by the speech activity detection model, the audio segment state can be determined as a Cartesian product of (start, end) x (present, absent). Obtaining the time boundaries of the audio segment can control the audio accumulation in the audio buffer of the automatic speech recognition model.

[0074] Audio segments can be trimmed based on their time boundaries. For example, an audio segment can be trimmed at 9600 (0.6s) sampling points.

[0075] The edited audio segments are converted into text using an automatic speech recognition model.

[0076] According to some embodiments, the automatic speech recognition model may employ the Paraformer ASR Online model.

[0077] According to the embodiments of this application, an end-to-end process from audio acquisition and input to speech-to-text conversion and output can be realized without involving network transmission of audio packets or limitations of cloud computing resources, thus solving the problem of high latency in cloud-based speech recognition. Furthermore, model acceleration can be directly achieved through the neural network processing unit (NPU) acceleration module of the device-side chip, without being constrained by system or framework limitations. It can be adapted to device chips, providing a high-performance, low-cost, and low-power real-time speech recognition foundation for various application scenarios.

[0078] Figure 3 A flowchart illustrating a model configuration method for a speech recognition system according to another embodiment of this application is provided.

[0079] like Figure 3 As shown, the model configuration method for the speech recognition system also includes step S500.

[0080] In step S500, the punctuation recovery model is pruned based on the vocabulary of the automatic speech recognition model.

[0081] First, determine the relationship between the vocabulary of the automatic speech recognition model and the original vocabulary of the punctuation recovery model.

[0082] When the vocabulary of the automatic speech recognition model is a fully inclusive subset of the original vocabulary of the punctuation recovery model, it is determined that the original vocabulary of the punctuation recovery model can be pruned.

[0083] When the vocabulary of the automatic speech recognition model is a semi-inclusive subset of the original vocabulary of the punctuation recovery model, obtain words that are included in the vocabulary of the automatic speech recognition model but not in the original vocabulary of the punctuation recovery model.

[0084] Furthermore, if words included in the vocabulary of the automatic speech recognition model but not in the original vocabulary of the punctuation recovery model are not valid, it is determined that the original vocabulary of the punctuation recovery model can be pruned.

[0085] According to some embodiments, words that are included in the vocabulary of the automatic speech recognition model but not in the original vocabulary of the punctuation recovery model may be rare characters or special English characters.

[0086] Given that the original vocabulary of the punctuation recovery model can be pruned, a new vocabulary for the punctuation recovery model is constructed, along with the mapping relationship between the original vocabulary and the new vocabulary.

[0087] For example, the original and new vocabularies of the punctuation recovery model are loaded and parsed into a bidirectional mapping from tokens to IDs. Furthermore, an index mapping table is constructed from the new vocabulary IDs to the old vocabulary IDs to mark existing and newly added tokens in the new vocabulary.

[0088] Based on the mapping relationship between the original vocabulary and the new vocabulary of the punctuation recovery model, vocabulary pruning is performed on the embedding layer and output layer of the punctuation recovery model.

[0089] For example, in the vocabulary pruning process of the embedding layer of the punctuation recovery model, the weights of the embedding layer need to be processed according to the mapping relationship between the original vocabulary and the new vocabulary. For tokens in the new vocabulary that exist in the original vocabulary, the corresponding rows are directly copied from the weights of the original vocabulary. For newly added tokens in the new vocabulary, they are populated through random initialization or optional UNK (unknown) vector initialization.

[0090] For example, in the vocabulary pruning process of the output layer of the punctuation recovery model, if the dimension of the weights / biases of the output layer is the same as the size of the original vocabulary, the same pruning and expansion logic as the embedding layer is used to prune the output layer. Otherwise, the output layer parameters remain unchanged (such as the punctuation classification head).

[0091] According to some embodiments, non-vocabulary-related parameters such as the encoder of the punctuation recovery model are preserved during vocabulary pruning.

[0092] The punctuation recovery model after vocabulary pruning was validated.

[0093] According to some embodiments, a test dataset can be used to compare and verify the punctuation recovery models without and with word pruning, so as to ensure that the inference results of the punctuation recovery model with word pruning are consistent with the inference results of the punctuation recovery model without word pruning.

[0094] According to the embodiments of this application, the vocabulary pruning of the punctuation recovery model is essentially a subset mapping and incremental initialization of the parameter space. While preserving the original semantic representation, it injects trainable initial embeddings into new tokens through a low-variance randomization strategy, thereby reducing the vocabulary size and avoiding model degradation caused by vocabulary expansion.

[0095] Performance statistics and comparative analysis were performed on two mobile devices using Qualcomm chips from embodiments of this application. The device information is shown in Table 1.

[0096] Table 1

[0097] Under the same environment and with the same voice input, the voice content is: "Hello, hello, how's the weather today?", the performance comparison is shown in Table 2.

[0098] Table 2

[0099] QNN HTP stands for NPU acceleration module in Snapdragon chips.

[0100] As shown in Tables 1 and 2, the two mobile devices have different configurations and their performance differences are negligible. Overall, the end-to-end latency can be kept within 100ms, and the total memory usage during model loading and inference will not exceed 100MB.

[0101] Figure 4 A schematic diagram of an electronic device according to an example embodiment of this application is shown.

[0102] like Figure 4 As shown, the electronic device 600 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0103] like Figure 4As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the methods described in this specification according to the various exemplary embodiments of this application. For example, the processing unit 610 can perform, for example... Figure 2 The method shown.

[0104] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include read-only memory (ROM) 6203.

[0105] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0106] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0107] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0108] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0109] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0110] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0111] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0112] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0113] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0114] The embodiments of this application have been described in detail above. These descriptions are solely for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, its specific implementation methods, and its application scope, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model configuration method for a speech recognition system, wherein the speech recognition system runs on the neural network processing unit of a target device's chip, and the speech recognition system includes a speech activity detection model, an automatic speech recognition model, and a punctuation recovery model, characterized in that, The method includes: Determine the basic parameters of the speech activity detection model; The speech activity detection model with determined basic parameters is transformed; The speech activity detection model that has undergone model conversion is encapsulated into text conversion so as to deploy the speech activity detection model on the neural network processing unit; The output data of the encapsulated speech activity detection model is obtained, and text conversion is performed based on the output data using the automatic speech recognition model.

2. The method according to claim 1, characterized in that, The basic parameters include the types of input and output data of the speech activity detection model, as well as the structure of the speech activity detection model.

3. The method according to claim 1, characterized in that, The speech activity detection model with determined basic parameters is transformed, including: The preset activation function in the speech activity detection model is decomposed into multiple basic operators; Graph optimization is performed on the aforementioned basic operators; The shape of the input data for the speech activity detection model is statically fixed; After graph optimization of the multiple basic operators and static fixation of the input data shape, the speech activity detection model is transformed using a preset transformation tool.

4. The method according to claim 3, characterized in that, The shape of the input data for the speech activity detection model is statically fixed, including: Obtain the dynamic range of the input data of the speech activity detection model; The static values ​​in the dynamic range are obtained according to preset rules; Based on the static value, the shape of the input data is statically fixed.

5. The method according to claim 4, characterized in that, The text-to-text conversion encapsulation of the speech activity detection model that has undergone model conversion includes: Feature extraction is performed on the input data to obtain the feature vector of the input data; Based on the statically fixed shape of the input data, feature alignment is performed on the feature vector; The feature vector that has been aligned is converted into an input tensor; Based on the input tensor and the preset text conversion inference rules, the speech activity detection model is used to perform inference and obtain the output data, which is the validity probability distribution data of the audio segment.

6. The method according to claim 5, characterized in that, Feature extraction is performed on the input data to obtain the feature vector of the input data, including: The input data is normalized, and the input data is real-time audio frame data; Using a preset feature extraction tool, the preset acoustic features corresponding to the input data that has been normalized are obtained, and the preset acoustic features are normalized for mean and variance. The input data that has been normalized is downsampled to splice together multiple consecutive audio frames in the input data to form the audio segment; The feature vector is generated based on the audio segment and the preset acoustic features that have undergone mean-variance normalization.

7. The method according to claim 5, characterized in that, Obtaining the output data of the encapsulated speech activity detection model, and performing text conversion based on the output data using the automatic speech recognition model, including: Based on the output data, the state of each audio frame in the input data is determined by preset parameters to determine the time boundary of the audio segment; The audio segment is trimmed according to the time boundary; The automatic speech recognition model is used to convert the cropped audio segments into text.

8. The method according to claim 1, characterized in that, The method further includes: Based on the vocabulary of the automatic speech recognition model, the vocabulary of the punctuation recovery model is pruned.

9. The method according to claim 8, characterized in that, Based on the vocabulary of the automatic speech recognition model, the punctuation recovery model is pruned, including: Determine whether the original vocabulary of the punctuation recovery model can be pruned. If the original vocabulary can be pruned, construct a new vocabulary for the punctuation recovery model and the mapping relationship between the original vocabulary and the new vocabulary; Based on the mapping relationship, the embedding layer and output layer of the punctuation recovery model are truncated. The punctuation recovery model after vocabulary pruning was validated.

10. The method according to claim 9, characterized in that, Determining whether the original vocabulary of the punctuation recovery model can be pruned includes: Determine the relationship between the vocabulary of the automatic speech recognition model and the original vocabulary of the punctuation recovery model; When the vocabulary of the automatic speech recognition model is a fully inclusive subset of the original vocabulary of the punctuation recovery model, it is determined that the original vocabulary of the punctuation recovery model can be truncated. When the vocabulary of the automatic speech recognition model is a semi-inclusive subset of the original vocabulary of the punctuation recovery model, words that are included in the vocabulary of the automatic speech recognition model but not included in the original vocabulary of the punctuation recovery model are obtained. If a word is included in the vocabulary of the automatic speech recognition model but not in the original vocabulary of the punctuation recovery model, it is determined that the original vocabulary of the punctuation recovery model can be pruned.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

12. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-10.

Images