A speech recognition method, device, system, storage medium and program product

Abstract

The embodiments of the present application provide a speech recognition method, device, system, storage medium and program product. In the system, the request processing module can receive multiple speech recognition requests and cache the audio files of the speech recognition requests to the request queue; the channel detection module can schedule the first audio file from the request queue, perform voice activity detection on the first audio file based on multiple channel queues to obtain the target voice segment in the first audio file, and cache the target voice segment to the segment queue; the speech recognition module can read the target voice segment from the segment queue, and use the speech recognition model to recognize the read target voice segment to obtain text information. By setting queues between streaming modules, the processing of adjacent modules is decoupled and each module is independent; if there is cached data in the queue, the subsequent module will not be affected by the state of the previous module, and the previous module will not be affected by the state of the subsequent module, thereby improving the utilization of hardware resources.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, system, storage medium and program product. Background Art

[0002] In existing speech recognition services, the traditional linear pipeline processing method is usually adopted, that is, the audio data is processed in a serial manner and in a fixed order through multiple steps such as voice activity detection and speech recognition, and the corresponding hardware resources are used for processing in each step. Although this method can work normally in small-scale data processing or low concurrency demand scenarios, when faced with a large number of concurrent requests or long-term audio data, due to the use of a serial processing flow, that is, each step depends on the result of the previous step, so any delay or resource overload in any link will affect the processing of subsequent links, resulting in delays in the entire processing flow, making it difficult to fully utilize hardware resources.

[0003] For example, if the audio feature extraction stage takes a long time, the subsequent model recognition stage will be blocked, resulting in low processing efficiency; the hardware resources that should have been used in the model recognition stage will be idle, and the hardware resources will be difficult to be fully utilized, resulting in resource waste. Summary of the invention

[0004] Multiple aspects of the present application provide a speech recognition method, device, system, storage medium and program product to improve hardware resource utilization.

[0005] An embodiment of the present application provides a speech recognition system, including: a request processing module, a channel detection module and a speech recognition module, wherein a request queue is provided between the request processing module and the channel detection module, and a segment queue is provided between the channel detection module and the speech recognition module: the request processing module is used to: receive multiple speech recognition requests, any speech recognition request at least includes: an audio file, and the audio file is cached in the request queue to wait for scheduling by the channel detection module; the channel detection module is used to: schedule a first audio file from the request queue, perform voice activity detection on the first audio file based on multiple parallel channel queues to obtain at least one target speech segment in the first audio file, and cache the at least one target speech segment in the segment queue; the speech recognition module is used to: read a target speech segment from the segment queue, perform speech recognition on the read target speech segment using a speech recognition model, and obtain text information corresponding to any target audio segment.

[0006] Optionally, any of the speech recognition requests also includes: priority information; the request processing module is specifically used to: cache the multiple speech recognition requests in sequence into the request queue according to the priority information of the multiple speech recognition requests, to wait for the channel detection module to schedule them.

[0007] Optionally, the request processing module is also used to: perform audio format detection and data integrity verification on the multiple voice recognition requests, and when the audio format of any of the voice recognition requests does not meet the first condition and / or the data integrity of any of the voice recognition requests does not meet the second condition, output a prompt message to the user to remind the user that there is a request error in any of the voice recognition requests.

[0008] Optionally, the channel detection module is specifically used to: perform channel separation on the first audio file to obtain multiple single-channel audios, and cache the multiple single-channel audios into multiple parallel channel queues; use the thread resources bound to each of the multiple channel queues to perform voice activity detection on the single-channel audios in the multiple channel queues to obtain at least one voice segment of each of the multiple single-channel audios; and perform channel merging on at least one voice segment of each of the multiple single-channel audios to obtain at least one target voice segment in the first audio file.

[0009] Optionally, the speech recognition system also includes: an asynchronous task scheduler; the asynchronous task scheduler is used to: determine the processing priority of each of the multiple channel queues based on the total duration of the single-channel audio accumulated in each of the multiple channel queues; the longer the total duration of the single-channel audio accumulated in any channel queue, the higher the processing priority; the channel detection module is specifically used to: utilize the thread resources bound to each of the multiple channel queues and their respective processing priorities to perform voice activity detection on the single-channel audio in the multiple channel queues in turn to obtain at least one voice segment of each of the multiple single-channel audios.

[0010] Optionally, the speech recognition module is specifically used to: allocate corresponding idle thread resources to the speech recognition model according to parameter information of the target speech segment read; the parameter information includes: the number and duration of the speech segments; use the speech recognition model and its corresponding idle thread resources to perform speech recognition on the target speech segment read to obtain text information corresponding to any target audio segment.

[0011] Optionally, it also includes: a response processing module; any voice recognition request also includes: priority information and identification information; the response processing module is used to: create a response queue for any voice recognition request; the response queue corresponds to the identification information of any voice recognition request; according to the priority information and identification information corresponding to any target audio segment, the text information corresponding to any target audio segment is sent to the response queue corresponding to any target audio segment to wait for scheduling; in response to the scheduling instruction, the scheduled target audio segment is sent from its corresponding response queue to the target device.

[0012] The embodiment of the present application also provides a speech recognition method, which is applicable to a speech recognition service; the speech recognition service includes: a request processing module, a channel detection module and a speech recognition module, a request queue is provided between the request processing module and the channel detection module, and a channel queue and a segment queue are provided between the channel detection module and the speech recognition module; the method includes: receiving multiple speech recognition requests through the request processing module, any speech recognition request includes at least: an audio file, caching the audio file in the request queue to wait for scheduling by the channel detection module; scheduling a first audio file from the request queue through the channel detection module, performing voice activity detection on the first audio file based on multiple parallel channel queues to obtain at least one target voice segment in the first audio file, and caching the at least one target voice segment in the segment queue; reading a target voice segment from the segment queue through the speech recognition module, performing speech recognition on the read target voice segment using a speech recognition model, and obtaining text information corresponding to any target audio segment.

[0013] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; wherein the memory is used to: store one or more computer instructions; and the processor is used to execute the one or more computer instructions to: execute the steps in the speech recognition method.

[0014] An embodiment of the present application also provides a computer-readable storage medium, which, when the computer program is executed by a processor, enables the processor to implement the steps in the speech recognition method.

[0015] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the speech recognition method.

[0016] In this embodiment, the request processing module can receive multiple speech recognition requests and cache the audio files of the speech recognition requests to the request queue; the channel detection module can schedule the first audio file from the request queue, perform speech activity detection on the first audio file based on multiple channel queues to obtain the target speech segment in the first audio file, and cache the target speech segment to the segment queue; the speech recognition module can read the target speech segment from the segment queue, and use the speech recognition model to recognize the read target speech segment to obtain text information. By setting queues between streaming modules, the processing of adjacent modules is decoupled and each module is independent; if there is cached data in the queue, the subsequent module will not be affected by the state of the previous module, and the previous module will not be affected by the state of the subsequent module, thereby improving the utilization of hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A schematic diagram of the architecture of a speech recognition system 10 provided for an exemplary embodiment of the present application;

[0019] Figure 2 A flowchart of a speech recognition method provided by an exemplary embodiment of the present application;

[0020] Figure 3 A flowchart of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0023] In existing speech recognition services, the traditional linear pipeline processing method is usually adopted, that is, the audio data is processed in a serial manner and in a fixed order through multiple steps such as voice activity detection and speech recognition, and the corresponding hardware resources are used for processing in each step. Although this method can work normally in small-scale data processing or low concurrency demand scenarios, when faced with a large number of concurrent requests or long-term audio data, due to the use of a serial processing flow, that is, each step depends on the result of the previous step, so any delay or resource overload in any link will affect the processing of subsequent links, resulting in delays in the entire processing flow, making it difficult to fully utilize hardware resources.

[0024] For example, if the audio feature extraction stage takes a long time, the subsequent model recognition stage will be blocked, resulting in low processing efficiency; the hardware resources that should have been used in the model recognition stage will be idle, and the hardware resources will be difficult to be fully utilized, resulting in resource waste.

[0025] In view of the above technical problems, a solution is provided in some embodiments of the present application. The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 The schematic diagram of the architecture of the speech recognition system 10 provided by an exemplary embodiment of the present application; Figure 1 As shown, the speech recognition system 10 may include the following streaming modules: a request processing module 101, a channel detection module 102, and a speech recognition module 103. A request queue is provided between the request processing module 101 and the channel detection module 102, and a segment queue is provided between the channel detection module 102 and the speech recognition module 103. A streaming module may also be referred to as a streaming component, which refers to a module in a software architecture used to process real-time or near real-time data streams. A streaming component is used to receive, process, and send continuous data streams, i.e., audio data. It should be noted that Figure 1 The voice recognition requests and text information shown in are only exemplary and do not limit the number of voice recognition requests and text information.

[0027] This embodiment does not limit the form and deployment location of the speech recognition system 10. For example, the speech recognition system 10 can be in the form of an application, a service platform, or a service in an operating system, or can be implemented as an application in a service platform or an operating system, etc.; the speech recognition system 10 can be deployed on a terminal device, such as a mobile phone, a computer, a tablet computer, or a smart wearable device, or can be deployed on any type of server, and this embodiment does not limit it.

[0028] In this embodiment, the request processing module 101 can receive multiple voice recognition requests, and any voice recognition request at least includes: an audio file, and the audio file is cached in the request queue to wait for the channel detection module to schedule. Among them, multiple voice recognition requests can be obtained from any device, for example, from a cloud server, from other terminal devices, or from the device where the voice recognition system 10 is located, and this embodiment does not limit it. Figure 1 The client shown in is only an example and does not limit the source of multiple speech recognition requests.

[0029] Based on the request queue, the speech recognition system can continue to receive and cache a large number of concurrent speech recognition requests more efficiently.

[0030] In this embodiment, the channel detection module 102 can schedule the first audio file from the request queue, perform voice activity detection on the first audio file based on multiple parallel channel queues to obtain at least one target voice segment in the first audio file, and cache the at least one target voice segment in the segment queue.

[0031] The first audio file refers to any audio file scheduled from the request queue. The multiple channel queues are parallel and independent of each other, and can be used to separate the channels of the first audio file, that is, each channel queue can perform voice activity detection on the single-channel audio of any channel separately. This embodiment does not limit the configuration position of the multiple parallel channel queues, which can be configured in the channel detection module 102 or between the channel detection module 102 and the speech recognition module 103, and the embodiment does not limit it. Figure 1 The multiple parallel channel queues shown in the figure are for illustrative purposes only and do not constitute a limitation on their configuration locations.

[0032] When performing voice activity detection on the first audio file based on multiple parallel channel queues, the channel detection module 102 can determine the voice segments containing voice and other segments of silence or background noise in the single-channel audio, and retain the voice segments containing voice, thereby playing the role of audio filtering and improving the accuracy and efficiency of the speech recognition system. Among them, the target voice segment refers to an audio segment containing voice. Optionally, the start and end points of the voice segments containing voice in the first audio file can be marked by a VAD (Voice Activity Detection) algorithm, thereby obtaining at least one target voice segment.

[0033] Channel separation of the first audio file based on multiple parallel channel queues can improve the concurrent processing capability of the speech recognition system. When processing more complex audio files, it can effectively reduce the processing load and ensure that the single-channel audio of each channel can enter the recognition stage independently, avoiding a single channel blocking the entire recognition process.

[0034] In this embodiment, the speech recognition module 103 can read the target speech segment from the segment queue, perform speech recognition on the read target speech segment using the speech recognition model, and obtain text information corresponding to any target audio segment. Among them, multiple target speech segments can be read from the segment queue at one time, and the speech recognition model can be used to perform speech recognition on the read target speech segments to obtain text information corresponding to multiple target audio segments; or one target speech segment can be read from the segment queue each time, and the speech recognition model can be used to perform speech recognition on the read target speech segment to obtain text information corresponding to the target audio segment.

[0035] By setting up queues between streaming modules, the processing of adjacent modules can be decoupled to a certain extent, making each module independent; if there is cached data in the queue, the subsequent modules will not be affected by the status of the previous modules, and the previous modules will not be affected by the status of the subsequent modules, thereby improving the utilization of hardware resources.

[0036] For example, even if the request processing module 101 encounters a freeze when caching the audio file to the request queue, the channel detection module 102 can still schedule the audio file already in the request queue from the request queue and use the hardware resources to perform corresponding processing. For another example, even if the channel detection module 102 encounters a freeze when performing voice activity detection, the speech recognition module 103 can still schedule the target voice segment already in the segment queue from the segment queue and use the hardware resources to perform corresponding processing.

[0037] In some optional embodiments, any speech recognition request may also include: priority information, the priority information of the speech recognition request is used to describe the priority of the speech recognition request, and the higher the priority, the more priority it needs. Based on this, when the request processing module 101 caches the audio file to the request queue, it can cache multiple speech recognition requests in the request queue in sequence according to the priority information of multiple speech recognition requests, waiting for the channel detection module to schedule. In this way, the speech recognition request can be more reasonably input into the request queue according to the priority of the speech recognition request.

[0038] Optionally, the request processing module may also perform audio format detection and data integrity verification on multiple speech recognition requests, wherein the audio format detection may be performed first, and then the data integrity verification may be performed, or the data integrity verification may be performed first, and then the audio format detection may be performed, or the data integrity verification and the audio format detection may be performed simultaneously, which is not limited in this embodiment. Audio format detection refers to identifying and determining the specific encoding format used by the audio file, such as MP3 (MPEG-1 Audio Layer III, an audio format), WAV (Waveform Audio File Format) or FLAC (Free Lossless Audio Codec), etc. Data integrity verification refers to the process of determining that the data has not been tampered with, lost or damaged during transmission or storage. Based on this, when the audio format of any speech recognition request does not meet the first condition, a prompt message can be output to the user to remind that any speech recognition request has a request error; or, when the data integrity of any speech recognition request does not meet the second condition, a prompt message can be output to the user to remind that any speech recognition request has a request error; or, when the audio format of any speech recognition request does not meet the first condition and the data integrity does not meet the second condition, a prompt message can be output to the user to remind that any speech recognition request has a request error. The first condition and the second condition can be set to any conditions according to actual design requirements. For example, the first condition can be that the audio format is MP3, and the second condition can be: the data integrity is 95% complete, or the data integrity is 100% complete.

[0039] In this way, the request processing module 101 can be used to more efficiently perform audio format detection and data integrity verification to ensure that the audio format of the audio file in the speech recognition request is correct and the data is not damaged.

[0040] In some optional embodiments, when the channel detection module 102 performs voice activity detection on the first audio file based on multiple parallel channel queues, the first audio file may be channel-separated to obtain multiple single-channel audios, and the multiple single-channel audios may be cached in multiple parallel channel queues. Wherein, channel separation of the first audio file refers to separating the various channels in the first audio file, such as the left channel and the right channel in stereo, or multiple channels in surround sound. Wherein, each of the mutually parallel channel queues may be pre-bound with some thread resources in the thread pool, thereby improving the concurrent processing capability of the channel queue. Based on this, the channel detection module 102 may use the thread resources bound to each of the multiple channel queues to perform voice activity detection on the single-channel audio in the multiple channel queues, respectively, to obtain at least one voice segment of each of the multiple single-channel audios. Afterwards, the at least one voice segment of each of the multiple single-channel audios may be channel-merged to obtain at least one target voice segment in the first audio file.

[0041] For example, suppose there are two parallel channel queues, one of which divides a single-channel audio in the first audio file into three voice segments, which are P1, P2 and P3 in chronological order; the other channel queue divides another single-channel audio in the first audio file into three voice segments, which are P4, P5 and P6 in chronological order; based on this, voice segments P1 and P4 can be merged through channels to obtain a target voice segment, voice segments P2 and P5 can be merged through channels to obtain a target voice segment, and voice segments P3 and P6 can be merged through channels to obtain a target voice segment.

[0042] In this way, voice activity detection can be performed on the first audio file more efficiently based on multiple parallel channel queues.

[0043] In some optional embodiments, such as Figure 1 As shown, the speech recognition system may further include an asynchronous task scheduler 104. The asynchronous task scheduler 104 may determine the processing priority of each of the multiple channel queues according to the total duration of the single-channel audio accumulated in each of the multiple channel queues, wherein the longer the total duration of the single-channel audio accumulated in any channel queue is, the higher the processing priority is.

[0044] Based on this, when the channel detection module 102 performs voice activity detection on the first audio file based on multiple parallel channel queues, it can utilize the thread resources bound to each of the multiple channel queues and their respective processing priorities to perform voice activity detection on the single-channel audios in the multiple channel queues in turn to obtain at least one voice segment of each of the multiple single-channel audios.

[0045] In this way, the asynchronous task scheduler 104 can dynamically adjust the processing priority of each channel queue according to the total duration of single-channel audio accumulated in each of the multiple channel queues, so that the channel detection module 102 can control multiple parallel channel queues to perform voice activity detection according to the processing priority, thereby more effectively avoiding the situation where the audio duration of a certain channel queue accumulates too long.

[0046] In some optional embodiments, when the speech recognition module 103 uses the speech recognition model to perform speech recognition on the read target speech segment to obtain the text information corresponding to any target audio segment, the speech recognition model can be allocated corresponding idle thread resources according to the parameter information of the read target speech segment. The parameter information may include: the number and duration of the speech segments. The idle thread resources in an idle state can be first determined from the thread pool using a load balancing algorithm, and the load balancing algorithm is used to allocate tasks or traffic between thread resources to ensure that no single thread resource becomes a bottleneck due to overload; then, the corresponding idle thread resources are allocated to the speech recognition model according to the parameter information of the read target speech segment and the correspondence between the preset parameter information and the thread resources. The more the number of speech segments, the greater the amount of idle thread resources; the longer the duration of the speech segment, the greater the amount of idle thread resources.

[0047] Afterwards, the speech recognition model and its corresponding idle thread resources can be used to perform speech recognition on the read target speech segment to obtain text information corresponding to any target audio segment. The speech recognition model may include a feature extraction layer and a recognition layer. When the speech recognition model is used to perform speech recognition on the read target speech segment to obtain text information corresponding to any target audio segment, the read target speech segment can be sent to the feature extraction layer to extract features from the target speech segment to obtain speech features; the speech features are input into the recognition layer to map the speech features to corresponding text information.

[0048] In this way, the speech recognition model can be used to more efficiently perform speech recognition on the read target speech segment to obtain text information corresponding to any target audio segment.

[0049] In some optional embodiments, such as Figure 1As shown, the speech recognition system 10 may also include a response processing module 105; any speech recognition request may also include: priority information and identification information. The identification information is used to identify the speech recognition request. Based on this, the response processing module 105 can create a corresponding response queue for any speech recognition request according to the identification information of any speech recognition request after the request processing module 101 receives multiple speech recognition requests; the response queue corresponds to the identification information of any speech recognition request. After the speech recognition module 103 obtains the text information corresponding to any target audio segment, the text information corresponding to any target audio segment can be sent to the response queue corresponding to any target audio segment to wait for scheduling according to the priority information and identification information corresponding to any target audio segment; in response to the scheduling instruction, the scheduled target audio segment is sent from its corresponding response queue to the target device. The target device can be any device, such as a server or other terminal device. In this way, the orderliness and consistency of responses in multi-request scenarios can be more effectively ensured.

[0050] Optionally, the speech recognition system 10 may also have an error handling and retry mechanism. Once the recognition result of the speech recognition, that is, the text information, is abnormal or fails, the speech recognition request can be re-put into the request queue and the various steps of the aforementioned embodiments can be re-executed based on the error handling and retry mechanism to re-perform speech recognition to ensure the reliability and integrity of the recognition result.

[0051] In addition to the above-mentioned speech recognition system 10, correspondingly, the embodiment of the present application also provides a speech recognition method, which is applicable to speech recognition services; the speech recognition service includes: a request processing module, a channel detection module and a speech recognition module, a request queue is provided between the request processing module and the channel detection module, and a channel queue and a segment queue are provided between the channel detection module and the speech recognition module. The method may include:

[0052] Step 11: Receive multiple speech recognition requests through the request processing module, where any speech recognition request at least includes: an audio file, and cache the audio file in a request queue to wait for scheduling by the channel detection module.

[0053] Step 12: Schedule the first audio file from the request queue through the channel detection module, perform voice activity detection on the first audio file based on multiple parallel channel queues to obtain at least one target voice segment in the first audio file, and cache the at least one target voice segment in the segment queue.

[0054] Step 13: read the target voice segment from the segment queue through the voice recognition module, perform voice recognition on the read target voice segment using the voice recognition model, and obtain text information corresponding to any target audio segment.

[0055] The optional implementation methods of the above steps have been described in detail in the previous text and can be continued here, so they will not be repeated here.

[0056] In this embodiment, the request processing module can receive multiple speech recognition requests and cache the audio files of the speech recognition requests to the request queue; the channel detection module can schedule the first audio file from the request queue, perform speech activity detection on the first audio file based on multiple channel queues to obtain the target speech segment in the first audio file, and cache the target speech segment to the segment queue; the speech recognition module can read the target speech segment from the segment queue, and use the speech recognition model to recognize the read target speech segment to obtain text information. By setting queues between streaming modules, the processing of adjacent modules is decoupled and each module is independent; if there is cached data in the queue, the subsequent module will not be affected by the state of the previous module, and the previous module will not be affected by the state of the subsequent module, thereby improving the utilization of hardware resources.

[0057] The above speech recognition method will be further explained below in combination with actual application scenarios.

[0058] In actual scenarios, speech recognition methods can be divided into the following four stages:

[0059] ① Request processing stage:

[0060] The main function of the request processing stage is to receive voice data requests sent by the client, that is, voice recognition requests, and assign them to the request queue. Each voice recognition request can contain an audio file, metadata (such as identification information and timestamp information) and priority information. Each voice recognition request will be stored in the request queue for subsequent processing. The request processing stage can also perform preliminary screening and cleaning of voice recognition requests, including audio format detection and data integrity verification, to ensure that the data in the voice recognition request is not damaged during transmission. The request processing stage is designed to continuously receive and cache a large number of concurrent requests, while reducing latency through multi-threading or asynchronous I / O (Input / Output).

[0061] ② Channel detection stage: In this stage, the speech recognition request can be taken out from the request queue to perform channel detection and separation of multi-channel audio for subsequent speech recognition processing. When performing channel detection, the start and end points of the speech segment can be marked by the VAD algorithm. In the case of multiple channels, the data of each audio channel can be stored in an independent "channel queue" for further parallel processing. This separation processing method can effectively reduce the processing load when facing complex audio signals (such as meeting records), and ensure that the audio of each channel can enter the recognition stage independently, avoiding a single channel blocking the entire recognition process.

[0062] The channel detection phase aims to achieve parallelization by using multi-threading or multi-process technology, and the processing priority of each channel queue can be dynamically adjusted through the asynchronous task scheduler. In the case of heavy load, the efficient operation of channel detection can be ensured by automatically expanding the thread pool or task queue to avoid system crash or overload.

[0063] ③ Speech recognition stage: In this stage, the target speech segments obtained after channel detection, separation and speech activity detection can be subjected to speech recognition processing, and the target audio segments can be converted into text information. This stage adopts an efficient neural network inference engine (i.e., the speech recognition model in the aforementioned embodiment), for example, based on ONNX Runtime (Open NeuralNetwork Exchange Runtime, a high-performance inference engine for accelerating machine learning model inference), which can perform fast speech recognition processing in a multi-threaded environment. Among them, each target speech segment in the segment queue can be distributed to the thread pool, and through the load balancing algorithm, each target speech segment in the segment queue can be allocated to the idle thread resources in the thread pool to maximize the utilization of hardware resources.

[0064] To further reduce latency, the speech recognition stage can also use small batch inference technology, that is, multiple target audio clips are batch-inputted into the speech recognition model, thereby improving the utilization efficiency of hardware such as GPU (Graphics Processing Unit) / TPU (Tensor Processing Unit). The recognized text information will be stored in the corresponding response queue, waiting to be returned to the requester, that is, the client. This speech recognition stage can ensure the real-time and high-precision of speech recognition, and can more effectively handle concurrent speech recognition requests for multiple channels and multiple requests.

[0065] ④Response processing and result feedback stage:

[0066] After the text is recognized, the text results are stored in the response queue for unified management and return after text post-processing. Afterwards, the text information corresponding to any speech recognition request can be returned to the corresponding client in order according to the priority information and identification information of any speech recognition request to ensure the orderliness and consistency of the response in the multi-request scenario.

[0067] The above method can effectively improve the concurrent processing capability, that is, by decomposing the voice processing process into multiple streaming components and through multiple hierarchical queues (request queues, channel queues, segment queues and response queues, etc.), the system can process multiple audio requests in parallel, significantly improving the concurrent processing capability of voice recognition. The above method can also reduce system latency, that is, through the mechanism design of channel parallelism, queue caching, and dynamic assembly of fragments, the system can smooth the load under high concurrent requests, reduce the waiting time for request processing, and reduce system latency. The above method can also improve the utilization of hardware resources, that is, each processing node works independently, and through reasonable configuration, hardware resources can be fully utilized to avoid idle or overloaded resources. Through the above method, it is easier to expand based on modular design, that is, the modules are decoupled through queues, have good modular characteristics, and are convenient for expansion and maintenance according to needs.

[0068] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 13 can be device A; for another example, the execution subject of steps 11 to 12 can be device A, and the execution subject of step 13 can be device B; and so on.

[0069] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 12, 13, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0070] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0071] Figure 3 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application, and the electronic device is applicable to the speech recognition method provided by the aforementioned embodiment, such as Figure 3 As shown, the electronic device may include: a memory 301 , a processor 302 and a communication component 303 .

[0072] The memory 301 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0073] In some exemplary embodiments, the processor 302 is coupled to the memory 301 and is used to execute a computer program in the memory 301, so as to: receive multiple speech recognition requests through the request processing module, and any speech recognition request at least includes: an audio file, and cache the audio file in the request queue to wait for scheduling by the channel detection module; schedule a first audio file from the request queue through the channel detection module, perform voice activity detection on the first audio file based on multiple parallel channel queues to obtain at least one target speech segment in the first audio file, and cache the at least one target speech segment in the segment queue; read the target speech segment from the segment queue through the speech recognition module, perform speech recognition on the read target speech segment using the speech recognition model, and obtain text information corresponding to any target audio segment.

[0074] Optionally, any of the speech recognition requests also includes priority information. The processor 302 is specifically configured to sequentially cache the multiple speech recognition requests into the request queue according to the priority information of the multiple speech recognition requests to wait for scheduling by the channel detection module.

[0075] Optionally, the processor 302 is also used to: perform audio format detection and data integrity verification on the multiple voice recognition requests, and when the audio format of any of the voice recognition requests does not meet the first condition and / or the data integrity of any of the voice recognition requests does not meet the second condition, output a prompt message to the user to remind the user that there is a request error in any of the voice recognition requests.

[0076] Optionally, the processor 302 is specifically used to: perform channel separation on the first audio file to obtain multiple single-channel audios, and cache the multiple single-channel audios into multiple parallel channel queues; use the thread resources bound to each of the multiple channel queues to perform voice activity detection on the single-channel audios in the multiple channel queues to obtain at least one voice segment of each of the multiple single-channel audios; and perform channel merging on at least one voice segment of each of the multiple single-channel audios to obtain at least one target voice segment in the first audio file.

[0077] Optionally, the processor 302 is also used to: determine, through an asynchronous task scheduler, the processing priority of each of the multiple channel queues according to the total duration of the single-channel audio accumulated in each of the multiple channel queues; the longer the total duration of the single-channel audio accumulated in any channel queue, the higher the processing priority; the processor 302 is specifically used to: utilize the thread resources bound to each of the multiple channel queues and their respective processing priorities to perform voice activity detection on the single-channel audio in the multiple channel queues in turn to obtain at least one voice segment of each of the multiple single-channel audios.

[0078] Optionally, the processor 302 is specifically used to: allocate corresponding idle thread resources to the speech recognition model according to parameter information of the read target speech segment; the parameter information includes: the number and duration of the speech segments; use the speech recognition model and its corresponding idle thread resources to perform speech recognition on the read target speech segment to obtain text information corresponding to any target audio segment.

[0079] Optionally, any one of the voice recognition requests also includes: priority information and identification information; the processor 302 is used to: create a response queue for any one of the voice recognition requests through a response processing module; the response queue corresponds to the identification information of any one of the voice recognition requests; according to the priority information and identification information corresponding to any one of the target audio segments, send the text information corresponding to any one of the target audio segments to the response queue corresponding to any one of the target audio segments to wait for scheduling; in response to the scheduling instruction, send the scheduled target audio segment from its corresponding response queue to the target device.

[0080] Further, if Figure 3 As shown, the electronic device also includes: a display 304, a power component 305, an audio component 306 and other components. Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0081] An embodiment of the present application also provides a computer-readable storage medium, which, when the computer program is executed by a processor, enables the processor to implement the steps in the speech recognition method.

[0082] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the steps in the speech recognition method are executed.

[0083] In this embodiment, the request processing module can receive multiple speech recognition requests and cache the audio files of the speech recognition requests to the request queue; the channel detection module can schedule the first audio file from the request queue, perform speech activity detection on the first audio file based on multiple channel queues to obtain the target speech segment in the first audio file, and cache the target speech segment to the segment queue; the speech recognition module can read the target speech segment from the segment queue, and use the speech recognition model to recognize the read target speech segment to obtain text information. By setting queues between streaming modules, the processing of adjacent modules is decoupled and each module is independent; if there is cached data in the queue, the subsequent module will not be affected by the state of the previous module, and the previous module will not be affected by the state of the subsequent module, thereby improving the utilization of hardware resources.

[0084] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0085] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0086] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0087] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0088] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0094] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0095] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0097] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A speech recognition system, characterized in that: include: A request processing module, a channel detection module and a speech recognition module, wherein a request queue is provided between the request processing module and the channel detection module, and a segment queue is provided between the channel detection module and the speech recognition module: The request processing module is used to: receive multiple speech recognition requests, each speech recognition request at least includes: an audio file, and cache the audio file in the request queue to wait for scheduling by the channel detection module; The channel detection module is used to: schedule a first audio file from the request queue, perform voice activity detection on the first audio file based on multiple parallel channel queues to obtain at least one target voice segment in the first audio file, and cache the at least one target voice segment in the segment queue; The speech recognition module is used to: read a target speech segment from the segment queue, perform speech recognition on the read target speech segment using a speech recognition model, and obtain text information corresponding to any target audio segment.

2. The system according to claim 1, characterized in that Any speech recognition request further includes: priority information; The request processing module is specifically used to: cache the multiple voice recognition requests in the request queue in sequence according to the priority information of the multiple voice recognition requests, so as to wait for the channel detection module to schedule them.

3. The system according to claim 2, characterized in that The request processing module is also used to: perform audio format detection and data integrity verification on the multiple voice recognition requests, and when the audio format of any of the voice recognition requests does not meet the first condition and / or the data integrity of any of the voice recognition requests does not meet the second condition, output a prompt message to the user to remind the user that there is a request error in any of the voice recognition requests.

4. The system according to claim 1, characterized in that The channel detection module is specifically used for: Perform channel separation on the first audio file to obtain multiple single-channel audios, and cache the multiple single-channel audios into multiple parallel channel queues; Utilizing the thread resources bound to each of the multiple channel queues, voice activity detection is performed on the single-channel audios in the multiple channel queues to obtain at least one voice segment of each of the multiple single-channel audios; channel merging is performed on the at least one voice segment of each of the multiple single-channel audios to obtain at least one target voice segment in the first audio file.

5. The system according to claim 4, characterized in that The speech recognition system further includes: an asynchronous task scheduler; the asynchronous task scheduler is used to: determine the processing priority of each of the multiple channel queues according to the total duration of the single-channel audio accumulated in each of the multiple channel queues; the longer the total duration of the single-channel audio accumulated in any channel queue is, the higher the processing priority is; The channel detection module is specifically used to: utilize the thread resources bound to each of the multiple channel queues and their respective processing priorities to perform voice activity detection on the single-channel audios in the multiple channel queues in turn to obtain at least one voice segment of each of the multiple single-channel audios.

6. The system according to claim 1, characterized in that The speech recognition module is specifically used for: Allocate corresponding idle thread resources to the speech recognition model according to the parameter information of the target speech segment read; the parameter information includes: the number and duration of the speech segments; The speech recognition model and its corresponding idle thread resources are used to perform speech recognition on the read target speech segment to obtain text information corresponding to any target audio segment.

7. The system according to claim 1, characterized in that Also includes: Response processing module; Any speech recognition request further includes: priority information and identification information; the response processing module is used to: Creating a response queue for any of the speech recognition requests; the response queue corresponds to identification information of any of the speech recognition requests; According to the priority information and identification information corresponding to the any target audio segment, the text information corresponding to the any target audio segment is sent to the response queue corresponding to the any target audio segment to wait for scheduling; In response to the scheduling instruction, the scheduled target audio segment is sent from its corresponding response queue to the target device.

8. A speech recognition method, characterized in that: Applicable to speech recognition service; the speech recognition service comprises: a request processing module, a channel detection module and a speech recognition module, a request queue is provided between the request processing module and the channel detection module, and a channel queue and a segment queue are provided between the channel detection module and the speech recognition module; the method comprises: Receiving multiple speech recognition requests through the request processing module, any speech recognition request at least includes: an audio file, caching the audio file in the request queue to wait for scheduling by the channel detection module; Scheduling a first audio file from the request queue through the channel detection module, performing voice activity detection on the first audio file based on a plurality of parallel channel queues to obtain at least one target voice segment in the first audio file, and caching the at least one target voice segment in the segment queue; The target voice segment is read from the segment queue by the voice recognition module, and the voice recognition model is used to perform voice recognition on the read target voice segment to obtain text information corresponding to any target audio segment.

9. An electronic device, characterized in that: include: A memory and a processor; wherein the memory is used to: store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: perform the steps in the method of claim 8.

10. A computer-readable storage medium, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to claim 8.

11. A computer program product, characterized in that The method comprises a computer program / instruction which, when executed by a processor, causes the processor to implement the steps of the method according to claim 8.

Images